path: root/documentation/dev-manual/packages.rst

.. SPDX-License-Identifier: CC-BY-SA-2.0-UK

Working with Packages
*********************

This section describes a few tasks that involve packages:

-  :ref:`dev-manual/packages:excluding packages from an image`

-  :ref:`dev-manual/packages:incrementing a package version`

-  :ref:`dev-manual/packages:handling optional module packaging`

-  :ref:`dev-manual/packages:using runtime package management`

-  :ref:`dev-manual/packages:generating and using signed packages`

-  :ref:`Setting up and running package test
   (ptest) <dev-manual/packages:testing packages with ptest>`

-  :ref:`dev-manual/packages:creating node package manager (npm) packages`

-  :ref:`dev-manual/packages:adding custom metadata to packages`

Excluding Packages from an Image
================================

You might find it necessary to prevent specific packages from being
installed into an image. If so, you can use several variables to direct
the build system to essentially ignore installing recommended packages
or to not install a package at all.

The following list introduces variables you can use to prevent packages
from being installed into your image. Each of these variables only works
with IPK and RPM package types, not for Debian packages.
Also, you can use these variables from your ``local.conf`` file
or attach them to a specific image recipe by using a recipe name
override. For more detail on the variables, see the descriptions in the
Yocto Project Reference Manual's glossary chapter.

-  :term:`BAD_RECOMMENDATIONS`:
   Use this variable to specify "recommended-only" packages that you do
   not want installed.

-  :term:`NO_RECOMMENDATIONS`:
   Use this variable to prevent all "recommended-only" packages from
   being installed.

-  :term:`PACKAGE_EXCLUDE`:
   Use this variable to prevent specific packages from being installed
   regardless of whether they are "recommended-only" or not. You need to
   realize that the build process could fail with an error when you
   prevent the installation of a package whose presence is required by
   an installed package.

Incrementing a Package Version
==============================

This section provides some background on how binary package versioning
is accomplished and presents some of the services, variables, and
terminology involved.

In order to understand binary package versioning, you need to consider
the following:

-  Binary Package: The binary package that is eventually built and
   installed into an image.

-  Binary Package Version: The binary package version is composed of two
   components --- a version and a revision.

   .. note::

      Technically, a third component, the "epoch" (i.e. :term:`PE`) is involved
      but this discussion for the most part ignores :term:`PE`.

   The version and revision are taken from the
   :term:`PV` and
   :term:`PR` variables, respectively.

-  :term:`PV`: The recipe version. :term:`PV` represents the version of the
   software being packaged. Do not confuse :term:`PV` with the binary
   package version.

-  :term:`PR`: The recipe revision.

-  :term:`SRCPV`: The OpenEmbedded
   build system uses this string to help define the value of :term:`PV` when
   the source code revision needs to be included in it.

-  :yocto_wiki:`PR Service </PR_Service>`: A
   network-based service that helps automate keeping package feeds
   compatible with existing package manager applications such as RPM,
   APT, and OPKG.

Whenever the binary package content changes, the binary package version
must change. Changing the binary package version is accomplished by
changing or "bumping" the :term:`PR` and/or :term:`PV` values. Increasing these
values occurs one of two ways:

-  Automatically using a Package Revision Service (PR Service).

-  Manually incrementing the :term:`PR` and/or :term:`PV` variables.

Given a primary challenge of any build system and its users is how to
maintain a package feed that is compatible with existing package manager
applications such as RPM, APT, and OPKG, using an automated system is
much preferred over a manual system. In either system, the main
requirement is that binary package version numbering increases in a
linear fashion and that there is a number of version components that
support that linear progression. For information on how to ensure
package revisioning remains linear, see the
":ref:`dev-manual/packages:automatically incrementing a package version number`"
section.

The following three sections provide related information on the PR
Service, the manual method for "bumping" :term:`PR` and/or :term:`PV`, and on
how to ensure binary package revisioning remains linear.

Working With a PR Service
-------------------------

As mentioned, attempting to maintain revision numbers in the
:term:`Metadata` is error prone, inaccurate,
and causes problems for people submitting recipes. Conversely, the PR
Service automatically generates increasing numbers, particularly the
revision field, which removes the human element.

.. note::

   For additional information on using a PR Service, you can see the
   :yocto_wiki:`PR Service </PR_Service>` wiki page.

The Yocto Project uses variables in order of decreasing priority to
facilitate revision numbering (i.e.
:term:`PE`,
:term:`PV`, and
:term:`PR` for epoch, version, and
revision, respectively). The values are highly dependent on the policies
and procedures of a given distribution and package feed.

Because the OpenEmbedded build system uses
":ref:`signatures <overview-manual/concepts:checksums (signatures)>`", which are
unique to a given build, the build system knows when to rebuild
packages. All the inputs into a given task are represented by a
signature, which can trigger a rebuild when different. Thus, the build
system itself does not rely on the :term:`PR`, :term:`PV`, and :term:`PE` numbers to
trigger a rebuild. The signatures, however, can be used to generate
these values.

The PR Service works with both ``OEBasic`` and ``OEBasicHash``
generators. The value of :term:`PR` bumps when the checksum changes and the
different generator mechanisms change signatures under different
circumstances.

As implemented, the build system includes values from the PR Service
into the :term:`PR` field as an addition using the form "``.x``" so ``r0``
becomes ``r0.1``, ``r0.2`` and so forth. This scheme allows existing
:term:`PR` values to be used for whatever reasons, which include manual
:term:`PR` bumps, should it be necessary.

By default, the PR Service is not enabled or running. Thus, the packages
generated are just "self consistent". The build system adds and removes
packages and there are no guarantees about upgrade paths but images will
be consistent and correct with the latest changes.

The simplest form for a PR Service is for a single host development system
that builds the package feed (building system). For this scenario, you can
enable a local PR Service by setting :term:`PRSERV_HOST` in your
``local.conf`` file in the :term:`Build Directory`::

   PRSERV_HOST = "localhost:0"

Once the service is started, packages will automatically
get increasing :term:`PR` values and BitBake takes care of starting and
stopping the server.

If you have a more complex setup where multiple host development systems
work against a common, shared package feed, you have a single PR Service
running and it is connected to each building system. For this scenario,
you need to start the PR Service using the ``bitbake-prserv`` command::

   bitbake-prserv --host ip --port port --start

In addition to
hand-starting the service, you need to update the ``local.conf`` file of
each building system as described earlier so each system points to the
server and port.

It is also recommended you use build history, which adds some sanity
checks to binary package versions, in conjunction with the server that
is running the PR Service. To enable build history, add the following to
each building system's ``local.conf`` file::

   # It is recommended to activate "buildhistory" for testing the PR service
   INHERIT += "buildhistory"
   BUILDHISTORY_COMMIT = "1"

For information on build
history, see the
":ref:`dev-manual/build-quality:maintaining build output quality`" section.

.. note::

   The OpenEmbedded build system does not maintain :term:`PR` information as
   part of the shared state (sstate) packages. If you maintain an sstate
   feed, it's expected that either all your building systems that
   contribute to the sstate feed use a shared PR service, or you do not
   run a PR service on any of your building systems.

   That's because if you had multiple machines sharing a PR service but
   not their sstate feed, you could end up with "diverging" hashes for
   the same output artefacts. When presented to the share PR service,
   each would be considered as new and would increase the revision
   number, causing many unnecessary package upgrades.

   For more information on shared state, see the
   ":ref:`overview-manual/concepts:shared state cache`"
   section in the Yocto Project Overview and Concepts Manual.

Manually Bumping PR
-------------------

The alternative to setting up a PR Service is to manually "bump" the
:term:`PR` variable.

If a committed change results in changing the package output, then the
value of the :term:`PR` variable needs to be increased (or "bumped") as part of
that commit. For new recipes you should add the :term:`PR` variable and set
its initial value equal to "r0", which is the default. Even though the
default value is "r0", the practice of adding it to a new recipe makes
it harder to forget to bump the variable when you make changes to the
recipe in future.

Usually, version increases occur only to binary packages. However, if
for some reason :term:`PV` changes but does not increase, you can increase
the :term:`PE` variable (Package Epoch). The :term:`PE` variable defaults to
"0".

Binary package version numbering strives to follow the `Debian Version
Field Policy
Guidelines <https://www.debian.org/doc/debian-policy/ch-controlfields.html>`__.
These guidelines define how versions are compared and what "increasing"
a version means.

Automatically Incrementing a Package Version Number
---------------------------------------------------

When fetching a repository, BitBake uses the
:term:`SRCREV` variable to determine
the specific source code revision from which to build. You set the
:term:`SRCREV` variable to
:term:`AUTOREV` to cause the
OpenEmbedded build system to automatically use the latest revision of
the software::

   SRCREV = "${AUTOREV}"

Furthermore, you need to reference :term:`SRCPV` in :term:`PV` in order to
automatically update the version whenever the revision of the source
code changes. Here is an example::

   PV = "1.0+git${SRCPV}"

The OpenEmbedded build system substitutes :term:`SRCPV` with the following:

.. code-block:: none

   AUTOINC+source_code_revision

The build system replaces the ``AUTOINC``
with a number. The number used depends on the state of the PR Service:

-  If PR Service is enabled, the build system increments the number,
   which is similar to the behavior of
   :term:`PR`. This behavior results in
   linearly increasing package versions, which is desirable. Here is an
   example:

   .. code-block:: none

      hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
      hello-world-git_0.0+git1+dd2f5c3565-r0.0_armv7a-neon.ipk

-  If PR Service is not enabled, the build system replaces the
   ``AUTOINC`` placeholder with zero (i.e. "0"). This results in
   changing the package version since the source revision is included.
   However, package versions are not increased linearly. Here is an
   example:

   .. code-block:: none

      hello-world-git_0.0+git0+b6558dd387-r0.0_armv7a-neon.ipk
      hello-world-git_0.0+git0+dd2f5c3565-r0.0_armv7a-neon.ipk

In summary, the OpenEmbedded build system does not track the history of
binary package versions for this purpose. ``AUTOINC``, in this case, is
comparable to :term:`PR`. If PR server is not enabled, ``AUTOINC`` in the
package version is simply replaced by "0". If PR server is enabled, the
build system keeps track of the package versions and bumps the number
when the package revision changes.

Handling Optional Module Packaging
==================================

Many pieces of software split functionality into optional modules (or
plugins) and the plugins that are built might depend on configuration
options. To avoid having to duplicate the logic that determines what
modules are available in your recipe or to avoid having to package each
module by hand, the OpenEmbedded build system provides functionality to
handle module packaging dynamically.

To handle optional module packaging, you need to do two things:

-  Ensure the module packaging is actually done.

-  Ensure that any dependencies on optional modules from other recipes
   are satisfied by your recipe.

Making Sure the Packaging is Done
---------------------------------

To ensure the module packaging actually gets done, you use the
``do_split_packages`` function within the ``populate_packages`` Python
function in your recipe. The ``do_split_packages`` function searches for
a pattern of files or directories under a specified path and creates a
package for each one it finds by appending to the
:term:`PACKAGES` variable and
setting the appropriate values for ``FILES:packagename``,
``RDEPENDS:packagename``, ``DESCRIPTION:packagename``, and so forth.
Here is an example from the ``lighttpd`` recipe::

   python populate_packages:prepend () {
       lighttpd_libdir = d.expand('${libdir}')
       do_split_packages(d, lighttpd_libdir, '^mod_(.*).so$',
                        'lighttpd-module-%s', 'Lighttpd module for %s',
                         extra_depends='')
   }

The previous example specifies a number of things in the call to
``do_split_packages``.

-  A directory within the files installed by your recipe through
   :ref:`ref-tasks-install` in which to search.

-  A regular expression used to match module files in that directory. In
   the example, note the parentheses () that mark the part of the
   expression from which the module name should be derived.

-  A pattern to use for the package names.

-  A description for each package.

-  An empty string for ``extra_depends``, which disables the default
   dependency on the main ``lighttpd`` package. Thus, if a file in
   ``${libdir}`` called ``mod_alias.so`` is found, a package called
   ``lighttpd-module-alias`` is created for it and the
   :term:`DESCRIPTION` is set to
   "Lighttpd module for alias".

Often, packaging modules is as simple as the previous example. However,
there are more advanced options that you can use within
``do_split_packages`` to modify its behavior. And, if you need to, you
can add more logic by specifying a hook function that is called for each
package. It is also perfectly acceptable to call ``do_split_packages``
multiple times if you have more than one set of modules to package.

For more examples that show how to use ``do_split_packages``, see the
``connman.inc`` file in the ``meta/recipes-connectivity/connman/``
directory of the ``poky`` :ref:`source repository <overview-manual/development-environment:yocto project source repositories>`. You can
also find examples in ``meta/classes-recipe/kernel.bbclass``.

Here is a reference that shows ``do_split_packages`` mandatory and
optional arguments::

   Mandatory arguments

   root
      The path in which to search
   file_regex
      Regular expression to match searched files.
      Use parentheses () to mark the part of this
      expression that should be used to derive the
      module name (to be substituted where %s is
      used in other function arguments as noted below)
   output_pattern
      Pattern to use for the package names. Must
      include %s.
   description
      Description to set for each package. Must
      include %s.

   Optional arguments

   postinst
      Postinstall script to use for all packages
      (as a string)
   recursive
      True to perform a recursive search --- default
      False
   hook
      A hook function to be called for every match.
      The function will be called with the following
      arguments (in the order listed):

      f
         Full path to the file/directory match
      pkg
         The package name
      file_regex
         As above
      output_pattern
         As above
      modulename
         The module name derived using file_regex
   extra_depends
      Extra runtime dependencies (RDEPENDS) to be
      set for all packages. The default value of None
      causes a dependency on the main package
      (${PN}) --- if you do not want this, pass empty
      string '' for this parameter.
   aux_files_pattern
      Extra item(s) to be added to FILES for each
      package. Can be a single string item or a list
      of strings for multiple items. Must include %s.
   postrm
      postrm script to use for all packages (as a
      string)
   allow_dirs
      True to allow directories to be matched -
      default False
   prepend
      If True, prepend created packages to PACKAGES
      instead of the default False which appends them
   match_path
      match file_regex on the whole relative path to
      the root rather than just the filename
   aux_files_pattern_verbatim
      Extra item(s) to be added to FILES for each
      package, using the actual derived module name
      rather than converting it to something legal
      for a package name. Can be a single string item
      or a list of strings for multiple items. Must
      include %s.
   allow_links
      True to allow symlinks to be matched --- default
      False
   summary
      Summary to set for each package. Must include %s;
      defaults to description if not set.



Satisfying Dependencies
-----------------------

The second part for handling optional module packaging is to ensure that
any dependencies on optional modules from other recipes are satisfied by
your recipe. You can be sure these dependencies are satisfied by using
the :term:`PACKAGES_DYNAMIC`
variable. Here is an example that continues with the ``lighttpd`` recipe
shown earlier::

   PACKAGES_DYNAMIC = "lighttpd-module-.*"

The name
specified in the regular expression can of course be anything. In this
example, it is ``lighttpd-module-`` and is specified as the prefix to
ensure that any :term:`RDEPENDS` and
:term:`RRECOMMENDS` on a package
name starting with the prefix are satisfied during build time. If you
are using ``do_split_packages`` as described in the previous section,
the value you put in :term:`PACKAGES_DYNAMIC` should correspond to the name
pattern specified in the call to ``do_split_packages``.

Using Runtime Package Management
================================

During a build, BitBake always transforms a recipe into one or more
packages. For example, BitBake takes the ``bash`` recipe and produces a
number of packages (e.g. ``bash``, ``bash-bashbug``,
``bash-completion``, ``bash-completion-dbg``, ``bash-completion-dev``,
``bash-completion-extra``, ``bash-dbg``, and so forth). Not all
generated packages are included in an image.

In several situations, you might need to update, add, remove, or query
the packages on a target device at runtime (i.e. without having to
generate a new image). Examples of such situations include:

-  You want to provide in-the-field updates to deployed devices (e.g.
   security updates).

-  You want to have a fast turn-around development cycle for one or more
   applications that run on your device.

-  You want to temporarily install the "debug" packages of various
   applications on your device so that debugging can be greatly improved
   by allowing access to symbols and source debugging.

-  You want to deploy a more minimal package selection of your device
   but allow in-the-field updates to add a larger selection for
   customization.

In all these situations, you have something similar to a more
traditional Linux distribution in that in-field devices are able to
receive pre-compiled packages from a server for installation or update.
Being able to install these packages on a running, in-field device is
what is termed "runtime package management".

In order to use runtime package management, you need a host or server
machine that serves up the pre-compiled packages plus the required
metadata. You also need package manipulation tools on the target. The
build machine is a likely candidate to act as the server. However, that
machine does not necessarily have to be the package server. The build
machine could push its artifacts to another machine that acts as the
server (e.g. Internet-facing). In fact, doing so is advantageous for a
production environment as getting the packages away from the development
system's :term:`Build Directory` prevents accidental overwrites.

A simple build that targets just one device produces more than one
package database. In other words, the packages produced by a build are
separated out into a couple of different package groupings based on
criteria such as the target's CPU architecture, the target board, or the
C library used on the target. For example, a build targeting the
``qemux86`` device produces the following three package databases:
``noarch``, ``i586``, and ``qemux86``. If you wanted your ``qemux86``
device to be aware of all the packages that were available to it, you
would need to point it to each of these databases individually. In a
similar way, a traditional Linux distribution usually is configured to
be aware of a number of software repositories from which it retrieves
packages.

Using runtime package management is completely optional and not required
for a successful build or deployment in any way. But if you want to make
use of runtime package management, you need to do a couple things above
and beyond the basics. The remainder of this section describes what you
need to do.

Build Considerations
--------------------

This section describes build considerations of which you need to be
aware in order to provide support for runtime package management.

When BitBake generates packages, it needs to know what format or formats
to use. In your configuration, you use the
:term:`PACKAGE_CLASSES`
variable to specify the format:

#. Open the ``local.conf`` file inside your :term:`Build Directory` (e.g.
   ``poky/build/conf/local.conf``).

#. Select the desired package format as follows::

      PACKAGE_CLASSES ?= "package_packageformat"

   where packageformat can be "ipk", "rpm",
   "deb", or "tar" which are the supported package formats.

   .. note::

      Because the Yocto Project supports four different package formats,
      you can set the variable with more than one argument. However, the
      OpenEmbedded build system only uses the first argument when
      creating an image or Software Development Kit (SDK).

If you would like your image to start off with a basic package database
containing the packages in your current build as well as to have the
relevant tools available on the target for runtime package management,
you can include "package-management" in the
:term:`IMAGE_FEATURES`
variable. Including "package-management" in this configuration variable
ensures that when the image is assembled for your target, the image
includes the currently-known package databases as well as the
target-specific tools required for runtime package management to be
performed on the target. However, this is not strictly necessary. You
could start your image off without any databases but only include the
required on-target package tool(s). As an example, you could include
"opkg" in your
:term:`IMAGE_INSTALL` variable
if you are using the IPK package format. You can then initialize your
target's package database(s) later once your image is up and running.

Whenever you perform any sort of build step that can potentially
generate a package or modify existing package, it is always a good idea
to re-generate the package index after the build by using the following
command::

   $ bitbake package-index

It might be tempting to build the
package and the package index at the same time with a command such as
the following::

   $ bitbake some-package package-index

Do not do this as
BitBake does not schedule the package index for after the completion of
the package you are building. Consequently, you cannot be sure of the
package index including information for the package you just built.
Thus, be sure to run the package update step separately after building
any packages.

You can use the
:term:`PACKAGE_FEED_ARCHS`,
:term:`PACKAGE_FEED_BASE_PATHS`,
and
:term:`PACKAGE_FEED_URIS`
variables to pre-configure target images to use a package feed. If you
do not define these variables, then manual steps as described in the
subsequent sections are necessary to configure the target. You should
set these variables before building the image in order to produce a
correctly configured image.

.. note::

   Your image will need enough free storage space to run package upgrades,
   especially if many of them need to be downloaded at the same time.
   You should make sure images are created with enough free space
   by setting the :term:`IMAGE_ROOTFS_EXTRA_SPACE` variable.

When your build is complete, your packages reside in the
``${TMPDIR}/deploy/packageformat`` directory. For example, if
``${``\ :term:`TMPDIR`\ ``}`` is
``tmp`` and your selected package type is RPM, then your RPM packages
are available in ``tmp/deploy/rpm``.

Host or Server Machine Setup
----------------------------

Although other protocols are possible, a server using HTTP typically
serves packages. If you want to use HTTP, then set up and configure a
web server such as Apache 2, lighttpd, or Python web server on the
machine serving the packages.

To keep things simple, this section describes how to set up a
Python web server to share package feeds from the developer's
machine. Although this server might not be the best for a production
environment, the setup is simple and straight forward. Should you want
to use a different server more suited for production (e.g. Apache 2,
Lighttpd, or Nginx), take the appropriate steps to do so.

From within the :term:`Build Directory` where you have built an image based on
your packaging choice (i.e. the :term:`PACKAGE_CLASSES` setting), simply start
the server. The following example assumes a :term:`Build Directory` of ``poky/build``
and a :term:`PACKAGE_CLASSES` setting of ":ref:`ref-classes-package_rpm`"::

   $ cd poky/build/tmp/deploy/rpm
   $ python3 -m http.server

Target Setup
------------

Setting up the target differs depending on the package management
system. This section provides information for RPM, IPK, and DEB.

Using RPM
~~~~~~~~~

The :wikipedia:`Dandified Packaging <DNF_(software)>` (DNF) performs
runtime package management of RPM packages. In order to use DNF for
runtime package management, you must perform an initial setup on the
target machine for cases where the ``PACKAGE_FEED_*`` variables were not
set as part of the image that is running on the target. This means if
you built your image and did not use these variables as part of the
build and your image is now running on the target, you need to perform
the steps in this section if you want to use runtime package management.

.. note::

   For information on the ``PACKAGE_FEED_*`` variables, see
   :term:`PACKAGE_FEED_ARCHS`, :term:`PACKAGE_FEED_BASE_PATHS`, and
   :term:`PACKAGE_FEED_URIS` in the Yocto Project Reference Manual variables
   glossary.

On the target, you must inform DNF that package databases are available.
You do this by creating a file named
``/etc/yum.repos.d/oe-packages.repo`` and defining the ``oe-packages``.

As an example, assume the target is able to use the following package
databases: ``all``, ``i586``, and ``qemux86`` from a server named
``my.server``. The specifics for setting up the web server are up to
you. The critical requirement is that the URIs in the target repository
configuration point to the correct remote location for the feeds.

.. note::

   For development purposes, you can point the web server to the build
   system's ``deploy`` directory. However, for production use, it is better to
   copy the package directories to a location outside of the build area and use
   that location. Doing so avoids situations where the build system
   overwrites or changes the ``deploy`` directory.

When telling DNF where to look for the package databases, you must
declare individual locations per architecture or a single location used
for all architectures. You cannot do both:

-  *Create an Explicit List of Architectures:* Define individual base
   URLs to identify where each package database is located:

   .. code-block:: none

      [oe-packages]
      baseurl=http://my.server/rpm/i586  http://my.server/rpm/qemux86 http://my.server/rpm/all

   This example
   informs DNF about individual package databases for all three
   architectures.

-  *Create a Single (Full) Package Index:* Define a single base URL that
   identifies where a full package database is located::

      [oe-packages]
      baseurl=http://my.server/rpm

   This example informs DNF about a single
   package database that contains all the package index information for
   all supported architectures.

Once you have informed DNF where to find the package databases, you need
to fetch them:

.. code-block:: none

   # dnf makecache

DNF is now able to find, install, and
upgrade packages from the specified repository or repositories.

.. note::

   See the `DNF documentation <https://dnf.readthedocs.io/en/latest/>`__ for
   additional information.

Using IPK
~~~~~~~~~

The ``opkg`` application performs runtime package management of IPK
packages. You must perform an initial setup for ``opkg`` on the target
machine if the
:term:`PACKAGE_FEED_ARCHS`,
:term:`PACKAGE_FEED_BASE_PATHS`,
and
:term:`PACKAGE_FEED_URIS`
variables have not been set or the target image was built before the
variables were set.

The ``opkg`` application uses configuration files to find available
package databases. Thus, you need to create a configuration file inside
the ``/etc/opkg/`` directory, which informs ``opkg`` of any repository
you want to use.

As an example, suppose you are serving packages from a ``ipk/``
directory containing the ``i586``, ``all``, and ``qemux86`` databases
through an HTTP server named ``my.server``. On the target, create a
configuration file (e.g. ``my_repo.conf``) inside the ``/etc/opkg/``
directory containing the following:

.. code-block:: none

   src/gz all http://my.server/ipk/all
   src/gz i586 http://my.server/ipk/i586
   src/gz qemux86 http://my.server/ipk/qemux86

Next, instruct ``opkg`` to fetch the
repository information:

.. code-block:: none

   # opkg update

The ``opkg`` application is now able to find, install, and upgrade packages
from the specified repository.

Using DEB
~~~~~~~~~

The ``apt`` application performs runtime package management of DEB
packages. This application uses a source list file to find available
package databases. You must perform an initial setup for ``apt`` on the
target machine if the
:term:`PACKAGE_FEED_ARCHS`,
:term:`PACKAGE_FEED_BASE_PATHS`,
and
:term:`PACKAGE_FEED_URIS`
variables have not been set or the target image was built before the
variables were set.

To inform ``apt`` of the repository you want to use, you might create a
list file (e.g. ``my_repo.list``) inside the
``/etc/apt/sources.list.d/`` directory. As an example, suppose you are
serving packages from a ``deb/`` directory containing the ``i586``,
``all``, and ``qemux86`` databases through an HTTP server named
``my.server``. The list file should contain:

.. code-block:: none

   deb http://my.server/deb/all ./
   deb http://my.server/deb/i586 ./
   deb http://my.server/deb/qemux86 ./

Next, instruct the ``apt`` application
to fetch the repository information:

.. code-block:: none

  $ sudo apt update

After this step,
``apt`` is able to find, install, and upgrade packages from the
specified repository.

Generating and Using Signed Packages
====================================

In order to add security to RPM packages used during a build, you can
take steps to securely sign them. Once a signature is verified, the
OpenEmbedded build system can use the package in the build. If security
fails for a signed package, the build system stops the build.

This section describes how to sign RPM packages during a build and how
to use signed package feeds (repositories) when doing a build.

Signing RPM Packages
--------------------

To enable signing RPM packages, you must set up the following
configurations in either your ``local.config`` or ``distro.config``
file::

   # Inherit sign_rpm.bbclass to enable signing functionality
   INHERIT += " sign_rpm"
   # Define the GPG key that will be used for signing.
   RPM_GPG_NAME = "key_name"
   # Provide passphrase for the key
   RPM_GPG_PASSPHRASE = "passphrase"

.. note::

   Be sure to supply appropriate values for both `key_name` and
   `passphrase`.

Aside from the ``RPM_GPG_NAME`` and ``RPM_GPG_PASSPHRASE`` variables in
the previous example, two optional variables related to signing are available:

-  *GPG_BIN:* Specifies a ``gpg`` binary/wrapper that is executed
   when the package is signed.

-  *GPG_PATH:* Specifies the ``gpg`` home directory used when the
   package is signed.

Processing Package Feeds
------------------------

In addition to being able to sign RPM packages, you can also enable
signed package feeds for IPK and RPM packages.

The steps you need to take to enable signed package feed use are similar
to the steps used to sign RPM packages. You must define the following in
your ``local.config`` or ``distro.config`` file::

   INHERIT += "sign_package_feed"
   PACKAGE_FEED_GPG_NAME = "key_name"
   PACKAGE_FEED_GPG_PASSPHRASE_FILE = "path_to_file_containing_passphrase"

For signed package feeds, the passphrase must be specified in a separate file,
which is pointed to by the ``PACKAGE_FEED_GPG_PASSPHRASE_FILE``
variable. Regarding security, keeping a plain text passphrase out of the
configuration is more secure.

Aside from the ``PACKAGE_FEED_GPG_NAME`` and
``PACKAGE_FEED_GPG_PASSPHRASE_FILE`` variables, three optional variables
related to signed package feeds are available:

-  *GPG_BIN* Specifies a ``gpg`` binary/wrapper that is executed
   when the package is signed.

-  *GPG_PATH:* Specifies the ``gpg`` home directory used when the
   package is signed.

-  *PACKAGE_FEED_GPG_SIGNATURE_TYPE:* Specifies the type of ``gpg``
   signature. This variable applies only to RPM and IPK package feeds.
   Allowable values for the ``PACKAGE_FEED_GPG_SIGNATURE_TYPE`` are
   "ASC", which is the default and specifies ascii armored, and "BIN",
   which specifies binary.

Testing Packages With ptest
===========================

A Package Test (ptest) runs tests against packages built by the
OpenEmbedded build system on the target machine. A ptest contains at
least two items: the actual test, and a shell script (``run-ptest``)
that starts the test. The shell script that starts the test must not
contain the actual test --- the script only starts the test. On the other
hand, the test can be anything from a simple shell script that runs a
binary and checks the output to an elaborate system of test binaries and
data files.

The test generates output in the format used by Automake::

   result: testname

where the result can be ``PASS``, ``FAIL``, or ``SKIP``, and
the testname can be any identifying string.

For a list of Yocto Project recipes that are already enabled with ptest,
see the :yocto_wiki:`Ptest </Ptest>` wiki page.

.. note::

   A recipe is "ptest-enabled" if it inherits the :ref:`ref-classes-ptest`
   class.

Adding ptest to Your Build
--------------------------

To add package testing to your build, add the :term:`DISTRO_FEATURES` and
:term:`EXTRA_IMAGE_FEATURES` variables to your ``local.conf`` file, which
is found in the :term:`Build Directory`::

   DISTRO_FEATURES:append = " ptest"
   EXTRA_IMAGE_FEATURES += "ptest-pkgs"

Once your build is complete, the ptest files are installed into the
``/usr/lib/package/ptest`` directory within the image, where ``package``
is the name of the package.

Running ptest
-------------

The ``ptest-runner`` package installs a shell script that loops through
all installed ptest test suites and runs them in sequence. Consequently,
you might want to add this package to your image.

Getting Your Package Ready
--------------------------

In order to enable a recipe to run installed ptests on target hardware,
you need to prepare the recipes that build the packages you want to
test. Here is what you have to do for each recipe:

-  *Be sure the recipe inherits the* :ref:`ref-classes-ptest` *class:*
   Include the following line in each recipe::

      inherit ptest

-  *Create run-ptest:* This script starts your test. Locate the
   script where you will refer to it using
   :term:`SRC_URI`. Here is an
   example that starts a test for ``dbus``::

      #!/bin/sh
      cd test
      make -k runtest-TESTS

-  *Ensure dependencies are met:* If the test adds build or runtime
   dependencies that normally do not exist for the package (such as
   requiring "make" to run the test suite), use the
   :term:`DEPENDS` and
   :term:`RDEPENDS` variables in
   your recipe in order for the package to meet the dependencies. Here
   is an example where the package has a runtime dependency on "make"::

      RDEPENDS:${PN}-ptest += "make"

-  *Add a function to build the test suite:* Not many packages support
   cross-compilation of their test suites. Consequently, you usually
   need to add a cross-compilation function to the package.

   Many packages based on Automake compile and run the test suite by
   using a single command such as ``make check``. However, the host
   ``make check`` builds and runs on the same computer, while
   cross-compiling requires that the package is built on the host but
   executed for the target architecture (though often, as in the case
   for ptest, the execution occurs on the host). The built version of
   Automake that ships with the Yocto Project includes a patch that
   separates building and execution. Consequently, packages that use the
   unaltered, patched version of ``make check`` automatically
   cross-compiles.

   Regardless, you still must add a ``do_compile_ptest`` function to
   build the test suite. Add a function similar to the following to your
   recipe::

      do_compile_ptest() {
          oe_runmake buildtest-TESTS
      }

-  *Ensure special configurations are set:* If the package requires
   special configurations prior to compiling the test code, you must
   insert a ``do_configure_ptest`` function into the recipe.

-  *Install the test suite:* The :ref:`ref-classes-ptest` class
   automatically copies the file ``run-ptest`` to the target and then runs make
   ``install-ptest`` to run the tests. If this is not enough, you need
   to create a ``do_install_ptest`` function and make sure it gets
   called after the "make install-ptest" completes.

Creating Node Package Manager (NPM) Packages
============================================

:wikipedia:`NPM <Npm_(software)>` is a package manager for the JavaScript
programming language. The Yocto Project supports the NPM
:ref:`fetcher <bitbake-user-manual/bitbake-user-manual-fetching:fetchers>`.
You can use this fetcher in combination with
:doc:`devtool </ref-manual/devtool-reference>` to create recipes that produce
NPM packages.

There are two workflows that allow you to create NPM packages using
``devtool``: the NPM registry modules method and the NPM project code
method.

.. note::

   While it is possible to create NPM recipes manually, using
   ``devtool`` is far simpler.

Additionally, some requirements and caveats exist.

Requirements and Caveats
------------------------

You need to be aware of the following before using ``devtool`` to create
NPM packages:

-  Of the two methods that you can use ``devtool`` to create NPM
   packages, the registry approach is slightly simpler. However, you
   might consider the project approach because you do not have to
   publish your module in the `NPM registry <https://docs.npmjs.com/misc/registry>`__,
   which is NPM's public registry.

-  Be familiar with
   :doc:`devtool </ref-manual/devtool-reference>`.

-  The NPM host tools need the native ``nodejs-npm`` package, which is
   part of the OpenEmbedded environment. You need to get the package by
   cloning the :oe_git:`meta-openembedded </meta-openembedded>`
   repository. Be sure to add the path to your local copy
   to your ``bblayers.conf`` file.

-  ``devtool`` cannot detect native libraries in module dependencies.
   Consequently, you must manually add packages to your recipe.

-  While deploying NPM packages, ``devtool`` cannot determine which
   dependent packages are missing on the target (e.g. the node runtime
   ``nodejs``). Consequently, you need to find out what files are
   missing and be sure they are on the target.

-  Although you might not need NPM to run your node package, it is
   useful to have NPM on your target. The NPM package name is
   ``nodejs-npm``.

Using the Registry Modules Method
---------------------------------

This section presents an example that uses the ``cute-files`` module,
which is a file browser web application.

.. note::

   You must know the ``cute-files`` module version.

The first thing you need to do is use ``devtool`` and the NPM fetcher to
create the recipe::

   $ devtool add "npm://registry.npmjs.org;package=cute-files;version=1.0.2"

The
``devtool add`` command runs ``recipetool create`` and uses the same
fetch URI to download each dependency and capture license details where
possible. The result is a generated recipe.

After running for quite a long time, in particular building the
``nodejs-native`` package, the command should end as follows::

   INFO: Recipe /home/.../build/workspace/recipes/cute-files/cute-files_1.0.2.bb has been automatically created; further editing may be required to make it fully functional

The recipe file is fairly simple and contains every license that
``recipetool`` finds and includes the licenses in the recipe's
:term:`LIC_FILES_CHKSUM`
variables. You need to examine the variables and look for those with
"unknown" in the :term:`LICENSE`
field. You need to track down the license information for "unknown"
modules and manually add the information to the recipe.

``recipetool`` creates a "shrinkwrap" file for your recipe. Shrinkwrap
files capture the version of all dependent modules. Many packages do not
provide shrinkwrap files but ``recipetool`` will create a shrinkwrap file as it
runs.

.. note::

   A package is created for each sub-module. This policy is the only
   practical way to have the licenses for all of the dependencies
   represented in the license manifest of the image.

The ``devtool edit-recipe`` command lets you take a look at the recipe::

   $ devtool edit-recipe cute-files
   # Recipe created by recipetool
   # This is the basis of a recipe and may need further editing in order to be fully functional.
   # (Feel free to remove these comments when editing.)

   SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network."
   # WARNING: the following LICENSE and LIC_FILES_CHKSUM values are best guesses - it is
   # your responsibility to verify that the values are complete and correct.
   #
   # NOTE: multiple licenses have been detected; they have been separated with &
   # in the LICENSE value for now since it is a reasonable assumption that all
   # of the licenses apply. If instead there is a choice between the multiple
   # licenses then you should change the value to separate the licenses with |
   # instead of &. If there is any doubt, check the accompanying documentation
   # to determine which situation is applicable.

   SUMMARY = "Turn any folder on your computer into a cute file browser, available on the local network."
   LICENSE = "BSD-3-Clause & ISC & MIT"
   LIC_FILES_CHKSUM = "file://LICENSE;md5=71d98c0a1db42956787b1909c74a86ca \
                       file://node_modules/accepts/LICENSE;md5=bf1f9ad1e2e1d507aef4883fff7103de \
                       file://node_modules/array-flatten/LICENSE;md5=44088ba57cb871a58add36ce51b8de08 \
   ...
                       file://node_modules/cookie-signature/Readme.md;md5=57ae8b42de3dd0c1f22d5f4cf191e15a"

   SRC_URI = " \
       npm://registry.npmjs.org/;package=cute-files;version=${PV} \
       npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
       "

   S = "${WORKDIR}/npm"

   inherit npm

   LICENSE:${PN} = "MIT"
   LICENSE:${PN}-accepts = "MIT"
   LICENSE:${PN}-array-flatten = "MIT"
   ...
   LICENSE:${PN}-vary = "MIT"

Three key points in the previous example are:

-  :term:`SRC_URI` uses the NPM
   scheme so that the NPM fetcher is used.

-  ``recipetool`` collects all the license information. If a
   sub-module's license is unavailable, the sub-module's name appears in
   the comments.

-  The ``inherit npm`` statement causes the :ref:`ref-classes-npm` class to
   package up all the modules.

You can run the following command to build the ``cute-files`` package::

   $ devtool build cute-files

Remember that ``nodejs`` must be installed on
the target before your package.

Assuming 192.168.7.2 for the target's IP address, use the following
command to deploy your package::

   $ devtool deploy-target -s cute-files root@192.168.7.2

Once the package is installed on the target, you can
test the application to show the contents of any directory::

  $ cd /usr/lib/node_modules/cute-files
  $ cute-files

On a browser,
go to ``http://192.168.7.2:3000`` and you see the following:

.. image:: figures/cute-files-npm-example.png
   :width: 100%

You can find the recipe in ``workspace/recipes/cute-files``. You can use
the recipe in any layer you choose.

Using the NPM Projects Code Method
----------------------------------

Although it is useful to package modules already in the NPM registry,
adding ``node.js`` projects under development is a more common developer
use case.

This section covers the NPM projects code method, which is very similar
to the "registry" approach described in the previous section. In the NPM
projects method, you provide ``devtool`` with an URL that points to the
source files.

Replicating the same example, (i.e. ``cute-files``) use the following
command::

   $ devtool add https://github.com/martinaglv/cute-files.git

The recipe this command generates is very similar to the recipe created in
the previous section. However, the :term:`SRC_URI` looks like the following::

   SRC_URI = " \
       git://github.com/martinaglv/cute-files.git;protocol=https;branch=master \
       npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \
       "

In this example,
the main module is taken from the Git repository and dependencies are
taken from the NPM registry. Other than those differences, the recipe is
basically the same between the two methods. You can build and deploy the
package exactly as described in the previous section that uses the
registry modules method.

Adding custom metadata to packages
==================================

The variable
:term:`PACKAGE_ADD_METADATA`
can be used to add additional metadata to packages. This is reflected in
the package control/spec file. To take the ipk format for example, the
CONTROL file stored inside would contain the additional metadata as
additional lines.

The variable can be used in multiple ways, including using suffixes to
set it for a specific package type and/or package. Note that the order
of precedence is the same as this list:

-  ``PACKAGE_ADD_METADATA_<PKGTYPE>:<PN>``

-  ``PACKAGE_ADD_METADATA_<PKGTYPE>``

-  ``PACKAGE_ADD_METADATA:<PN>``

-  :term:`PACKAGE_ADD_METADATA`

`<PKGTYPE>` is a parameter and expected to be a distinct name of specific
package type:

-  IPK for .ipk packages

-  DEB for .deb packages

-  RPM for .rpm packages

`<PN>` is a parameter and expected to be a package name.

The variable can contain multiple [one-line] metadata fields separated
by the literal sequence '\\n'. The separator can be redefined using the
variable flag ``separator``.

Here is an example that adds two custom fields for ipk
packages::

   PACKAGE_ADD_METADATA_IPK = "Vendor: CustomIpk\nGroup:Applications/Spreadsheets"