diff options
Diffstat (limited to 'bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst')
| -rw-r--r-- | bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst | 151 |
1 files changed, 89 insertions, 62 deletions
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst index 14c342a6a7..d58fbb32ea 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst @@ -40,7 +40,7 @@ the BitBake command and its options, see ":ref:`The BitBake Command the number of processors, which takes into account hyper-threading. Thus, a quad-core build host with hyper-threading most likely shows eight processors, which is the value you would then assign to - ``BB_NUMBER_THREADS``. + :term:`BB_NUMBER_THREADS`. A possibly simpler solution is that some Linux distributions (e.g. Debian and Ubuntu) provide the ``ncpus`` command. @@ -65,13 +65,13 @@ data itself is of various types: The ``layer.conf`` files are used to construct key variables such as :term:`BBPATH` and :term:`BBFILES`. -``BBPATH`` is used to search for configuration and class files under the -``conf`` and ``classes`` directories, respectively. ``BBFILES`` is used +:term:`BBPATH` is used to search for configuration and class files under the +``conf`` and ``classes`` directories, respectively. :term:`BBFILES` is used to locate both recipe and recipe append files (``.bb`` and ``.bbappend``). If there is no ``bblayers.conf`` file, it is assumed the -user has set the ``BBPATH`` and ``BBFILES`` directly in the environment. +user has set the :term:`BBPATH` and :term:`BBFILES` directly in the environment. -Next, the ``bitbake.conf`` file is located using the ``BBPATH`` variable +Next, the ``bitbake.conf`` file is located using the :term:`BBPATH` variable that was just constructed. The ``bitbake.conf`` file may also include other configuration files using the ``include`` or ``require`` directives. @@ -79,8 +79,8 @@ directives. Prior to parsing configuration files, BitBake looks at certain variables, including: -- :term:`BB_ENV_WHITELIST` -- :term:`BB_ENV_EXTRAWHITE` +- :term:`BB_ENV_PASSTHROUGH` +- :term:`BB_ENV_PASSTHROUGH_ADDITIONS` - :term:`BB_PRESERVE_ENV` - :term:`BB_ORIGENV` - :term:`BITBAKE_UI` @@ -104,7 +104,7 @@ BitBake first searches the current working directory for an optional contain a :term:`BBLAYERS` variable that is a space-delimited list of 'layer' directories. Recall that if BitBake cannot find a ``bblayers.conf`` file, then it is assumed the user has -set the ``BBPATH`` and ``BBFILES`` variables directly in the +set the :term:`BBPATH` and :term:`BBFILES` variables directly in the environment. For each directory (layer) in this list, a ``conf/layer.conf`` file is @@ -114,7 +114,7 @@ files automatically set up :term:`BBPATH` and other variables correctly for a given build directory. BitBake then expects to find the ``conf/bitbake.conf`` file somewhere in -the user-specified ``BBPATH``. That configuration file generally has +the user-specified :term:`BBPATH`. That configuration file generally has include directives to pull in any other metadata such as files specific to the architecture, the machine, the local environment, and so forth. @@ -135,7 +135,7 @@ The ``base.bbclass`` file is always included. Other classes that are specified in the configuration using the :term:`INHERIT` variable are also included. BitBake searches for class files in a ``classes`` subdirectory under the paths -in ``BBPATH`` in the same way as configuration files. +in :term:`BBPATH` in the same way as configuration files. A good way to get an idea of the configuration files and the class files used in your execution environment is to run the following BitBake @@ -184,13 +184,13 @@ Locating and Parsing Recipes During the configuration phase, BitBake will have set :term:`BBFILES`. BitBake now uses it to construct a list of recipes to parse, along with any append files (``.bbappend``) to -apply. ``BBFILES`` is a space-separated list of available files and +apply. :term:`BBFILES` is a space-separated list of available files and supports wildcards. An example would be:: BBFILES = "/path/to/bbfiles/*.bb /path/to/appends/*.bbappend" BitBake parses each -recipe and append file located with ``BBFILES`` and stores the values of +recipe and append file located with :term:`BBFILES` and stores the values of various variables into the datastore. .. note:: @@ -201,18 +201,18 @@ For each file, a fresh copy of the base configuration is made, then the recipe is parsed line by line. Any inherit statements cause BitBake to find and then parse class files (``.bbclass``) using :term:`BBPATH` as the search path. Finally, BitBake -parses in order any append files found in ``BBFILES``. +parses in order any append files found in :term:`BBFILES`. One common convention is to use the recipe filename to define pieces of metadata. For example, in ``bitbake.conf`` the recipe name and version are used to set the variables :term:`PN` and :term:`PV`:: - PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}" - PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}" + PN = "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}" + PV = "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}" In this example, a recipe called "something_1.2.3.bb" would set -``PN`` to "something" and ``PV`` to "1.2.3". +:term:`PN` to "something" and :term:`PV` to "1.2.3". By the time parsing is complete for a recipe, BitBake has a list of tasks that the recipe defines and a set of data consisting of keys and @@ -228,7 +228,7 @@ and then reload it. Where possible, subsequent BitBake commands reuse this cache of recipe information. The validity of this cache is determined by first computing a checksum of the base configuration data (see -:term:`BB_HASHCONFIG_WHITELIST`) and +:term:`BB_HASHCONFIG_IGNORE_VARS`) and then checking if the checksum matches. If that checksum matches what is in the cache and the recipe and class files have not changed, BitBake is able to use the cache. BitBake then reloads the cached information about @@ -260,21 +260,21 @@ Providers Assuming BitBake has been instructed to execute a target and that all the recipe files have been parsed, BitBake starts to figure out how to -build the target. BitBake looks through the ``PROVIDES`` list for each -of the recipes. A ``PROVIDES`` list is the list of names by which the -recipe can be known. Each recipe's ``PROVIDES`` list is created +build the target. BitBake looks through the :term:`PROVIDES` list for each +of the recipes. A :term:`PROVIDES` list is the list of names by which the +recipe can be known. Each recipe's :term:`PROVIDES` list is created implicitly through the recipe's :term:`PN` variable and explicitly through the recipe's :term:`PROVIDES` variable, which is optional. -When a recipe uses ``PROVIDES``, that recipe's functionality can be -found under an alternative name or names other than the implicit ``PN`` +When a recipe uses :term:`PROVIDES`, that recipe's functionality can be +found under an alternative name or names other than the implicit :term:`PN` name. As an example, suppose a recipe named ``keyboard_1.0.bb`` contained the following:: PROVIDES += "fullkeyboard" -The ``PROVIDES`` +The :term:`PROVIDES` list for this recipe becomes "keyboard", which is implicit, and "fullkeyboard", which is explicit. Consequently, the functionality found in ``keyboard_1.0.bb`` can be found under two different names. @@ -284,12 +284,12 @@ in ``keyboard_1.0.bb`` can be found under two different names. Preferences =========== -The ``PROVIDES`` list is only part of the solution for figuring out a +The :term:`PROVIDES` list is only part of the solution for figuring out a target's recipes. Because targets might have multiple providers, BitBake needs to prioritize providers by determining provider preferences. A common example in which a target has multiple providers is -"virtual/kernel", which is on the ``PROVIDES`` list for each kernel +"virtual/kernel", which is on the :term:`PROVIDES` list for each kernel recipe. Each machine often selects the best kernel provider by using a line similar to the following in the machine configuration file:: @@ -309,10 +309,10 @@ specify a particular version. You can influence the order by using the :term:`DEFAULT_PREFERENCE` variable. By default, files have a preference of "0". Setting -``DEFAULT_PREFERENCE`` to "-1" makes the recipe unlikely to be used -unless it is explicitly referenced. Setting ``DEFAULT_PREFERENCE`` to -"1" makes it likely the recipe is used. ``PREFERRED_VERSION`` overrides -any ``DEFAULT_PREFERENCE`` setting. ``DEFAULT_PREFERENCE`` is often used +:term:`DEFAULT_PREFERENCE` to "-1" makes the recipe unlikely to be used +unless it is explicitly referenced. Setting :term:`DEFAULT_PREFERENCE` to +"1" makes it likely the recipe is used. :term:`PREFERRED_VERSION` overrides +any :term:`DEFAULT_PREFERENCE` setting. :term:`DEFAULT_PREFERENCE` is often used to mark newer and more experimental recipe versions until they have undergone sufficient testing to be considered stable. @@ -394,7 +394,7 @@ ready to run, those tasks have all their dependencies met, and the thread threshold has not been exceeded. It is worth noting that you can greatly speed up the build time by -properly setting the ``BB_NUMBER_THREADS`` variable. +properly setting the :term:`BB_NUMBER_THREADS` variable. As each task completes, a timestamp is written to the directory specified by the :term:`STAMP` variable. On subsequent @@ -435,7 +435,7 @@ BitBake writes a shell script to executes the script. The generated shell script contains all the exported variables, and the shell functions with all variables expanded. Output from the shell script goes to the file -``${T}/log.do_taskname.pid``. Looking at the expanded shell functions in +``${``\ :term:`T`\ ``}/log.do_taskname.pid``. Looking at the expanded shell functions in the run file and the output in the log files is a useful debugging technique. @@ -477,7 +477,7 @@ changes because it should not affect the output for target packages. The simplistic approach for excluding the working directory is to set it to some fixed value and create the checksum for the "run" script. BitBake goes one step better and uses the -:term:`BB_HASHBASE_WHITELIST` variable +:term:`BB_BASEHASH_IGNORE_VARS` variable to define a list of variables that should never be included when generating the signatures. @@ -523,7 +523,7 @@ it cannot figure out dependencies. Thus far, this section has limited discussion to the direct inputs into a task. Information based on direct inputs is referred to as the "basehash" in the code. However, there is still the question of a task's -indirect inputs - the things that were already built and present in the +indirect inputs --- the things that were already built and present in the build directory. The checksum (or signature) for a particular task needs to add the hashes of all the tasks on which the particular task depends. Choosing which dependencies to add is a policy decision. However, the @@ -534,11 +534,11 @@ At the code level, there are a variety of ways both the basehash and the dependent task hashes can be influenced. Within the BitBake configuration file, we can give BitBake some extra information to help it construct the basehash. The following statement effectively results -in a list of global variable dependency excludes - variables never +in a list of global variable dependency excludes --- variables never included in any checksum. This example uses variables from OpenEmbedded to help illustrate the concept:: - BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \ + BB_BASEHASH_IGNORE_VARS ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR \ SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL \ USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \ PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \ @@ -552,8 +552,8 @@ through dependency chains are more complex and are generally accomplished with a Python function. The code in ``meta/lib/oe/sstatesig.py`` shows two examples of this and also illustrates how you can insert your own policy into the system if so -desired. This file defines the two basic signature generators -OpenEmbedded-Core uses: "OEBasic" and "OEBasicHash". By default, there +desired. This file defines the basic signature generator +OpenEmbedded-Core uses: "OEBasicHash". By default, there is a dummy "noop" signature handler enabled in BitBake. This means that behavior is unchanged from previous versions. ``OE-Core`` uses the "OEBasicHash" signature handler by default through this setting in the @@ -561,14 +561,13 @@ behavior is unchanged from previous versions. ``OE-Core`` uses the BB_SIGNATURE_HANDLER ?= "OEBasicHash" -The "OEBasicHash" ``BB_SIGNATURE_HANDLER`` is the same as the "OEBasic" -version but adds the task hash to the stamp files. This results in any -metadata change that changes the task hash, automatically causing the -task to be run again. This removes the need to bump -:term:`PR` values, and changes to metadata automatically -ripple across the build. +The main feature of the "OEBasicHash" :term:`BB_SIGNATURE_HANDLER` is that +it adds the task hash to the stamp files. Thanks to this, any metadata +change will change the task hash, automatically causing the task to be run +again. This removes the need to bump :term:`PR` values, and changes to +metadata automatically ripple across the build. -It is also worth noting that the end result of these signature +It is also worth noting that the end result of signature generators is to make some dependency and hash information available to the build. This information includes: @@ -578,10 +577,7 @@ the build. This information includes: - ``BB_BASEHASH_``\ *filename:taskname*: The base hashes for each dependent task. -- ``BBHASHDEPS_``\ *filename:taskname*: The task dependencies for - each task. - -- ``BB_TASKHASH``: The hash of the currently running task. +- :term:`BB_TASKHASH`: The hash of the currently running task. It is worth noting that BitBake's "-S" option lets you debug BitBake's processing of signatures. The options passed to -S allow different @@ -590,10 +586,11 @@ or possibly those defined in the metadata/signature handler itself. The simplest parameter to pass is "none", which causes a set of signature information to be written out into ``STAMPS_DIR`` corresponding to the targets specified. The other currently available parameter is -"printdiff", which causes BitBake to try to establish the closest +"printdiff", which causes BitBake to try to establish the most recent signature match it can (e.g. in the sstate cache) and then run -``bitbake-diffsigs`` over the matches to determine the stamps and delta -where these two stamp trees diverge. +compare the matched signatures to determine the stamps and delta +where these two stamp trees diverge. This can be used to determine why +tasks need to be re-run in situations where that is not expected. .. note:: @@ -648,13 +645,6 @@ compiled binary. To handle this, BitBake calls the each successful setscene task to know whether or not it needs to obtain the dependencies of that task. -Finally, after all the setscene tasks have executed, BitBake calls the -function listed in -:term:`BB_SETSCENE_VERIFY_FUNCTION2` -with the list of tasks BitBake thinks has been "covered". The metadata -can then ensure that this list is correct and can inform BitBake that it -wants specific tasks to be run regardless of the setscene result. - You can find more information on setscene metadata in the :ref:`bitbake-user-manual/bitbake-user-manual-metadata:task checksums and setscene` section. @@ -667,7 +657,7 @@ builds are when execute, bitbake also supports user defined configuration of the `Python logging <https://docs.python.org/3/library/logging.html>`__ facilities through the :term:`BB_LOGCONFIG` variable. This -variable defines a json or yaml `logging +variable defines a JSON or YAML `logging configuration <https://docs.python.org/3/library/logging.config.html>`__ that will be intelligently merged into the default configuration. The logging configuration is merged using the following rules: @@ -701,9 +691,9 @@ logging configuration is merged using the following rules: adds a filter called ``BitBake.defaultFilter``, both filters will be applied to the logger -As an example, consider the following user logging configuration file -which logs all Hash Equivalence related messages of VERBOSE or higher to -a file called ``hashequiv.log`` :: +As a first example, you can create a ``hashequiv.json`` user logging +configuration file to log all Hash Equivalence related messages of ``VERBOSE`` +or higher priority to a file called ``hashequiv.log``:: { "version": 1, @@ -732,3 +722,40 @@ a file called ``hashequiv.log`` :: } } } + +Then set the :term:`BB_LOGCONFIG` variable in ``conf/local.conf``:: + + BB_LOGCONFIG = "hashequiv.json" + +Another example is this ``warn.json`` file to log all ``WARNING`` and +higher priority messages to a ``warn.log`` file:: + + { + "version": 1, + "formatters": { + "warnlogFormatter": { + "()": "bb.msg.BBLogFormatter", + "format": "%(levelname)s: %(message)s" + } + }, + + "handlers": { + "warnlog": { + "class": "logging.FileHandler", + "formatter": "warnlogFormatter", + "level": "WARNING", + "filename": "warn.log" + } + }, + + "loggers": { + "BitBake": { + "handlers": ["warnlog"] + } + }, + + "@disable_existing_loggers": false + } + +Note that BitBake's helper classes for structured logging are implemented in +``lib/bb/msg.py``. |