diff options
Diffstat (limited to 'bitbake/doc')
| -rw-r--r-- | bitbake/doc/README | 4 | ||||
| -rw-r--r-- | bitbake/doc/_templates/footer.html | 9 | ||||
| -rw-r--r-- | bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst | 69 | ||||
| -rw-r--r-- | bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst | 53 | ||||
| -rw-r--r-- | bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.rst | 137 | ||||
| -rw-r--r-- | bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst | 85 | ||||
| -rw-r--r-- | bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables-context.rst | 91 | ||||
| -rw-r--r-- | bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst | 129 | ||||
| -rw-r--r-- | bitbake/doc/index.rst | 1 | ||||
| -rw-r--r-- | bitbake/doc/releases.rst | 82 |
10 files changed, 516 insertions, 144 deletions
diff --git a/bitbake/doc/README b/bitbake/doc/README index cdbb23776e..d4f56afa37 100644 --- a/bitbake/doc/README +++ b/bitbake/doc/README @@ -47,8 +47,8 @@ To install all required packages run: To build the documentation locally, run: - $ cd documentation - $ make -f Makefile.sphinx html + $ cd doc + $ make html The resulting HTML index page will be _build/html/index.html, and you can browse your own copy of the locally generated documentation with diff --git a/bitbake/doc/_templates/footer.html b/bitbake/doc/_templates/footer.html new file mode 100644 index 0000000000..1398f20d7e --- /dev/null +++ b/bitbake/doc/_templates/footer.html @@ -0,0 +1,9 @@ +<footer> + <hr/> + <div role="contentinfo"> + <p>© Copyright {{ copyright }} + <br>Last updated on {{ last_updated }} from the <a href="https://git.openembedded.org/bitbake/">bitbake</a> git repository. + </p> + </div> +</footer> + diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst index 7a22e96edf..d58fbb32ea 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.rst @@ -552,8 +552,8 @@ through dependency chains are more complex and are generally accomplished with a Python function. The code in ``meta/lib/oe/sstatesig.py`` shows two examples of this and also illustrates how you can insert your own policy into the system if so -desired. This file defines the two basic signature generators -OpenEmbedded-Core uses: "OEBasic" and "OEBasicHash". By default, there +desired. This file defines the basic signature generator +OpenEmbedded-Core uses: "OEBasicHash". By default, there is a dummy "noop" signature handler enabled in BitBake. This means that behavior is unchanged from previous versions. ``OE-Core`` uses the "OEBasicHash" signature handler by default through this setting in the @@ -561,14 +561,13 @@ behavior is unchanged from previous versions. ``OE-Core`` uses the BB_SIGNATURE_HANDLER ?= "OEBasicHash" -The "OEBasicHash" :term:`BB_SIGNATURE_HANDLER` is the same as the "OEBasic" -version but adds the task hash to the stamp files. This results in any -metadata change that changes the task hash, automatically causing the -task to be run again. This removes the need to bump -:term:`PR` values, and changes to metadata automatically -ripple across the build. +The main feature of the "OEBasicHash" :term:`BB_SIGNATURE_HANDLER` is that +it adds the task hash to the stamp files. Thanks to this, any metadata +change will change the task hash, automatically causing the task to be run +again. This removes the need to bump :term:`PR` values, and changes to +metadata automatically ripple across the build. -It is also worth noting that the end result of these signature +It is also worth noting that the end result of signature generators is to make some dependency and hash information available to the build. This information includes: @@ -587,10 +586,11 @@ or possibly those defined in the metadata/signature handler itself. The simplest parameter to pass is "none", which causes a set of signature information to be written out into ``STAMPS_DIR`` corresponding to the targets specified. The other currently available parameter is -"printdiff", which causes BitBake to try to establish the closest +"printdiff", which causes BitBake to try to establish the most recent signature match it can (e.g. in the sstate cache) and then run -``bitbake-diffsigs`` over the matches to determine the stamps and delta -where these two stamp trees diverge. +compare the matched signatures to determine the stamps and delta +where these two stamp trees diverge. This can be used to determine why +tasks need to be re-run in situations where that is not expected. .. note:: @@ -657,7 +657,7 @@ builds are when execute, bitbake also supports user defined configuration of the `Python logging <https://docs.python.org/3/library/logging.html>`__ facilities through the :term:`BB_LOGCONFIG` variable. This -variable defines a json or yaml `logging +variable defines a JSON or YAML `logging configuration <https://docs.python.org/3/library/logging.config.html>`__ that will be intelligently merged into the default configuration. The logging configuration is merged using the following rules: @@ -691,9 +691,9 @@ logging configuration is merged using the following rules: adds a filter called ``BitBake.defaultFilter``, both filters will be applied to the logger -As an example, consider the following user logging configuration file -which logs all Hash Equivalence related messages of VERBOSE or higher to -a file called ``hashequiv.log`` :: +As a first example, you can create a ``hashequiv.json`` user logging +configuration file to log all Hash Equivalence related messages of ``VERBOSE`` +or higher priority to a file called ``hashequiv.log``:: { "version": 1, @@ -722,3 +722,40 @@ a file called ``hashequiv.log`` :: } } } + +Then set the :term:`BB_LOGCONFIG` variable in ``conf/local.conf``:: + + BB_LOGCONFIG = "hashequiv.json" + +Another example is this ``warn.json`` file to log all ``WARNING`` and +higher priority messages to a ``warn.log`` file:: + + { + "version": 1, + "formatters": { + "warnlogFormatter": { + "()": "bb.msg.BBLogFormatter", + "format": "%(levelname)s: %(message)s" + } + }, + + "handlers": { + "warnlog": { + "class": "logging.FileHandler", + "formatter": "warnlogFormatter", + "level": "WARNING", + "filename": "warn.log" + } + }, + + "loggers": { + "BitBake": { + "handlers": ["warnlog"] + } + }, + + "@disable_existing_loggers": false + } + +Note that BitBake's helper classes for structured logging are implemented in +``lib/bb/msg.py``. diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst index 9c269ca837..fb4f0a23d7 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst @@ -424,8 +424,8 @@ This fetcher supports the following parameters: - *"nobranch":* Tells the fetcher to not check the SHA validation for the branch when set to "1". The default is "0". Set this option for - the recipe that refers to the commit that is valid for a tag instead - of the branch. + the recipe that refers to the commit that is valid for any namespace + (branch, tag, ...) instead of the branch. - *"bareclone":* Tells the fetcher to clone a bare clone into the destination directory without checking out a working tree. Only the @@ -476,6 +476,14 @@ Here are some example URLs:: easy to share metadata without removing passwords. SSH keys, ``~/.netrc`` and ``~/.ssh/config`` files can be used as alternatives. +Using tags with the git fetcher may cause surprising behaviour. Bitbake needs to +resolve the tag to a specific revision and to do that, it has to connect to and use +the upstream repository. This is because the revision the tags point at can change and +we've seen cases of this happening in well known public repositories. This can mean +many more network connections than expected and recipes may be reparsed at every build. +Source mirrors will also be bypassed as the upstream repository is the only source +of truth to resolve the revision accurately. For these reasons, whilst the fetcher +can support tags, we recommend being specific about revisions in recipes. .. _gitsm-fetcher: @@ -688,6 +696,41 @@ Here is an example URL:: It can also be used when setting mirrors definitions using the :term:`PREMIRRORS` variable. +.. _gcp-fetcher: + +GCP Fetcher (``gs://``) +-------------------------- + +This submodule fetches data from a +`Google Cloud Storage Bucket <https://cloud.google.com/storage/docs/buckets>`__. +It uses the `Google Cloud Storage Python Client <https://cloud.google.com/python/docs/reference/storage/latest>`__ +to check the status of objects in the bucket and download them. +The use of the Python client makes it substantially faster than using command +line tools such as gsutil. + +The fetcher requires the Google Cloud Storage Python Client to be installed, along +with the gsutil tool. + +The fetcher requires that the machine has valid credentials for accessing the +chosen bucket. Instructions for authentication can be found in the +`Google Cloud documentation <https://cloud.google.com/docs/authentication/provide-credentials-adc#local-dev>`__. + +If it used from the OpenEmbedded build system, the fetcher can be used for +fetching sstate artifacts from a GCS bucket by specifying the +``SSTATE_MIRRORS`` variable as shown below:: + + SSTATE_MIRRORS ?= "\ + file://.* gs://<bucket name>/PATH \ + " + +The fetcher can also be used in recipes:: + + SRC_URI = "gs://<bucket name>/<foo_container>/<bar_file>" + +However, the checksum of the file should be also be provided:: + + SRC_URI[sha256sum] = "<sha256 string>" + .. _crate-fetcher: Crate Fetcher (``crate://``) @@ -740,7 +783,7 @@ Here is an example URL with both fetchers:: " See :yocto_docs:`Creating Node Package Manager (NPM) Packages -</dev-manual/common-tasks.html#creating-node-package-manager-npm-packages>` +</dev-manual/packages.html#creating-node-package-manager-npm-packages>` in the Yocto Project manual for details about using :yocto_docs:`devtool <https://docs.yoctoproject.org/ref-manual/devtool-reference.html>` to automatically create a recipe from an NPM URL. @@ -777,7 +820,7 @@ the package which has such dependencies, for example:: Such a file can automatically be generated using :yocto_docs:`devtool <https://docs.yoctoproject.org/ref-manual/devtool-reference.html>` as described in the :yocto_docs:`Creating Node Package Manager (NPM) Packages -</dev-manual/common-tasks.html#creating-node-package-manager-npm-packages>` +</dev-manual/packages.html#creating-node-package-manager-npm-packages>` section of the Yocto Project. Other Fetchers @@ -791,6 +834,8 @@ Fetch submodules also exist for the following: - OSC (``osc://``) +- S3 (``s3://``) + - Secure FTP (``sftp://``) - Secure Shell (``ssh://``) diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.rst index 722dc5a2cc..654196ca24 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.rst +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-hello.rst @@ -18,28 +18,32 @@ it. Obtaining BitBake ================= -See the :ref:`bitbake-user-manual/bitbake-user-manual-hello:obtaining bitbake` section for +See the :ref:`bitbake-user-manual/bitbake-user-manual-intro:obtaining bitbake` section for information on how to obtain BitBake. Once you have the source code on your machine, the BitBake directory appears as follows:: $ ls -al - total 100 - drwxrwxr-x. 9 wmat wmat 4096 Jan 31 13:44 . - drwxrwxr-x. 3 wmat wmat 4096 Feb 4 10:45 .. - -rw-rw-r--. 1 wmat wmat 365 Nov 26 04:55 AUTHORS - drwxrwxr-x. 2 wmat wmat 4096 Nov 26 04:55 bin - drwxrwxr-x. 4 wmat wmat 4096 Jan 31 13:44 build - -rw-rw-r--. 1 wmat wmat 16501 Nov 26 04:55 ChangeLog - drwxrwxr-x. 2 wmat wmat 4096 Nov 26 04:55 classes - drwxrwxr-x. 2 wmat wmat 4096 Nov 26 04:55 conf - drwxrwxr-x. 3 wmat wmat 4096 Nov 26 04:55 contrib - -rw-rw-r--. 1 wmat wmat 17987 Nov 26 04:55 COPYING - drwxrwxr-x. 3 wmat wmat 4096 Nov 26 04:55 doc - -rw-rw-r--. 1 wmat wmat 69 Nov 26 04:55 .gitignore - -rw-rw-r--. 1 wmat wmat 849 Nov 26 04:55 HEADER - drwxrwxr-x. 5 wmat wmat 4096 Jan 31 13:44 lib - -rw-rw-r--. 1 wmat wmat 195 Nov 26 04:55 MANIFEST.in - -rw-rw-r--. 1 wmat wmat 2887 Nov 26 04:55 TODO + total 108 + drwxr-xr-x 9 fawkh 10000 4096 feb 24 12:10 . + drwx------ 36 fawkh 10000 4096 mar 2 17:00 .. + -rw-r--r-- 1 fawkh 10000 365 feb 24 12:10 AUTHORS + drwxr-xr-x 2 fawkh 10000 4096 feb 24 12:10 bin + -rw-r--r-- 1 fawkh 10000 16501 feb 24 12:10 ChangeLog + drwxr-xr-x 2 fawkh 10000 4096 feb 24 12:10 classes + drwxr-xr-x 2 fawkh 10000 4096 feb 24 12:10 conf + drwxr-xr-x 5 fawkh 10000 4096 feb 24 12:10 contrib + drwxr-xr-x 6 fawkh 10000 4096 feb 24 12:10 doc + drwxr-xr-x 8 fawkh 10000 4096 mar 2 16:26 .git + -rw-r--r-- 1 fawkh 10000 31 feb 24 12:10 .gitattributes + -rw-r--r-- 1 fawkh 10000 392 feb 24 12:10 .gitignore + drwxr-xr-x 13 fawkh 10000 4096 feb 24 12:11 lib + -rw-r--r-- 1 fawkh 10000 1224 feb 24 12:10 LICENSE + -rw-r--r-- 1 fawkh 10000 15394 feb 24 12:10 LICENSE.GPL-2.0-only + -rw-r--r-- 1 fawkh 10000 1286 feb 24 12:10 LICENSE.MIT + -rw-r--r-- 1 fawkh 10000 229 feb 24 12:10 MANIFEST.in + -rw-r--r-- 1 fawkh 10000 2413 feb 24 12:10 README + -rw-r--r-- 1 fawkh 10000 43 feb 24 12:10 toaster-requirements.txt + -rw-r--r-- 1 fawkh 10000 2887 feb 24 12:10 TODO At this point, you should have BitBake cloned to a directory that matches the previous listing except for dates and user names. @@ -52,7 +56,7 @@ directory to where your local BitBake files are and run the following command:: $ ./bin/bitbake --version - BitBake Build Tool Core version 1.23.0, bitbake version 1.23.0 + BitBake Build Tool Core version 2.3.1 The console output tells you what version you are running. @@ -130,23 +134,8 @@ Following is the complete "Hello World" example. directory. Run the ``bitbake`` command and see what it does:: $ bitbake - The BBPATH variable is not set and bitbake did not - find a conf/bblayers.conf file in the expected location. + ERROR: The BBPATH variable is not set and bitbake did not find a conf/bblayers.conf file in the expected location. Maybe you accidentally invoked bitbake from the wrong directory? - DEBUG: Removed the following variables from the environment: - GNOME_DESKTOP_SESSION_ID, XDG_CURRENT_DESKTOP, - GNOME_KEYRING_CONTROL, DISPLAY, SSH_AGENT_PID, LANG, no_proxy, - XDG_SESSION_PATH, XAUTHORITY, SESSION_MANAGER, SHLVL, - MANDATORY_PATH, COMPIZ_CONFIG_PROFILE, WINDOWID, EDITOR, - GPG_AGENT_INFO, SSH_AUTH_SOCK, GDMSESSION, GNOME_KEYRING_PID, - XDG_SEAT_PATH, XDG_CONFIG_DIRS, LESSOPEN, DBUS_SESSION_BUS_ADDRESS, - _, XDG_SESSION_COOKIE, DESKTOP_SESSION, LESSCLOSE, DEFAULTS_PATH, - UBUNTU_MENUPROXY, OLDPWD, XDG_DATA_DIRS, COLORTERM, LS_COLORS - - The majority of this output is specific to environment variables that - are not directly relevant to BitBake. However, the very first - message regarding the :term:`BBPATH` variable and the - ``conf/bblayers.conf`` file is relevant. When you run BitBake, it begins looking for metadata files. The :term:`BBPATH` variable is what tells BitBake where @@ -179,20 +168,14 @@ Following is the complete "Hello World" example. ``bitbake`` command again:: $ bitbake - ERROR: Traceback (most recent call last): - File "/home/scott-lenovo/bitbake/lib/bb/cookerdata.py", line 163, in wrapped - return func(fn, *args) - File "/home/scott-lenovo/bitbake/lib/bb/cookerdata.py", line 173, in parse_config_file - return bb.parse.handle(fn, data, include) - File "/home/scott-lenovo/bitbake/lib/bb/parse/__init__.py", line 99, in handle - return h['handle'](fn, data, include) - File "/home/scott-lenovo/bitbake/lib/bb/parse/parse_py/ConfHandler.py", line 120, in handle - abs_fn = resolve_file(fn, data) - File "/home/scott-lenovo/bitbake/lib/bb/parse/__init__.py", line 117, in resolve_file - raise IOError("file %s not found in %s" % (fn, bbpath)) - IOError: file conf/bitbake.conf not found in /home/scott-lenovo/hello - - ERROR: Unable to parse conf/bitbake.conf: file conf/bitbake.conf not found in /home/scott-lenovo/hello + ERROR: Unable to parse /home/scott-lenovo/bitbake/lib/bb/parse/__init__.py + Traceback (most recent call last): + File "/home/scott-lenovo/bitbake/lib/bb/parse/__init__.py", line 127, in resolve_file(fn='conf/bitbake.conf', d=<bb.data_smart.DataSmart object at 0x7f22919a3df0>): + if not newfn: + > raise IOError(errno.ENOENT, "file %s not found in %s" % (fn, bbpath)) + fn = newfn + FileNotFoundError: [Errno 2] file conf/bitbake.conf not found in <projectdirectory> + This sample output shows that BitBake could not find the ``conf/bitbake.conf`` file in the project directory. This file is @@ -226,12 +209,12 @@ Following is the complete "Hello World" example. .. note:: - Without a value for PN , the variables STAMP , T , and B , prevent more - than one recipe from working. You can fix this by either setting PN to + Without a value for :term:`PN`, the variables :term:`STAMP`, :term:`T`, and :term:`B`, prevent more + than one recipe from working. You can fix this by either setting :term:`PN` to have a value similar to what OpenEmbedded and BitBake use in the default - bitbake.conf file (see previous example). Or, by manually updating each - recipe to set PN . You will also need to include PN as part of the STAMP - , T , and B variable definitions in the local.conf file. + ``bitbake.conf`` file (see previous example). Or, by manually updating each + recipe to set :term:`PN`. You will also need to include :term:`PN` as part of the :term:`STAMP`, + :term:`T`, and :term:`B` variable definitions in the ``local.conf`` file. The ``TMPDIR`` variable establishes a directory that BitBake uses for build output and intermediate files other than the cached @@ -254,18 +237,14 @@ Following is the complete "Hello World" example. exists, you can run the ``bitbake`` command again:: $ bitbake - ERROR: Traceback (most recent call last): - File "/home/scott-lenovo/bitbake/lib/bb/cookerdata.py", line 163, in wrapped - return func(fn, *args) - File "/home/scott-lenovo/bitbake/lib/bb/cookerdata.py", line 177, in _inherit - bb.parse.BBHandler.inherit(bbclass, "configuration INHERITs", 0, data) - File "/home/scott-lenovo/bitbake/lib/bb/parse/parse_py/BBHandler.py", line 92, in inherit - include(fn, file, lineno, d, "inherit") - File "/home/scott-lenovo/bitbake/lib/bb/parse/parse_py/ConfHandler.py", line 100, in include - raise ParseError("Could not %(error_out)s file %(fn)s" % vars(), oldfn, lineno) - ParseError: ParseError in configuration INHERITs: Could not inherit file classes/base.bbclass - - ERROR: Unable to parse base: ParseError in configuration INHERITs: Could not inherit file classes/base.bbclass + ERROR: Unable to parse /home/scott-lenovo/bitbake/lib/bb/parse/parse_py/BBHandler.py + Traceback (most recent call last): + File "/home/scott-lenovo/bitbake/lib/bb/parse/parse_py/BBHandler.py", line 67, in inherit(files=['base'], fn='configuration INHERITs', lineno=0, d=<bb.data_smart.DataSmart object at 0x7fab6815edf0>): + if not os.path.exists(file): + > raise ParseError("Could not inherit file %s" % (file), fn, lineno) + + bb.parse.ParseError: ParseError in configuration INHERITs: Could not inherit file classes/base.bbclass + In the sample output, BitBake could not find the ``classes/base.bbclass`` file. You need @@ -284,7 +263,10 @@ Following is the complete "Hello World" example. $ mkdir classes Move to the ``classes`` directory and then create the - ``base.bbclass`` file by inserting this single line: addtask build + ``base.bbclass`` file by inserting this single line:: + + addtask build + The minimal task that BitBake runs is the ``do_build`` task. This is all the example needs in order to build the project. Of course, the ``base.bbclass`` can have much more depending on which build @@ -328,10 +310,19 @@ Following is the complete "Hello World" example. BBFILES += "${LAYERDIR}/*.bb" BBFILE_COLLECTIONS += "mylayer" BBFILE_PATTERN_mylayer := "^${LAYERDIR_RE}/" + LAYERSERIES_CORENAMES = "hello_world_example" + LAYERSERIES_COMPAT_mylayer = "hello_world_example" For information on these variables, click on :term:`BBFILES`, - :term:`LAYERDIR`, :term:`BBFILE_COLLECTIONS` or :term:`BBFILE_PATTERN_mylayer <BBFILE_PATTERN>` - to go to the definitions in the glossary. + :term:`LAYERDIR`, :term:`BBFILE_COLLECTIONS`, :term:`BBFILE_PATTERN_mylayer <BBFILE_PATTERN>` + or :term:`LAYERSERIES_COMPAT` to go to the definitions in the glossary. + + .. note:: + + We are setting both ``LAYERSERIES_CORENAMES`` and :term:`LAYERSERIES_COMPAT` in this particular case, because we + are using bitbake without OpenEmbedded. + You should usually just use :term:`LAYERSERIES_COMPAT` to specify the OE-Core versions for which your layer + is compatible, and add the meta-openembedded layer to your project. You need to create the recipe file next. Inside your layer at the top-level, use an editor and create a recipe file named @@ -389,12 +380,14 @@ Following is the complete "Hello World" example. target:: $ bitbake printhello + Loading cache: 100% | + Loaded 0 entries from dependency cache. Parsing recipes: 100% |##################################################################################| - Time: 00:00:00 Parsing of 1 .bb files complete (0 cached, 1 parsed). 1 targets, 0 skipped, 0 masked, 0 errors. NOTE: Resolving any missing task queue dependencies - NOTE: Preparing RunQueue - NOTE: Executing RunQueue Tasks + Initialising tasks: 100% |###############################################################################| + NOTE: No setscene tasks + NOTE: Executing Tasks ******************** * * * Hello, World! * diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst index 337821612c..58975f4c88 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst @@ -319,6 +319,10 @@ The variable ``D`` becomes "dvaladditional data". You must control all spacing when you use the override syntax. +.. note:: + + The overrides are applied in this order, ":append", ":prepend", ":remove". + It is also possible to append and prepend to shell functions and BitBake-style Python functions. See the ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:shell functions`" and ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:bitbake-style python functions`" sections for examples. @@ -330,7 +334,8 @@ Removal (Override Style Syntax) You can remove values from lists using the removal override style syntax. Specifying a value for removal causes all occurrences of that -value to be removed from the variable. +value to be removed from the variable. Unlike ":append" and ":prepend", +there is no need to add a leading or trailing space to the value. When you use this syntax, BitBake expects one or more strings. Surrounding spaces and spacing are preserved. Here is an example:: @@ -351,6 +356,28 @@ The variable ``FOO`` becomes Like ":append" and ":prepend", ":remove" is applied at variable expansion time. +.. note:: + + The overrides are applied in this order, ":append", ":prepend", ":remove". + This implies it is not possible to re-append previously removed strings. + However, one can undo a ":remove" by using an intermediate variable whose + content is passed to the ":remove" so that modifying the intermediate + variable equals to keeping the string in:: + + FOOREMOVE = "123 456 789" + FOO:remove = "${FOOREMOVE}" + ... + FOOREMOVE = "123 789" + + This expands to ``FOO:remove = "123 789"``. + +.. note:: + + Override application order may not match variable parse history, i.e. + the output of ``bitbake -e`` may contain ":remove" before ":append", + but the result will be removed string, because ":remove" is handled + last. + Override Style Operation Advantages ----------------------------------- @@ -421,6 +448,12 @@ documentation to a BitBake variable as follows:: CACHE[doc] = "The directory holding the cache of the metadata." +.. note:: + + Variable flag names starting with an underscore (``_``) character + are allowed but are ignored by ``d.getVarFlags("VAR")`` + in Python code. Such flag names are used internally by BitBake. + Inline Python Variable Expansion -------------------------------- @@ -1463,12 +1496,35 @@ functionality of the task: directory listed is used as the current working directory for the task. +- ``[file-checksums]``: Controls the file dependencies for a task. The + baseline file list is the set of files associated with + :term:`SRC_URI`. May be used to set additional dependencies on + files not associated with :term:`SRC_URI`. + + The value set to the list is a file-boolean pair where the first + value is the file name and the second is whether or not it + physically exists on the filesystem. :: + + do_configure[file-checksums] += "${MY_DIRPATH}/my-file.txt:True" + + It is important to record any paths which the task looked at and + which didn't exist. This means that if these do exist at a later + time, the task can be rerun with the new additional files. The + "exists" True or False value after the path allows this to be + handled. + - ``[lockfiles]``: Specifies one or more lockfiles to lock while the task executes. Only one task may hold a lockfile, and any task that attempts to lock an already locked file will block until the lock is released. You can use this variable flag to accomplish mutual exclusion. +- ``[network]``: When set to "1", allows a task to access the network. By + default, only the ``do_fetch`` task is granted network access. Recipes + shouldn't access the network outside of ``do_fetch`` as it usually + undermines fetcher source mirroring, image and licence manifests, software + auditing and supply chain security. + - ``[noexec]``: When set to "1", marks the task as being empty, with no execution required. You can use the ``[noexec]`` flag to set up tasks as dependency placeholders, or to disable tasks defined @@ -1922,6 +1978,33 @@ looking at the source code of the ``bb`` module, which is in the commonly used functions ``bb.utils.contains()`` and ``bb.utils.mkdirhier()``, which come with docstrings. +Extending Python Library Code +----------------------------- + +If you wish to add your own Python library code (e.g. to provide +functions/classes you can use from Python functions in the metadata) +you can do so from any layer using the ``addpylib`` directive. +This directive is typically added to your layer configuration ( +``conf/layer.conf``) although it will be handled in any ``.conf`` file. + +Usage is of the form:: + + addpylib <directory> <namespace> + +Where <directory> specifies the directory to add to the library path. +The specified <namespace> is imported automatically, and if the imported +module specifies an attribute named ``BBIMPORTS``, that list of +sub-modules is iterated and imported too. + +Testing and Debugging BitBake Python code +----------------------------------------- + +The OpenEmbedded build system implements a convenient ``pydevshell`` target which +you can use to access the BitBake datastore and experiment with your own Python +code. See :yocto_docs:`Using a Python Development Shell +</dev-manual/python-development-shell.html#using-a-python-development-shell>` in the Yocto +Project manual for details. + Task Checksums and Setscene =========================== diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables-context.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables-context.rst new file mode 100644 index 0000000000..e9c454ba11 --- /dev/null +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables-context.rst @@ -0,0 +1,91 @@ +.. SPDX-License-Identifier: CC-BY-2.5 + +================ +Variable Context +================ + +| + +Variables might only have an impact or can be used in certain contexts. Some +should only be used in global files like ``.conf``, while others are intended only +for local files like ``.bb``. This chapter aims to describe some important variable +contexts. + +.. _ref-varcontext-configuration: + +BitBake's own configuration +=========================== + +Variables starting with ``BB_`` usually configure the behaviour of BitBake itself. +For example, one could configure: + +- System resources, like disk space to be used (:term:`BB_DISKMON_DIRS`), + or the number of tasks to be run in parallel by BitBake (:term:`BB_NUMBER_THREADS`). + +- How the fetchers shall behave, e.g., :term:`BB_FETCH_PREMIRRORONLY` is used + by BitBake to determine if BitBake's fetcher shall search only + :term:`PREMIRRORS` for files. + +Those variables are usually configured globally. + +BitBake configuration +===================== + +There are variables: + +- Like :term:`B` or :term:`T`, that are used to specify directories used by + BitBake during the build of a particular recipe. Those variables are + specified in ``bitbake.conf``. Some, like :term:`B`, are quite often + overwritten in recipes. + +- Starting with ``FAKEROOT``, to configure how the ``fakeroot`` command is + handled. Those are usually set by ``bitbake.conf`` and might get adapted in a + ``bbclass``. + +- Detailing where BitBake will store and fetch information from, for + data reuse between build runs like :term:`CACHE`, :term:`DL_DIR` or + :term:`PERSISTENT_DIR`. Those are usually global. + + +Layers and files +================ + +Variables starting with ``LAYER`` configure how BitBake handles layers. +Additionally, variables starting with ``BB`` configure how layers and files are +handled. For example: + +- :term:`LAYERDEPENDS` is used to configure on which layers a given layer + depends. + +- The configured layers are contained in :term:`BBLAYERS` and files in + :term:`BBFILES`. + +Those variables are often used in the files ``layer.conf`` and ``bblayers.conf``. + +Recipes and packages +==================== + +Variables handling recipes and packages can be split into: + +- :term:`PN`, :term:`PV` or :term:`PF` for example, contain information about + the name or revision of a recipe or package. Usually, the default set in + ``bitbake.conf`` is used, but those are from time to time overwritten in + recipes. + +- :term:`SUMMARY`, :term:`DESCRIPTION`, :term:`LICENSE` or :term:`HOMEPAGE` + contain the expected information and should be set specifically for every + recipe. + +- In recipes, variables are also used to control build and runtime + dependencies between recipes/packages with other recipes/packages. The + most common should be: :term:`PROVIDES`, :term:`RPROVIDES`, :term:`DEPENDS`, + and :term:`RDEPENDS`. + +- There are further variables starting with ``SRC`` that specify the sources in + a recipe like :term:`SRC_URI` or :term:`SRCDATE`. Those are also usually set + in recipes. + +- Which version or provider of a recipe should be given preference when + multiple recipes would provide the same item, is controlled by variables + starting with ``PREFERRED_``. Those are normally set in the configuration + files of a ``MACHINE`` or ``DISTRO``. diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst index 12aef3cbb7..899e584f91 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.rst @@ -40,8 +40,7 @@ overview of their function and contents. Azure Storage Shared Access Signature, when using the :ref:`Azure Storage fetcher <bitbake-user-manual/bitbake-user-manual-fetching:fetchers>` This variable can be defined to be used by the fetcher to authenticate - and gain access to non-public artifacts. - :: + and gain access to non-public artifacts:: AZ_SAS = ""se=2021-01-01&sp=r&sv=2018-11-09&sr=c&skoid=<skoid>&sig=<signature>"" @@ -100,10 +99,26 @@ overview of their function and contents. the path of the build. BitBake's output should not (and usually does not) depend on the directory in which it was built. + :term:`BB_CACHEDIR` + Specifies the code parser cache directory (distinct from :term:`CACHE` + and :term:`PERSISTENT_DIR` although they can be set to the same value + if desired). The default value is "${TOPDIR}/cache". + :term:`BB_CHECK_SSL_CERTS` Specifies if SSL certificates should be checked when fetching. The default value is ``1`` and certificates are not checked if the value is set to ``0``. + :term:`BB_HASH_CODEPARSER_VALS` + Specifies values for variables to use when populating the codeparser cache. + This can be used selectively to set dummy values for variables to avoid + the codeparser cache growing on every parse. Variables that would typically + be included are those where the value is not significant for where the + codeparser cache is used (i.e. when calculating variable dependencies for + code fragments.) The value is space-separated without quoting values, for + example:: + + BB_HASH_CODEPARSER_VALS = "T=/ WORKDIR=/ DATE=1234 TIME=1234" + :term:`BB_CONSOLELOG` Specifies the path to a log file into which BitBake's user interface writes output during the build. @@ -344,6 +359,14 @@ overview of their function and contents. For example usage, see :term:`BB_GIT_SHALLOW`. + :term:`BB_GLOBAL_PYMODULES` + Specifies the list of Python modules to place in the global namespace. + It is intended that only the core layer should set this and it is meant + to be a very small list, typically just ``os`` and ``sys``. + :term:`BB_GLOBAL_PYMODULES` is expected to be set before the first + ``addpylib`` directive. + See also ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:extending python library code`". + :term:`BB_HASHCHECK_FUNCTION` Specifies the name of the function to call during the "setscene" part of the task's execution in order to validate the list of task hashes. @@ -409,6 +432,15 @@ overview of their function and contents. ``ConfigParsed`` event can set the variable to trigger the re-parse. You must be careful to avoid recursive loops with this functionality. + :term:`BB_LOADFACTOR_MAX` + Setting this to a value will cause BitBake to check the system load + average before executing new tasks. If the load average is above the + the number of CPUs multipled by this factor, no new task will be started + unless there is no task executing. A value of "1.5" has been found to + work reasonably. This is helpful for systems which don't have pressure + regulation enabled, which is more granular. Pressure values take + precedence over loadfactor. + :term:`BB_LOGCONFIG` Specifies the name of a config file that contains the user logging configuration. See @@ -483,13 +515,64 @@ overview of their function and contents. You must set this variable in the external environment in order for it to work. + :term:`BB_PRESSURE_MAX_CPU` + Specifies a maximum CPU pressure threshold, above which BitBake's + scheduler will not start new tasks (providing there is at least + one active task). If no value is set, CPU pressure is not + monitored when starting tasks. + + The pressure data is calculated based upon what Linux kernels since + version 4.20 expose under ``/proc/pressure``. The threshold represents + the difference in "total" pressure from the previous second. The + minimum value is 1.0 (extremely slow builds) and the maximum is + 1000000 (a pressure value unlikely to ever be reached). + + This threshold can be set in ``conf/local.conf`` as:: + + BB_PRESSURE_MAX_CPU = "500" + + :term:`BB_PRESSURE_MAX_IO` + Specifies a maximum I/O pressure threshold, above which BitBake's + scheduler will not start new tasks (providing there is at least + one active task). If no value is set, I/O pressure is not + monitored when starting tasks. + + The pressure data is calculated based upon what Linux kernels since + version 4.20 expose under ``/proc/pressure``. The threshold represents + the difference in "total" pressure from the previous second. The + minimum value is 1.0 (extremely slow builds) and the maximum is + 1000000 (a pressure value unlikely to ever be reached). + + At this point in time, experiments show that IO pressure tends to + be short-lived and regulating just the CPU with + :term:`BB_PRESSURE_MAX_CPU` can help to reduce it. + + :term:`BB_PRESSURE_MAX_MEMORY` + + Specifies a maximum memory pressure threshold, above which BitBake's + scheduler will not start new tasks (providing there is at least + one active task). If no value is set, memory pressure is not + monitored when starting tasks. + + The pressure data is calculated based upon what Linux kernels since + version 4.20 expose under ``/proc/pressure``. The threshold represents + the difference in "total" pressure from the previous second. The + minimum value is 1.0 (extremely slow builds) and the maximum is + 1000000 (a pressure value unlikely to ever be reached). + + Memory pressure is experienced when time is spent swapping, + refaulting pages from the page cache or performing direct reclaim. + This is why memory pressure is rarely seen, but setting this variable + might be useful as a last resort to prevent OOM errors if they are + occurring during builds. + :term:`BB_RUNFMT` Specifies the name of the executable script files (i.e. run files) saved into ``${``\ :term:`T`\ ``}``. By default, the :term:`BB_RUNFMT` variable is undefined and the run filenames get created using the following form:: - run.{task}.{pid} + run.{func}.{pid} If you want to force run files to take a specific name, you can set this variable in a configuration file. @@ -846,9 +929,9 @@ overview of their function and contents. section. :term:`BBPATH` - Used by BitBake to locate class (``.bbclass``) and configuration - (``.conf``) files. This variable is analogous to the ``PATH`` - variable. + A colon-separated list used by BitBake to locate class (``.bbclass``) + and configuration (``.conf``) files. This variable is analogous to the + ``PATH`` variable. If you run BitBake from a directory outside of the build directory, you must be sure to set :term:`BBPATH` to point to the build directory. @@ -940,7 +1023,7 @@ overview of their function and contents. ``bblayers.conf`` configuration file. To exclude a recipe from a world build using this variable, set the - variable to "1" in the recipe. + variable to "1" in the recipe. Set it to "0" to add it back to world build. .. note:: @@ -998,6 +1081,11 @@ overview of their function and contents. environment variable. The value is a colon-separated list of directories that are searched left-to-right in order. + :term:`FILE_LAYERNAME` + During parsing and task execution, this is set to the name of the + layer containing the recipe file. Code can use this to identify which + layer a recipe is from. + :term:`GITDIR` The directory in which a local copy of a Git repository is stored when it is cloned. @@ -1046,6 +1134,29 @@ overview of their function and contents. variable is not available outside of ``layer.conf`` and references are expanded immediately when parsing of the file completes. + :term:`LAYERSERIES_COMPAT` + Lists the versions of the OpenEmbedded-Core (OE-Core) for which + a layer is compatible. Using the :term:`LAYERSERIES_COMPAT` variable + allows the layer maintainer to indicate which combinations of the + layer and OE-Core can be expected to work. The variable gives the + system a way to detect when a layer has not been tested with new + releases of OE-Core (e.g. the layer is not maintained). + + To specify the OE-Core versions for which a layer is compatible, use + this variable in your layer's ``conf/layer.conf`` configuration file. + For the list, use the Yocto Project release name (e.g. "kirkstone", + "mickledore"). To specify multiple OE-Core versions for the layer, use + a space-separated list:: + + LAYERSERIES_COMPAT_layer_root_name = "kirkstone mickledore" + + .. note:: + + Setting :term:`LAYERSERIES_COMPAT` is required by the Yocto Project + Compatible version 2 standard. + The OpenEmbedded build system produces a warning if the variable + is not set for any given layer. + :term:`LAYERVERSION` Optionally specifies the version of a layer as a single number. You can use this variable within @@ -1068,8 +1179,8 @@ overview of their function and contents. order. :term:`OVERRIDES` - BitBake uses :term:`OVERRIDES` to control what variables are overridden - after BitBake parses recipes and configuration files. + A colon-separated list that BitBake uses to control what variables are + overridden after BitBake parses recipes and configuration files. Following is a simple example that uses an overrides list based on machine architectures: OVERRIDES = "arm:x86:mips:powerpc" You can diff --git a/bitbake/doc/index.rst b/bitbake/doc/index.rst index 3ff8b1580f..ee1660ac15 100644 --- a/bitbake/doc/index.rst +++ b/bitbake/doc/index.rst @@ -13,6 +13,7 @@ BitBake User Manual bitbake-user-manual/bitbake-user-manual-intro bitbake-user-manual/bitbake-user-manual-execution bitbake-user-manual/bitbake-user-manual-metadata + bitbake-user-manual/bitbake-user-manual-ref-variables-context bitbake-user-manual/bitbake-user-manual-fetching bitbake-user-manual/bitbake-user-manual-ref-variables bitbake-user-manual/bitbake-user-manual-hello diff --git a/bitbake/doc/releases.rst b/bitbake/doc/releases.rst index 6635032c01..b38b1c0652 100644 --- a/bitbake/doc/releases.rst +++ b/bitbake/doc/releases.rst @@ -1,61 +1,63 @@ .. SPDX-License-Identifier: CC-BY-2.5 -=========================== - Supported Release Manuals -=========================== +================================= +BitBake Supported Release Manuals +================================= -****************************** -Release Series 3.4 (honister) -****************************** +******************************* +Release Series 4.2 (mickledore) +******************************* -- :yocto_docs:`3.4 BitBake User Manual </3.4/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.4.1 BitBake User Manual </3.4.1/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.4.2 BitBake User Manual </3.4.2/bitbake-user-manual/bitbake-user-manual.html>` +- :yocto_docs:`BitBake 2.4 User Manual </bitbake/2.4/>` ****************************** -Release Series 3.3 (hardknott) +Release Series 4.0 (kirkstone) ****************************** -- :yocto_docs:`3.3 BitBake User Manual </3.3/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.3.1 BitBake User Manual </3.3.1/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.3.2 BitBake User Manual </3.3.2/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.3.3 BitBake User Manual </3.3.3/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.3.4 BitBake User Manual </3.3.4/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.3.5 BitBake User Manual </3.3.5/bitbake-user-manual/bitbake-user-manual.html>` +- :yocto_docs:`BitBake 2.0 User Manual </bitbake/2.0/>` **************************** Release Series 3.1 (dunfell) **************************** -- :yocto_docs:`3.1 BitBake User Manual </3.1/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.1.1 BitBake User Manual </3.1.1/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.1.2 BitBake User Manual </3.1.2/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.1.3 BitBake User Manual </3.1.3/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.1.4 BitBake User Manual </3.1.4/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.1.5 BitBake User Manual </3.1.5/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.1.6 BitBake User Manual </3.1.6/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.1.7 BitBake User Manual </3.1.7/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.1.8 BitBake User Manual </3.1.8/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.1.9 BitBake User Manual </3.1.9/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.1.10 BitBake User Manual </3.1.10/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.1.11 BitBake User Manual </3.1.11/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.1.12 BitBake User Manual </3.1.12/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.1.13 BitBake User Manual </3.1.13/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.1.14 BitBake User Manual </3.1.14/bitbake-user-manual/bitbake-user-manual.html>` - -========================== - Outdated Release Manuals -========================== +- :yocto_docs:`BitBake 1.46 User Manual </bitbake/1.46/>` + +================================ +BitBake Outdated Release Manuals +================================ + +***************************** +Release Series 4.1 (langdale) +***************************** + +- :yocto_docs:`BitBake 2.2 User Manual </bitbake/2.2/>` + +****************************** +Release Series 3.4 (honister) +****************************** + +- :yocto_docs:`BitBake 1.52 User Manual </bitbake/1.52/>` + +****************************** +Release Series 3.3 (hardknott) +****************************** + +- :yocto_docs:`BitBake 1.50 User Manual </bitbake/1.50/>` ******************************* Release Series 3.2 (gatesgarth) ******************************* -- :yocto_docs:`3.2 BitBake User Manual </3.2/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.2.1 BitBake User Manual </3.2.1/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.2.2 BitBake User Manual </3.2.2/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.2.3 BitBake User Manual </3.2.3/bitbake-user-manual/bitbake-user-manual.html>` -- :yocto_docs:`3.2.4 BitBake User Manual </3.2.4/bitbake-user-manual/bitbake-user-manual.html>` +- :yocto_docs:`BitBake 1.48 User Manual </bitbake/1.48/>` + +******************************************* +Release Series 3.1 (dunfell first versions) +******************************************* + +- :yocto_docs:`3.1 BitBake User Manual </3.1/bitbake-user-manual/bitbake-user-manual.html>` +- :yocto_docs:`3.1.1 BitBake User Manual </3.1.1/bitbake-user-manual/bitbake-user-manual.html>` +- :yocto_docs:`3.1.2 BitBake User Manual </3.1.2/bitbake-user-manual/bitbake-user-manual.html>` +- :yocto_docs:`3.1.3 BitBake User Manual </3.1.3/bitbake-user-manual/bitbake-user-manual.html>` ************************* Release Series 3.0 (zeus) |