diff options
Diffstat (limited to 'bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst')
| -rw-r--r-- | bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst | 195 |
1 files changed, 176 insertions, 19 deletions
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst index 4396830a2f..fb4f0a23d7 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst @@ -84,18 +84,18 @@ fetcher does know how to use HTTP as a transport. Here are some examples that show commonly used mirror definitions:: PREMIRRORS ?= "\ - bzr://.*/.\* http://somemirror.org/sources/ \\n \ - cvs://.*/.\* http://somemirror.org/sources/ \\n \ - git://.*/.\* http://somemirror.org/sources/ \\n \ - hg://.*/.\* http://somemirror.org/sources/ \\n \ - osc://.*/.\* http://somemirror.org/sources/ \\n \ - p4://.*/.\* http://somemirror.org/sources/ \\n \ - svn://.*/.\* http://somemirror.org/sources/ \\n" + bzr://.*/.\* http://somemirror.org/sources/ \ + cvs://.*/.\* http://somemirror.org/sources/ \ + git://.*/.\* http://somemirror.org/sources/ \ + hg://.*/.\* http://somemirror.org/sources/ \ + osc://.*/.\* http://somemirror.org/sources/ \ + p4://.*/.\* http://somemirror.org/sources/ \ + svn://.*/.\* http://somemirror.org/sources/" MIRRORS =+ "\ - ftp://.*/.\* http://somemirror.org/sources/ \\n \ - http://.*/.\* http://somemirror.org/sources/ \\n \ - https://.*/.\* http://somemirror.org/sources/ \\n" + ftp://.*/.\* http://somemirror.org/sources/ \ + http://.*/.\* http://somemirror.org/sources/ \ + https://.*/.\* http://somemirror.org/sources/" It is useful to note that BitBake supports cross-URLs. It is possible to mirror a Git repository on an @@ -167,6 +167,9 @@ govern the behavior of the unpack stage: - *dos:* Applies to ``.zip`` and ``.jar`` files and specifies whether to use DOS line ending conversion on text files. +- *striplevel:* Strip specified number of leading components (levels) + from file names on extraction + - *subdir:* Unpacks the specific URL to the specified subdirectory within the root directory. @@ -226,6 +229,11 @@ downloaded file is useful for avoiding collisions in :term:`DL_DIR` when dealing with multiple files that have the same name. +If a username and password are specified in the ``SRC_URI``, a Basic +Authorization header will be added to each request, including across redirects. +To instead limit the Authorization header to the first request, add +"redirectauth=0" to the list of parameters. + Some example URLs are as follows:: SRC_URI = "http://oe.handhelds.org/not_there.aac" @@ -388,6 +396,19 @@ This fetcher supports the following parameters: protocol is "file". You can also use "http", "https", "ssh" and "rsync". + .. note:: + + When ``protocol`` is "ssh", the URL expected in :term:`SRC_URI` differs + from the one that is typically passed to ``git clone`` command and provided + by the Git server to fetch from. For example, the URL returned by GitLab + server for ``mesa`` when cloning over SSH is + ``git@gitlab.freedesktop.org:mesa/mesa.git``, however the expected URL in + :term:`SRC_URI` is the following:: + + SRC_URI = "git://git@gitlab.freedesktop.org/mesa/mesa.git;branch=main;protocol=ssh;..." + + Note the ``:`` character changed for a ``/`` before the path to the project. + - *"nocheckout":* Tells the fetcher to not checkout source code when unpacking when set to "1". Set this option for the URL where there is a custom routine to checkout code. The default is "0". @@ -403,17 +424,17 @@ This fetcher supports the following parameters: - *"nobranch":* Tells the fetcher to not check the SHA validation for the branch when set to "1". The default is "0". Set this option for - the recipe that refers to the commit that is valid for a tag instead - of the branch. + the recipe that refers to the commit that is valid for any namespace + (branch, tag, ...) instead of the branch. - *"bareclone":* Tells the fetcher to clone a bare clone into the destination directory without checking out a working tree. Only the raw Git metadata is provided. This parameter implies the "nocheckout" parameter as well. -- *"branch":* The branch(es) of the Git tree to clone. If unset, this - is assumed to be "master". The number of branch parameters much match - the number of name parameters. +- *"branch":* The branch(es) of the Git tree to clone. Unless + "nobranch" is set to "1", this is a mandatory parameter. The number of + branch parameters must match the number of name parameters. - *"rev":* The revision to use for the checkout. The default is "master". @@ -436,8 +457,9 @@ This fetcher supports the following parameters: Here are some example URLs:: - SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1" - SRC_URI = "git://git.oe.handhelds.org/git/vip.git;protocol=http" + SRC_URI = "git://github.com/fronteed/icheck.git;protocol=https;branch=${PV};tag=${PV}" + SRC_URI = "git://github.com/asciidoc/asciidoc-py;protocol=https;branch=main" + SRC_URI = "git://git@gitlab.freedesktop.org/mesa/mesa.git;branch=main;protocol=ssh;..." .. note:: @@ -454,6 +476,14 @@ Here are some example URLs:: easy to share metadata without removing passwords. SSH keys, ``~/.netrc`` and ``~/.ssh/config`` files can be used as alternatives. +Using tags with the git fetcher may cause surprising behaviour. Bitbake needs to +resolve the tag to a specific revision and to do that, it has to connect to and use +the upstream repository. This is because the revision the tags point at can change and +we've seen cases of this happening in well known public repositories. This can mean +many more network connections than expected and recipes may be reparsed at every build. +Source mirrors will also be bypassed as the upstream repository is the only source +of truth to resolve the revision accurately. For these reasons, whilst the fetcher +can support tags, we recommend being specific about revisions in recipes. .. _gitsm-fetcher: @@ -666,6 +696,133 @@ Here is an example URL:: It can also be used when setting mirrors definitions using the :term:`PREMIRRORS` variable. +.. _gcp-fetcher: + +GCP Fetcher (``gs://``) +-------------------------- + +This submodule fetches data from a +`Google Cloud Storage Bucket <https://cloud.google.com/storage/docs/buckets>`__. +It uses the `Google Cloud Storage Python Client <https://cloud.google.com/python/docs/reference/storage/latest>`__ +to check the status of objects in the bucket and download them. +The use of the Python client makes it substantially faster than using command +line tools such as gsutil. + +The fetcher requires the Google Cloud Storage Python Client to be installed, along +with the gsutil tool. + +The fetcher requires that the machine has valid credentials for accessing the +chosen bucket. Instructions for authentication can be found in the +`Google Cloud documentation <https://cloud.google.com/docs/authentication/provide-credentials-adc#local-dev>`__. + +If it used from the OpenEmbedded build system, the fetcher can be used for +fetching sstate artifacts from a GCS bucket by specifying the +``SSTATE_MIRRORS`` variable as shown below:: + + SSTATE_MIRRORS ?= "\ + file://.* gs://<bucket name>/PATH \ + " + +The fetcher can also be used in recipes:: + + SRC_URI = "gs://<bucket name>/<foo_container>/<bar_file>" + +However, the checksum of the file should be also be provided:: + + SRC_URI[sha256sum] = "<sha256 string>" + +.. _crate-fetcher: + +Crate Fetcher (``crate://``) +---------------------------- + +This submodule fetches code for +`Rust language "crates" <https://doc.rust-lang.org/reference/glossary.html?highlight=crate#crate>`__ +corresponding to Rust libraries and programs to compile. Such crates are typically shared +on https://crates.io/ but this fetcher supports other crate registries too. + +The format for the :term:`SRC_URI` setting must be:: + + SRC_URI = "crate://REGISTRY/NAME/VERSION" + +Here is an example URL:: + + SRC_URI = "crate://crates.io/glob/0.2.11" + +.. _npm-fetcher: + +NPM Fetcher (``npm://``) +------------------------ + +This submodule fetches source code from an +`NPM <https://en.wikipedia.org/wiki/Npm_(software)>`__ +Javascript package registry. + +The format for the :term:`SRC_URI` setting must be:: + + SRC_URI = "npm://some.registry.url;ParameterA=xxx;ParameterB=xxx;..." + +This fetcher supports the following parameters: + +- *"package":* The NPM package name. This is a mandatory parameter. + +- *"version":* The NPM package version. This is a mandatory parameter. + +- *"downloadfilename":* Specifies the filename used when storing the downloaded file. + +- *"destsuffix":* Specifies the directory to use to unpack the package (default: ``npm``). + +Note that NPM fetcher only fetches the package source itself. The dependencies +can be fetched through the `npmsw-fetcher`_. + +Here is an example URL with both fetchers:: + + SRC_URI = " \ + npm://registry.npmjs.org/;package=cute-files;version=${PV} \ + npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \ + " + +See :yocto_docs:`Creating Node Package Manager (NPM) Packages +</dev-manual/packages.html#creating-node-package-manager-npm-packages>` +in the Yocto Project manual for details about using +:yocto_docs:`devtool <https://docs.yoctoproject.org/ref-manual/devtool-reference.html>` +to automatically create a recipe from an NPM URL. + +.. _npmsw-fetcher: + +NPM shrinkwrap Fetcher (``npmsw://``) +------------------------------------- + +This submodule fetches source code from an +`NPM shrinkwrap <https://docs.npmjs.com/cli/v8/commands/npm-shrinkwrap>`__ +description file, which lists the dependencies +of an NPM package while locking their versions. + +The format for the :term:`SRC_URI` setting must be:: + + SRC_URI = "npmsw://some.registry.url;ParameterA=xxx;ParameterB=xxx;..." + +This fetcher supports the following parameters: + +- *"dev":* Set this parameter to ``1`` to install "devDependencies". + +- *"destsuffix":* Specifies the directory to use to unpack the dependencies + (``${S}`` by default). + +Note that the shrinkwrap file can also be provided by the recipe for +the package which has such dependencies, for example:: + + SRC_URI = " \ + npm://registry.npmjs.org/;package=cute-files;version=${PV} \ + npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \ + " + +Such a file can automatically be generated using +:yocto_docs:`devtool <https://docs.yoctoproject.org/ref-manual/devtool-reference.html>` +as described in the :yocto_docs:`Creating Node Package Manager (NPM) Packages +</dev-manual/packages.html#creating-node-package-manager-npm-packages>` +section of the Yocto Project. + Other Fetchers -------------- @@ -675,10 +832,10 @@ Fetch submodules also exist for the following: - Mercurial (``hg://``) -- npm (``npm://``) - - OSC (``osc://``) +- S3 (``s3://``) + - Secure FTP (``sftp://``) - Secure Shell (``ssh://``) |