| Age | Commit message (Collapse) | Author |
|---|
|
When a command is run with a non-null log, errors are only output to the
log and are not returned in the exception. In that case direct users to
that logfile instead of telling the command had no output.
(Bitbake rev: 944fe0a77932a5559e01ae6035c4bffa5185ea6a)
Signed-off-by: Viswanath Kraleti <quic_vkraleti@quicinc.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Whilst typically the URI query is a list of key-value pairs, that's not
actually required by the URI specification.
For example: http://example.com/foo?bar is a valid query, but this will
result in the fetcher raising an exception:
File "bitbake/lib/bb/fetch2/__init__.py", line 265, in __init__
self.query = self._param_str_split(urlp.query, "&")
File "bitbake/lib/bb/fetch2/__init__.py", line 293, in _param_str_split
for k, v in [x.split(kvdelim, 1) for x in string.split(elmdelim) if x]:
ValueError: not enough values to unpack (expected 2, got 1)
In this case the query is just "bar", but the fetcher is trying to split
it into a key-value pair.
The URI object exposes the parsed query explicitly as a dictionary of
key-value pairs, so we have to be a little creative here: if a value is
None then it isn't a key-value pair, but a bare key.
Fix this by handling elements without the deliminator in _param_str_split()
(by assigning the value to None), and handle a None value when formatting
the query in _param_str_join().
(Bitbake rev: eac583bd4c46f3bb9661852cb6a1448f16147ff1)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently when git fetches fail, it destroys all the existing local clone data.
For large repositories this can introduce long build delays when for example,
you just typo'd the git revision hash.
The git fetcher should be able to recover most directories so when the fetch is
for a git repo, avoid removing things unless clean is explicitly called
(e.g. a -c cleanall task).
(Bitbake rev: 1b3cd039fe19b24bd4be9a0202a98cdcbb0e9443)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Support AWS_SESSION_TOKEN which is used in temporary
AWS credentials.
Fixes [YOCTO #15384].
(Bitbake rev: ae1e4c90bbc2002cb2728c64649c095c00220ceb)
Signed-off-by: Toni Lammi <toni.lammi@kone.com>
Reported-by: Toni Lammi <toni.lammi@tl-software.fi>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Previously, there was duplicate clutter in the output, particularly if the
URI points to sstate cache items:
bb.fetch2.FetchError: Fetcher failure for URL: {uri}. URL {uri} doesn't work
(Bitbake rev: 61537b8a98b963e4af265e046d41407b32fa5935)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This patch adds the following variables to the allow-list, which are
used in the "IAM roles for AWS when using the GitLab chart":
- AWS_ROLE_ARN
- AWS_WEB_IDENTITY_TOKEN_FILE
These variables are set in the CI job environment and are needed to
access the sstate cache artifacts in a connected S3 bucket.
[1] https://docs.gitlab.com/charts/advanced/external-object-storage/aws-iam-roles.html
Reported-by: Zhi Bin Dong <zhibin.dong@siemens.com>
(Bitbake rev: c534526ea73805ee7cc16f3168b05ece10e0c03c)
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This patch adds an API to bb.fetch2 to enable users to plug in an unpack
tracer that can trace each source file back to its corresponding
upstream source url, even when multiple upstream sources are combined
together in the same unpack directory. This may be required for software
composition analysis, license compliance, and detailed SBoM generation.
This patch provides only the needed hooks in bb.fetch2 code and a dummy
abstract class defining the API; users may load their own unpack tracer
class by setting the BB_UNPACK_TRACER_CLASS config parameter.
(Bitbake rev: 05051152cc42acc52bcf9af9a696f632fac4307f)
Signed-off-by: Alberto Pianon <alberto@pianon.eu>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Visual studio code is injecting a git credential helper
into every dev container. This helper is forwarding the git credentials
from the host into the container.
In order for this helper to work the REMOTE_CONTAINERS_IPC
needs to be available in the environment for all git operations
that require authentication.
(Bitbake rev: 612bdf09416ff96d08b22cf92b077516d23d0963)
Signed-off-by: Matthias Schnelte <develop@schnelte.de>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add new functions to return some of the get_srcrev data in new and different
ways. We need two different forms of the data, one is a string to inject into
PKGV, the other is the full revisions as a string to include in hash computations
so that the hash changes when the input revisions change.
This allows us to clean up and simplify the code in OE-Core and move the
version information from PV to PKGV.
(Bitbake rev: ae4dfa2a31c74c0c6c2b14cece822ed1f3d79723)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This fetcher allows BitBake to fetch from a Google Cloud Storage
bucket. The fetcher expects a gs:// URI of the following form:
SSTATE_MIRRORS = "file://.* gs://<bucket name>/PATH"
The fetcher uses the Google Cloud Storage Python Client, and
expects it to be installed, configured, and authenticated prior
to use.
If accepted, this patch should merge in with the corresponding oe-core
patch titled "Add GCP fetcher to list of supported protocols".
Some comments on the patch:
There is also documentation for the fetcher added to the User
Manual. I'm still not completely sure about the recommends_checksum()
being set to True. As I've noted in the mailing list, it will throw
warnings if the fetcher is used in recipes without specifying a
checksum. Please let me know if this is intended behavior or if it
should be modified.
Here is how this fetcher conforms to the fetcher expectations
described at this link:
https://git.yoctoproject.org/poky/tree/bitbake/lib/bb/fetch2/README
a) Yes, network fetching only happens in the fetcher
b) The fetcher has nothing to do with the unpack phase so there is no
network access there
c) This change doesn't affect the behavior of DL_DIR. The GCP fetcher
only downloads to the DL_DIR in the same way that other fetchers,
namely the S3 and Azure fetchers do.
d) The fetcher is identical to the S3 and Azure fetchers in this
context
e) Yes, the fetcher output is deterministic because it is downloading
tarballs from a bucket and not modifying them in any way.
f) I set up a local proxy using tinyproxy and set the http_proxy
variable to test whether the Python API respected the proxy. It
appears that it did as I could see traffic passing through the
proxy. I also did some searching online and found posts indicating
that the Google Cloud Python APIs supported the classic Linux proxy
variables, namely:
- https://github.com/googleapis/google-api-python-client/issues/1260
g) Access is minimal, only checking if the file exists and downloading
it if it does.
h) Not applicable, BitBake already knows which version it wants and
the version infomation is encoded in the filename. The fetcher has no
concept of versions.
i) Not applicable
j) Not applicable
k) No tests were added as part of this change. I didn't see any tests
for the S3 or Azure changes either, is that OK?
l) I'm not 100% familiar but I don't believe this fetcher is using any
tools during parse time. Please correct me if I'm wrong.
(Bitbake rev: 8e7e5719c1de79eb488732818871add3a6fc238b)
Signed-off-by: Emil Ekmečić <eekmecic@snap.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add one more verification that checks if localpath is None, because
we can't compute checksum of a None.
(Bitbake rev: 47ab9d21171a834cbac3d1ce368d59fd71d14452)
Signed-off-by: BELOUARGA Mohamed <m.belouarga@technologyandstrategy.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Set the maxsplit value to match the expected number of variables.
This also avoids an unnecessary split as the parameters are in the form 'key=value'
and the 'value' could contain the '=' character.
(Bitbake rev: 3b17a7ed9bf6cd2808946c2d9c3ed9961af11f19)
Signed-off-by: Dit Kozmaj <dit.kozmaj@kynetics.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
To avoid confusion, rename the local fetcher localpaths function to
something more specific so it doesn't appear similar to the localpaths
top level function.
(Bitbake rev: ada5ea712cc42fff8c55bd261ee6d7579efd4a68)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Instead of only display the first and stop.
For recipe (crago based) that can contains several artifacts
to fetch with their checksum, it will particularly handy
to display all of missing one.
An example of error message would be
NOTE: Executing Tasks
ERROR: zvariant-3.12.0-r0 do_fetch: Missing SRC_URI checksum, please add those to the recipe:
SRC_URI[anyhow-1.0.70.sha256sum] = "7de8ce5e0f9f8d88245311066a578d72b7af3e7088f32783804676302df237e4"
SRC_URI[chrono-0.4.24.sha256sum] = "4e3c5919066adf22df73762e50cffcde3a758f2a848b113b586d1f86728b673b"
SRC_URI[serde-1.0.158.sha256sum] = "771d4d9c4163ee138805e12c710dd365e4f44be8be0503cb1bb9eb989425d9c9"
SRC_URI[system-deps-1.3.2.sha256sum] = "0f3ecc17269a19353b3558b313bba738b25d82993e30d62a18406a24aba4649b"
ERROR: zvariant-3.12.0-r0 do_fetch: Bitbake Fetcher Error: BBFetchException('There was some missing checksums in the recipe')
ERROR: Logfile of failure stored in: /home/jenkins/yocto-poky-master/poky/build/tmp/work/core2-64-poky-linux/zvariant/3.12.0-r0/temp/log.do_fetch.1025
ERROR: Task (/home/jenkins/yocto-poky-master/poky/meta-selftest/recipes-extended/zvariant/zvariant_3.12.0.bb:do_fetch) failed with exit code '1'
(Bitbake rev: dafa07c080e05975b6319b5adf78a9691c6b6643)
Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This change brings checksum verification of each crate
in a recipe, e.g
| SRC_URI += " \
| crate://crates.io/aho-corasick/0.7.20 \
| crate://crates.io/atomic-waker/1.1.0 \
| crate://crates.io/cc/1.0.79 \
| "
|
| SRC_URI[aho-corasick.sha256sum] = "cc936419f96fa211c1b9166887b38e5e40b19958e5b895be7c1f93adec7071ac"
| SRC_URI[atomic-waker.sha256sum] = "debc29dde2e69f9e47506b525f639ed42300fc014a3e007832592448fa8e4599"
| SRC_URI[cc.sha256sum] = "50d30906286121d95be3d479533b458f87493b30a4b5f79a607db8f5d11aa91f"
That will require to move the checksum initialization
after the possible call to urldata_init method in order
for the crate fetcher to parse the url.
Another way of doing could have been implementing a decodeurl
method that would have been specific for each fetcher class.
(Bitbake rev: 4920686c13dd66f9bfa4f7dd38d6e955f153eeec)
Signed-off-by: Frederic Martinsons <frederic.martinsons@gmail.com>
Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Bitbake expects a consistent metadata environment when it parses. There
are plenty of ways you can set a recipe to autorev at a point where parts
of the fetcher have already been triggered leading to obsure bugs which
I struggled to debug, let alone anyone not familar with the code.
If anyone is running into the message from the commit, the issue is
likely one of timing. Keep in mind that the anonymous python code
in base.bbclass will expand variables like FILESPATH, WORKDIR and others
which contain PV. The recipe needs to be set to AUTOREV before that
anonymous python runs.
In particular, that means you can't set SRCREV = "${AUTOREV}" in other
anonymous python, it needs to happen earlier.
(Bitbake rev: 4d9ec332d5bfc8b60b54f8ec2a17d34e35aa903a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
For readability of following patches, rename this internal variable to
allow for others in a similar format.
(Bitbake rev: 450b4a00895ed1f9396a8ff859dc1cc0eccc838f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
These environment variables are needed by gclient and needed to be
passed into fetcher.
(Bitbake rev: 9414ba62454c6b911addf6b0bc02af2afc69b926)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If you are behind a corporate proxy, the npm fetcher uses
the proxy IP address already passed in the list of exports.
However, it will error if the proxy injects its own root
CA certificate. Thus, the NODE_EXTRA_CA_CERTS environment
variable must be passed so the user can include their
company's root CA as a trusted CA inside node's
certificate store.
(Bitbake rev: 5daecc25a59ce5bf6c31f5085e6caa7cb543517e)
Signed-off-by: George Kelly <george.kelly1097@gmail.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
location
Currently the codeparser cache is set from CACHE, which is typically in
bitbake.conf which means we can't read/write any cache until it is found/read.
We may well have python expressions to parse before that happens.
The net result is suboptimal functioning of the codeparser cache since it will
often be invalidated by data that is never written.
This patch changes the codeparser and filechecksum caches to use BB_CACHE as
their setting and defaults it to ${TOPDIR}/cache.
The patch doesn't change where the "persistent" data such as prserver and
hash-equiavalance resides (PERSISTENT_DIR) or where the metadata parsing
cache resists (still currently CACHE). I've left those for a later patch.
The patch does ensure data parsed by the core datastore parsing calls is
written back since this is now much more useful after this change.
(Bitbake rev: ee89ade5b5a4cf9c53f336d8b800e06fbe436628)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fixes:
File "/bitbake/./lib/bb/fetch2/__init__.py", line 446, in uri_replace
if uri_decoded[loc][k] != uri_find_decoded[loc][k]:
KeyError: 'module'
(Bitbake rev: 80421abd5bf3f1f751219c050fa93970cf1f2088)
Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
local file fetches now validate checksums. The checksums for mirror
tarballs of repositories will not match so ignore these checksums.
(Bitbake rev: 6424f4b7e9c1ba8db81346e8b3a806dd035d4551)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If two recipes have conflicting checksums for a file, the code will currently
remove the existing file when a mismatch is downloaded, even if another task
successfully fetched it.
This changes the code to verify the checksum (if possible) before replacing
the file. This removes a potential race window and stops builds failing
everywhere from one incorrect checksum.
To make this work, we need to be able to override localpath and avoid
NoChecksum errors being logged.
(Bitbake rev: 4b8de2e7d12667d69d86ffe6e9f85a7932c4c9a5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
PN is correct here, bitbake has no knowledge of BPN.
This reverts commit d613e48c07d4b12219270c1359cbf2f390b848dd.
(Bitbake rev: cffcfacb747d41304c857b17bfea646e220b2389)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When checking for the non-existing file, BPN is actually the acutal recipe
name. And we should use BPN for the error message and it also fix the below
test when multilib is enabled.
$ oe-selftest -r bbtests.BitbakeTests.test_invalid_recipe_src_uri
(Bitbake rev: d613e48c07d4b12219270c1359cbf2f390b848dd)
Signed-off-by: Mingli Yu <mingli.yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
A long time ago, we made DL_DIR a final fallback for the local fetcher.
Since then we added checksum support and task hashes and the world
has changed. There were warnings added some time ago if this fallback
triggers and it is now time to drop it entirely.
The original use case was for sstate however the sstate code now sets
FILESPATH correctly so DL_DIR is no longer needed.
There have been a few small bugs exposed by this change, missing mkdir
calls and some minor test issues that needed tweaks. In general this
simplifies and improves the fetcher code flow though.
This completes a cleanup that ensures local files are correctly covered
at parse time which ensures rebuilds and reparses happen at the right
times.
(Bitbake rev: 3e1444e536c71d3885ef6b9d428807163c309640)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If the mirrors code is trying to create a symlink and the
parent directory doesn't exist, as might be the case for sstate
mirrors where the fetch is into a subdir, it can silently fail.
Ensure the directory exists in this case to avoid issues.
(Bitbake rev: eff16e474ee7dc49ae433420a4c8d15d3314a618)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If the local fetcher was not able to find the file anywhere but it
was included in the SRC_URI for checksumming just make it a fatal
error. Ensure a list of searched locations is included too to match
the runtime error that would have resulted.
(Bitbake rev: 5e3b2ad90d9cd0f248b1cb740637caa24442d101)
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
GIT_SSH_COMMAND is more convinient to use if arguments have to be passed
and the user doesn't want to create a wrapper script around ssh.
(Bitbake rev: 5e746cb9d26ce87d6c9d52d9022122081a9811c5)
Signed-off-by: Pavel Zhukov <pavel.zhukov@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In line with the inclusive language migration defined at:
https://wiki.yoctoproject.org/wiki/Inclusive_language
replace the remaining non-backwards-compatibility related usage
of "blacklist"/"whitelist" with "allowed"/"disallowed" in a few
places as appropriate.
(Bitbake rev: f579fb8c23d2919d25641fa4234f8a1e9c06a922)
Signed-off-by: Scott Murray <scott.murray@konsulko.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The changing of the environment inside the wget fetcher can race if
threading is used, such as with sstate in OE-Core. Abstract the function
so the environment can be correct before the function is called, removing
the race since the enviroment is then no longer changed.
(Bitbake rev: c73bb6023c73f003a160bb02aa4da1b580b86c23)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(Bitbake rev: e1c13a820f9e3154699ec32d84cce502464e4f6b)
Signed-off-by: Zygmunt Krynicki <zygmunt.krynicki@huawei.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Where a git url uses a tag instead of a full source revision, breakage
can currently occur in builds. Issues include:
* the revision being looked up in multiple tasks (fetch and unpack)
* the risk a different revision may be obtained in those tasks
* that some tasks may not be allowed to access the network
* that a revision may not be consistent throughout a given build
* rerunning a specific task may given inconsistent results
To fix this, stop the workers from cleaning out the source revision store. This
should only be done in the cooker itself (based on current policy).
Also, where the code "sees" an upstream access, mark the recipe as not to be
cached. The reparse re-triggers the upstream lookup by the server.
Add a test to ensure that if get_srcrev isn't called, the user is told they're
using a configuration that is known to break.
(Bitbake rev: 4b5eed1626709ef3dc06b32fd55d40a2a6edd179)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The solution implementated in commit 96c30007 (fetch2: fix
downloadfilename issue with premirror) missed two corner cases. The
first is if the basename of the original URI also appears somewhere
else in the replacement URI, in which case it would also be replaced.
The second is if the basename of the original URI partially matches
the basename of the replacement URI, in which case the inital part of
the basename from the replacement URI would be left behind.
The second case caused test_npm_premirrors_with_specified_filename to
fail.
The solution is to prefix the basename with a slash when matching to
avoid partial matches, and only replace the basename at the end of the
URI.
This also adds two test cases that test for these problems. Before
this they would give the following errors:
- ['file:///mirror/example/1.0.0/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz']
+ ['file:///mirror/some-example-1.0.0.tgz/1.0.0/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz']
? +++++ ++++++++++
- ['file:///mirror/some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz']
+ ['file:///mirror/some-some-example-1.0.0.tgz;downloadfilename=some-example-1.0.0.tgz']
? +++++
(Bitbake rev: 5924c6f007519cd8ea6cc8b316814d17b43048ca)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This imports the crate fetcher from OE-Core to resolve various module issues
and adds some very very basic tests of that new fetcher.
(Bitbake rev: 1f06f326fa8b47e2a4dce756d57a9369a2225201)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add a parameter `striplevel` to the SRC_URI to strip NUMBER leading
components (levels) from file names on extraction.
For example, if the archive `archive.tar.gz` contains `some/file`,
the SRC_URI `https://.../archive.tar.gz;subdir=other;striplevel=1`
will extract `some/file` to `other/file`.
This is useful to extract archives to a specified directory instead of
the original root component of the archive. The feature is required for
the npm support. The npm package contents should reside in a subfolder
inside a npm archive (usually it is called package/). npm strips one
directory layer when installing the package.
(Bitbake rev: aa4926e5d9c92f33b4434e2da709ff0bf3049f5b)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The tar command and its arguments are repeated for many archive types in
the unpack function. Unify the common parts in a variable to prepare
further extension.
(Bitbake rev: a08e57c9eaec1d9740a96149bf4843e576da4e5c)
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Using "" as a target for .replace() is a really bad idea as it duplicates the replacement
for every character in the string. Add a testcase which triggered this and correct the
code to return the correct result.
(Bitbake rev: 3af1ecf049d2eed56f6d319dc7df6eb4a3d4eebc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Username or password replacements in URIs were being appended rather than
replaced in mirror url remapping. Fix this and add a test case.
[YOCTO #13823]
(Bitbake rev: 66ad58bb87e5158aced572be4f1d5726bc97fcce)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Python scales badly when concatinating strings in loops. Most of these
references aren't problematic but at least one (in data.py) is probably
a performance issue as the issue is compounded as strings become large.
The way to handle this in python is to create lists which don't reconstruct
all the objects when appending to them. We may as well fix all the references
since it stops them being copy/pasted into something problematic in the future.
This patch was based on issues highligthted by a report from AWS Codeguru.
(Bitbake rev: d654139a833127b16274dca0ccbbab7e3bb33ed0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(Bitbake rev: bbbc843e86639604d00d76b1949b94a78cf1d95d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The following commit to fix [Yocto #13039] causes regression of
the behavior of PREMIRRORS.
"bitbake: fetch2: fix premirror URI when downloadfilename defined"
Take meta-openembedded/meta-networking/recipes-protocols/freediameter/freediameter_1.4.0.bb
as an example.
SRC_URI = "\
http://www.freediameter.net/hg/${fd_pkgname}/archive/${PV}.tar.gz;downloadfilename=${fd_pkgname}-${PV}.tar.gz \
...
"
With the above commit, it now tries to fetch 1.4.0.tar.gz instead of
freeDiameter-1.4.0.tar.gz. This makes https://downloads.yoctoproject.org/mirror/sources
not work for freediameter, as it holds freeDiameter-1.4.0.tar.gz.
The commit above tries to avoid fetching from invalid url such as:
https://<some_mirror>/1.4.0.tar.gz/freeDiameter-1.4.0.tar.gz.
And its solution is to make basename to be 1.4.0.tar.gz, thus causing the
regression.
This patch fixes the above regression. For Yocto #13039, it now tries
to fetch from url: https://<some_mirror>/freeDiameter-1.4.0.tar.gz.
(Bitbake rev: 96c30007dc0b32eee2b15771daec7948bc9bfd97)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
npmsw(and maybe other fetchers) may try to download same files simulatenously.
if a premirror is set, download method tries premirror location first. If file
is not found, done is set to false. In the meantime, other fetchers may download
the files required and verify_donestamp might return true.
In this scenerio, fetcher doesn't try to download again and fails. Do not
attempt to verify_donestamp if done is not set. By this way, we ensure that
fetcher attempts to do another download after premirror failure.
(Bitbake rev: 4fa25714916e84f99ecd22cb76cb5adada01e5e8)
Signed-off-by: Caner Altinbasak <cal@brightsign.biz>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The checkstatus semantics are unconventional, so document them.
(Bitbake rev: efb949020215b580e7dc3694c377b18df2fd7e9c)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If runfetchcmd() fails with bb.process.NotFoundError, the message output
is simply "Fetch command" which doesn't really explain what the problem
is.
Add "not found" to clarify what happened.
(Bitbake rev: 8de9dc02ed6a73b47f2ab10be30d1aed7954bc72)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Make usage of Bitbake's s3 fetcher more flexible with different AWS profiles
and switch between profiles as export of the AWS_PROFILE environment variable
(Bitbake rev: 0f35dc4dfc829fe9c51c239d15567f86c5c14e58)
Signed-off-by: Valentin Danaila <drlv85@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Users sometimes put ${S} references in ${SRC_URI} without realising this can be
problematic. Improve the error messages if they accidentally do.
[YOCTO #11593]
(Bitbake rev: 89e0b19ec0b245a6cd414088904c91808e8814ab)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When downloadfilename is defined in a recipe's SRC_URI and PREMIRRORS is also
defined using the same URI, the downloadfilename is appended to the mirror
URI and it should not be.
[YOCTO #13039]
(Bitbake rev: 8a3ff9f3eaf19d4258eb070c5dc230dface269b2)
Signed-off-by: Scott Weaver <weaverjs@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There is a list of environment variable names that need to be exported
when executing external commands, such as 'http_proxy'. To avoid
duplication, make this a top-level variable.
Also add SSL_CERT_FILE, which is used by OpenSSL to locate the
certificate bundle. This is needed in buildtools environments where the
default path isn't valid.
(Bitbake rev: 5c2cf57fd11d91f749a3b562f6f0a863f15013ed)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fix some references that missed during the overrides syntax migration or
were incorrect. Thanks to Quentin Schulz <foss@0leil.net> for the patch.
(Bitbake rev: 6184cb07dfa44f5f76f1c423533b4547d80b20ab)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Viswanath Kraleti
Ross Burton
Richard Purdie
Toni Lammi
Alexander Kanavin
Felix Moessbauer
Alberto Pianon
Matthias Schnelte
Emil Ekmečić
BELOUARGA Mohamed
Dit Kozmaj
Frederic Martinsons
Caner Altinbasak
George Kelly
Pavel Zhukov
Mingli Yu
Paulo Neves
Scott Murray
Zygmunt Krynicki
Peter Kjellerstedt
Stefan Herbrechtsmeier
Chen Qi
Ross Burton
Valentin Danaila
Scott Weaver