| Age | Commit message (Collapse) | Author |
|---|
|
(Bitbake rev: 67a1aa8dbb3cb3a30fa7d697431ebb30323e4f28)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(Bitbake rev: c86466d51e8ff14e57a734c1eec5bb651fdc73ef)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Now that the bitbake hash server supports SSL connections, we need to
capture a few environment variables which can affect the ability to
connect via SSL. Note that the variables are only put in place to affect
the environment while actually invoking the server
[RP: Tweak to use BB_ORIGENV as well]
[RP: Tweak to handle os.environ restore correctly]
(Bitbake rev: 0bacf6551821beb8915513b120ae672ae8eb1612)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
A function accidentally defined as:
somefunction() {
:
}
which is unclosed due to the space at the end, would currently silently
cause breakage. Have the parser throw and error for this.
[YOCTO #15470]
(Bitbake rev: a7dce72da6be626734486808f1b731247697e638)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Moving the code and related definitions from
hashserv/__init__.py to asyncrpc/client.py,
allowing this function to be used in other asyncrpc clients.
(Bitbake rev: b67bb05e431414866b8e8c6a4c88d20b9cdb44a3)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Suggested-by: Joshua Watt <JPEWhacker@gmail.com>
Cc: Tim Orling <ticotimo@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fedora 40 introduces wget2 as a drop-in replacement for wget. This
rewrite does not currently have support for FTP. This causes
the wget fetcher to fail complaining about an unrecognized option.
Making --passive-ftp conditional based on the protocol used in
the SRC_URI limits the scope of the problem. It also gives us
an opportunity to build the older wget as a host tool.
(Bitbake rev: f10e630fd7561746d835a4378e8777e78f56e44a)
Signed-off-by: Rob Woolley <rob.woolley@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Adds support for hashserver credentials to be specified in the
SignatureGenerator
(Bitbake rev: 741bef3755fde7bae1386aad575ea704d9fe0969)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Git uses a lock file to prevent concurrent modifications to the global
config, so if unpack tasks for different recipes try to run "git lfs
install" simultaneously the operation can fail:
error: could not lock config file /home/build/.gitconfig: File exists exit status 255
Run `git lfs install --force` to reset Git configuration.
Adding "--local" sets the smudge and clean filters in the local
repository's config instead of modifying the user's global config.
(Bitbake rev: 328ca4de8422be514fa0d0c9e3cfd36bb9d3e9a7)
Signed-off-by: Derek Erdmann <derek.erdmann@sonos.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When running in a cgroup which is limited to a subset of cpus (via
cpuset.cpus), cpu_count() should return the number of cpus that can be
used instead of the number of cpus the system has.
This also aligns the semantics with nproc.
(Bitbake rev: a029bfe96c6542f178720c72a772b7ede9898118)
Signed-off-by: Felix Moessbauer <felix.moessbauer@siemens.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When a command is run with a non-null log, errors are only output to the
log and are not returned in the exception. In that case direct users to
that logfile instead of telling the command had no output.
(Bitbake rev: 944fe0a77932a5559e01ae6035c4bffa5185ea6a)
Signed-off-by: Viswanath Kraleti <quic_vkraleti@quicinc.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Whilst typically the URI query is a list of key-value pairs, that's not
actually required by the URI specification.
For example: http://example.com/foo?bar is a valid query, but this will
result in the fetcher raising an exception:
File "bitbake/lib/bb/fetch2/__init__.py", line 265, in __init__
self.query = self._param_str_split(urlp.query, "&")
File "bitbake/lib/bb/fetch2/__init__.py", line 293, in _param_str_split
for k, v in [x.split(kvdelim, 1) for x in string.split(elmdelim) if x]:
ValueError: not enough values to unpack (expected 2, got 1)
In this case the query is just "bar", but the fetcher is trying to split
it into a key-value pair.
The URI object exposes the parsed query explicitly as a dictionary of
key-value pairs, so we have to be a little creative here: if a value is
None then it isn't a key-value pair, but a bare key.
Fix this by handling elements without the deliminator in _param_str_split()
(by assigning the value to None), and handle a None value when formatting
the query in _param_str_join().
(Bitbake rev: eac583bd4c46f3bb9661852cb6a1448f16147ff1)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Red text on a black background can make it difficult for people with
visual impairments to read the text of error messages. Respect the
presence of a non-empty NO_COLOR environment variable as an indication
the user doesn't want colorization to be automatically enabled.
See: https://no-color.org/
(Bitbake rev: d9986c54cd3d67ed1f7cb636b17696c8d0d4db85)
Signed-off-by: Peter A. Bigot <pab@pabigot.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
FIXES [YOCTO #15404]
When using git fetcher on a repo with parentheses in its URL, the
invocation of the git clone command will fail. The clone directory
is not quoted thus the shell will return an error and won't execute
the command.
(Bitbake rev: b5624ee5643d881afa004571a096a189ab5389b5)
Cc: Philippe Rivest <privest@genetec.com>
(Bitbake rev: 12f9738577934ad7c99f0770f1392a9d6050e7d6)
Signed-off-by: Philippe Rivest <technophil98@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Limit the split to key & value (2 items) instead of the n items one
can get if there are '=' characters in the value.
Fixes [YOCTO #15447]
(Bitbake rev: 86315961829ab1d137a0265cc246c44d3929e1fb)
Signed-off-by: david d zuhn <david.zuhn@sonos.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(Bitbake rev: becf88c2250a47102c8d36ad8b40839e0bfa9137)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
preserved_envvars() is used when the BB_ENV_PASSTHROUGH
environment variable is not set. Therefore, its code shouldn't
return this variable.
(Bitbake rev: 0a33b560233b983456178541603ab96fea22238b)
Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Reported-by: Robert P. J. Day <rpjday@crashcourse.ca>
Tested-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently when git fetches fail, it destroys all the existing local clone data.
For large repositories this can introduce long build delays when for example,
you just typo'd the git revision hash.
The git fetcher should be able to recover most directories so when the fetch is
for a git repo, avoid removing things unless clean is explicitly called
(e.g. a -c cleanall task).
(Bitbake rev: 1b3cd039fe19b24bd4be9a0202a98cdcbb0e9443)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Drop the ".py" file extension from the "-u" option's argument that has been
overlooked while applying the original patch (see [1]) to make the example work.
While at it sort the recipes' names consistently with respect to what is found
in the self-test examples below.
[1] https://lore.kernel.org/bitbake-devel/6f2645a7c4db2ae149d387544d2b94209cfed3f4.camel@linuxfoundation.org/
(Bitbake rev: 1f4d517b7a0389f78d1f791135f8dc9120e9912b)
Signed-off-by: Ulrich Ölmann <u.oelmann@pengutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Adds support for asyncrpc servers to send connection headers to clients
on connection. Since this is a breaking protocol change, clients must
opt-in to expect headers from the server, corresponding to a version
bump in the client protocol.
(Bitbake rev: 1cb2b8be6cc5269553f549285592e47b7d29db03)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The current implementation only performs a git lfs fetch alongside of a
regular git fetch. This causes issues when the downloaded revision is
already part of the fetched repository (e.g. because of moving back in
history or the updated revision already being part of the repository at
the time of the initial clone).
Fix this by explicitly checking whether the required LFS objects are
available in the downloade directory before confirming that a downloaded
repository is up-to-date.
This issue previously went unnoticed as git lfs would silently fetch the
missing objects during the `unpack` task. With network isolation turned
on, this no longer works, and unpacking fails.
(Bitbake rev: cfae1556bf671acec119a6c8bbc4b667a856b9ae)
Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Some ditros don't enable /proc/pressure and it tends to be those which we
see bitbake timeout issues on, seemingly as load gets too high and the bitbake
processes don't get scheduled in for minutes at a time.
Add support for stopping running extra tasks if the system load average goes
above a certain threshold by setting BB_LOADFACTOR_MAX.
The value used is scaled by CPU number, so a value of 1 would be when
the load average equals the number of cpu cores of the system, under one
only starts tasks when the load average is below the number of cores.
This means you can centrally set a value such as 1.5 which will then
scale correctly to different sized machines with differing numbers
of CPUs.
The pressure regulation is probably more accurate and responsive, however
our graphs do show singificant load spikes on some workers and this
patch is aimed at trying to avoid those.
Pressure regulation is used where available in preference to this load
factor regulation when both are set.
(Bitbake rev: 14a27306f6dceb4999c2804ccae5a09cc3d8dd49)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This reverts commit 987ab2a446aab235d6e86e97c05f25fb800d7acc.
There's been a report that this breaks downloads from Jfrog Artifactory
as self.user_agent is set to 'Mozilla Firefox', and when Artifactory
sees that, it sends a response tailored for showing in an interactive browser
(which in my opinion it has every right to).
If we're using wget, we should say so via wget's default; handling uncooperative
servers should be done on per-recipe basis, and ideally with tickets
to admins of those servers.
(Bitbake rev: feef5cd12e877f42ffcace168d44b0e6eb80a907)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
correctly
Before, everything up to the last slash was removed when extracting the
names of the tags. This would lead to that a tag such as "agent/11.0.0"
would be incorrectly identified as "11.0.0", which would then be treated
as a correct version matching "^(?P<pver>\d+(\.\d+)+)".
(Bitbake rev: 8b21024b9966d5158ac4a77e87ffb935c2a57764)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This is mostly preparations for the next commit.
(Bitbake rev: dcd2abfde55cc59d9869a7c97620b6fc30a52047)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In some cases, the version found by latest_versionstring() may be higher
than the real version. Make it possible to specify a maximum version so
that this case can be detected.
(Bitbake rev: 9134d4777109bc78410c3e641420d9a78b485e33)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The purpose of ensuring 'incremental fetch' is not easy to see from
the codes. So add comments to explain this.
(Bitbake rev: 8b890b87e30cd05ec92ed71ee3691a47b4d77253)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We had issue when BB_ALLOWED_NETWORKS is set and `bitbake grpc-native -c fetch'
failed even with all contents available in PREMIRRORS.
Add a test case to ensure no regression in the future.
(Bitbake rev: 80c91ceb81b1cae203067af58d3f1fe9c619ae83)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
For gitsm recipes, it's possible that some URL is used more than once.
e.g.,
A -> B:rev1 (B is a submodule of A)
A -> C (C is a submodule of A)
C -> B:rev2 (B is a submodule of C)
A anc C are both using B as submodules, but on different revs.
Now if we have:
B:rev1 -> D
B:rev2 -> E
Then, the mirror will not be fully used.
Say we have all repo mirrors for A, B, C, D, E, then in theory it's not
necessary to reach out to any network for downloading. But it's not the
case. After downloading B(rev1) and its submodule D from mirrors, the fetch
process continues to download C, thus B(rev2) and E. Now it finds that B
needs an update because its submodule E needs an update. Of course this is
true because E is not downloaded yet. Now the problem comes to whether to
use mirror or not. The git.py defines try_premirror to return 'False' when
the ud.clonedir exists. As B has been cloned, the ud.clonedir exists and
try_mirror returns False, resulting in not using mirror and going to upstream
directly.
We can see that the mirrors are not fully used. This is usually not problem,
as the cost is only some network download. But in case the following two
settings are there, we get errors.
BB_NO_NETWORK = "0"
BB_ALLOWED_NETWORKS = "*.some.allowed.domain"
In such case, the gitsm recipe A will fail to fetch. Note that all contents
that A needs are in mirrors and now it's failing to fetch. This is unexpected.
Note that the different revs of the same repo in gitsm recipe is not the only
way to reveal this problem. For example, there might be a recipe call B that
uses B:rev3. Check the protobuf and grpc recipes as an example.
For now, we can use the following steps to reproduce this issue. To be clear,
the grpc recipe in meta-oe is now 1.60.0.
1. Add in local.conf:
DL_DIR = "${TOPDIR}/downloads-premirror"
bitbake grpc -c fetch
2. Comment out the DL_DIR setting in local.conf and add the following lines:
PREMIRRORS:append = " \
git://.*/.* git://${TOPDIR}/downloads-premirror/git2/MIRRORNAME;protocol=file \n \
gitsm://.*/.* gitsm://${TOPDIR}/downloads-premirror/git2/MIRRORNAME;protocol=file \n \
"
3. Set BB_NO_NETWORK = "1" and then 'bitbake grpc -c fetch'.
This command succeeds and this shows that the premirror holds everything we need.
4. Add the following lines and then 'bitbake grpc -c fetch'.
BB_NO_NETWORK = "0"
BB_ALLOWED_NETWORKS = "*.some.domain"
After step 4, the error message is as below:
ERROR: grpc-1.60.0-r0 do_fetch: The URL: 'gitsm://github.com/protocolbuffers/protobuf.git;protocol=https;name=third_party/protobuf;subpath=third_party/protobuf;nobranch=1;lfs=True;bareclone=1;nobranch=1' is not trusted and cannot be used
This patch fixes this problem by handling this corner case, that is, if the URL is
not trusted from the settings of BB_NO_NETWORK and BB_ALLOWED_NETWORKS, then we should
try premirrors because trying to reach upstream is destined to fail.
(Bitbake rev: e1be272ad105b47d3131b77168d9172386993fcb)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fetch from premirror few times to emulate multiple machines sharing same
clonedir or few rebuilds of the package from (pre)mirror
Regression test for [Yocto #15369]
(Bitbake rev: 7fcbac574c68f16b95ab7abb2874931d168d3c9e)
Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fix the issue with using of (pre)mirror in case if clonedir exists but
outdated.
Previous version of the code fetched new mirror content into FETCH_HEAD
instead of branch which caused refetch from the upstream. Add new remote
add fetch from it instead so the ref can be found by "_contains_ref"
Fixes [Yocto #15369]
(Bitbake rev: 69588e2a5c7c200e47b02b2391498dcb72388bd2)
Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When git is configured with safe.bareRepository=explicit [1], the
bitbake selftests fail miserably. LWN has an article about the
problem that this configuration option addresses and why it is useful
in [2].
It also seems that it is being rolled out in some environments as a
default for users.
In order to allow having this configuration turned on for a user's
environment in general, the fetcher has to be tought to use --git-dir=
for all relevent git operations.
The alternative, implemented here, is to forcibly turn off that option
for all git operations. In the future, we could look into converting
these to using the --git-dir= command line argument instead.
Link: https://git.kernel.org/pub/scm/git/git.git/tree/Documentation/config/safe.txt#n1 [1]
Link: https://lwn.net/Articles/892755/ [2]
(Bitbake rev: a45e14a7343e36101e45639931322e5649587f57)
Signed-off-by: André Draszik <andre.draszik@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When git is configured with safe.bareRepository=explicit [1], the
bitbake git fetcher fails miserably. LWN has an article about the
problem that this configuration option addresses and why it is useful
in [2].
It also seems that it is being rolled out in some environments as a
default for users.
In order to allow having this configuration turned on for a user's
environment in general, the fetcher has to be tought to use --git-dir=
for all relevent git operations.
The alternative, implemented here, is to forcibly turn off that option
for all git operations. In the future, we could look into converting
these to using the --git-dir= command line argument instead.
While at it, fix one open-coded invocation of git that wasn't using
ud.basecmd
Link: https://git.kernel.org/pub/scm/git/git.git/tree/Documentation/config/safe.txt#n1 [1]
Link: https://lwn.net/Articles/892755/ [2]
(Bitbake rev: 5f3b1d8dc9ee70e707536bd75ee845b547440c97)
Signed-off-by: André Draszik <andre.draszik@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(Bitbake rev: c1e0a0b6ddc9667c9d62319bd9ccd4eb8c64c2a6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Adds API to query if unihashes are known to the server in parallel
(Bitbake rev: 7e2479109b40ce82507f73b4f935903f7f79fb06)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Implements a new API called get_unihashes() that allows for querying
multiple unihashes in parallel.
The API is also reworked to make it easier for derived classes to
interface with the new API in a consistent manner. Instead of overriding
get_unihash() to add custom handling for local hash calculating (e.g.
caches) derived classes should now override get_cached_unihash(), and
return the local unihash or None if there isn't one.
(Bitbake rev: 6faf48c09a4003a31b32e450779fb8ac9cc5e946)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Adds an abstract base class that can be used to implement a pool of
client connections. The class implements a thread that runs an async
event loop, and allows derived classes to schedule work on the loop and
wait for the work to be finished.
(Bitbake rev: f113456417f9ac0a4b44b291a6e22ea8219c3a5f)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
* Create an ncurses version of the GTK app "taskexp.py".
* Add these additional features:
- Sort tasks in recipes by their dependency order
- Print individual and/or recipe-wide dependencies to a file
- Add a wild card filter
- Show the target recipes on BOLD
* Provide a GUI self test
* Provide a non-ncurses self test for ptest
(Bitbake rev: f49bec66ad51c8cddeceafbbe2445c46e396ee8b)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
dependencies
"bitbake world -n --setscene-only" shows poor performance as the numbers of tasks
increases. Analysys shows this is due to the "deferred" hard dependencies being
repeatedly processed which doesn't allow much forward progress in overall task
execution.
To avoid this, mark when it has been done and don't reprocess until dependencies
are updated. We have to be careful as we've seen bugs where these dependencies
aren't processed at the right time. They also have to be reproceseed during task
migration/rehashing so the code has to "self heal" the data structures.
(Bitbake rev: e5609bac06c17dabcf6286b47b1a3f19f5a1160f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
A quick profile of bitbake world showed 147 million calls to taskname_from_tid().
The next_buildable_task function is performance senstive so move the call
inside the if block to reduce the number of calls and speed the code up.
(Bitbake rev: 8b332c16a7b6b85c5cbe1919dd8cae45fda6adf9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Now that runqueue performance profiling works again we can see a lot
of time is lost in build_taskdepdata. Whilst we can't compute that
in advance, we can compute the individual entries.
Therefore put a cache in place to compute those and save overhead in
starting up tasks.
(Bitbake rev: c4519b542702ba25023e53d77b275a6fa571ec50)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Ensure the message matches the filenames the code actually uses.
(Bitbake rev: deb7db2e2b125c6a6732db4f185f4de5926494fd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add the "--user-agent" paramater in the wget base command to
perform all wget commands with this parameter, because a few
HTTP servers block requests with the default wget user agent.
For example, "hg.openjdk.org" never send a response to requests
have been sent with wget:
wget https://hg.openjdk.org/jdk8u/jdk8u/archive/jdk8u272-ga.tar.bz2
https://hg.openjdk.org/jdk8u/jdk8u/archive/jdk8u272-ga.tar.bz2
Resolving hg.openjdk.org (hg.openjdk.org)... 23.54.129.73
Connecting to hg.openjdk.org (hg.openjdk.org)|23.54.129.73|:443... connected.
HTTP request sent, awaiting response...
(Bitbake rev: d6fa261a9603677f0b3abbd309c1ca6073b63f4c)
Signed-off-by: Thomas Perrot <thomas.perrot@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When the idle/main loop was added, we didn't include profiling information
for it. There is a performance issue in there, add logging for it.
(Bitbake rev: d8d5cd43a60560f67e86f4f625113b0f73b944c0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Support AWS_SESSION_TOKEN which is used in temporary
AWS credentials.
Fixes [YOCTO #15384].
(Bitbake rev: ae1e4c90bbc2002cb2728c64649c095c00220ceb)
Signed-off-by: Toni Lammi <toni.lammi@kone.com>
Reported-by: Toni Lammi <toni.lammi@tl-software.fi>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
* github no longer provides support for subversion clients:
https://docs.github.com/en/enterprise-server@3.11/get-started/working-with-subversion-on-github/support-for-subversion-clients
it was shut down on 2024-01-08:
https://github.blog/2023-01-20-sunsetting-subversion-support/
and this test was now failing with:
======================================================================
ERROR: test_external_svn (bb.tests.fetch.SVNTest.test_external_svn)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/OE/layers/bitbake/lib/bb/tests/fetch.py", line 1287, in test_external_svn
fetcher.download()
File "/OE/layers/bitbake/lib/bb/fetch2/__init__.py", line 1896, in download
raise FetchError("Unable to fetch URL from any source.", u)
bb.fetch2.FetchError: Fetcher failure for URL: 'svn:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project;module=trunk;protocol=file;externals=allowed;rev=2'. Unable to fetch URL from any source.
Stdout:
Fetch svn:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project;module=trunk;protocol=file;externals=allowed;rev=2
Failed to fetch URL svn:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project;module=trunk;protocol=file;externals=allowed;rev=2, attempting MIRRORS if available
Fetcher failure: Fetch command export PSEUDO_DISABLED=1; /usr/bin/env svn --non-interactive --trust-server-cert co --no-auth-cache -r 2 file:///tmp/bitbake-fetch-zdvedqt_/svnfetch_localrepo_wup8mgn6/project/trunk@2 trunk failed with exit code 1, output:
A trunk/README.md
U trunk
Checked out revision 2.
svn: warning: W205011: Error handling externals definition for 'trunk/bitbake':
svn: warning: W170013: Unable to connect to a repository at URL 'https://github.com/PhilipHazel/pcre2.git'
svn: E205011: Failure occurred processing one or more externals definitions
in the rare cases where subversion was still installed on the host
running bitbake-selftest :).
to avoid this use still alive repository from https://svn.apache.org/
and pick something rather small and only the trunk subdirectory which
is fast to fetch:
svn co https://svn.apache.org/repos/asf/serf/trunk
takes just 2 sec here
adjust expected dir/file to use "protocols/fcgi_buckets.h" instead of
"trunk/README"
(Bitbake rev: a735898abcf056f897c9350bb128a5637e6b4617)
Signed-off-by: Martin Jansa <martin.jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
error
* when comparing 2 tmp/stamps/*do_configure.sigdata* I got
TypeError("filename must be a str or bytes object, or a file")
but both files I was comparing were
Zstandard compressed data (v0.8+), Dictionary ID: None
according to "file"
with TypeError catched to show which file it failed to open I got better
error which shows it was trying to read "do_prepare_recipe_sysroot.sigdata"
file now and after a while you might notice that it's not just the expected
file, but a dict with 'path', 'sstate', 'time'.
Fix that in bitbake-diffsigs but keep the TypeError and add OSError
in case it will eventually walk on file which isn't zstd compressed
pipecompress throws CompressionError.
ERROR: Failed to open {'path': '5.15.do_prepare_recipe_sysroot.sigdata.99b12a401341a0df7c3553cb00c87a7674295496bd5c25ed71764ee0d0fb8eb8', 'sstate': False, 'time': 1707136354.991718}: filename must be a str or bytes object, or a file
Traceback (most recent call last):
File "bitbake/bin/bitbake-diffsigs", line 192, in <module>
output = bb.siggen.compare_sigfiles(options.sigdatafile1, options.sigdatafile2, recursecb, color=color)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "bitbake/lib/bb/siggen.py", line 1039, in compare_sigfiles
recout = recursecb(dep, a[dep], b[dep])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "bitbake/bin/bitbake-diffsigs", line 102, in recursecb
out2 = bb.siggen.compare_sigfiles(hashfiles[hash1], hashfiles[hash2], recursecb, color=color)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "bitbake/lib/bb/siggen.py", line 857, in compare_sigfiles
raise err
File "bitbake/lib/bb/siggen.py", line 853, in compare_sigfiles
with bb.compress.zstd.open(a, "rt", encoding="utf-8", num_threads=1) as f:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "bitbake/lib/bb/compress/zstd.py", line 12, in open
return bb.compress._pipecompress.open_wrap(ZstdFile, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "bitbake/lib/bb/compress/_pipecompress.py", line 59, in open_wrap
raise TypeError("filename must be a str or bytes object, or a file")
TypeError: filename must be a str or bytes object, or a file
* if I replace zstd file with just plaintext it fails with:
$ echo foo > foo
$ echo foo > bar
$ bitbake-diffsigs foo bar
zstd: /*stdin*\: unsupported format
ERROR: Process died with 1
sys:1: ResourceWarning: unclosed file <_io.BufferedReader name='foo'>
with this change it shows the name of the file which it failed to uncompress:
$ bitbake-diffsigs foo bar
zstd: /*stdin*\: unsupported format
ERROR: Failed to open sigdata file 'foo': Process died with 1
ERROR: Process died with 1
sys:1: ResourceWarning: unclosed file <_io.BufferedReader name='foo'>
(Bitbake rev: f3f843a4fd6e0a9e8f6edef5dd3cf1fce29c50ba)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We've added a new statement, inherit_defer, so bump the version so this
can be checked.
(Bitbake rev: 191e6eb2bceb467c97e315301f1f64722cf0e976)
Signed-off-by: Ross Burton <ross.burton@arm.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add support for an inherit_defer statement which works as
inherit does but is only evaulated at the end of parsing.
This allows conditional expressions to be evaulated 'late' meaning changes
to PACKAGECONFIG from bbappends can be applied for example. This addresses
a usability/confusion issue users have with the current conditional
inherit mechanism since they don't realise the condition has to be
expanded immediately with the current model.
There is a commented out warning we could enable in future which
warns about the use of a conditional inherit that isn't deferred.
There is a behaviour difference in the placement of the inherit,
particularly around variables set using ?= which means wen swapping
from one to the other, caution must be used as values can change.
(Bitbake rev: 5c2e840eafeba1f0f754c226b87bfb674f7bea29)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Where chains of RDEPENDS are multiple levels deep, the runall code was not
accounting for this and recursing deeply enough to gather all dependencies.
Fix this by iterating over the result until no more dependencies are found.
Tested-by: Jonas Gorski <jonas.gorski@bisdn.de>
Reported-by: Jonas Gorski <jonas.gorski@bisdn.de>
(Bitbake rev: 966f25dfc23a6d17b2b6d3e0100e9ae264f99025)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Previously printdiff code would iterate over tasks that were reported as invalid or absent,
trying to follow dependency chains that would reach the most basic invalid items in the tree.
While this works in tightly controlled local builds, it can lead to bizarre reports
against industrial-sized sstate caches, as the code would not consider whether the
overall target can be fulfilled from valid sstate objects, and instead report
missing sstate signature files that perhaps were never even created due to hash
equivalency providing shortcuts in builds.
This commit reworks the logic in two ways:
- start the iteration over final targets rather than missing objects
and try to recursively arrive at the root of the invalid object dependency.
A previous version of this patch relied relies on finding the most 'recent'
signature in stamps or sstate in a different function later, and recursively
comparing that to the current signature, which is unreliable on real world caches.
- if a given object can be fulfilled from sstate, recurse only into
its setscene dependencies; bitbake wouldn't care if dependencies
for the actual task are absent, and neither should printdiff
I wrote a recursive function for following dependencies, as
doing recursive algorithms non-recursively can result in write-only
code, as was the case here.
[YOCTO #15289]
(Bitbake rev: aadeca63da5d96160ce4d6d71da556e2e033f9b7)
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Richard Purdie
Joshua Watt
Michael Opdenacker
Rob Woolley
Derek Erdmann
Felix Moessbauer
Viswanath Kraleti
Ross Burton
Peter A. Bigot
Philippe Rivest
david d zuhn
Alexander Kanavin
Ulrich Ölmann
Philip Lorenz
Peter Kjellerstedt
Chen Qi
Pavel Zhukov
André Draszik
David Reyna
Thomas Perrot
Toni Lammi
Martin Jansa