| Age | Commit message (Collapse) | Author |
|---|
|
directories
The new BB_SIGNATURE_LOCAL_DIRS_EXCLUDE allows you to specify a list
of directories to exclude when making taskhash, our specific case
is using SRC_URI that points local VCS directory.
Use bb.fetch.module to set default to: "CVS .bzr .git .hg .osc .p4 .repo .svn"
(Bitbake rev: 923aff060d8aba8456979c35b16d300ba7c13ff9)
Signed-off-by: Aníbal Limón <anibal.limon@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We need to ensure the correct working directory is used for this
tar command to succeed.
(Bitbake rev: e1a2a0e29a75dead3426ae083387181ad52da91f)
Signed-off-by: Christian Herzig <cherzig@gauselmann.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We have bb.debug(2, xxx) messages in cooker which are useful for debugging
but have really bad effects on performance, 640,000 calls on recent profile
graphs taking tens of seconds.
Rather than commenting out debug which can be useful for debugging, don't
create events for debug log messages from cooker which would never be seen.
We already stop the messages hitting the IPC but this avoids the overhead
of creating the log messages too, which has been shown to be signficiant
on the profiles. This allows the code to perform whilst allowing debug
messages to be availble when wanted/enabled.
(Bitbake rev: f04cd931091fb0508badf3e002d70a6952700495)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This doesn't save much time but does make the profile counts for the
function more accurate which is in itself useful.
(Bitbake rev: d446fa89d206fbc6d098215163c968ea5a8cf4a9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We were seeing this running thousands of times with hashequiv, do
the filtering where it makes more sense and make it persist.
(Bitbake rev: 2cfeb9998a8ad5b1dcda0bb4e192c5e4306dab17)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(Bitbake rev: 105d1f0748edde7753a4063e6fdc758ffc8a8a9e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Move the calls to difference_update out a code level which improves efficiency
significantly.
Also further combine the outer loop for efficiency too.
These two changes remove a bottleneck from the performance charts.
(Bitbake rev: e28ec69356f1797de3e4e3fca0fef710bc4564de)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Creating a new sorted list of sstate tasks each iteration through runqueue is
extremely ineffecient and was compounded by the recent change from a list to set.
Create one sorted list instead of recreating it each time.
(Bitbake rev: de18824996841c3f35f54ff5ad12f94f6dc20d88)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There are two operations happening in get_taskhash, the building of the
underlying data and the calculation of the hash.
Split these into two funtions since the preparation part doesn't need
to rerun when unihash changes, only the calculation does.
This split allows sigificant performance improvements for hashequiv
in builds where many hashes are equivalent and many hashes are changing.
(Bitbake rev: 6a32af2808d748819f4af55c443578c8a63062b3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The "no dependencies" task case was not being correctly considered in this
code and seemed to be the cause of occasionaly task hash mismatch errors
that were being seen as the dependencies were never accounted for properly.
(Bitbake rev: 608b9f821539de813bfbd9e65950dbc56a274bc2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The looping over multiple changed hashes causes many calls to get_taskhash
and get_unihash which are potentially slow and then overwritten.
Instead, batch up all the tasks which have changed unihashes and then
do one big loop over the changed tasks rather than each in turn.
This makes worlds of difference to the performance graphs and should speed
up build where many tasks are being rehashed.
(Bitbake rev: c9c68d898985cf0bec6fc95f54c151cc50255cac)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Discovered with a recipe under devtool. The ${S}/singletask.lock file (added by
externalsrc.bbclass) was leaked, giving a warning like:
WARNING: <PN>+git999-r0 do_populate_lic: /home/laplante/yocto/sources/poky/bitbake/lib/bb/build.py:582: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/laplante/yocto/build/workspace/sources/<PN>/singletask.lock' mode='a+' encoding='UTF-8'>
exec_func(task, localdata)
(Bitbake rev: 6beddf6214e22b4002626761031a9e9d34fb04db)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The local hash equivalence server used by bitbake requires python 3.5,
so bump up the minimum required version.
[YOCTO #13678]
(Bitbake rev: 1412dcc077b1bea10b72fc8b525d6258dca46d97)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There is a significant performance issue in get_unihash(). The issue turns out
to be the lookups of setscene tasks. We can fix this by using a set() instead of
the current list.
(Bitbake rev: 1e561672d039ebfb8cd0e0654a44dcf48513317c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This reverts commit c4b8440f730c33eaf9f818b856ae81b2f1017fec.
The logic in this change is flawed and needs to be re-thought.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We're seeing huge slowdowns on large builds on the autobuilder. A qemux86 world
build was running really slowly, a primary feature was lots of rehashing going on
due to an sstate change which caused a rebuild when all output should be identical.
This was traced to the hundreds of thousands of calls to get_unihash() from
get_taskash(). If we simplify the unitaskhashes data structure, we can bypass
the function call and access the data directly. In local profile charts,
this significanly sped up process_possible_migrations(), both on profiles
and visually.
Whilst this change doesn't aid readability, it does solve an otherwise
huge performance issue.
(Bitbake rev: c4b8440f730c33eaf9f818b856ae81b2f1017fec)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Batch all updates to scenequeue data together in a single invocation
instead of checking each task serially. This allows the checks for
sstate object to happen in parallel, and also makes sure the log
statement only happens once (per set of rehashes).
(Bitbake rev: db033a8f8a276d864bdb2e1eef159ab5794a0658)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Its possible the new unihash may not exist in sstate. Currently the code
would create an sstate object with the old hash however this updates it to
create the object with the new unihash.
(Bitbake rev: abcaa1398031fa5338a43859c661e6d4a9ce863d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
dependencies rehash
An example:
NOTE: recipe binutils-cross-testsuite-2.32.0-r0: task do_check: Started
ERROR: Taskhash mismatch b074da4334aff8aa06572e7a8725c941fa6b08de4ce714a65a90c0c0b680abea versus 17375278daed609a7129769b74a1336a37bdef14b534ae85189ccc033a9f2db4 for /home/pokybuild/yocto-worker/qemux86-64/build/meta/recipes-devtools/binutils/binutils-cross-testsuite_2.32.bb:do_check
NOTE: recipe binutils-cross-testsuite-2.32.0-r0: task do_check: Succeeded
Is caused by a rehash in a dependency happening somewhere earlier in the build
and the taint being reset.
Change the code so that nostamp taints are preserved to avoid the issue.
(Bitbake rev: 61624a3fc38e8546e01356d5ce7a09f21e7094ab)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We've seen cases where a task can execute with a given pid, complete
and a new task can start using the same pid before the UI handler has
had time to adapt.
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/qemux86-alt/build/bitbake/lib/bb/ui/knotty.py", line 484, in main
helper.eventHandler(event)
File "/home/pokybuild/yocto-worker/qemux86-alt/build/bitbake/lib/bb/ui/uihelper.py", line 30, in eventHandler
del self.running_tasks[event.pid]
KeyError: 13490
This means using pids to match up events on the UI side is a bad
idea. Change the code to use task ids instead. There is a small
amount of fuzzy matching for the progress information since there
is no task information there and we don't want the overhead of a task
ID in every event, however since pid reuse is unlikely, we can live
with a progress bar not quite working properly in a corner case like
this.
[YOCTO #13667]
(Bitbake rev: e427eafa1bb04008d12100ccc5c862122bba53e0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(Bitbake rev: 3aad9978be2a40d4c535a5ae092f374ba2a5f627)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Based on the hashserv's new ability to accept hash mappings, update runqueue
to use this through a helper function in siggen.
This addresses problems with meta-extsdk-toolchain and its dependency on
gdb-cross which caused errors when building eSDK. See the previous commit
for more details.
(Bitbake rev: 39098b4ba2133f4d9229a0aa4fcf4c3e1291286a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When follow the instructions of multiconfig from Yocto dev manual that
set in core-image-sato recipe:
do_image[mcdepends] = "multiconfig:x86:arm:core-image-minimal:do_rootfs"
it show too many annoying warnings look like:
| WARNING: Deferring mc:x86:virtual:native:/buildarea6/kkang/poky/meta/recipes-support/libxslt/libxslt_1.1.33.bb:do_populate_sysroot
| after mc:arm: virtual:native:/buildarea6/kkang/poky/meta/recipes-support/libxslt/libxslt_1.1.33.bb:do_populate_sysroot
Treat them as infomations rather than warnings.
(Bitbake rev: 0fb4b5153237af4a13b2c65711ab798b0de06c2f)
Signed-off-by: Kai Kang <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(Bitbake rev: 35784582fdbb2092eddec094deb6ab9c87666b5e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If a hash is reported to the hash server, the stamp written out by the
current task didn't account for any new hash the server may have provided.
Fix this so the correct stamp is written. This means "bitbake X; bitbake X"
no longer rebuilds lots of things when hashequiv is active.
(Bitbake rev: 9dc5178f56ab8ae37e1a646c09117c503e48d072)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
accounted for in migrations
Setscene hard dependencies were not being correctly handled during task migration.
For example, do_package of recipe X might become valid due to hashequiv yet we're
still rebuilding pseudo-native, a harddep of do_package. This would mean
it would try to execute that setscene task despite pseudo not being present.
Fix this by ignoring tasks with failed harddependencies. This does mean
stlightly more rebuilds than is optimal but it avoids build crashes. Ultimately
the new runqueue model can likely better handle these cases than the older codebase
could but that is for another more invasive patch.
(Bitbake rev: 0dc61e8b9964f7fe41d51b6a481ca7aeaacd8bce)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The key problem is that currently running setscene tasks are not
accounted for when processing task migrations. This means can allow
two of the same task to execute at the same time with unpredictable
effects.
This change allows us to stop doing that and refactor the code slightly
to make it clearer that these conditions don't arrive even with
deferred tasks.
(Bitbake rev: ac1922d348613ca46a1047ad5ddf755eac16d568)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
bitbake/lib/bb/fetch2/clearcase.py:148: SyntaxWarning: "is" with a literal. Did you mean "=="?
if command is 'mkview':
bitbake/lib/bb/fetch2/clearcase.py:155: SyntaxWarning: "is" with a literal. Did you mean "=="?
elif command is 'rmview':
bitbake/lib/bb/fetch2/clearcase.py:159: SyntaxWarning: "is" with a literal. Did you mean "=="?
elif command is 'setcs':
Python 3.8 is quite correct and we so mean "==" here, fix it to
avoid the warnings.
(Bitbake rev: 1fab03f6e10eaa13b8a89ce0b2f9fe8ce5157189)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
md5_file() uses a mmap() window to improve performance when hashing files, so
refactor the code and do the same for SHA1 and SHA256.
(Bitbake rev: ecf87437ff796e17c3e4f210b5803b0136a9e8a4)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add a basic test for bb.utils.md5_file() etc.
(Bitbake rev: e944d02fe678f7c6b05c62419f8bceb0709f3037)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This should have been removed together with expanded_data in commit
e3694e73 (cooker/command: Drop expanded_data).
(Bitbake rev: 33197db8abf82be240d7c1c5c9d2484a08a90849)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fix mercurial fetching after breakage from changes to the core fetcher.
Fix username and password usage and setting moddir needed by setup_revisions.
(Bitbake rev: 8962b27abc68427eae085624f26300108ae88aa1)
Signed-off-by: Volker Vogelhuber <v.vogelhuber@digitalendoscopy.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Bibake is currently too 'chatty' when hash equivalence is enabled. Fix
this by only printing the log output if a rehash happens and it matches
an sstate object.
Also, pass a summary option to the hash checking function. This was
already changed to a mechanism which allows addition of new parameters
so this should be backwards and forwards compatible.
(Bitbake rev: 0c4515603ad08775e3b0404cba5374367e49f236)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There was a weird error in OE-Core where "devtool modify virtual/kernel"
was showing basehash mismatch errors. This was due to SRCPV sometimes being:
AUTOINC+b867b78b50_47b80ef7bd and sometimes AUTOINC+b867b78b50_255a750d28.
The latter hash comes from KBRANCH and meant sometimes the correct branch
was seen, sometimes it was not. The issue was complicated by the execution
using a remote datastore over tinfoil.
The problem turns out to be a fetcher caching error. If the datastore
changes, the cached url data may not be valid.
We therefore ensure we match cached url data against the datastore that
generated it, which appears to fix this issue.
(Bitbake rev: 0540abd338ed2ffd822edbd2947cab2f18873422)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
[YOCTO #13586]
(Bitbake rev: f7a973604fbfd1059875ff1fabb3f885df9c5b95)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When bitbake determines if a git clone needs updating, it only checks for the
needed srcrevs, not the revs listed in BB_GIT_SHALLOW_REVS, which will fail if
using shallow and the needed rev was added to the upstream git repo after a
previous fetch. Ensure that we also check for shallow revs.
[YOCTO #13586]
(Bitbake rev: 93b65e3c6ca64f644946953980595c44fbbcc748)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(Bitbake rev: 459ad524756a3f9b50feeedf31e33502dceae8d5)
Signed-off-by: Ola x Nilsson <olani@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(Bitbake rev: 5d83d828cacb58ccb7c464e799c85fd2d2a50ccc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This allows metadata to depend on SignatureGeneratorUniHashMixIn which was recently added.
(Bitbake rev: f0f814407fdd2fffa7071c36c011b489bfcd53da)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Some of the objects that bitbake reports to Toaster as dependencies to packages
are known objects that are not packages, for example library files and kernel
modules. In the Toaster logs, mark these as "Info" instead of "Warning".
[YOCTO #13386]
(Bitbake rev: 0d66f644d647900e8f5afa526a6d9cee687c41cc)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The hashserve can delete its socket whilst the cleanup us happening leading to
backtraces and test failures.
Add code to avoid this race condition.
[YOCTO #13542]
(Bitbake rev: efd7b025cee25d0ee668c09476395d08fcf5ae1a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The full paths make the cache useless in the sdk. They also bloat the
cache size. They're for human debugging benefit only so compromise and
reduce this to the filename.
(Bitbake rev: 3b275c4083eae1d3781f0862919af9de83932b0f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
A minor performance optmisation to keep lists smaller when running large
builds. We can do this since once a task has been built, we don't need
to worry about it. This improves a major bottleneck that shows up on
performance profile charts in dryruns.
(Bitbake rev: cd6b89230823707c3c9bb9e6883bf5a971916581)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There are some runqueue code paths where the unihash cache would not be
saved where for example only parsing or an occurred. Save the cache at the
end of runqueue generation to ensure entries are cached.
(Bitbake rev: 9eee0d36870c11dd303894a6151c33a83bd3a1bc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Use the unihash in the output filename of sigdata files else the contents
of stamp directories is misleading. Write the unihash into the singature to
make it clear what happened.
(Bitbake rev: feb01ee54d3706fe93768f332054c7532f7209e4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We need to set the setscene tasklist before we call into the
taskhash/unihash code else the behaviour is inconsistent.
Avoid reporting hashes for non setscene tasks since we'd never
query that.
(Bitbake rev: 419a7840b8627278db694029c25df00214d01d96)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently runqueue will rerun setscene tasks multiple times as hashes
change. This has caused numerous problems since a setscene task may
become "unavailable" for some future signature combination and the code
then can't easily "unskip" tasks its already passed into the execution
queue.
At least for now, only run setscene once and assume they're equivalent
at that point. In practise that has been much more stable in testing.
Tweak the test to match the change in behaviour.
(Bitbake rev: 4205a3ef23834f317642bba155d67cd772176fb6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The setscenetasks list needs to be available in the worker contexts
else the signature behaviour there mismatches what the server does.
Add the data to get/set_taskdata to ensure this happens.
(Bitbake rev: 632980ef90fe126b7ba3d138f4d574ae05914779)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Tasks were not migrating consistently, particularly:
* if a task was rehashed which had already run
* if a task which was valid became invalid due to a rehash
We need to always run the migration code for rehashed tasks and then
reprocess them for hash validity. This means rearranging the code.
It also means several tests are no longer correct and can't be written
correctly to work on all possible workflows so those are removed.
(Bitbake rev: 8443989ee41e9b162972935513e437b5c66ea74d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The hash server process is terminated and waited on with join(), so it
should not be a daemon. Daemonizing it cause races with the server
cleanup, especially in the selftest because the process may not have
terminated and cleanup up its socket before the test cleanup runs and
tries to do it.
[YOCTO #13542]
(Bitbake rev: 7c829675581818f92d57056b57fbd3880829b6bd)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Aníbal Limón
CHerzig@Gauselmann.de
Richard Purdie
Chris Laplante via bitbake-devel
Joshua Watt
Kai Kang
Ross Burton
Peter Kjellerstedt
Volker Vogelhuber
Christopher Larson
Ola x Nilsson
David Reyna