| Age | Commit message (Collapse) | Author |
|---|
|
We've seen PR Server timeouts on the autobuilder, this is likely from the
journal being blocked on disk IO generated by the build.
Since we're running with synchronous off, we may as well put the journal
into memory and avoid any IO related stalls.
(Bitbake rev: ee3fc6030e653f3244b065fc89aafd2a7c36ae04)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We're seeing performance problems with hashserv running on a normal build
system. The cause seems to be the large amounts of file IO that builds involve
blocking writes to the database. Since sqlite blocks on the sync calls, this
causes a significant problem.
Since if we lose power we have bigger problems, run with synchronous=off
to avoid locking and put the jounral into memory to avoid any write issues
there too.
This took writes from 120s down to negligible in my tests, which means
hashserv then responds promptly to requests.
(Bitbake rev: 7ae56a4d4fcf66e1da1581c70f75e30bfdf3ed83)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
BB_HASHSERVE
Its useful, particularly in the local developer model of usage, for
bitbake to start and stop a hash equivalence server on local port,
rather than relying on one being started by the user before the build.
The new BB_HASHSERVE variable supports this.
The database handling is moved internally into the hashserv code so that
different threads/processes can be used for the server without errors.
(Bitbake rev: a4fa8f1bd88995ae60e10430316fbed63d478587)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Whilst this isn't strictly necessary, its helpful if the log output is
consistent and its also helpful if bugs either appear or don't appear
for a specific configuration. Ensuring the various iterations we make
are deterministic (sorted) helps with this.
(Bitbake rev: 6a901bb904a97ca90d88be2c6901d3d32346282f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There is a compelling usecase for tasks being able to notify runqueue
that their "unihash" has changed. When this is recieved, the hashes of
all subsequent tasks should be recomputed and their new hashes checked
against existing setscene validity. Any newly available setscene tasks
should then be executed.
Making this work effectively needs several pieces. An event is added
which the cooker listen for. If a new hash becomes available it can
send an event to notify of this.
When such an event is seen, hash recomputations are made. A setscene
task can't be run until all the tasks it "covers" are stopped. The
notion of "holdoff" tasks is therefore added, these are removed from
the buildable list with the assumption that some setscene task will
run and cover them.
The workers need to be notified when taskhashes change to update their
own internal siggen data stores. A new worker command is added to do this
which will affect all newly spawned worker processes from that worker.
An example workflow which tests this code is:
Configuration:
BB_SIGNATURE_HANDLER = "OEEquivHash"
SSTATE_HASHEQUIV_SERVER = "http://localhost:8686"
$ bitbake-hashserv &
$ bitbake automake-native
$ bitbake autoconf-native automake-native -c clean
$ bitbake m4-native -c install -f
$ bitbake automake-native
with the test being whether automake-native is installed from sstate.
(Bitbake rev: 1f630fdf0260db08541d3ca9f25f852931c19905)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Rather than metadata driven sqlite databases for communication, use
bitbake's unitaskhashes variable instead.
(Bitbake rev: a0d941c787cf3ef030d190903279d311bc05d752)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We need to preserve unihash task hashes between runs. Use the new SimpleCache
class to create such a class within the signature generator which is loaded
at init time and saved when builds complete. The default is unpopulated but
metadata sig handlers can populate this cache.
(Bitbake rev: 1f326f2c29c2664a5daaeeb0c1fd332630efbdba)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Rather than a special copy of the data structure which we change, compute
the logic using set operations from other data we have. This means
we can add tasks back into the scenequeue without having to worry about
reversing operations on this variable with all the potential bugs that
might involve.
(Bitbake rev: b707d0cbc25fa336a1e95ff588f1ea37eee063eb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This adds a simple version of the MultiProcessCache which can be used to
save and load cache data, useful for a new usecase we have in
sigdata/runqueue.
(Bitbake rev: 19a6e35600ae6d2d1bcecca6e68ab8c37674774e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This code is closely tied with the hash server in bitbake and also means
we can't relibably test the hashserv runqueue functionality without OE
metadata. Moving this to bitbake as a MixIn class makes most sense
and encourages code collaboration and reuse as well as enabling easier
and more accurate testing of the APIs.
(Bitbake rev: 7bb79099a6c1b463d6ae9226c4cab5e76a965675)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Implements a number of optimizations to the SQL used in the hash
equivalence server:
1) Two indexes are created for the two methods (method, taskhash and
method outhash) by which rows are found in order to speed up the
lookup
2) An extra SELECT to lookup the just inserted row was removed. This
SELECT is unnecessary since all of the information about the newly
inserted row is already available.
3) A uniqueness constraint was added to the table. This should allow
the server to be multithreaded in the future since duplicate inserts
can be detected (and ignored). This change requires bumping the
database version to '2', since a uniqueness constraint can't be
added to an existing table.
4) Some comments are added to clarify the trick SELECT statement used
when inserting new equivalent hashes
(Bitbake rev: 7aec8632e67b4f0ab7b72692c40a42f6926608c3)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
handling 403 codes
The code says that some servers respond with 403 codes when they really
mean 405 codes. But we still need to account for legitimate 403 codes.
Before this change, I noticed that sstate mirror checking was taking a
very long time when I purposely entered incorrect credentials into my
.netrc file for our sstate mirror. Instrumenting the code, I discovered
tracebacks like the following for every mirror access attempt:
File "/home/laplante/yocto/sources/poky/meta/classes/sstate.bbclass", line 839, in checkstatus
fetcher.checkstatus()
File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/__init__.py", line 1736, in checkstatus
ret = try_mirrors(self, self.d, ud, mirrors, True)
File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/__init__.py", line 1077, in try_mirrors
ret = try_mirror_url(fetch, origud, uds[index], ld, check)
File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/__init__.py", line 979, in try_mirror_url
found = ud.method.checkstatus(fetch, ud, ld)
File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/wget.py", line 337, in checkstatus
opener.open(r)
File "/usr/lib/python3.5/urllib/request.py", line 472, in open
response = meth(req, response)
File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.5/urllib/request.py", line 504, in error
result = self._call_chain(*args)
File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
result = func(*args)
File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/wget.py", line 280, in http_error_405
unverifiable=True))
File "/usr/lib/python3.5/urllib/request.py", line 472, in open
response = meth(req, response)
File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.5/urllib/request.py", line 504, in error
result = self._call_chain(*args)
File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
result = func(*args)
File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/fetch2/wget.py", line 280, in http_error_405
unverifiable=True))
... (repeats until recursion depth is reached)
Solution is to make sure we only attempt the GET request once when handling 403/405 error codes.
(Bitbake rev: 18d4a31fdcec1f0e5d2199d6142f0ce833fca1a7)
Signed-off-by: Chris Laplante <chris.laplante@agilent.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Using npm pack to download the main node module and its dependencies
allow for the use of private npm modules and access to them via .npmrc
(Bitbake rev: e5eda3871893e4eadeb311aeb997e183675598f4)
Signed-off-by: Mads Andreasen <mads@andreasen.cc>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(Bitbake rev: 1dddfe3512b6390958abb91b21f074568ae4e8db)
Signed-off-by: Daniel Ammann <daniel.ammann@bytesatwork.ch>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The addition of some debugging code meant that comparisions between sig
files with a taint and without a taint weren't working. Tweak the logic
to avoid tracebacks if one side doesn't have a taint.
(Bitbake rev: f5ea06fc2b6713c9f8e85ecf7cb981ae9a84d896)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
So that people or other tools can easily know which one is being used, just
like what we did for run.do_task and log.do_task, otherwise, we have no way
to know it. I usually use "ls -t", but it isn't reliable since the one which
is being used may not the latest one.
(Bitbake rev: cf286dff653eed542bf347ca46234c224944d5b0)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If you specify both setscene and non-setscene tasks on the commandline, the
non-setscene tasks could be missed, e.g. "bitbake X:do_patch X:do_populate_sysroot"
and do_patch would fail to run.
Fix the problem in runqueue and add a testcase.
(Bitbake rev: 75292fdec5d9c0b5b3c554c4b7474a63656f7e12)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(Bitbake rev: 9a5152fa4613a1164cbf2a0248460e75207b2624)
Signed-off-by: Christian Herzig <cherzig@gauselmann.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
As the logic in bitbake improves, the logic in the tests needs to as well.
Afer we built a task for the first time, allow its setscene hash verification
status to change, mirroring what would happen in a multiconfig build.
(Bitbake rev: 27ec2e69ab3e32972caf8b072b2945736696d83d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently if a multiconfig build contains different configurations which
have overlapping sstate artefacts, it will build them multiple times.
This is clearly suboptimal and not what users want/expect.
This adds code to detect this and stall all but one of the setscne tasks
so that once its built, it can be found by the other tasks.
We take care to iterate the multiconfigs in order so try and avoid
dependency loops. We also match on PN+taskname+taskhash since this is
what we know sstate in OE-Core would use. There are some tasks even within
a multiconfig which match hashes (mostly do_populate_lic tasks) but those
have a much higher chance of circular dependency so aren't work attempting
to optimise.
If a deadlock does occur the build will be slower but there is code to
unbreak such a deadlock so it hopefully doens't break anything.
Comments are injected into the test tasks so they have different task
hashes and a new test for this optimisation is added.
(Bitbake rev: a75c5fd6d4ec56836de0be2fe679c81297a080ad)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Pull the common pieces of the hash verification code into a single function
and reduce code duplication.
(Bitbake rev: d0c39e05cef841c6f29cc6c919df6cbf271a9bda)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fix some unwanted extra indentation.
(Bitbake rev: 460a5c2e3e1d72f2da16fbc96832fadc82e72c52)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This was left from when task IDs complicated the code, simplify.
(Bitbake rev: ae36b5c693bb9f13c88199e78e3c31616852eafb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
With the merge of the scenequeue with real tasks, this now confuses the
statistics. The real tasks are the definitive progress so monitor only
those.
(Bitbake rev: 20956b508a082224139c8f56b68299edff6e0443)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We need some tests for runqueue, its been something which has been hard to test
for a long time. Add some dummy metadata to allow this, mirroring the OE
structure in spirit.
(Bitbake rev: 37564d7440c5d7aa05ec537f3b79026b1c83bb68)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This combines the scqenequeue and normal task execution into one function
and simplifies the state engine accordingly.
This is the final set of cleanup to fully merge things without adding the
extra noise to the previous commits.
(Bitbake rev: 56f3396d8c7cfbebd175877c9d773e4e35f8dea1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Since there is now parallel execution of setscene and normal tasks, the way
setscenewhitelist handling worked can't function the way it did. Paul and I
never liked its error output anyway.
This code tries a different approach, checking the task at execution time
but printing the uncovered task list.
This code may need improvement after real world usage but can
work with the new task flows.
(Bitbake rev: a08d8ba5f5194a09391b1904ee31c04c5f0b1e28)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
parallel
This is the serious functionality change in this runqueue patch series of
changes.
Rather than two phases of execution, the scenequeue setscene phase, followed
by normal task exeuction, this change allows them to execute in parallel
together.
To do this we need to handle marking of tasks as covered/uncovered in a piecemeal
fashion on a task by task basis rather than in a single function.
The code will block normal task exeuction until any setcene task which could
cover that task is executed and its status is known. There is a slight
optimisation which could be possible here at the risk of races but that
doesn't seem worthwhile.
The state engine isn't entirely cleaned up in this commit (see FIXME) and
the setscenewhitelist functionality is broken by it (see following patches)
however its good enough to test with normal workflows.
(Bitbake rev: 58b3f0847cc2d47e76f74d59dcbbf78fe41b118b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
It wasn't clear whether the variable contained just setscene covered
tasks or all covered tasks. We need both sets of data so lets just have
two clearly named variables.
(Bitbake rev: a9fb55627762e7c8b3df30b335ad0b2f1adc080e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The code for setting up buildable tasks can be simplified.
(Bitbake rev: ce3cd2df5b034f8dbdcf9834e8b9a393b6b01aad)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Its now clear a variable is pointless, remove it and tweak the logic
so the data structure of the existing variable matches what we need.
(Bitbake rev: c257c7b93b86dd794d31307e820215301c7ccf3b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Simplfy some looping code which no longer has any purpose.
(Bitbake rev: 01dfc37095e5c661f275917d22aa1c1ad7f24d8d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add some extra comments to build_scenequeue_data() and fix the debug code
so it actually works.
(Bitbake rev: 8ea6d8193fc89b4596da69e400fbc50e5a443f9f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The existing code to compute the 'unskippable' setscene task list is overcomlicated,
so replace it with something functionally equivalent but simpler and more efficient.
We don't need to process all chains, just the 'top' ones to the first setscene tasks.
This also makes the code more readable.
(Bitbake rev: 06982c82f10cbdbea0b601e5cf0450a2a99c14c2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Its useful to have a list of all the tasks a given setscene task covers
and we can easily generate this data whilst doing other data processing.
This is used in later changes to runqueue rather than trying to compute it
on the fly which is difficult.
(Bitbake rev: 63ddc2fec40bd1b456702b97091f9dc5ef70a941)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Whilst this class has existed for years, it doesn't have any
users and has a questionable interface. Drop it to allow for further
simplification and changes.
(Bitbake rev: 3ab51764f7965d696bb2c5a872bf161473df4289)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Also move the scheduler init over, apart for the builtable tasks part which need
to remain called later.
(Bitbake rev: ad30a16cd30f9eab0224eb271f98f9a24516b621)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Cleanup to the _execute_runqueue logic to reduce indentation, drop the
dummy executor class concept and prepare for further changes.
(Bitbake rev: 726e3c61a69fef16e605ba9b911a17cd99f1a2c3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Replace the remains of the Tasks and Scenequeue Tasks classes with simple
function calls. Also drop the dummy version of the execution class to
simplify further changes as its not needed.
(Bitbake rev: 33805394310046cd58c2194f6d063b3946811014)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Some tasks were not being marked as covered/notcovered since internal
calls were being made without using the external call points.
Fix the accounting issues by using the correct external call points.
(Bitbake rev: fe0a7be03e8baed22f6b0915cd5f7956ba3fbf83)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Merge the unique functions from the Tasks and Scenequeue Tasks classes
into the common base class.
(Bitbake rev: 7539fe22bc831bb835901e3aca77985ab4ebc4c7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Use a seperate stats class for scenequeue tasks and move the setup
into the base class. Update references accordingly.
(Bitbake rev: 32f39bbd5d3b7394689da9ba05be2c15b4523b27)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In preparation for merging the setscene and normal task execution,
uniquely namespace the scenequeue specific functions.
For the one shared function, add the "sq_live" variable so we know
which functions to send the results to.
(Bitbake rev: 2cbe9399902ba67dca566c7344b2247412cf4d5c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
For ease of refactoring, move this code to its own separate function
until it becomes clear what we should do with it.
(Bitbake rev: 4b96b204f986dd62fba485876b7208665c14268d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The function is only used by setscene code so the parameter is pointless,
remove it.
(Bitbake rev: b52dbf5e9cb327f8434213d286ad333f5dbad1d3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Move the bulk of the scenequeue data generation to its own function
allowing for refactoring of the code.
Create the start of an object to represent this data.
(Bitbake rev: 68326e0426f25a1bbfd5ae3aa278656a3744053e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
With the removal of the setcene verify code, this additional code block
is also now unneeded since tasks can't be forced at this point in the code
any move. This effectively reverts f21910157d873c030b149c4cdc5b57c5062ab5a6.
(Bitbake rev: 4514fe4f045d595cc9b938f9326f66f2b3e99f71)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Nothing in OE-Core uses this and hasn't since 2017. It wasn't needed by core
metadata since the switch to recipe specific sysroots.
Since this function would be hard to implement with the planned changes to
runqueue, drop it which allows simplification and further code cleanup.
(Bitbake rev: 5deaa5df730a8a846f3192b4a639b7a2a72c1b71)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Work off a copy of the 'buildable' class variable, allowing easier
future code changes.
(Bitbake rev: e851169acfebba404514135bf512e6f045739a13)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Now that runqueue optimises based on task hash, we need to ensure
tasks have unique hashes even in the simplest siggen mode. Use
the task name to calculate a unique hash.
This fixes runqueue tests when hash optimisations are added.
(Bitbake rev: 8ede873ef4ef492fbaf01474685c1ca8b34d80d5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Richard Purdie
Joshua Watt
Chris Laplante via bitbake-devel
Mads Andreasen
Daniel Ammann
Robert Yang
CHerzig@Gauselmann.de