summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/cooker.py
AgeCommit message (Collapse)Author
2023-11-27bitbake: cooker: Avoid eventlog variable listing lockupsRichard Purdie
If the event log is enabled and parsing the metadata triggers log messages, the event code and deadlock. Iterating the variables inside the event handling code causes this. SOURCE_DATE_EPOCH triggers a python function which calls bb.debug() and can trigger a lockup as one example. Move the code around and add it to the BuildStarted events explictly. This does mean runs without builds no longer get variables added to the eventlog however we can look into a more targetted version of data if/as/where neded. (Bitbake rev: 4135a617ae16d509362b5bf56378139cdc0876d2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-11-23bitbake: cooker: Avoid sideeffects for autorev from getAllKeysWithFlagsRichard Purdie
If we expand the variable AUTOREV in OE-Core, it triggers side effects in the fetcher. The situation isn't ideal and needs improvement but this breaks and is blocking enabling BB_DEFAULT_EVENTLOG. Hack around the issue for now so we unblock things until we can work out a better plan for how to improve AUTOREV support. (Bitbake rev: cb9b6530f3d12c56a8b48847af2e7461924205d2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-11-23bitbake: cooker: Add support for BB_DEFAULT_EVENTLOGRichard Purdie
Currently it is only possible to specify an eventlog on the bitbake commandline. Add a variable that can be used in bitbake.conf so that we can log data by default more easily. (Bitbake rev: ee174b231897a53cdde0f68769518342e53210cf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-11-14bitbake: asyncrpc: Add option to set log level when running as a processJoshua Watt
When running an asyncrpc server as a subprocess, it is often desired to run it with a lower logging level since the normal logging of clients connecting and disconnecting is not desired. As such, add an option to set the logging level of the server when running as a subprocess and set the level to WARNING when starting a local hashserver or PRserver (Bitbake rev: 61dac7b99ad6d2a858f85d8ed1b5524d558be6c8) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-10-23bitbake: Fix find_bbfiles string endswith callBELHADJ SALEM Talel
(Bitbake rev: 5f742591b251b6a5302ab07fe6b809c2863c3c70) Signed-off-by: Talel BELHAJSALEM <bhstalel@gmail.com> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-09-28bitbake: bitbake: cooker: add a new function to retrieve task signaturesJulien Stephan
adding a new command in cooker to compute and get task signatures this commit also add the associated command and event needed to get the signatures using tinfoil (Bitbake rev: 05c15162de90c41dad67e37a95ec9fdb440a7864) Signed-off-by: Julien Stephan <jstephan@baylibre.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-09-20bitbake: cooker: Drop unneeded flush callsRichard Purdie
Since the flush calls have significant effects for bitbake timeout issues, drop the remaining ones from cooker. These aren't in as critical paths as the other issues but it makes sense to clean up. (Bitbake rev: dd682363341bae3b060e284d73f000813964dc05) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-09-18bitbake: lib: Drop inotify support and replace with mtime checksRichard Purdie
With the flush in serverlog() removed and a memory resident bitbake with a 60s timeout, the following could fail in strange ways: rm bitbake-cookerdaemon.log bitbake-layers add-layer ../meta-virtualization/ bitbake-layers add-layer ../meta-openembedded/meta-oe/ bitbake -m specifically that it might error adding meta-oe with an error related to meta-virt. This clearly shows that whilst bblayers.conf was modified, bitbake was not recognising that. This would fit with the random autobuilder issues seen when the serverlog flush() call was removed. The issue appears to be that you have no way to "sync()" the inotify events with the command stream coming over the socket. There is no way to know if there are changes in the IO queue which bitbake needs to wait for before proceeding with the next command. I did experiment with os.sync() and fsync on the inotify fd, however nothing addressed the issue. Since it is extremely important we have accurate cache data, the only realistic thing to do is to switch to stat() calls and check mtime. For bitbake commands, this is straightforward since we can revalidate the cache upon new connections/commands. For tinfoil this is problematic and we need to introduce and explict command "revalidateCaches" that the code can use to force bitbake to re-check it's cache validity. I've exposed this through tinfoil with a new "modified_files" function. So, this patch: a) drops inotify support within bitbake's cooker/server and switch to using mtime b) requires a new function call in tinfoil when metadata has been modified (Bitbake rev: da3ec3801bdb80180b3f1ac24edb27a698415ff7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-08-16bitbake: cooker: Fix error messageJoshua Watt
"Not" should be "No" (Bitbake rev: a06619951a43acb80b80d92e0caac560657ca249) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-06-30bitbake: cooker: Log when parsing starts in server logRichard Purdie
It is unclear from the server logs when parsing starts. We know that timeouts sometimes happen when parsing but it is unclear where in the code the delays are from. Adding this debug message to the server log should help narrow that down. (Bitbake rev: a5c145f436d68f090b113cfb9b82857adc95b546) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-05-25bitbake: cooker: Add FILE_LAYERNAME variable containing the layername for a ↵Richard Purdie
recipe There are times when it would be useful for code to know which layer (or collection in old bitbake terms) it is contained within. Add support for FILE_LAYERNAME to be set by bitbake when parsing a recipe so that it is possible to determine this. To do it, we need to pass data from the cooker into the recipe endpoints, since only the top level cooker information knows about the layer structure which makes the patch a bit painful. The idea is that this would make layer overrides possible: OVERRIDES .= ":layer-${FILE_LAYERNAME}" which then opens possibilities like: WARN_QA:append:layer-core = " patch-fuzz" as an example where OE-Core could enable specific QA tests only for that specific layer. (Bitbake rev: 7090a14b0035842112d073acf7f2ed1a01fdeccf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-05-25bitbake: cooker: Fix/improve collections handlingRichard Purdie
Code changes for FILE_LAYERNAME exposed lifecycle issues around the collections object in Cooker which only appeared in devtool usage in eSDK. Move the collections setup to an earlier stage after parsing completes to avoid any kind of race around it. Also stop overwriting the code variable in MatchFiles. Ultimately we need to combine these codepaths but that is for another patch. (Bitbake rev: 27b872ed4fbe73b3b61e14cb885bb7c16c039cdb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-05-02bitbake: cooker: Log config and parse cache status changesRichard Purdie
It can be hard to tell why bitbake does some things slowly. Log the changes to the base configuration and parse cache status so that it becomes clear from the logs when the cache invalidation causes a slowdown. (Bitbake rev: 6e99d89f3c00a5f53c24d687eaef24f52fe0ef99) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-20bitbake: cooker: do not abort on single ctrl-cEnrico Scholz
After b7ed7e9a815c4e10447fd499508be3dbb47f06e8 bitbake aborts immediately when a single ctrl-c is pressed. Patch restores the previous behavior where a single ctrl-c waits for active tasks to finish. https://bugzilla.yoctoproject.org/show_bug.cgi?id=15094 (Bitbake rev: 66131fa6a3e12c28710d09e1dbf3c03f2981280d) Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-04-13bitbake: event: add bb.event.ParseErrorMingli Yu
Add bb.event.ParseError to let error-report can catch this kind of error. (Bitbake rev: 316524ab59a5e738c25e062923ee5717d88ae5c7) Signed-off-by: Mingli Yu <mingli.yu@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-24bitbake: cooker: Fix memory resident cache invalidation issueRichard Purdie
We've been seeing weird PRServ failures on the autobuilder. These had one failure always followed by a second. Whilst I can't reproduce the first, if I made that test fail, I could reproduce the second with memory resident bitbake. This was with the tests: prservice.BitbakePrTests.test_import_export_replace_db and then prservice.BitbakePrTests.test_pr_service_deb_arch_dep which was giving a strange looking error: NOTE: Running task 1053 of 1055 (/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-devtools/m4/m4_1.4.19.bb:do_package_write_rpm) NOTE: Running task 1054 of 1055 (/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-devtools/m4/m4_1.4.19.bb:do_package_qa) ERROR: No such task: do_package_write_rpm ERROR: Task (/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/recipes-devtools/m4/m4_1.4.19.bb:do_package_write_rpm) failed with exit code '1' where the issue is that selftest.inc written by the test framework and containing PACKAGE_CLASSES = "package_deb" was being ignored. The issue is the cached_statements{} within BBHandler() is not being invalidated at the right time. This patch changes the code to ensure base configuration is not parsed until inotify updates have been processed. (Bitbake rev: cada37c6b9e5862ca2c5a54ad6fd1e1f1939cd9c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-23bitbake: event/cooker/runqueue: Add ability to interrupt longer running codeRichard Purdie
Bitbake is now able to understand when a UI wants it to stop the current processing. There are some long running functions which currently have no mechanism to interrupt them however. This patch adds a call, bb.event.check_for_interrupts(d) which can be placed in long running code and allows an internal state flag within bitbake to be checked. If set, that flag will trigger an exit. This means that Ctrl+C can be made to work in a variety of places where it currently would not. Long running event handlers in OE-Core can also then benefit from this new approach with the addition of the function call as well. (Bitbake rev: b7ed7e9a815c4e10447fd499508be3dbb47f06e8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-02-20bitbake: cooker: Ensure lock is held with changing notifierRichard Purdie
We've seen a couple of cases which bitbake hangs due to an inotifer exception such as: 3323260 21:48:31.554468 Running command ['getVariable', 'BBINCLUDELOGS'] Exception in thread Thread-1 (idle_thread): Traceback (most recent call last): File "/usr/lib64/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() File "/usr/lib64/python3.10/threading.py", line 953, in run self._target(*self._args, **self._kwargs) File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/server/process.py", line 408, in idle_thread self.cooker.process_inotify_updates() File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/bb/cooker.py", line 256, in process_inotify_updates n.read_events() File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/bitbake/lib/pyinotify.py", line 1207, in read_events if fcntl.ioctl(self._fd, termios.FIONREAD, buf_, 1) == -1: OSError: [Errno 9] Bad file descriptor 3323260 21:48:32.206995 Command Completed (socket: True) Ensure we don't destory the inotifier when the idle thread is reading is by holding the lock during setup/teardown. (Bitbake rev: 8fc5c50c2e23017833f93bcd514d708a14fa4266) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-26bitbake: cache/codeparser: Switch to a new BB_CACHEDIR variable for cache ↵Richard Purdie
location Currently the codeparser cache is set from CACHE, which is typically in bitbake.conf which means we can't read/write any cache until it is found/read. We may well have python expressions to parse before that happens. The net result is suboptimal functioning of the codeparser cache since it will often be invalidated by data that is never written. This patch changes the codeparser and filechecksum caches to use BB_CACHE as their setting and defaults it to ${TOPDIR}/cache. The patch doesn't change where the "persistent" data such as prserver and hash-equiavalance resides (PERSISTENT_DIR) or where the metadata parsing cache resists (still currently CACHE). I've left those for a later patch. The patch does ensure data parsed by the core datastore parsing calls is written back since this is now much more useful after this change. (Bitbake rev: ee89ade5b5a4cf9c53f336d8b800e06fbe436628) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-24bitbake: cooker: Fix parsing race around cache handlingRichard Purdie
When draining the result queue from the parsing processes, cache objects can be created even if they are then immediately destroyed. The reset in the sync code needs to happen after any objects have been created. Change the ordering to fix this. This ordering has caused various cache errors, particularly when interrupting parsing with Ctrl+C. (Bitbake rev: f45a94e6720dacf7f51ac147c115a6f608769093) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-24bitbake: cooker: Fix siggen recipe cache race issueRichard Purdie
We need to reset the cache before the start() call which starts the parsing processs, tweak the code to ensure this is the case. (Bitbake rev: 4712d6237df5d2188238294fbdb9f107b068a3a7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-11bitbake: cooker: Fix exit handling issuesRichard Purdie
The shutdown() call should definitely not be waiting for idle, since we expect execution and events to continue even after setting either shutdown state. This was causing the UI to appear to hang at Ctrl+C interrupts. Also ensure that if we reset cooker, we stop any active parser processes since these currently appear to be being left behind. (Bitbake rev: 4f73c2eb12ee074f3b7d4717380649f6ca8f3def) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-11bitbake: process/cooker/command: Fix currentAsyncCommand locking/racesRichard Purdie
currentAsyncCommand currently doesn't have any locking and we have a conflict in "idle" conditions since the idle functions count needs to be zero *and* there needs to be no active command. Move the changes/checks of currentAsyncCommand to within the lock and then we can add it to the condition for idle, simplifying some of the code. (Bitbake rev: b5215887d2f8ea3f28f1ebda721bd5b8f93ec7f3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-11bitbake: cooker/command: Drop async command handler indirection via cookerRichard Purdie
Indirecting the async command handler via cooker is confusing and no longer needed. Drop it to make things slightly clearer. (Bitbake rev: 4a41a7d0594e6a84a67b9de70a505858aebcd84a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-06bitbake: cooker: Clean up inotify idle handlerRichard Purdie
We no longer need to abstract the inotify callback handler, remove the abstraction and simplify/clean up the code. (Bitbake rev: af4ccab8acc49e91bf7647f209d69f4858618466) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-06bitbake: cooker: Rework the parsing results submissionRichard Purdie
The loop submitting the parser results was not efficient on many core systems as may of the parsers could block for a long time waiting to send the results. While a result was pending, the code wouldn't parse further jobs. Change the code to parse further jobs even if the results can't be submitted and lower the result submission timeout to keep the parsers busy. (Bitbake rev: 9ece4a2e406509bd89dbce8dac80a02e600b0ecf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2023-01-05bitbake: lib/bb: Update thread/process locks to use a timeoutRichard Purdie
The thread/process locks we use translate to futexes in Linux. If a process dies holding the lock, anything else trying to take the lock will hang indefinitely. An example would be the OOM killer taking out a parser process. To avoid bitbake processes just hanging indefinitely, add a timeout to our lock calls using a context manager. If we can't obtain the lock after waiting 5 minutes, hard exit out using os._exit(1). Use _exit() to avoid locking in any other places trying to write error messages to event handler queues (which also need locks). Whilst a bit harsh, this should mean we stop having lots of long running processes in cases where things are never going to work out and also avoids hanging builds on the autobuilder. (Bitbake rev: d2a3f662b0eed900fc012a392bfa0a365df0df9b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-31bitbake: server/process: Run idle commands in a separate idle threadRichard Purdie
When bitbake is off running heavier "idle" commands, it doesn't service it's command socket which means stopping/interrupting it is hard. It also means we can't "ping" from the UI to know if it is still alive. For those reasons, split idle command execution into it's own thread. The commands are generally already self containted so this is easier than expected. We do have to be careful to only handle inotify poll() from a single thread at a time. It also means we always have to use a thread lock when sending events since both the idle thread and the command thread may generate log messages (and hence events). The patch depends on previous fixes to the builtins locking in event.py and the heartbeat enable/disable changes as well as other locking additions. We use a condition to signal from the idle thread when other sections of code can continue, thanks to Joshua Watt for the review and tweaks squashed into this patch. We do have some sync points where we need to ensure any currently executing commands have finished before we can start a new async command for example. (Bitbake rev: 67dd9a5e84811df8869a82da6a37a41ee8fe94e2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-31bitbake: server/process: Improve idle loop exit codeRichard Purdie
When idle handlers want to exit, returning "False" isn't very clear and also causes challenges with the ordering of the removing the idle handler and marking that no async command is running. Use a specific class to signal the exit condition allowing clearer code and allowing the async command to be cleared after the handler has been removed, reducing any opportunity for races. (Bitbake rev: 102e8d0d4c5c0dd8c7ba09ad26589deec77e4308) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-31bitbake: cooker: Ensure commands clean up any parser processesRichard Purdie
When finishing a command, we need to ensure any parsing processes that may have been started are cleaned up before we reset the cooker state. (Bitbake rev: 6569ab64bea35de21acc89053ba76e2828163f3f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-31bitbake: cooker/cookerdata: Rework the way the datastores are resetRichard Purdie
As far as I could tell, the current code could result in some strange situations where some data was set back to the original data store copy but the multiconfig data was not restored. There are also some changes made to the datastore which did not persist. The data store was also being reset at every client reset, which seems a little excessive if we can reset it to the original condition properly. Move the __depends -> __base_depends rename into databuilder along with the __bbclasstype change so these are saved in the original data. Tweak the databuilder code to be clearer and easier to follow on which copies are where, then save copies of all the mc datastores. Finally, drop the cache invalidation upon reset for the base config as we shouldn't need that now (oe-selftest -r tinfoil works with memory resident bitbake which was the original reproducer requiring that change). (Bitbake rev: 5f8b9b9c35b4ec0c8c539bf54ca85f068f4a5094) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-31bitbake: event: Add enable/disable heartbeat codeRichard Purdie
Currently heartbeat events are always generated by the server whilst it is active. Change this so they only appear when builds are running, which is when most code would expect to be executed. This removes a number of races around changes in the datastore which can happen outside of builds. (Bitbake rev: 8c36c90afc392980d999a981a924dc7d22e2766e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-29bitbake: cooker: Ensure cache is cleared for partial resetsRichard Purdie
We're still seeing occasional SiggenRecipeInfo coherency issues, add some further reset points into the parsing code to ensure the cache is cleared before reparsing. (Bitbake rev: 26ed783caf11dc9ebf53d3790681eb44c0c360f0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-29bitbake: cache/siggen: Fix cache issues with signature handlingRichard Purdie
There is a bug in the current cache code where the restored data structures were "inverted" leading to some very weird issues, not sure how anything worked like that, this patch fixes it. Also fix some issues with multiconfig cache ordering problems by resetting the stream counters when appropriate. (Bitbake rev: cd06beb948eff5eaf2d474f5b127d51a61b0b2ef) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-21bitbake: cache: Allow compression of the data in SiggenRecipeInfoRichard Purdie
The data in SiggenRecipeInfo is large and has a lot of duplication. The size causes a few problems, impacting: - bitbake's overall memory usage - the amount of data sent over IPC between parsing processes and the server - the size of the cache files on disk - the size of "sigdata" hash information files on disk The data consists of strings (some large) or frozenset lists of variables. To reduce the impact we can: a) deplicate the data b) pass references to the object on the second usage (e.g. over IPC or saving into pickle). This patch does this for SiggenRecipeInfo mostly behind the scenes but we do need a couple of reset points so that streamed data is written correctly on the second usage. (Bitbake rev: 9a2b13af483c20763d6559a823310954884f6ab1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-17bitbake: build/siggen: Rework stamps functionsRichard Purdie
The current method of passing either a task's datastore, or dataCaches and a filename into the stamp functions is rather horrible. Due to the different contexts, fixing this is hard but we do control the bitbake side of the API usage so we can migrate those to use other functions and then only support a datastore in the public bb.build API which is only called from task context in OE-Core. (Bitbake rev: c79ecec580e4c2a141ae483ec0f6448f70593dcf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-17bitbake: siggen: Add dummy dataCaches from task context/datastoreRichard Purdie
One of the challenges in maintaining the code is that it sometimes uses a datacaches structure and sometimes a datastore. Rather than continue the current dual API madness, have the worker contexts create a dummy datacaches structure with the entries we need. Whilst this does need to be kept in sync with the real structure, that doesn't change and this allows the code to be simplified. With this new approach, we can unify the stamps dependency code again. (Bitbake rev: c6d325fc9b53e9d588ab273ee3c2a99a70fba42c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-08bitbake: cooker: Start sync thread a little earlierJoshua Watt
Starts the sync thread slightly earlier to give it some extra time to dump out the caches while the main process tears down the parsing processes (Bitbake rev: 105f2897b0618713b036fc0f7a6e0f3e78d1707a) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-08bitbake: cooker: Use event to terminate parser threadsJoshua Watt
Uses an event to terminate the parser threads instead of a queue. This is not only simpler, but saves about 500ms on shutdown time (Bitbake rev: 2aed34e1d4bf24bba6263f168ff31b55b5fbe982) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-12-08bitbake: siggen/cache: Optionally allow adding siggen hash data to the ↵Richard Purdie
bitbake cache Being able to track siggen hash construction data can be useful for cache debugging. For now, add an extra cache class which contains this information. It can be enabled in the same way as the hob data cache through a feature flag to cooker. This allows us to experiment with the data without carrying larger patches around and ultimately may allow use to have a hash mismatch debugging mode that is more easily enabled. (Bitbake rev: 0736a8a03da8b774fafbd28f746bef4705378049) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-11-20bitbake: cache: Drop broken/unused codeRichard Purdie
Some parts of functions in Cache() were broken and unused, there was also a totally unused function. This was historical as a result of the cooker parsing process needing to handle cached entries in the main thread but parsing actions in seperate processes. Document the way it works, update the function name to be clear about what it now does and drop the old code which was unused. (Bitbake rev: af83ee32df85c8e4144f022a1f58493eb72cb18e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-11-20bitbake: cache/cookerdata: Move recipe parsing functions from cache to ↵Richard Purdie
databuilder When 'NoCache' was written, databuilder/cookerdata didn't exist. It does now and the recipe parsing functionality contained in NoCache clearly belongs there, it isn't a cache function. Move those functions, renaming to match the style in databuilder but otherwise not changing functionality for now. Fix up the callers to match (which make it clear this is the right move). (Bitbake rev: 783879319c6a4cf3639fcbf763b964e42f602eca) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-10-25bitbake: cooker: fix a typoOliver Lang
(Bitbake rev: cf23612f4e8946b9ed4c9f87b451f32b8c471df2) Signed-off-by: Oliver Lang <quantenkeks@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-09-21bitbake: bitbake: Fix a few more logger debug() callsJoshua Watt
f68682a7 ("logging: Make bitbake logger compatible with python logger") replaced several .debug() calls to make them comply with the standard python logging API, but a few were missed. (Bitbake rev: eb25cd4d64b9a4e8e2b2ce7fbccc123d00b2fc2b) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-08-12bitbake: BBHandler/cooker: Implement recipe and global classesRichard Purdie
We have some confusion for users since some classes are meant to work in the configuration space (or "globally") and some are meant to be selected by recipes individually. The cleanest way I could find to clarify this is to create "classes-global" and "classes-recipe" directories which contain the approproate classes and have bitbake switch scope between them at the appropriate point during parsing. The existing "classes" directory is always searched as a fallback. Once a class is moved to a specific directory, it will no longer be found in the incorrect context. A good example from OE is that INHERIT += "testimage" will no longer work but IMAGE_CLASSES += "testimage" will, which makes the global scope cleaner by only including it where it is useful and intended to be used (images). (Bitbake rev: f33ce7e742f46635658c400b82558cf822690b5e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-06-11bitbake: cooker: Drop sre_constants usageRichard Purdie
As reported by Martin Jansa <Martin.Jansa@gmail.com>: bitbake/lib/bb/cooker.py:16: DeprecationWarning: module 'sre_constants' is deprecated import sre_constants it's deprecated since 3.11 with: https://github.com/python/cpython/issues/91308 The correct replacement for our usage is re.error so use that instead. (Bitbake rev: 3c0cd401472ffee06d5a93bdba566cb033851fcf) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-04-21bitbake: cooker: Drop unused loopRichard Purdie
(Bitbake rev: 1c811ad6f10560e7a7fb6830cf83707551ba04bd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-04-03bitbake: cooker: Reset and rebuild inotify watchesRichard Purdie
The recent inotify changes can cause entire build trees to be monitored which is suboptimal for performance. Rather than trying increasingly convoluted tricks to try and handle add/removed directories, rebuild the inotify watch when we reparse the configuration or metadata. (Bitbake rev: 3df322a200c28b45af1f2c92478c85eb7d20c38b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-04-01bitbake: cooker: Avoid error if siggen wasn't setupRichard Purdie
In some cases we'd never have setup a siggen so don't error in that case. (Bitbake rev: bbaaf2cf7b5a9339d3790610e622020c19d52f5a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2022-04-01bitbake: cooker: Ensure any existing hashserv connection is closedRichard Purdie
Ensure any exiting hash server connection is terminated before we start a new bitbake session. This avoids errors seen with memory resident bitbake when the asyncio event loop isn't closed correctly. (Bitbake rev: 42ff9de77f24e2a0bec48a14b64c4b538e00b4af) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>