summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/cooker.py
AgeCommit message (Collapse)Author
2017-01-11bitbake: bitbake: cooker: Handle inofity queue overflows more gracefullyRichard Purdie
If many files change and the inotify queue overflows, rather than print a traceback, invalidate the caches and warn the user. [YOCTO #10676] (Bitbake rev: 058f8517c041b80e8b591ad7d34a68281b2d03fc) (Bitbake rev: 4fafb6c6d261de78dd1bc3824a1389d191b70321) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-11bitbake: bitbake: cooker: Fix world taskgraph generation issueRichard Purdie
The processing of the "do_" prefix to tasks is currently inconsistent and has resulted in "bitbake world -g" being broken as task prefixes don't get handled correctly. Make the "do_" task prefix handling consistent through various codepaths. [YOCTO #10651] (Bitbake rev: 3d7186353e804c9410096c408bc337a98c8b33fe) (Bitbake rev: 100439e715841ecfd6460d59cd51c831184b328d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-09bitbake: runqueue: Optimise task id string manipulationsRichard Purdie
Some task id manipulations were suboptimal: * taskfn_fromtid and fn_from_tid were effectively the same function * many calls to split_tid(), then taskfn_fromtid() * taskfn_fromtid() called split_tid() internally This patch adds split_tid_mcfn() to replace split_tid() and returns the "taskfn" variant being used in many places. We update all core calls to the new function and ignore the return values we don't need since the function call overhead of the split_tid wrapper is higher than ignoring a return value. The one remaining standalone use of taskfn_fromtid is replaced with fn_from_tid. I couldn't see any external usage so it was dropped. There is external usage of split_tid so a wrapper remains for it. Combined together these changes should improve some of the runqueue task manipulation performance. (Bitbake rev: 1bf2ef874fbe47f1320007efa0bdeef8d630b8a1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-22bitbake: cooker/providers: Only add target to world build if task existsRichard Purdie
A "bitbake world -c unpack" currently breaks as not all tasks have an unpack task. This change allows addition of world targets only if the specified task exists which makes certain commands possible when otherwise you just get errors which can't easily be avoided. (Bitbake rev: ca4f5e6d01b5c8cf315f59bc86194d63c0d3d042) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-21bitbake: cooker.py: add LAYERRECOMMENDS processingJoe Slater
Add recommended layers to collection_depends[] so that dynamic priority assignment will work for both depends and recommends. Recommended layers do not cause an error or warning if they are not in the collection list, but debug messages are output for level 3 and above. explode_dep_versions2 returns a dictionary, so we change the variable deplist to depDict. The dictionary values are lists which are either empty or contain only one version specification. (Bitbake rev: 20cdc3d609f8aea992f97c3db336574d3a549973) Signed-off-by: Joe Slater <jslater@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-16bitbake: cooker: check if target contains taskEd Bartosh
Task name was incorrectly added to the targets that already contained :task suffix and fired with BuildInit event. This caused Toaster to create incorrect Target objects and show them in UI. [YOCTO #10221] (Bitbake rev: b7faf1af3bd3110fba347fbe6e432fc4ee66590a) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-08bitbake: cooker: record events on cooker exitEd Bartosh
Bitbake collects all events in special event queue when called with -w option. However, it starts to write events to the eventlog only after BuildStarted event is received. In some cases this event is not received at all, e.g. when bitbake is run with --parse-only command line option. It makes sense to write all collected events when CookerExit event received to make sure all events are written into the eventlog even if BuildStarted event is not fired. [YOCTO #10145] (Bitbake rev: 57912de63fa83550c0ae658eb99b76e9cc91a8d1) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-02bitbake: cooker.py: Catch when stdout doesn't have a file descriptorMariano Lopez
Currently, there is a check to remove the TOSTOP attribute from a tty to avoid hangs. It assumes that sys.stdout will have a file descriptor and this is not always true, some IO classes will throw exceptions when trying to get its file descriptor. This will add a check for such cases and avoid throwing an exception. [YOCTO #10162] (Bitbake rev: cb4f8f6efa28ef2b13bc738a0118b876baa15b3e) Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18bitbake: bitbake: Initial multi-config supportRichard Purdie
This patch adds the notion of supporting multiple configurations within a single build. To enable it, set a line in local.conf like: BBMULTICONFIG = "configA configB configC" This would tell bitbake that before it parses the base configuration, it should load conf/configA.conf and so on for each different configuration. These would contain lines like: MACHINE = "A" or other variables which can be set which can be built in the same build directory (or change TMPDIR not to conflict). One downside I've already discovered is that if we want to inherit this file right at the start of parsing, the only place you can put the configurations is in "cwd", since BBPATH isn't constructed until the layers are parsed and therefore using it as a preconf file isn't possible unless its located there. Execution of these targets takes the form "bitbake multiconfig:configA:core-image-minimal core-image-sato" so similar to our virtclass approach for native/nativesdk/multilib using BBCLASSEXTEND. Implementation wise, the implication is that instead of tasks being uniquely referenced with "recipename/fn:task" it now needs to be "configuration:recipename:task". We already started using "virtual" filenames for recipes when we implemented BBCLASSEXTEND and this patch adds a new prefix to these, "multiconfig:<configname>:" and hence avoid changes to a large part of the codebase thanks to this. databuilder has an internal array of data stores and uses the right one depending on the supplied virtual filename. That trick allows us to use the existing parsing code including the multithreading mostly unchanged as well as most of the cache code. For recipecache, we end up with a dict of these accessed by multiconfig (mc). taskdata and runqueue can only cope with one recipecache so for taskdata, we pass in each recipecache and have it compute the result and end up with an array of taskdatas. We can only have one runqueue so there extensive changes there. This initial implementation has some drawbacks: a) There are no inter-multi-configuration dependencies as yet b) There are no sstate optimisations. This means if the build uses the same object twice in say two different TMPDIRs, it will either load from an existing sstate cache at the start or build it twice. We can then in due course look at ways in which it would only build it once and then reuse it. This will likely need significant changes to the way sstate currently works to make that possible. (Bitbake rev: 5287991691578825c847bac2368e9b51c0ede3f0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18bitbake: cache: Build datastores from databuilder objectRichard Purdie
Rather than passing in a datastore to build on top of, use the data builder object in the cache and base the parsed recipe from this. This turns things into proper objects building from one another rather than messy mixes of static and class functions. This sets things up so we can support parsing and building multiple configurations. (Bitbake rev: fef18b445c0cb6b266cd939b9c78d7cbce38663f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18bitbake: cache/cooker: Pass databuilder into bb.cache.Cache()Richard Purdie
Rather that the current mix of static and class methods, refactor so that the cache has the databuilder object internally. This becomes useful for the following patches for multi config support. It effectively completes some of the object oriented work we've been working towards in the bitbake core for a while. (Bitbake rev: 7da062956bf40c1b9ac1aaee222a13f40bba9b19) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18bitbake: cache: Make virtualfn2realfn/realfn2virtual standalone functionsRichard Purdie
Needing to access these static methods through a class doesn't make sense. Move these to become module level standalone functions. (Bitbake rev: 6d06e93c6a2204af6d2cf747a4610bd0eeb9f202) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-11bitbake: cooker: add BuildInit eventElliot Smith
In situations where a bitbake run fails before the build properly starts and BuildStarted is fired, a UI has no way to get at the targets passed to the build. This makes it difficult for the UI to report on the targets which failed. Fire a BuildInit event before running buildTargets() or buildFile(). This enables a UI to capture targets passed to buildTargets(), even if the build fails (e.g. the targets themselves are invalid). [YOCTO #8440] (Bitbake rev: ac02fda870965bf7d44ff5688eda54d2d11ab9c7) Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-20bitbake: cooker: Fix incorrect dot file generationRichard Purdie
In the runqueue cleanup/conversion, "dep" was mistakenly used where "tid" should be leading to incorrect task-depends.dot files and causing general confusion. Fix this, its clearly incorrect looking at the code. (Bitbake rev: 689730dbb068c5ea3593e7b92fe5d5e5c0c3760a) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-20bitbake: cooker: clean up EvertWriterEd Bartosh
Restructured EventWriter code to make it more readable: - got rid of init_file method as it's called only once - renamed exception variable e -> err - renamed event variable e -> evt - simplified main 'if' structure of send method (Bitbake rev: 31977e7bb98f676197c6cee66f6ab4c12d4dcbde) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-20bitbake: cooker: replace EventLogWriteHandler with namedtupleEd Bartosh
class EventLogWriteHandler is a simple wrapper class with only one class member. Replacing it with namedtuple makes code less nested and more readable. (Bitbake rev: 7c5b6812d32d173df36e7f9fc1d877329e79f994) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-20bitbake: cooker: don't remove event fileEd Bartosh
There is no need to remove output file as it gets rewritten by open(self.eventfile, 'w') anyway. (Bitbake rev: 1fc9957837b7038dfb983217a3fcd880f143e3a4) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-20bitbake: cooker: encode event objects to base64Ed Bartosh
pickle converts python objects into the binary form that can't be decoded to text and therefore can't be converted to JSON format. Attempt to convert event objects raises this error: TypeError: b'\x80\x03cbb.runqueue\nrunQueueTaskSkipped\nq\x00)... is not JSON serializable Encoded pickled event objects to base64 to be able to convert data structure to JSON. [YOCTO #9803] (Bitbake rev: f18055237e6084f90f6221442e3ba021dcc59c50) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-20bitbake: cooker: move EventLogWriteHandler to the top module levelEd Bartosh
EventLogWriteHandler object was created and used in BBCooker.initConfigurationData. This causes creation of multiple EventLogWriteHandler objects and results in duplicated entries in the output event file as BBCooker.initConfigurationData is called multiple times. Added eventlogfile parameter to EventLogWriteHandler to avoid using global variable DEFAULT_EVENTFILE. Moved EventLogWriteHandler to the module level. Created EventLogWriteHandler object in BBCooker.__init__ to ensure that only one handler object is created. (Bitbake rev: d3ad8eee850ec2df54aa09fae44cc7e69c12f32a) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-16bitbake: bitbake: fix wrong usage of format_excEd Bartosh
First parameter of traceback.format_exc is a 'limit' - a number of stracktraces to format. Passing exception object to format_exc is incorrect, but it works in Python 2 as this code from traceback module works: while tb is not None and (limit is None or n < limit): Comparing integer counter n with the exception object in Python 2 always results in True. However, in Python 3 it throws exception: TypeError: unorderable types: int() < <Exception type>() As format_exc is used in except block of handling another exception this can cause hard to find and debug bugs. (Bitbake rev: a9509949d7e2adba6e3cd89f97daa19a955855b5) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-15bitbake: taskdata/runqueue: Rewrite without use of ID indirectionRichard Purdie
I'm not sure what possesed me when I wrote this code originally but its indirection of everyting to use numeric IDs and position dependent lists is horrific. Given the way python internals work, its completely and utterly pointless from performance perspective. It also makes the code hard to understand and debug since any numeric ID has to be translated into something human readable. The hard part is that the IDs are infectous and spread from taskdata into runqueue and even partly into cooker for the dependency graph processing. The only real way to deal with this is to convert everything to use a more sane data structure. This patch: * Uses "<fn>:<taskname>" as the ID for tasks rather than a number * Changes to dict() based structures rather than position dependent lists * Drops the build name, runtime name and filename ID indexes On the most part there shouldn't be user visible changes. Sadly we did leak datastructures to the setscene verify function which has to be rewritten. To handle this, the variable name used to specifiy the version changes from BB_SETSCENE_VERIFY_FUNCTION to BB_SETSCENE_VERIFY_FUNCTION2 allowing multiple versions of bitbake to work with suitably written metadata. Anyone with custom schedulers may also need to change them. I believe the benefits in code readability and easier debugging far outweigh those issues though. It also means we have a saner codebase to add multiconfig support on top of. During development, I did have some of the original code coexisting with the new data stores to allow comparision of the data and check it was working correcty, particuarly for taskdata. I have also compared task-depends.dot files before and after the change. There should be no functionality changes in this patch, its purely a data structure change and that is visible in the patch. (Bitbake rev: 2c88afb60da54e58f555411a7bd7b006b0c29306) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-13bitbake: cooker.py: fix warn() -> warning()Robert Yang
Fixed: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead (Bitbake rev: a3f464d202dafef4538e66c008cdecb7b8709ed1) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02bitbake: bitbake: Convert to python 3Richard Purdie
Various misc changes to convert bitbake to python3 which don't warrant separation into separate commits. (Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02bitbake: bitbake: Drop futures usage since we're python 3Richard Purdie
(Bitbake rev: bf25f05ce4db11466e62f134f9a6916f886a93d9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-21bitbake: cooker: Fix parse progress for python3Richard Purdie
Under python the type conversions can mean there are float values used for triggering the parse progress events which then fails. Add an explict int() conversion to ensure the parse events are generated under python3. (Bitbake rev: 138329c58e92744c56aae3ab70ceeef09613250c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-19bitbake: cooker: Improve taskgraph file handlingRichard Purdie
Use open() instead of file() and close files when finished with them. (Bitbake rev: 033c5a16ff19781ed793c2d97d285884017a2a4e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-19bitbake: bb.cooker: show limited traceback for parsing ExpansionErrorChristopher Larson
It's useful to see tracebacks for ExpansionErrors, but only if we skip the leading bitbake-internal elements, otherwise we see elements of the expansion process. As one example: Before: ERROR: ExpansionError during parsing /scratch/yocto-new/external-as-needed/poky/meta/recipes-core/glibc/glibc-locale_2.23.bb: Failure expanding variable PV[:=], expression was ${@get_external_libc_version(d)} which triggered exception AttributeError: 'module' object has no attribute 'external' After: ERROR: ExpansionError during parsing /scratch/yocto-new/external-as-needed/poky/meta/recipes-core/glibc/glibc-locale_2.23.bb Traceback (most recent call last): File "PV[:=]", line 1, in <module> File "/scratch/yocto-new/external-as-needed/meta-sourcery/recipes-external/glibc/glibc-external-version.inc", line 3, in get_external_libc_version(d=<bb.data_smart.DataSmart object at 0x7f05d2566950>): sopattern = os.path.join(d.getVar('base_libdir', True), 'libc-*.so') > found_paths = oe.external.find_sysroot_files([sopattern], d) if found_paths: ExpansionError: Failure expanding variable PV[:=], expression was ${@get_external_libc_version(d)} which triggered exception AttributeError: 'module' object has no attribute 'external' (Bitbake rev: 7ff5b9eed82b7f4fd138fc6d746a0b79efbea98a) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-14bitbake: cooker/toasterui: Drop SEND_DEPENDS_TREE UI featureRichard Purdie
Now the event is sent unconditionally we can drop this feature as its no longer needed. (Bitbake rev: 473deeb0fc6065693e1fcfcbb8b79753103db537) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-13bitbake: cooker: Fix log message syntaxRichard Purdie
Ensure we pass the string parameter correctly. (Bitbake rev: 7ed82bd1fe7bdd93b0614119c42eb218dc5d83e6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-06bitbake: bb.{cooker, data}: only emit a var as python if 'func' is setChristopher Larson
This avoids a common issue where PACKAGECONFIG is emitted as a function in bitbake -e when the 'python' flag exists. It isn't a python function unless both 'func' and 'python' are set. This aligns with the behavior of emit_func_python. (Bitbake rev: c5e45063cb3ae17bbe3304ea5e712bd76e686c4a) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-15bitbake: cooker/knotty: Prefix parse logs with filename being parsedRichard Purdie
We now prefix log messages coming from worker task context with the PF and task info, however parsing messages all have to be manually prefixed which is ugly and error prone. This change modifies the log handler filter so this happens automatically, meaning we don't have to change every message to include that information. This makes error messages longer but more usable. (Bitbake rev: 1af0ccaac81e182c4ca520037dda362d180e5605) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-15bitbake: cooker: pass exception to finishAsyncCommandRichard Purdie
An invalid task causes bitbake to exit incorrectly, firing a CommandCompleted event rather than a CommandFailed one. This means that clients listening for CommandFailed events are unable to detect the build failure even though one occurred. Passing an exception string to finishAsyncCommand when a task fails causes the CommandFailed event to be fired correctly. [YOCTO #9087] (Bitbake rev: 98a2c37e077b16e3bc8bb102bd18b293130d15a4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-31bitbake: cooker: fix CookerParser.shutdown()Markus Lehtonen
Prevent a hang when shutdown() is called during parsing (e.g. after SIGINT). We must not append 'None' to the jobs queue. Otherwise the worker loop inside Parser.realrun() may break out at the wrong point, causing the results queue thread blocking bitbake indefinitely. [YOCTO #9319] (Bitbake rev: 7ebea3e9a60232222efa8a546a0ff28a53029949) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-30bitbake: cooker: Ensure bbappend order is deterministicRichard Purdie
Currently bbappend files in a layer are applied in the order they're found on disk (as reported by glob) which means things are not deterministic. By sorting the glob results, the order becomes deterministic, the parsing order for .bb files also should be deterministic as a result of this change. [YOCTO #9138] (Bitbake rev: 3f8febc4212fbd3485ac9bdd4ac71b8fb0a05693) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18bitbake: bb/cache: drop some unused argumentsMarkus Lehtonen
Drop unused 'd' argument from the cache save methods, simplifying the API. (Bitbake rev: 81bc1f20662c39ee8db1da45b1e8c7eb64abacf3) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-15bitbake: cooker: Improve cache handlingRichard Purdie
We're seeing issues where the self test, which uses tinfoil doesn't notice the changed contents of include files. The issue is cached_statements in the parser being reused when the files have changed. Whilst looking at this, I realised there were some other issues: * We need to also invalidate the mtime cache when cooker restarts * We should pass full filenames to the file invalidation code * We should process cached_statements as part of inotify invalidation With these fixes, the caching is more reliable for memory resident bitbake too. It does raise some questions about cache validation and lifecycles and indicates bitbake does need more work in the area, preferably with the removal of the globals. This at least highlights and works around some of the current issues. (Bitbake rev: 3f507ff8bc467fba936cf3f31bb8ea8e02f168e8) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-11bitbake: cooker: gracefully shutdown parsersEd Bartosh
CookerParser.shutdown code doesn't do all required work to shutdown parser processes. As a result bitbake hangs if interrupted during parsing. Putting None into the parser_quit queue should fix this issue as it makes parsers to quit main loop. (Bitbake rev: f67307977e8f089ce6d208d3e9de2a6a1768757e) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10bitbake: build/data: Don't expand python functions before execution [API change]Richard Purdie
Right now, if you have some python code like: X = "a" def somefunction(d): d.setVar("X", "b") d.setVar("Y", "${X}") then any sane person would expect that Y = "b" at the end of the function. This is not the case, Y = "a". This is due to the python function being expanded before execution, the executed code would read d.setVar("Y", "a"). This understandably confuses people, it also makes it near impossible to write ${} in a python function without unintended things happening. I think there is general agreement we should fix this and standardise on non-expansion of python functions. We already don't expand anonymous python (mostly). I've checked OE-Core with buildhistory before and after this change and there were a small number of issues this exposed which I've sent patches for. I propose we default to not expanding python code and then deal with any consequences from that if/as/where identified. This will improve new user understanding and usability of the system, it also allows several long standing weird expansion issues to be fixed. (Bitbake rev: 8bf33a8e92c0e188fa392030025756196c96fcbb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10bitbake: cooker: Don't expand python functions in variable dumpsRichard Purdie
We don't want to expand python functions since they aren't expanded at execution time (e.g. anonymous python). They can also have side effects. This function is primarily used by toaster for variable dumps for later display. The lack of expansion of python functions won't matter in this case and actively helps some variable handling (e.g. SRCPV). (Bitbake rev: 3f5520b4844a4bdd615046479ba08ed192bdc8cd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-10bitbake: cooker: extended dot stylingSchroeder, Henning
Extended the dot styling of dependencies created by bitbake -g in dot syntax to differentiate between the various kinds. depends: solid rdepends: dashed rrecommends: dotted The change observed is that depends get an explicit style which is the same as dot default behavior and the runtime recommends get dotted while before they were dashed. This helps to distinguish them graphically as well as eases post processing by script. (Bitbake rev: 86e78e0ca7aa5452411f35239942ecee3d8824ec) Signed-off-by: Henning Schroeder <henning.schroeder@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-06bitbake: cooker: shutdown cooker parser on shutdownEd Bartosh
Currently any not catched exception in cooker causes bitbake to hang because of not terminated children of CookerParser. Long term solution would be to reimplement Cooker as a context manager and terminate parser children in its __exit__ method. Partial fix is to call CookerParser.shutdown in Cooker.shutdown in hope that all Cooker exceptions are caught and shutdown method is called. [YOCTO #8900] (Bitbake rev: 3f67600dc3292bc8208644ce89e8bf7ab95cf2e7) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-01bitbake: cooker, bitbake-worker: Fix spelling of "received"Phil Blundell
I before E, except after C... (Bitbake rev: 14c9593265f7469cb8a205a46f845ac7491246df) Signed-off-by: Phil Blundell <pb@pbcl.net> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-01bitbake: cooker: Only start as many parse threads as we needRichard Purdie
If we're only going to parse one recipe, no point in starting a large number of threads. (Bitbake rev: b977faf59dc08050a44a16032fe52d1bbb80f2a1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-30bitbake: bitbake: Set process names to be meaninfulRichard Purdie
This means that when you view the process tree, the processes have meaningful names, aiding debugging: $ pstree -p 30021 bash(30021)───KnottyUI(115579)───Cooker(115590)─┬─PRServ(115592)───{PRServ Handler}(115593) ├─Worker(115630)───bash:sleep(115631)───run.do_sleep.11(115633)───sleep(115634) └─{ProcessEQueue}(115591) $ pstree -p 30021 bash(30021)───KnottyUI(117319)───Cooker(117330)─┬─Cooker(117335) ├─PRServ(117332)───{PRServ Handler}(117333) ├─Parser-1:2(117336) └─{ProcessEQueue}(117331) Applies to parse threads, PR Server, cooker, the workers and execution threads, working within the 16 character limit as best we can. Needed to tweak the bitbake-worker magic values to tell the workers apart. (Bitbake rev: 539726a3b2202249a3f148d99e08909cb61902a5) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-29bitbake: cooker: Allow BBMASK to contain multiple regular expressionsPeter Kjellerstedt
Before, BBMASK was only permitted to contain one regular expression. This made it hard to add to the BBMASK in multiple places as one was supposed to separate the different regular expressions with a "|" rather than with whitespace as is customary in BitBake variables. Now one can specify any number of regular expressions in BBMASK. This makes it possible to, e.g., mask out recipes in another layer from the layer.conf file. This also properly ignores any regular expressions that do not compile (before an invalid regular expression would cause a ParseError in the first bbappend file found stating that it was not a BitBake file...) (Bitbake rev: 2c778ad50aceaffb855baf5f4aa0fed98c880870) Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-22bitbake: taskdata: add the ability to access world targets listPaul Eggleton
In certain circumstances it can be useful to get access to the world targets list from a recipe in order to add dependencies on some or all of the items in it. If a special function, 'calculate_extra_depends' is defined in the recipe, and the recipe is to be built, then call it at the right point before we calculate which tasks should be run. The function can append items to the "deps" list in order to add dependencies. This is not as tidy a solution as I would have liked, but it does at least do the job. As part of this change, the buildWorldTargets function was moved to bb.providers to make it possible to call from taskdata. Part of the implementation of [YOCTO #8600]. (Bitbake rev: aba0dce57c889495ec5c13919991a060aeff65d2) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11bitbake: cooker: add providermap to dep_treeEd Bartosh
Added providermap information to the result of buildDependTree API. This will be used by Toaster to map virtual dependencies to recipes. (Bitbake rev: d3e07368549f30265f59846a260efa8230a225ca) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Elliot Smith <elliot.smith@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-08bitbake: cooker: fix findFilesMatchingInDir documentationRoss Burton
The documentation for findFilesMatchingInDir() was inconsistant with the implementation: the regex was escaped before searching so effectively it's a pure textual substring, and the machine example was broken. (Bitbake rev: 6bef981488ec94b46dbe3797acfecf9c4b6ecbbc) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-08bitbake: cooker: use in instead of countRoss Burton
No point counting all instances when we just want to know if there's any or not. (Bitbake rev: 3369072efb653339da8dbd1ca864ff8e1ff899ca) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-06bitbake: cooker: add state.get_name methodEd Bartosh
Added method to convert state code into the human readable name. It will be used in logging and error reporting. (Bitbake rev: 9ec6379b27d210214d0b3f2e55962f721b7f5f51) Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>