summaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/cache.py
AgeCommit message (Collapse)Author
2019-08-06bitbake: siggen: Clean up task reference formatsRichard Purdie
Currently siggen uses the format "<filename>.<taskname>" for referencing tasks whilst runqueue uses "<filename>:<taskname>". This converts to use ":" as the separator everywhere. This is an API breaking change since the cache is affected, as are siginfo files and any custom signature handlers such as those in OE-Core. Ultimately this will let us clean up and the accessor functions from runqueue, removing all the ".rsplit(".", 1)[0]" type code currently all over the place. Once a standard is used everwhere we can update the code over time to be more optimal. (Bitbake rev: 07e539e1c566ca3434901e1a00335cb76c69d496) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-08-06bitbake: cache: Add SimpleCache classRichard Purdie
This adds a simple version of the MultiProcessCache which can be used to save and load cache data, useful for a new usecase we have in sigdata/runqueue. (Bitbake rev: 19a6e35600ae6d2d1bcecca6e68ab8c37674774e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-19bitbake: cache: Create a symlink for current cachefileRobert Yang
So that people or other tools can easily know which one is being used, just like what we did for run.do_task and log.do_task, otherwise, we have no way to know it. I usually use "ls -t", but it isn't reliable since the one which is being used may not the latest one. (Bitbake rev: cf286dff653eed542bf347ca46234c224944d5b0) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-15bitbake: cache: Set packages for skipped recipesRobert Yang
The provides and rprovides had been set for skipped recipes, packages are similar to them (all of them provide something), so also set it. This makes it easier to figure out the RDEPENDS issues, for example, lmsensors (lmsensors_3.5.0.bb) RRECOMMENDS lmsensors-config-fancontrol (lmsensors-config_1.0.bb), but lmsensors-config is skipped for some reasons, then if we run: $ bitbake lmsensors ERROR: Nothing RPROVIDES 'lmsensors-config-fancontrol' (but /path/to/lmsensors_3.5.0.bb RDEPENDS on or otherwise requires it) NOTE: Runtime target 'lmsensors-config-fancontrol' is unbuildable, removing... Missing or unbuildable dependency chain was: ['lmsensors-config-fancontrol'] ERROR: Required build target 'lmsensors' has no buildable providers. Missing or unbuildable dependency chain was: ['lmsensors', 'lmsensors-config-fancontrol'] We had no way to know who rprovides lmsensors-config-fancontrol, we can figure it out by bitbake/contrib/dump_cache.py after this patch. (Bitbake rev: 9cf7a5e5a28e676427970a821893e9d930973969) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-07-15bitbake: cache: Remove duplicated lines for provides and rprovidesRobert Yang
Whether skip or not, they are always set, so move the lines ahead to avoid duplicated lines. (Bitbake rev: c1a8ebb8f83e5108b667f291c924fc2fbd2ac769) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-10bitbake: multiconfig: Switch from 'multiconfig' -> 'mc'Richard Purdie
After real world use its clear the "multiconfig:" prefix to multiconfig tasks, whilst clear, is also clumbersome. Switch to use the short version instead. mcdepends will continue to work with "multiconfig:" for now as well. The commandline will only accept mc: going forward. [YOCTO #11168] (Bitbake rev: 821daf093b76504067a8b77dfa4b181af6ec92b4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04bitbake: bitbake: Strip old editor directives from file headersRichard Purdie
There are much better ways to handle this and most editors shouldn't need this in modern times, drop the noise from the files. Its not consitently applied anyway. (Bitbake rev: 5e43070e3087d09aea2f459b033d035c5ef747d0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04bitbake: bitbake: Drop duplicate license boilerplace textRichard Purdie
With the introduction of SPDX-License-Identifier headers, we don't need a ton of header boilerplate in every file. Simplify the files and rely on the top level for the full licence text. (Bitbake rev: 695d84397b68cc003186e22f395caa378b06bc75) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-04bitbake: bitbake: Add initial pass of SPDX license headers to source codeRichard Purdie
This adds the SPDX-License-Identifier license headers to the majority of our source files to make it clearer exactly which license files are under. The bulk of the files are under GPL v2.0 with one found to be under V2.0 or later, some under MIT and some have dual license. There are some files which are potentially harder to classify where we've imported upstream code and those can be handled specifically in later commits. The COPYING file is replaced with LICENSE.X files which contain the full license texts. (Bitbake rev: ff237c33337f4da2ca06c3a2c49699bc26608a6b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-12-07bitbake: cache: Cleanup deprecated function usageRichard Purdie
Remove a deprecated warning and stop using our own deprecated API! (Bitbake rev: 83ece2c6f4b000e906fec9148f25bd1dff66cfb0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-07-30bitbake: codeparser: Track oe.utils.conditional usageMartin Jansa
* base_conditional was already removed from oe-core: http://git.openembedded.org/openembedded-core/commit/?id=0391fcad9103abca0796a068f957d0df63ab4776 after the usage was migrated to oe.utils.conditional: http://git.openembedded.org/openembedded-core/commit/?id=c97acbd034532895ce57c6717ed1b3ccc7900b0d so we can handle just ".conditional" version * add 1st parameter to variable dependencies, that way when you use FOO = "${@oe.utils.conditional('VAR', 'VALUE', 'true', 'false')" FOO variable will have dependency on VAR variable and you don't need to add FOO[vardeps] += "VAR" manually every time you use oe.utils.conditional * this is similar to contains tracking from: https://bugzilla.yoctoproject.org/show_bug.cgi?id=3890 http://git.openembedded.org/bitbake/commit/?id=ed2d0a22a80299de0cfd377999950cf4b26c512e http://git.openembedded.org/bitbake/commit/?id=0b9d117631ce909312d53b93289e61defc6be01c but conditional is simpler, we don't need to handle the first parameter as a set (Bitbake rev: 5156b4bb6876dac636be9726df22c8ee792714dd) Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2018-02-14bitbake: bitbake: cache: improve debug messageRobert Yang
* Print message when cachefile is found/not can help debug. * Update "Using cache in" to "Cache dir:" since it was the same as the debug message of "codeparser & file checksum caches", which caused confusion. And whether the cache file will be used or not is still unknown at that time, so just print the cache dir. (Bitbake rev: c8d3a2016f432e8ed9e99d9c28850149ab6fd6d8) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-07-18bitbake: cache.py: fix whitespacePaul Eggleton
(Bitbake rev: e22b1f1c0e57c6ada4fb044791159546e2260dad) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-28bitbake: cache: don't insert PN into PACKAGESRoss Burton
The cache code currently inserts PN into the package list if it isn't already present. Whilst this ensures that the package list contains something which is important for native recipes that don't set PACKAGES, it causes confusing behaviour where a normal recipe doesn't have PN in PACKAGES: for example adding dhcp to IMAGE_INSTALL will parse successfully but fail at rootfs time as the dhcp recipe doesn't generate a dhcp package. Solve this by only adding PN to the cache's package list if the package list is empty. This results in the package list for recipes such as DHCP being correct, but native recipes continue to have just PN in the list as before. [ YOCTO #5533 ] (Bitbake rev: df31a88786ce5bd7708ff14e1379dc2a58a8c0cf) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-11bitbake: cache: prevent division-by-zero in load_cahefile()Markus Lehtonen
[YOCTO 11315] (Bitbake rev: 227c5acd4b40154bc61202e7bb67a13818a7d727) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-05bitbake: codeparser: improve handling of contains_any() and filter()Paul Eggleton
Ensure we handle bb.utils.contains_any() as separate items, rather than how we handle contains() where every item must be in the list. Additionally, enable handling bb.utils.filter() which for the purposes of looking at dependencies is the same as contains_any(). Additionally bump the codeparser cache and recipe cache versions to invalidate the user's existing caches (ensuring that the changes take effect and avoiding "taskhash mismatch" errors respectively). (Bitbake rev: 496e3c84820a2a889d99d3604659e47a550941d5) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30bitbake: bitbake: remove True option to getVarFlag callsJoshua Lock
getVarFlag() now defaults to expanding by default, thus remove the True option from getVarFlag() calls with a regex search and replace. Search made with the following regex: getVarFlag ?\(( ?[^,()]*, ?[^,()]*), True\) (Bitbake rev: c19baa8c19ea8ab9b9b64fd30298d8764c6fd2cd) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30bitbake: bitbake: remove True option to getVar callsJoshua Lock
getVar() now defaults to expanding by default, thus remove the True option from getVar() calls with a regex search and replace. Search made with the following regex: getVar ?\(( ?[^,()]*), True\) (Bitbake rev: 3b45c479de8640f92dd1d9f147b02e1eecfaadc8) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-02bitbake: cache: allow parsing a recipe with a custom config datastorePaul Eggleton
To accommodate the OpenEmbedded layer index recipe parsing, we have to have the ability to pass in a custom config datastore since it constructs a synthetic one. To make this possible after the multi-config changes, rename the internal _load_bbfile() function to parse_recipe(), make it a function at the module level (since it doesn't actually need to access any members of the class or instance) and move setting __BBMULTICONFIG inside it since other code will expect that to be set. Part of the fix for [YOCTO #10192]. (Bitbake rev: 5b3fedfe0822dd7effa4b6d5e96eaf42669a71df) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18bitbake: bitbake: Initial multi-config supportRichard Purdie
This patch adds the notion of supporting multiple configurations within a single build. To enable it, set a line in local.conf like: BBMULTICONFIG = "configA configB configC" This would tell bitbake that before it parses the base configuration, it should load conf/configA.conf and so on for each different configuration. These would contain lines like: MACHINE = "A" or other variables which can be set which can be built in the same build directory (or change TMPDIR not to conflict). One downside I've already discovered is that if we want to inherit this file right at the start of parsing, the only place you can put the configurations is in "cwd", since BBPATH isn't constructed until the layers are parsed and therefore using it as a preconf file isn't possible unless its located there. Execution of these targets takes the form "bitbake multiconfig:configA:core-image-minimal core-image-sato" so similar to our virtclass approach for native/nativesdk/multilib using BBCLASSEXTEND. Implementation wise, the implication is that instead of tasks being uniquely referenced with "recipename/fn:task" it now needs to be "configuration:recipename:task". We already started using "virtual" filenames for recipes when we implemented BBCLASSEXTEND and this patch adds a new prefix to these, "multiconfig:<configname>:" and hence avoid changes to a large part of the codebase thanks to this. databuilder has an internal array of data stores and uses the right one depending on the supplied virtual filename. That trick allows us to use the existing parsing code including the multithreading mostly unchanged as well as most of the cache code. For recipecache, we end up with a dict of these accessed by multiconfig (mc). taskdata and runqueue can only cope with one recipecache so for taskdata, we pass in each recipecache and have it compute the result and end up with an array of taskdatas. We can only have one runqueue so there extensive changes there. This initial implementation has some drawbacks: a) There are no inter-multi-configuration dependencies as yet b) There are no sstate optimisations. This means if the build uses the same object twice in say two different TMPDIRs, it will either load from an existing sstate cache at the start or build it twice. We can then in due course look at ways in which it would only build it once and then reuse it. This will likely need significant changes to the way sstate currently works to make that possible. (Bitbake rev: 5287991691578825c847bac2368e9b51c0ede3f0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18bitbake: cache: Build datastores from databuilder objectRichard Purdie
Rather than passing in a datastore to build on top of, use the data builder object in the cache and base the parsed recipe from this. This turns things into proper objects building from one another rather than messy mixes of static and class functions. This sets things up so we can support parsing and building multiple configurations. (Bitbake rev: fef18b445c0cb6b266cd939b9c78d7cbce38663f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18bitbake: cache: Split Cache() into a NoCache() parent objectRichard Purdie
There are some cases we want to parse recipes without any cache setup or involvement. Split out the standalone functions into a NoCache variant which the Cache is based upon, setting the scene for further cleanup and restructuring. (Bitbake rev: 120b64ea6a0c0ecae7af0fd15d989934fa4f1c36) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18bitbake: cache/cooker: Pass databuilder into bb.cache.Cache()Richard Purdie
Rather that the current mix of static and class methods, refactor so that the cache has the databuilder object internally. This becomes useful for the following patches for multi config support. It effectively completes some of the object oriented work we've been working towards in the bitbake core for a while. (Bitbake rev: 7da062956bf40c1b9ac1aaee222a13f40bba9b19) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18bitbake: cache: Make virtualfn2realfn/realfn2virtual standalone functionsRichard Purdie
Needing to access these static methods through a class doesn't make sense. Move these to become module level standalone functions. (Bitbake rev: 6d06e93c6a2204af6d2cf747a4610bd0eeb9f202) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-18bitbake: cache/ast: Move __VARIANTS handling to parse cache functionRichard Purdie
Simple refactoring to allow for multiconfig support. (Bitbake rev: 266b848da40904446eb1d084bbdc5307a9b45197) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26bitbake: cache: Don't interleave pickle cache file writingRichard Purdie
For some reason the data written in this way is coming back out the files out of order. I've not been able to simplify the test case to a point where this was standalone reproducible. Simplify the code and write out the cache files sequentially since this seems to avoid the errors and makes the code more readable. (Bitbake rev: 14ec47f5f0566dbd280fae8a03160c8500ad3929) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26bitbake: cache: Add better cache loading sanity checksRichard Purdie
We've seen cache corruption where the pairs come out in a different order to the way we saved them for unknown reasons. Add better sanity checking to give a more user friendly error rather than a crash/traceback. Also allows the system to reparse and recover. (Bitbake rev: 4be4a15491530bd6dc018033ad3d4b2562ab6e23) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26bitbake: cache: Drop/simplify pointless type checkingRichard Purdie
Since we no longer have random data like version fields in these structures and we can assume any extra cache data subclasses our class, simplify the code. This is mostly reindenting after removal of the pointless type checks. (Bitbake rev: 5eb36278ac9975de1945f6da8161187320d90ba7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26bitbake: cache: Improve versions fields handlingRichard Purdie
Firstly, don't store the versions fields in memory in the cache objects data store. This just complicates the code for no good reason. Secondly, write the version fields to all cache files, not just the core one. This makes everything consistent and easier. (Bitbake rev: cb666262b2f986b5d9331dfb30458ef1a151fa4d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26bitbake: cache: Correctly handle missing extra cachesRichard Purdie
If an "extras" cache file is corrupted, the system would not notice and later fail with errors about missing entries. Add a test for this which means we can fall back to re-parsing in those cases. [YOCTO #9902] (Bitbake rev: 51843d8f2bbe2e54db7593ca61984abe70423ef6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-26bitbake: cache: Move the parsing message to a more logical placeRichard Purdie
Otherwise you can look at the log and wonder why parsing isn't happening when it really is due to other code paths. (Bitbake rev: b48d95677a4d285a77cda2892179965f7f8f06dd) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02bitbake: bitbake: Convert to python 3Richard Purdie
Various misc changes to convert bitbake to python3 which don't warrant separation into separate commits. (Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-19bitbake: cache: Increase cache version after task checksum calculation changesRichard Purdie
(Bitbake rev: 6df0425a9d5c4c520eb7845d8f6175d9641779a7) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-13bitbake: contrib/dump_cache.py, cache: Fix to use python 3 syntaxRichard Purdie
Some tweaks to use python 3 syntax in a python 2 compatible way. (Bitbake rev: 322949c77dbaa4db01b5a43d85b39a2af67ba7b2) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-23bitbake: cache: Make BB_DONT_CACHE variable externalMarkus Lehtonen
This makes it possible to prevent a recipe to be cached, and thus, parsed every time. Use with care. [YOCTO #8853] (Bitbake rev: 78335c1fbe5266116700c2413aac28b00423a75b) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18bitbake: MultiProcessCache: make cache filename configurableMarkus Lehtonen
If no cache file name is given a default from class variable is used, like before. (Bitbake rev: 2602a312818f564961de7dfa63c429d45ff9e5ac) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-18bitbake: bb/cache: drop some unused argumentsMarkus Lehtonen
Drop unused 'd' argument from the cache save methods, simplifying the API. (Bitbake rev: 81bc1f20662c39ee8db1da45b1e8c7eb64abacf3) Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-22bitbake: taskdata: add the ability to access world targets listPaul Eggleton
In certain circumstances it can be useful to get access to the world targets list from a recipe in order to add dependencies on some or all of the items in it. If a special function, 'calculate_extra_depends' is defined in the recipe, and the recipe is to be built, then call it at the right point before we calculate which tasks should be run. The function can append items to the "deps" list in order to add dependencies. This is not as tidy a solution as I would have liked, but it does at least do the job. As part of this change, the buildWorldTargets function was moved to bb.providers to make it possible to call from taskdata. Part of the implementation of [YOCTO #8600]. (Bitbake rev: aba0dce57c889495ec5c13919991a060aeff65d2) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-22bitbake: cache.py: check existence before add to cachedata.rprovidersRobert Yang
The rprovides maybe contain duplicated lines when parse again, we need check it before add to cachedata.rproviders, similar to what we had done to cachedata.providers. (Bitbake rev: 6c488afb0fe30a9655ec62a1d22f9f388365f012) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-11bitbake: bitbake: clean up stamp-base related codesChen Qi
The 'stamp-base' and 'stamp-base-clean' related codes are no longer useful, clean them up. [YOCTO #8468] (Bitbake rev: 7b4c42b315d4a26dd8f2ceb874a94737bf9f183e) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-11-16bitbake: cache: Don't try to expand __inherit_dataRichard Purdie
Trying to expand a variable which isn't a string doesn't make sense. (Bitbake rev: 54d0ddd166a6707b4f8c8535639e3055b783363b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-18bitbake: cache: Handle spaces and colons in directory names for file-checksumsRichard Purdie
If there is a space in a directory name containing a file in file-checksums (e.g. from a file:// url), you currently get tracebacks from bitbake. This improves the code to handle colons and spaces in the file-checksums names since it possible to figure out the correct names. [YOCTO #8267] (Bitbake rev: 87282b283921a58426f24fb21151db457c5bca66) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-05-26bitbake: cache: Clean up getVar usage to modern syntax/styleRichard Purdie
No functional change. (Bitbake rev: 0eb75a34bd9731e9de7bc9600a7418a927561fdb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-01-14bitbake: cooker/cache/parse: Implement pyinofity based reconfigureRichard Purdie
Memory resident bitbake has one current flaw, changes in the base configuration are not noticed by bitbake. The parsing cache is also refreshed on each invocation of bitbake (although the mtime cache is not cleared so its pointless). This change adds in pyinotify support and adds two different watchers, one for the base configuration and one for the parsed recipes. Changes in the latter will trigger a reparse (and an update of the mtime cache). The former will trigger a complete reload of the configuration. Note that this code will also correctly handle creation of new configuration files since the __depends and __base_depends variables already track these for cache correctness purposes. We could be a little more clever about parsing cache invalidation, right now we just invalidate the whole thing and recheck. For now, its better than what we have and doesn't seem to perform that badly though. For education and QA purposes I can document a workflow that illustrates this: $ source oe-init-build-env-memres $ time bitbake bash [base configuration is loaded, recipes are parsed, bash builds] $ time bitbake bash [command returns quickly since all caches are valid] $ touch ../meta/classes/gettext.bbclass $ time bitbake bash [reparse is triggered, time is longer than above] $ echo 'FOO = "1"' >> conf/local.conf $ time bitbake bash [reparse is triggered, but with a base configuration reload too] As far as changes go, I like this one a lot, it makes memory resident bitbake truly usable and may be the tweak we need to make it the default. The new pyinotify dependency is covered in the previous commit. (Bitbake rev: 0557d03c170fba8d7efe82be1b9641d0eb229213) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-01-08bitbake: cache/fetch2/siggen: Ensure we track include history for file checksumsRichard Purdie
Currently, if you reference a file url, its checksum is included in the task hash, however if you change to a different file at a different location, perhaps taking advantage of the FILESPATH functionality, the system will not reparse the file in question and change its checksum to match the new file. To correctly handle this, the system not only needs to know if the existing file still exists or not, but also check the existance of every file it would have looked at when computing the original file. We already do this in the bitbake parsing code for class inclusion. This change uses the same technique to log the file list we looked at and if files in these locations exist when they previously did not, to invalidate and reparse the file. Since data stored in the cache is flattened text, we have to use a string form of the data and split on the ":" character which is ugly, but is an internal detail we can improve later if a better method is found. The cache version changes to trigger a reparse since the previous cache data is now incompatible. [YOCTO #7019] (Bitbake rev: 6c0706a28d72c591f1b75b6e3f3b645859387c7e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-08-25bitbake: lib/bb/*.py: Typo fixes/grammar/comment fixes, nothing functional.Robert P. J. Day
(Bitbake rev: 587b144ee409d444494d8d7f2d1c53ede8f7c953) Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-07-26bitbake: codeparser cache improvementsRichard Purdie
It turns out the codeparser cache is the bottleneck I've been observing when running bitbake commands, particularly as it grows. There are some things we can do about this: * We were processing the cache with "intern()" at save time. Its actually much more memory efficient to do this at creation time. * Use hashable objects such as frozenset rather than set so that we can compare objects * De-duplicate the cache objects, link duplicates to the same object saving memory and disk usage and improving speed * Using custom setstate/getstate to avoid the overhead of object attribute names in the cache file To make this work, a global cache was needed for the list of set objects as this was the only way I could find to get the data in at setstate object creation time :(. Parsing shows a modest improvement with these changes, cache load time is significantly better, cache save time is reduced since there is now no need to reprocess the data and cache is much smaller. We can drop the compress_keys() code and internSet code from the shared cache core since its no longer used and replaced by codeparser specific pieces. (Bitbake rev: 4aaf56bfbad4aa626be8a2f7a5f70834c3311dd3) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-07-26bitbake: cache: Optimise invalid cache file handlingRichard Purdie
If there is a corrupt/invalid cache file, we'd keep trying to reopen it. This is pointless, simplify the code paths and delete the dead file. (Bitbake rev: c22441f7025be012ad2e62a51ccb993c3a0e16c9) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-07-26bitbake: cache: Don't reload the cache file since we already have this data ↵Richard Purdie
in memory If we're writing out merged data to disk, its safe to assume that either we loaded the data or couldn't. Loading it again is relatively pointless and time consuming. (Bitbake rev: 14c31e18e4a4a52ee54056aeefb09ff8c295b475) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-06-12bitbake: cache.py: print debug info when EXCLUDE_FROM_WORLDRobert Yang
This gives us an easy way to find out which recipes have been excluded from world when there are many layers. (Bitbake rev: cf33d4bfc53ae8dc6353fa295e1acd789d4fc2d2) Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>