aboutsummaryrefslogtreecommitdiffstats
path: root/bitbake/lib/bb/data_smart.py
AgeCommit message (Collapse)Author
2012-07-05bitbake: data_smart: Fix multiple override interaction with append and ↵Richard Purdie
prepend operators Variables which used multiple overrides and the append/prepend operators were not functioning correctly. This change fixes that. This fixes the testcase: OVERRIDES = "linux:x86" TESTVAR = "original" TESTVAR_append_x86 = " x86" TESTVAR_append_x86_linux = " x86+linux" TESTVAR_append_linux_x86 = " linux+x86" [YOCTO #2672] (Bitbake rev: dc35a2e506e15fb7ddbf74c3b3280e9e83ab33bb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-05-30lib/bb/data_smart.py: don't report variable in ExpansionError if not setPaul Eggleton
If the variable name is not specified then don't confuse the error message by starting off with "Failure expanding variable None...". (Bitbake rev: 9cb16f3c73751e7cf6d495586a6193f06eb97b1f) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-04-17data_smart: Improve the calculation of config hashDongxiao Xu
For config hash, we put the keys in structure of "set()", which is not order sensitive. Therefore when calculating the md5 value for config hash, we need to identify the order of the keys. (Bitbake rev: 0f1b142a3f6b8125bf023c2e5ec269618869abf7) Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-03-05meta/classes: Convert to use appendVar and appendVarFlagsRichard Purdie
(From OE-Core rev: 3b57de68e70e77dbc03c0616a83a29a2e99e40b4) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-02-27data_smart.py: Uses BB_HASHCONFIG_WHITELIST to filter unnecessary variablesDongxiao Xu
Adopt the BB_HASHCONFIG_WHITELIST as a exclusion list for variables that are not needed in cache hash calculation. (Bitbake rev: ae8cf138b5eb8f1f28a7143b8d67ad06cbe43061) Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com> CC: Christopher Larson <kergoth@gmail.com> CC: Martin Jansa <martin.jansa@gmail.com> CC: Andreas Oberritter <obi@opendreambox.org> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-02-23cache: Use configuration's hash value to validate cacheDongxiao Xu
Previously we use the file time stamp to judge if a cache is valid. Here this commit introduce a new method, which calculates the total hash value for a certain configuration's key/value paris, and tag it into cache filename, for example, bb_cache.dat.xxxyyyzzz. This mechanism also ensures the cache's correctness if user dynamically setting variables from some frontend GUI, like HOB. (Bitbake rev: 1c1df03a6c4717bfd5faab144c4f8bbfcbae0b57) Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-11-27bitbake: Update users of getVar/setVar to use the data store functions directlyRichard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-11-10data_smart: Add appendVar/prependVar functionsRichard Purdie
This patch adds appendVar and prependVar functions to the data store meaning python code would no longer have to do the getVar, append and the setVar dance that much of the current python code does. It also adds corresponding variants for flags. Currently there is no spacing added by these functions. That could be added as a parameter if desired. If these functions turn out to be hotspots in the code, there are tricks that could potentially be used to increase the speed of these specific operations within the datastore. (Bitbake rev: 4a4046268f84b85559eea2c4b6a6004ad8cccb77) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-11-10codeparser: silence non-literal warnings for vardepsChristopher Larson
If the vardeps flag is not None, we now silence the warnings about non-literal usage for that variable. (Bitbake rev: e724b9f417d1baf898f5afc6376c73c1a2ad8db9) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-11-10codeparser: accept a name for better messagesChristopher Larson
- If a name is passed to the parser, prepend the messages with "while parsing <name>:". This gives a bit more context. - Tweak the warning messages slightly (they had to be altered anyway to inject the variable being parsed). Before: DEBUG: Warning: in call to 'bb.data.getVar': argument ''%s' % var' is \ not a literal After: DEBUG: while parsing emit_pkgdata, in call of bb.data.getVar, argument \ ''%s' % var' is not a string literal (Bitbake rev: 1060193ae4d54e667735dbff5d1d2be49a3f95c9) Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-08-29data_smart.py: make use of expand cache in getVar()Dongxiao Xu
Currently if passing expand=True to getVar() function, it will pass the handling to getVarFlag(), which doesn't get any benefit from the expand cache. Call the expand() function separately in getVar() to make use of the expand cache, which can decrease the parsing time by 40%. (from current 49s to 27s) (Bitbake rev: 6555a77c199f41bf35460138764e03e30c56d29f) Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-07-05bitbake/ast: Fix ??= vs. ?= handlingRichard Purdie
As the code stands, setting a variable with ??= could result in a ?= variable not overriding it. This patch fixes the issue by allowing the ast to make lookups that ignore any ??= set variables. (Bitbake rev: 32fee2e650dfdd3aa9a7572dad1251e0c24ca34b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-06-28bitbake/data_smart: Don't track overrides in deleted variable namesRichard Purdie
When we delete a variable we no longer expect it to override other variables. To do this we remove it from the list of active overrides at deletion time. It turns out we already had to do this at override expansion time so this cleans up the code to be more consistent as an added bonus. (Bitbake rev: d924ff9ede57c3dea6e1c738ba3633f18d460b14) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-06-28bitbake/data_smart: Don't export deleted/empty entries in the list of keysRichard Purdie
If you d.delVar(), you expect the variable to be gone. Even empty variables continue to exist in the datastore and are still user visible unfortunately. The COW siutation means you can't just remove it since it might unmask a variable from an inner copy. This patch therefore stops empty variables from appearing in key lists exposed to the external world making empty variables an internal implementation detail only. (Bitbake rev: 2b5548c591d4cfde9238d2cc0959c42cfc08f09c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-06-28bitbake/data_smart: Optimise the data store iteratorRichard Purdie
Since we're going to creat the seen set() anyway, we might as well use it directly. If we don't do this, we see thousands of function calls with associated overhead on profiles. (Bitbake rev: 9d43e3279895639ee4899df635f2546c7ee13737) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-06-09bitbake/data_smart: Change overrides behaviour to remove expanded variables ↵Richard Purdie
from the datastore Currently if you do: OVERRIDES = "z" DEPENDS_prepend = "a " DEPENDS = "b" DEPENDS_z = "c" d.update_data() d.getVar("DEPENDS") gives "a c" d.update_data() d.getVar("DEPENDS") then gives "c" This patch changes the behaviour such that at the time bitbake expands the DEPENDS_z override, it removes "DEPENDS_z" from the data store. In the above example this would mean that it wouldn't matter how often you call d.update_data(), you'd always get "a c" back. See the bitbake-devel mailing list for further discussion and analysis of the potential impact of this change. (Bitbake rev: 899d45b90061eb3cf3e71029072eee42cd80930c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-06-01data_smart: Adding overrides of zero length is pointless so lets notRichard Purdie
(Bitbake rev: 67434496108efc3aba9cb1e3640bc712658b1408) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-03-31More 'is' fixupsChris Larson
(Bitbake rev: 967cd1aa2c59f15d805862bd9935f507c635c762) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-02-23bitbake/data_smart: Improve Variable expansion error handlingRichard Purdie
If expanding a variable triggers an exception the caller currently has no way to supress the error message or otherwise handle the siutation. An example of where this is a problem is "bitbake -e" showing tracebacks and errors for variables like SRCPV in OE/Poky. Secondly in a chained expansion fails, log mesages are recorded for every step of the expansion, not just the innermost error which is where the real failure occured. To fix this we introduce a new exception ExpansionError which callers can handle as appropriate. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-02-23bitbake/data_smart: Improve the way lazyassignment worksRichard Purdie
Currently, if a variable has been set with ??= and the code looks it up before the data finalisation phase, no value is found. This is causes serious problems for anonymous python functions which manipulate data, or for the fetcher revision handling code where revisions can be set with ??=. There is also a significant performance implication for processing lazy assignment in finalise. Moving the check for a default value into getVarFlag addresses both the timing issue and the performace. This change gives a 7% real time performance improvement to parsing the Poky metadata. The cost of the check at this point is minimal since we have all the data flags available. This should also fix Yocto bug 752. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-01-04bitbake: Sync a load of whitespace and other non-functionality changes with ↵Richard Purdie
bitbake uptream Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04Move LAYERDIR expansion hack into DataSmartChris Larson
(Bitbake rev: 40778a6e9e82c7ea4673a74fc19574430fa63e8d) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04build: use bb.process instead of os.systemChris Larson
(Bitbake rev: 53740977521bc81ffa37adfa7bbeb8f2a80ea165) build: write logfiles per task, not per function Based on d14f9bf6 from poky, reworked for master and other cleanup. (Bitbake rev: beadff2eca1eb95f0411115dd72ddb4c3c44c604) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04Access metadata vars as locals in python snippetsChris Larson
Example: FOO = "bar" BAR = "${@FOO + '/baz'}" ${BAR} == "bar/baz" (Bitbake rev: 606fa1fd97cbd47a6a7ebdc7a2e6aa93a8f65cf5) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04Fix __getitem__ for DataSmartChris Larson
Ensure it raises KeyError for a missing key, this is required to use this as a mapping in various places, e.g. as locals in an eval. (Bitbake rev: 8d661ce0c303e8d69f17c1d095545d5ed086d1d5) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04Switch bitbake internals to use logging directly rather than bb.msgChris Larson
We use a custom Logger subclass for our loggers This logger provides: - 'debug' method which accepts a debug level - 'plain' method which bypasses log formatting - 'verbose' method which is more detail than info, but less than debug (Bitbake rev: 3b2c1fe5ca56daebb24073a9dd45723d3efd2a8d) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2011-01-04Make DataSmart inherit the MutableMapping ABCChris Larson
Provide __len__, __iter__, and the getitem/setitem/delitem methods, and its mixed in versions of keys(), values(), items(), etc will automatically behave, making the DataSmart act more like a real mapping. (Bitbake rev: 89b5351c656d263b0ce513cee043bc046d20a01e) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-12-14bitbake: add optional expansion to getVarFlag()Paul Eggleton
Add a parameter to getVarFlag() to auto-expand the value of the flag. This makes getVarFlag() more consistent with getVar(), and allows expansion of vardeps and vardepsexclude (which has been done in this commit). Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2010-12-07bitbake/data_smart: Refactor _append/_prepend code to remove duplicationRichard Purdie
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-12-07bitbake/data_smart: Fix append/prepend/override ordering issueRichard Purdie
Where a variable name consisted of an append/prepend combined with an override and there was also an append/prepend to the variable, the override could be lost if the override was not in OVERRIDES. For example: FOO = "A" FOO_append = "B" FOO_append_virtclass-native = "C" could result in "AB" even though virtclass-native was in OVERRIDES. With this patch applied, the result is "ABC" as would be expected. The problem was the deletion of the _append/_prepend flag was happening if *any* append/prepend was procesed, the result should really be that it should contain any unprocessed append/prepend. Kevin Tian deserves credit for looking into this and working out the problem here. Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-08-31bitbake: Add codeparser for parsing shell and python functionsRichard Purdie
This commit is derived from Chris Larson's checksum work, turned into a standalone piece of code for parsing python and shell functions. The deindent code has been replaced with code to work around indentation for speed. The original NodeVisitor in the ast was replaced with a faster class walk call. Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-08-31bitbake/data_smart.py: Allow the data expand function to keep track of ↵Richard Purdie
references (including those from python code) Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-07-02Revert "In expand, drop the unnecessary second regular expression match"Chris Larson
This reverts commit 05462fa7908fc22988b3dc9d376798d0a46ccb5a. Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-07-02In expand, drop the unnecessary second regular expression matchChris Larson
(Bitbake rev: 05462fa7908fc22988b3dc9d376798d0a46ccb5a) Signed-off-by: Imran Mehmood <imran_mehmood@mentor.com> Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-07-02Simplify _findVar a bitChris Larson
(Bitbake rev: b017acd39b811a00305002a8044e7d02e79f41d7) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-07-02Fix major bug that slipped in when moving update_dataChris Larson
The root cause is, I was testing the use of renameVar() from finalize, in order to get flags copied over when an override is applied, but renameVar removes the original, whereas the old code did not do so. Going back to the old method, will revisit the override/flags later on. (Bitbake rev: 2f7c498abcf675e5b8de197d8056a0581670c2bd) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-07-02For ${@}, run str() on the resultChris Larson
(Bitbake rev: cf1a51eab11f645c95e5fa278221f78c129b9dd1) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-07-02Kill unnecessary usages of the types moduleChris Larson
types.IntType -> int types.StringType -> basestring ... Also moves our ImmutableTypes tuple into our own namespace. (Bitbake rev: 83674a3a5564ecb1f9d2c9b2d5b1eeb3c31272ab) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-07-02Apply some 2to3 transforms that don't cause issues in 2.6Chris Larson
(Bitbake rev: d39ab776e7ceaefc8361150151cf0892dcb70d9c) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-07-02Exception handling fixupsChris Larson
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-07-02Import fixupsChris Larson
(Bitbake rev: 4fa052f426e3205ebace713eaa22deddc0420e8a) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-07-02Formatting cleanupsChris Larson
(Bitbake rev: 2caf134b43a44dad30af4fbe33033b3c58deee57) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-07-02Move update_data into the DataSmart class as a finalize() methodChris Larson
(Bitbake rev: ff801397785567cb84b3615de86bff764d65decf) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-07-02Consolidate the exec/eval bits, switch anonfunc to better_exec, etcChris Larson
The methodpool, ${@} expansions, anonymous python functions, event handlers now all run with the same global context, ensuring a consistent environment for them. Added a bb.utils.better_eval function which does an eval() with the same globals as better_exec. (Bitbake rev: 424d7e267b009cc19b8503eadab782736d9597d0) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-07-02Pass an empty locals dict in expansion, since list comprehensions apparently ↵Chris Larson
leak their '_' helper variable (Bitbake rev: 97da0cebbaf4dd1b46e58bd2e80cab6c007ae7c9) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-07-02Two minor changes to the way python snippet expansion happensChris Larson
- Use a single dictionary for the context, both global & local, since for some reason it chokes wanting a global "d" rather than a local in the metadata. - First compile the string into a code object before running eval, so we can include the variable name in an evaluation error. (Bitbake rev: 49534d928a37e0804ca84eed186cd22363023b2e) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-03-25data_smart: be explicit about what we make available to eval'd python codeChris Larson
(Bitbake rev: 6a73dda60f50e9b3e5513795d1ec7207d1446de2) Signed-off-by: Chris Larson <chris_larson@mentor.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2010-03-22Avoid unnecessary calls to keys() when iterating over dictionaries.Chris Larson
dict objects provide an __iter__ method for the iteration which gives you the keys, so calling keys directly is unnecessary, and isn't really a best practice. The only time you really need to call the keys is if there's a danger of the dict changing out from underneith you, either due to external forces or due to modification of the iterable in the loop. Iterations over os.environ are apparently subject to such changes, so they must continue to use keys(). As an aside, also switches a couple spots to using sorted() rather than creating a temporary list with keys() and sorting that. (Bitbake rev: 5b6ccb16c6e71e23dac6920cd2df994d67c2587b) Signed-off-by: Chris Larson <clarson@mvista.com> Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2009-12-16bitbake/data_smart.py: Fix error where update-rc.d would not get added to ↵Richard Purdie
the dependency tree If there was a variable such as: X_${Y}_append = "Z" The "Z" would be lost if X_${Y} was unset. This was due to a bug in the renameVar function used by expandKeys(). Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
2009-05-12bitbake: Update to work without warnings with python 2.6Richard Purdie
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
generated by cgit 1.2.3-korg (git 2.39.0) at 2024-05-14 07:41:07 +0000