| Age | Commit message (Collapse) | Author |
|---|
|
configuration files
Added a new command in bitbake to save a variable in a file; added a function
in cooker which is called by this command.
Added new command in bitbake to enable/disable data tracking.
The function saveConfigurationVar from cooker.py saves a variable in the file that
is received by argument. It checks all the operations made on that variable, using the history.
If it's the first time when it does some changes on a variable,it comments the lines where
an operation is made on it, and it sets it in a line to the end of file. If it's not
the first time(it has a comment before), it replaces the line.
Made some changes in hob to save the variables from bblayers.conf and local.conf
using the bitbake command.
[YOCTO #2934]
(Bitbake rev: 55b814ccfa413d461d12956896364ab63eed70a8)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This patch adds tracking of the history of variable assignments.
The changes are predominantly localized to data_smart.py and
parse/ast.py. cooker.py and data.py are altered to display the
recorded data, and turn tracking on for the bitbake -e case.
The data.py update_data() function warns DataSmart.finalize()
to report the caller one further back up the tree.
In general, d.setVar() does what it used to do. Optionally,
arguments describing an operation may be appended; if none
are present, the operation is implicitly ignored. If it's
not ignored, it will attempt to infer missing information
(name of variable, value assigned, file and line) by examining
the traceback. This slightly elaborate process eliminates a
category of problems in which the 'var' member of the keyword
arguments dict is set, and a positional argument corresponding
to 'var' is also set. It also makes calling much simpler for
the common cases.
The resulting output gives you a pretty good picture of what
values got set, and how they got set.
RP Modifications:
a) Split from IncludeHistory to separate VariableHistory
b) Add dedicated copy function instead of deepcopy
c) Use COW for variables dict
d) Remove 'value' loginfo value and just use 'details'
e) Desensitise code for calling order (set 'op' before/after
infer_caller_details was error prone)
f) Fix bug where ?= "" wasn't shown correctly
g) Log more set operations as some variables mysteriously acquired
values previously
h) Standardise infer_caller_details to be triggered from .record()
where at all possible to reduce overhead in non-enabled cases
i) Rename variable parameter names to match inference code
j) Add VariableHistory emit() function to match IncludeHistory
k) Fix handling of appendVar, prependVar and matching flag ops
l) Use ignored=True to stop logging further events where appropriate
(Bitbake rev: f00524a3729000cbcb3317fee933ac448fae5e2d)
Signed-off-by: Peter Seebach <peter.seebach@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
bitbake -e
This code adds inclusion history to bitbake -e output, showing
which files were included, in what order. This doesn't completely
resolve timing questions, because it doesn't show you which lines
of a file were processed before or after a given include, but it
does let you figure out what the path was by which a particular
file ended up in your build at all.
How it works: data_smart acquires a .history member, which is an
IncludeHistory; this represents the inclusion of a file and all its
inclusions, recursively. It provides methods for including files,
for finishing inclusion (done as an __exit__), and for
dumping the whole tree.
The parser is modified to run includes inside a with() to push
and pop the include filename.
RP Modifications:
a) Split Include and Variable tracking
b) Replace deepcopy usage with dedicated copy function
c) Simplify some variable and usage
(Bitbake rev: b2dda721262da8abb7dc32d019e18fbc32ed8860)
Signed-off-by: Peter Seebach <peter.seebach@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If SkipParse is raised from something which isn't anonymous python, it wasn't
being handled correctly. This improves the handling for example from within inline
python.
(Bitbake rev: 7467d7d66b24cc8f43ab168e65895e7c4aee6092)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
An issue was uncovered where changing:
IMAGE_INSTALL_append = "X"
to
IMAGE_INSTALL_append = "X Y"
in local.conf would not get noticed by bitbake. The issue is that
the configuration hash doesn't account for overrides or key expansion.
This patch improves get_hash to account for these. This means the hash
does account for changes like the above.
[YOCTO #3503]
(Bitbake rev: 86bf1f5603e8f98019544e45f51bd0db9a48112a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Instead of logging the function/variable separately as a NOTE when
failing to expand, re-raise ExpansionError with more contextual
information. This means that the full details are reported in Hob as
well as actually reporting the original error message in any UI where
we previously did not. For example, we used to get this with tab/space
indentation issues in a python function:
NOTE: Error expanding variable populate_packages
ERROR: Unable to parse /path/to/recipename.bb
Now, we will get this:
ERROR: ExpansionError during parsing /path/to/recipename.bb: Failure
expanding variable populate_packages: IndentationError: unindent does
not match any outer indentation level (<string>, line 4)
Fixes [YOCTO #3162].
(Bitbake rev: ce5c7a95a359cdaecab7c4a519ad4f9df029da82)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(Bitbake rev: 9f631e29a2eebb96a8291839dd8b39aa9126a10e)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(Bitbake rev: 684cf09aed09ec82c8afb99895f92d73cd0519df)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If this regular expression is unanchored, it would accept strings like:
do_install_append1
do_install_appendsomelongstring
and treat them like they were do_install_append. Clearly this isn't desirable.
Only one instance of this type of issue was found in OE-Core and has been fixed
so correcting the regexp should be safe to do.
(Bitbake rev: 23bd5300b4a99218a15f4f6b0ab4091d63a602a5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
prepend operators
Variables which used multiple overrides and the append/prepend operators were
not functioning correctly. This change fixes that.
This fixes the testcase:
OVERRIDES = "linux:x86"
TESTVAR = "original"
TESTVAR_append_x86 = " x86"
TESTVAR_append_x86_linux = " x86+linux"
TESTVAR_append_linux_x86 = " linux+x86"
[YOCTO #2672]
(Bitbake rev: dc35a2e506e15fb7ddbf74c3b3280e9e83ab33bb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If the variable name is not specified then don't confuse the error message
by starting off with "Failure expanding variable None...".
(Bitbake rev: 9cb16f3c73751e7cf6d495586a6193f06eb97b1f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
For config hash, we put the keys in structure of "set()", which is not
order sensitive. Therefore when calculating the md5 value for config
hash, we need to identify the order of the keys.
(Bitbake rev: 0f1b142a3f6b8125bf023c2e5ec269618869abf7)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(From OE-Core rev: 3b57de68e70e77dbc03c0616a83a29a2e99e40b4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Adopt the BB_HASHCONFIG_WHITELIST as a exclusion list for variables that are
not needed in cache hash calculation.
(Bitbake rev: ae8cf138b5eb8f1f28a7143b8d67ad06cbe43061)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
CC: Christopher Larson <kergoth@gmail.com>
CC: Martin Jansa <martin.jansa@gmail.com>
CC: Andreas Oberritter <obi@opendreambox.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Previously we use the file time stamp to judge if a cache is valid.
Here this commit introduce a new method, which calculates the total
hash value for a certain configuration's key/value paris, and tag
it into cache filename, for example, bb_cache.dat.xxxyyyzzz.
This mechanism also ensures the cache's correctness if user
dynamically setting variables from some frontend GUI, like HOB.
(Bitbake rev: 1c1df03a6c4717bfd5faab144c4f8bbfcbae0b57)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This patch adds appendVar and prependVar functions to the data store
meaning python code would no longer have to do the getVar, append and
the setVar dance that much of the current python code does.
It also adds corresponding variants for flags.
Currently there is no spacing added by these functions. That could be
added as a parameter if desired.
If these functions turn out to be hotspots in the code, there are tricks
that could potentially be used to increase the speed of these specific
operations within the datastore.
(Bitbake rev: 4a4046268f84b85559eea2c4b6a6004ad8cccb77)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If the vardeps flag is not None, we now silence the warnings about
non-literal usage for that variable.
(Bitbake rev: e724b9f417d1baf898f5afc6376c73c1a2ad8db9)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
- If a name is passed to the parser, prepend the messages with "while
parsing <name>:". This gives a bit more context.
- Tweak the warning messages slightly (they had to be altered anyway to
inject the variable being parsed).
Before:
DEBUG: Warning: in call to 'bb.data.getVar': argument ''%s' % var' is \
not a literal
After:
DEBUG: while parsing emit_pkgdata, in call of bb.data.getVar, argument \
''%s' % var' is not a string literal
(Bitbake rev: 1060193ae4d54e667735dbff5d1d2be49a3f95c9)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently if passing expand=True to getVar() function, it will pass the
handling to getVarFlag(), which doesn't get any benefit from the expand
cache.
Call the expand() function separately in getVar() to make use of the
expand cache, which can decrease the parsing time by 40%.
(from current 49s to 27s)
(Bitbake rev: 6555a77c199f41bf35460138764e03e30c56d29f)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
As the code stands, setting a variable with ??= could result in a ?=
variable not overriding it. This patch fixes the issue by allowing
the ast to make lookups that ignore any ??= set variables.
(Bitbake rev: 32fee2e650dfdd3aa9a7572dad1251e0c24ca34b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When we delete a variable we no longer expect it to override other
variables.
To do this we remove it from the list of active overrides at deletion
time. It turns out we already had to do this at override expansion time
so this cleans up the code to be more consistent as an added bonus.
(Bitbake rev: d924ff9ede57c3dea6e1c738ba3633f18d460b14)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If you d.delVar(), you expect the variable to be gone. Even empty
variables continue to exist in the datastore and are still user visible
unfortunately. The COW siutation means you can't just remove it
since it might unmask a variable from an inner copy.
This patch therefore stops empty variables from appearing in key lists
exposed to the external world making empty variables an internal
implementation detail only.
(Bitbake rev: 2b5548c591d4cfde9238d2cc0959c42cfc08f09c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Since we're going to creat the seen set() anyway, we might as well use
it directly. If we don't do this, we see thousands of function calls
with associated overhead on profiles.
(Bitbake rev: 9d43e3279895639ee4899df635f2546c7ee13737)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
from the datastore
Currently if you do:
OVERRIDES = "z"
DEPENDS_prepend = "a "
DEPENDS = "b"
DEPENDS_z = "c"
d.update_data()
d.getVar("DEPENDS")
gives "a c"
d.update_data()
d.getVar("DEPENDS")
then gives "c"
This patch changes the behaviour such that at the time bitbake expands the DEPENDS_z
override, it removes "DEPENDS_z" from the data store. In the above example this would
mean that it wouldn't matter how often you call d.update_data(), you'd always get
"a c" back.
See the bitbake-devel mailing list for further discussion and analysis of the
potential impact of this change.
(Bitbake rev: 899d45b90061eb3cf3e71029072eee42cd80930c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(Bitbake rev: 67434496108efc3aba9cb1e3640bc712658b1408)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(Bitbake rev: 967cd1aa2c59f15d805862bd9935f507c635c762)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If expanding a variable triggers an exception the caller currently has no
way to supress the error message or otherwise handle the siutation. An
example of where this is a problem is "bitbake -e" showing tracebacks and
errors for variables like SRCPV in OE/Poky.
Secondly in a chained expansion fails, log mesages are recorded for
every step of the expansion, not just the innermost error which is
where the real failure occured.
To fix this we introduce a new exception ExpansionError which callers
can handle as appropriate.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently, if a variable has been set with ??= and the code looks it up
before the data finalisation phase, no value is found. This is causes
serious problems for anonymous python functions which manipulate data, or
for the fetcher revision handling code where revisions can be set with
??=.
There is also a significant performance implication for processing lazy
assignment in finalise.
Moving the check for a default value into getVarFlag addresses both
the timing issue and the performace. This change gives a 7% real time
performance improvement to parsing the Poky metadata. The cost of the
check at this point is minimal since we have all the data flags available.
This should also fix Yocto bug 752.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
bitbake uptream
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
(Bitbake rev: 40778a6e9e82c7ea4673a74fc19574430fa63e8d)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
(Bitbake rev: 53740977521bc81ffa37adfa7bbeb8f2a80ea165)
build: write logfiles per task, not per function
Based on d14f9bf6 from poky, reworked for master and other cleanup.
(Bitbake rev: beadff2eca1eb95f0411115dd72ddb4c3c44c604)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
Example:
FOO = "bar"
BAR = "${@FOO + '/baz'}"
${BAR} == "bar/baz"
(Bitbake rev: 606fa1fd97cbd47a6a7ebdc7a2e6aa93a8f65cf5)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
Ensure it raises KeyError for a missing key, this is required to use this as a
mapping in various places, e.g. as locals in an eval.
(Bitbake rev: 8d661ce0c303e8d69f17c1d095545d5ed086d1d5)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
We use a custom Logger subclass for our loggers
This logger provides:
- 'debug' method which accepts a debug level
- 'plain' method which bypasses log formatting
- 'verbose' method which is more detail than info, but less than debug
(Bitbake rev: 3b2c1fe5ca56daebb24073a9dd45723d3efd2a8d)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
Provide __len__, __iter__, and the getitem/setitem/delitem methods, and its
mixed in versions of keys(), values(), items(), etc will automatically behave,
making the DataSmart act more like a real mapping.
(Bitbake rev: 89b5351c656d263b0ce513cee043bc046d20a01e)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
Add a parameter to getVarFlag() to auto-expand the value of the flag. This
makes getVarFlag() more consistent with getVar(), and allows expansion of
vardeps and vardepsexclude (which has been done in this commit).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
Where a variable name consisted of an append/prepend combined with an override
and there was also an append/prepend to the variable, the override could be lost
if the override was not in OVERRIDES.
For example:
FOO = "A"
FOO_append = "B"
FOO_append_virtclass-native = "C"
could result in "AB" even though virtclass-native was in OVERRIDES.
With this patch applied, the result is "ABC" as would be expected.
The problem was the deletion of the _append/_prepend flag was happening
if *any* append/prepend was procesed, the result should really be that
it should contain any unprocessed append/prepend.
Kevin Tian deserves credit for looking into this and working out the
problem here.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
This commit is derived from Chris Larson's checksum work, turned into a
standalone piece of code for parsing python and shell functions.
The deindent code has been replaced with code to work around indentation
for speed. The original NodeVisitor in the ast was replaced with a faster
class walk call.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
references (including those from python code)
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
This reverts commit 05462fa7908fc22988b3dc9d376798d0a46ccb5a.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
(Bitbake rev: 05462fa7908fc22988b3dc9d376798d0a46ccb5a)
Signed-off-by: Imran Mehmood <imran_mehmood@mentor.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
(Bitbake rev: b017acd39b811a00305002a8044e7d02e79f41d7)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
The root cause is, I was testing the use of renameVar() from finalize, in
order to get flags copied over when an override is applied, but renameVar
removes the original, whereas the old code did not do so. Going back to the
old method, will revisit the override/flags later on.
(Bitbake rev: 2f7c498abcf675e5b8de197d8056a0581670c2bd)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
(Bitbake rev: cf1a51eab11f645c95e5fa278221f78c129b9dd1)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
types.IntType -> int
types.StringType -> basestring
...
Also moves our ImmutableTypes tuple into our own namespace.
(Bitbake rev: 83674a3a5564ecb1f9d2c9b2d5b1eeb3c31272ab)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
(Bitbake rev: d39ab776e7ceaefc8361150151cf0892dcb70d9c)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
|
(Bitbake rev: 4fa052f426e3205ebace713eaa22deddc0420e8a)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
|
Cristiana Voicu
Peter Seebach
Richard Purdie
Paul Eggleton
Ross Burton
Dongxiao Xu
Christopher Larson
Richard Purdie