| Age | Commit message (Collapse) | Author |
|---|
|
Add some tests to verify that the new tinfoil API is operating
correctly.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
* Check that man .in file actually gets modified, since sed -i doesn't
fail if it it doesn't
* Use a variable for man file path
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
If we don't catch this then attempting to run devtool in non-memres mode
when bitbake is already running will produce a traceback instead of just
an error message.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Using the setVariable commands here followed by buildFile will result in
"basehash mismatch" errors, and that's expected since we are deviating
*at runtime* from what was previously seen by changing these variable
values. Set BB_HASH_IGNORE_MISMATCH to turn off the errors.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
This was intended to be used with tinfoil, but tinfoil now has its own
parse_recipe() method to do this which works properly in the memres
case.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Extracting the source for a recipe (as used by devtool's extract, modify
and upgrade subcommands) requires us to run do_fetch, do_unpack,
do_patch and any tasks that the recipe has inserted inbetween, and do so
with a modified datastore primarily so that we can redirect WORKDIR and
STAMPS_DIR in order to have the files written out to a place of our
choosing and avoid stamping the tasks as having executed in a real build
context respectively. However, this all gets much more difficult when in
memres mode since we can't call internal functions such as
bb.build.exec_func() directly - instead we need to execute the tasks on
the server. To do this we use the buildFile command which already exists
for the purpose of supporting bitbake -b, and setVariable commands to
set up the appropriate datastore.
(I did look at passing the modified datastore to the buildFile command
instead of using setVar() on the main datastore, however its use of
databuilder makes that very difficult, and we'd also need a different
method of getting the changes in the datastore over to the worker as
well.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
The autotools code imports oe.package; we weren't experiencing a problem
with this probably due to OE itself adding that path previously.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Move patch_path(), src_patches() and should_apply() to oe.patch, making
them easier to call from elsewhere (particularly across the
UI/server boundary).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
If PATCHTOOL is "git", and PATCH_COMMIT_FUNCTIONS is set to "1", for
additional tasks between do_unpack and do_patch, make a git commit. This
logic was previously implemented in devtool itself, but it makes more
sense for it to be implemented in the patch class since that's where the
rest of the logic is for this (or in lib/oe/patch.py). It also makes
it possible for this to work with tinfoil2.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
With the changes to the code for extracting source for a recipe, we are
properly executing the tasks for a recipe, which means their stamps (and
therefore signatures) are important. When running devtool extract on
the lsof recipe I noticed that do_fetch and do_unpack were executing a
second time when we called for do_patch, and this turned out to be
because LIC_FILES_CHKSUM in that recipe contains an entry which
is an absolute path (has ${S} at the start). Normally this wouldn't be
an issue since S is under TMPDIR and thus the existing code would ignore
it, however devtool's extraction code extracts to a temporary directory
which is not under TMPDIR; the result was the path to this file was not
being ignored and the second time around when the license file had been
extracted it was incorporated into the signature. We don't want this, so
explicitly exclude S as well as B and WORKDIR for good measure.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Use Tinfoil.parse_recipe_file() and Tinfoil.parse_recipe() instead of
the recipeutils equivalents, and replace any local duplicate
implementations. This not only tidies up the code but also allows these
calls to work in memres mode.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Use tinfoil.parse_recipe() in order to allow oe-selftest to be used in
memres mode.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
We don't need to see the parsing/cache loading message in the
oe-selftest output, so use the newly added quiet option to disable it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
We now have a function in cooker itself that can do this lookup;
additionally, the rewritten tinfoil's cooker adapter has its own
implementation that can work remotely, so if we use it then this
function can work in that scenario as well.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
On terminating the connection to the server, we were disabling SIGINT -
and this is executed on the UI side. I'm not sure whether the intention
here was to undo the SIGINT disabling we did in the server, and it was
just a mistake that it disabled rather than restored and it's run on the
wrong side, or whether we wanted to stop the user from breaking out of
the shutdown code - the commit message provides no clues either way.
Regardless, we do not want to permanently disable Ctrl+C here - it's
legitimate to terminate the connection to the server and then
re-establish it within the same process; at least currently, devtool
modify by virtue of using tinfoil in two separate parts of the code does
this, and the result of this disabling is that during the second tinfoil
usage we can potentially be parsing all recipes without the ability to
easily interrupt the process.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
If you run the setVariable command to set variables then you end up
causing the basehash to not match the previously computed values, which
triggers error messages. These mismatches are expected, so add a means
of disabling them.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Allow the client to set variables with the setVariable command and have
those changes take effect when running tasks. This is accomplished by
collecting changes made by setVariable separately and pass these to the
worker so it can be applied on top of the datastore it creates.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
The COW object used within VariableHistory can't be serialised itself,
but we can convert it to a dict when serializing and then back when
deserialising. This finally allows DataSmart objects to be serialized.
NOTE: "serialisation" here means pickling, not over XMLRPC or any other
transport.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
If we want to use this function/command internally, we don't want this
warning shown.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
If you're expanding a value that refers to the value of a variable in
python code, we need to ensure that the datastore that gets used to get
the value of that variable is the client-side datastore and not just the
part of it that's on the server side. For example, suppose you are in
client code doing the following:
d.setVar('HELLO', 'there')
result = d.expand('${@d.getVar("HELLO", True)}')
result should be "there" but if the client part wasn't taken into
account, it would be whatever value HELLO had in the server portion of
the datastore (if any).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
It's not really practical for us to parse recipes on the client side, we
need to do it on the server because that's where we have the full python
environment (including any "pure" python functions defined in classes).
Thus, add some functions to tinfoil do this including a few shortcut
functions.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
For the purposes of server-side parsing and expansion allowing for
client-side use of the datastore, we need a means of sending a datastore
from the client back to the server, where the datastore probably
consists of a remote (server-side) original plus some client-side
modifications. To do this we need to take care of a couple of things:
1) xmlrpc can't handle nested dicts, so if you enable memres and simply
try passing a serialised datastore then things break. Instead of
serialising the entire datastore, just take the naive option of
transferring the internal dict alone (as a list of tuples) for now.
2) Change the TinfoilDataStoreConnector object into simply the handle
(number) when transmitting; it gets substituted with the real
datastore when the server receives it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Rewrite tinfoil as a wrapper around the UI, instead of the earlier
approach of starting up just enough of cooker to do what we want. This
has several advantages:
* It now works when bitbake is memory-resident instead of failing with
"ERROR: Only one copy of bitbake should be run against a build
directory".
* We can now connect an actual UI, thus you get things like the recipe
parsing / cache loading progress bar and parse error handling for free
* We can now handle events generated by the server if we wish to do so
* We can potentially extend this to do more stuff, e.g. actually running
build operations - this needs to be made more practical before we can
use it though (since you effectively have to become the UI yourself
for this at the moment.)
The downside is that tinfoil no longer has direct access to cooker, the
global datastore, or the cache. To mitigate this I have extended
data_smart to provide remote access capability for the datastore, and
created "fake" cooker and cooker.recipecache / cooker.collection adapter
objects in order to avoid breaking too many tinfoil-using scripts that
might be out there (we've never officially documented tinfoil or
BitBake's internal code, but we can still make accommodations where
practical). I've at least gone far enough to support all of the
utilities that use tinfoil in OE-Core with some changes, but I know
there are scripts such as Chris Larson's "bb" out there that do make
other calls into BitBake code that I'm not currently providing access to
through the adapters.
Part of the fix for [YOCTO #5470].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
In memory resident mode we don't really want to actually shut down since
it's only the client going away.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
This allows you to maintain a local reference to a remote datastore. The
actual implementation of the remote connection is delegated to a
connector object that the caller must define and supply. There is
support for getting variable values and expanding python references
(i.e. ${@...} remotely, however setting variables remotely is not
supported - any variable setting is done locally as if the datastore
were a copy (which it kind of is).
Loosely based on an earlier prototype implementation by Qing He.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
If you send back a string here you get "TypeError: 'str' does not
support the buffer interface" errors in bitbake-cookerdaemon.log and
"IncompleteRead(0 bytes read, 22 more expected)" errors on the client
side.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
If we're in observe-only mode then we cannot run commands that would
affect the server's state, including getSetVariable, so prevent that
from being called in observe-only mode.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
Allow you to specify -q / --quiet more than once to reduce the messages
even further. It will now operate as follows:
Level Option Result
----- ------ ----------------------------------------
0 Print usual output
1 -q Only show progress and warnings or above
2 -qq Only show warnings or above
3+ -qqq Only show errors
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
There is no "datasmart" member, only dataroot. This dates back to the
original implementation of variable history support - it's surprising we
haven't noticed the issue until now, but I guess it's rare to change a
copy of a datastore in a manner which using the old reference would
cause an issue.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
BBVERSIONS is moderately horrible and it doesn't appear to be actually used by
anyone, so remove it to simplify the finalise codepaths.
(Bitbake rev: 0bb188f01e396052b127e170a25246d79a6d6741)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
For shell constructs like
echo hello & wait $!
the process_tokens() method ended up with a situation where "token"
in the "name, value = token" assignment was a list of tuples
and not the expected tuple, causing the assignment to fail.
There were already two for loops (one in _parse_shell(), one in
process_tokens()) which iterated over token lists. Apparently the
actual nesting can also be deeper.
Now there is just one such loop in process_token_list() which calls
itself recursively when it detects that a list entry is another list.
As a side effect (improvement?!) of the loop removal in
_parse_shell(), the local function definitions in process_tokens() get
executed less often.
Fixes: [YOCTO #10668]
(Bitbake rev: d18a74de9ac75ba32f84c40620ca9d47c1ef96a3)
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
[YOCTO #10508]
(Bitbake rev: ddd3bc2d64d7240ecb6b6e4a1ae29b1faef6cc22)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(Bitbake rev: 5accd6c4d1dcdf6609b4ed25c2b5e4faaf7f0909)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This function is simplified by not trying to handle replacing the regex
and just compiling and using it for matching.
- Fix typo in logger output with undefined variable
- Fix pyflake errors
(Bitbake rev: ea298ece8d678889cd5bcde46e00545e9a73edb9)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Simplify the layer event information to layer version object in toaster
function. Previously this attempted many different methods of trying to
obtain the correct layer from toaster by manipulating the data from the
event or the data from the known layers to try and match them together.
We speed up and simplify this process by making better use of django's
orm methods and by working down the most likely matching methods in order
of accuracy.
[YOCTO #10220]
(Bitbake rev: 6935cc06974ea94c9971ede89b6e8f0eae9c195b)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
function
Move the custom image file creation (i.e. create the layer file
structure, conf and recipe file) into it's own function and remove the
creation of the BRLayer as this is done at schedule_build just like all
the other layers.
Fix a bug where the toaster-custom-images layer was always being appened
to the layer list if the directory exists.
(Bitbake rev: 15a42b36c01fccd79e5aa0788dea5640b253982b)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The schedule_build function on the project object is where the BRLayers
are created for the build. Instead of creating the BRLayer for the
CustomImageRecipe in the localhostbbcontroller create it here so that
all that mechanism is in one place.
Also fix a number of pyflake errors.
(Bitbake rev: f8d3ea784937b6e416d3e5a4feb1283c478e4caa)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(Bitbake rev: 69f33397083f54f977fa0cd4dd621731f32fd034)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This fixes the unidentified layers issue by making the
toaster-custom-images layer a local layer. By doing this we also fix the
git assumptions made for the local layers which stop recipes and other
meta data being associated with them. This also removed some of the
special casing previously needed when we didn't have the concept of a
local (non git) layer.
Also rename created flag var to a have a different var for each returned
value so that the same value isn't used multiple times.
[YOCTO #10220]
(Bitbake rev: ba5332d4960d7f4f79aef63136796e2fa67284e3)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
toaster-bblayers.conf
Instead of updating conf/bblayers, here we update toaster-bblayers.conf
file. So extra effort to update bblayers.conf can be removed safely.
(Bitbake rev: f3e99d820f3798869a2a1d1604709c1c324dbbab)
Signed-off-by: Sujith H <sujith.h@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When passed variable CUSTOM_BUILD_INIT_SCRIPT to toaster
setting, it would be nice to use it. Else toaster
can use oe-init script. This gives an oppurtunity to
use customized build init scritps.
(Bitbake rev: 9d168bb5f05453bdb7156793eea25da0a3119b4d)
Signed-off-by: Sujith H <sujith.h@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There is a missing close square bracket.
[YOCTO #10631]
(Bitbake rev: f3da9f85e6036e6f43377172cbcfe701e0efca7f)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Limit the recursion (to say 20 levels) when processing layer dependencies
so that circular dependecies do not cause infinite decent and an
out-of-memory failure. The duplicate found layers are already immediately
filtered in the code.
[YOCTO #10630]
(Bitbake rev: e9efef0bdb8068984c3013b87aac9e872ffb38ae)
Signed-off-by: David Reyna <David.Reyna@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Consume the click event on the download recipe link if it's disabled. To
prevent the link from sending user to an error page.
See http://getbootstrap.com/css/#forms-disabled-fieldsets and a link
caveat.
[YOCTO #10151]
(Bitbake rev: bc8401e78cea140349bded228d38f72f628b3980)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add an example settings that can be used for the basis of the production
instance of Toaster.
[YOCTO #10581]
(Bitbake rev: 1a7c356491b88c8decced39fb2039ef90065f2d2)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add some simple sanity tests for the management commands that we use for
Toaster.
Can be executed with ./manage.py test tests.commands
For faster execution use the test settings and keepdb flag:
DJANGO_SETTINGS_MODULE=toastermain.settings_test ./manage.py test
tests.commands --keepdb
(Bitbake rev: 161ea71519e7f70d4aadaafc9c3294a12612f0cb)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Write the pid file out in the start up of this management command. This
ensures this has happened instead of relying on the shell command having
been run which may or may not be the case. This also makes it simpler for
testing.
Couple of clean ups of runbuilds as identified by pyflake
(Bitbake rev: 999e980ee1a58d16f33ef6c0e41aecdcd0206f39)
Signed-off-by: Michael Wood <michael.g.wood@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
getVarFlag() now defaults to expanding by default, thus remove the
True option from getVarFlag() calls with a regex search and
replace.
Search made with the following regex:
getVarFlag ?\(( ?[^,()]*, ?[^,()]*), True\)
(Bitbake rev: c19baa8c19ea8ab9b9b64fd30298d8764c6fd2cd)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.
Search made with the following regex: getVar ?\(( ?[^,()]*), True\)
(Bitbake rev: 3b45c479de8640f92dd1d9f147b02e1eecfaadc8)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(From OE-Core rev: 11063a01d4511b2688ea7ba2d7359e4e07328c66)
Signed-off-by: Edwin Plauchu <edwin.plauchu.camacho@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Paul Eggleton
Ross Burton
Patrick Ohly
Markus Lehtonen
Michael Wood
Sujith H
Reyna, David
Joshua Lock
Edwin Plauchu