summaryrefslogtreecommitdiffstats
AgeCommit message (Collapse)Author
2016-12-13lib/bb/build: enable access to logger within taskspaule/bb-logger-accessPaul Eggleton
In certain circumstances it can be useful to get access to BitBake's logger within a task; the main example is in OpenEmbedded's image construction code where we want to be able to check the log file for errors and warnings, but we don't want to see any errors or warnings that were emitted through the logger; so we need a way to exclude those. In order to do this, pass the logger object into the task via a BB_TASK_LOGGER variable, and add a logging handler class to bb.utils that can be added to it in order to keep a list of warnings/errors that have been emitted. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
2016-12-12toaster: views Remove old code that converts template context to JSONMichael Wood
Remove the template context to JSON decorator function as this is deprecated by having a proper REST API. Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-12toaster: js test Update js tests to use correct url for layer RESTMichael Wood
Update js tests so that we use the new Layer REST API. Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-12toaster: Switch front end to use Layer get REST APIMichael Wood
Switch the front end to use the proper REST API for retrieving layer information. Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-12toaster: typeaheads Layers add url to layer REST API to the layerMichael Wood
Add the url to the Layer typeahead so that this can be used later on by the front end code to look up layer details. Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-12toaster: api Add GET REST API for Layer informationMichael Wood
Add a get API for returning information about layers. Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-12toaster: importlayer Add git revision typeahead to that input fieldMichael Wood
Add the front end mechanism to load the typeahead for the git revision field on importing a layer. Also fix one indentation issue and update the js test. Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-12toaster: typeaheads Add a git revisions suggestionsMichael Wood
When we're importing a layer it's useful to suggest available git revisions of the layers. This uses git ls-remote to fetch the revisions and then filter on this. Caching is added for this typeahead to avoid having to fetch this information multiple times in a single session. [YOCTO #8429] Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-12toaster: typeaheads widgets Fix flake8 issues and remove redundant codeMichael Wood
Fix flake8 issues and remove redundant __init__ function definitions from typeaheads (likely a copy and paste error). Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-12toaster: tests Update import layer test to use new Layer add apiMichael Wood
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-12toaster: importlayer Use new layer add APIMichael Wood
Switch the importlayer.js to use the new REST API for importing a new layer. Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-12toaster: api Add layer Add apiMichael Wood
Add layer adding REST api and remove old views method. Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-12toaster: models Layer_Version get_detailspage_url optional project_idMichael Wood
Allow passing none as the project id, this is convenient for layers which belong to projects already and therefore have their own project field. Add documentation string to function Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-12toaster: models Layer fix whitespaceMichael Wood
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-12toaster: tablejs Add visual indicator for table data loadingMichael Wood
Add a visual indicator when the data is changing in the table, for instance if it's being re-ordered, searched or paginated. [YOCTO #10104] Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-08bitbake-user-manual: Added new "Line Joining" section.Scott Rifenbark
Fixes [YOCTO #10444] Added a new section about syntax called "Line Joining". The section describes how you can use the backslash character (\) to joing stuff. Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
2016-12-08bitbake-user-manual: Refreshed the 'bitbake --help' output.Scott Rifenbark
Signed-off-by: Scott Rifenbark <srifenbark@gmail.com>
2016-12-08toaster: Reference pip3 not pipRichard Purdie
Now that we use python3, we should refer to pip3, not pip. [YOCTO #10774] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-08cooker: Handle inofity queue overflows more gracefullyRichard Purdie
If many files change and the inotify queue overflows, rather than print a traceback, invalidate the caches and warn the user. [YOCTO #10676] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-08cooker: Fix world taskgraph generation issueRichard Purdie
The processing of the "do_" prefix to tasks is currently inconsistent and has resulted in "bitbake world -g" being broken as task prefixes don't get handled correctly. Make the "do_" task prefix handling consistent through various codepaths. [YOCTO #10651] Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07utils: Avoid traceback errorsRichard Purdie
Avoid errors like: ERROR: Exception handler error: 'NoneType' object has no attribute 'decode' Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07runqueue: Send BB_TASKDEPDATA for setscene tasksRichard Purdie
We now have code in OE that needs BB_TASKDEPDATA for setscene tasks. Therefore generate and send this data. In this case its a "pre collapsed" tree but that is fine for the use cases in question. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07runqueue: Add the taskhash to BB_TASKDEPDATARichard Purdie
Its useful to know the task hash in code using TASKDEPDATA so add this data to the data structure. The recipe specific sysroots in OE need this data. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07bitbake-worker: Further IO performance tweaksRichard Purdie
Looking further at the CPU loads on systems running large numbers of tasks, the following things helps performance: * Loop on waitpid until there are no processes still waiting * Using select to wait for the cooker pipe to be writable before writing avoiding pointless 100% cpu usage * Only reading from worker pipes that select highlights are readable Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07lib/bs4: Fix imports from html5lib >= 0.9999999/1.0b8Daniel Lublin
As of html5lib 0.9999999/1.0b8 (released on July 14, 2016), some modules have moved from _base to base. Handle this, while staying compatible with earlier versions. Signed-off-by: Daniel Lublin <daniel@lublin.se> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07cooker: convert type which needs to be marshalledSujith H
We assume that the value taken by variable v can be string, integer or any type which can be marshalled by xmlrpc. This change would help us to convert the non marshallable types to string. So that we don't get exception from xmlrpc. [YOCTO #10740] Signed-off-by: Sujith H <sujith.h@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07toaster: browser tests - add Selenium Docker container as driverbrian avery
Adds the ability to specify a Selenium Docker container server as a driver. This allows for repeatable tests independent of host. Currently we assume you are using the Firefox container. Instructions are located in the README in tests/browser. Signed-off-by: brian avery <brian.avery@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07depexp: fix string formatting.Ismo Puustinen
The parameters to Python string formatting need to be inside a tuple. Signed-off-by: Ismo Puustinen <ismo.puustinen@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07monitordisk: add eventPatrick Ohly
The current disk usage is interesting and may be worth logging over time as part of the build statistics. Instead of re-implementing the code and the configuration option (BB_DISKMON_DIRS), the information gathered by monitordisk.py is made available to buildstats.bbclass via a new event. This has pros and cons: - there is already a useful default configuration for "interesting" directories - no code duplication - on the other hand, users cannot configure recording separately from monitoring (probably not that important) Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07monitordisk.py: minor code and comment cleanupPatrick Ohly
There's no need to encode and decode the hash key as a single string, a tuple works just fine. Iterating over entries can be written more concisely. Entries in the stat results are integers, not floating point values. Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07runqueue.py: monitor disk space at regular time intervalsPatrick Ohly
Hooking the disk monitor into the regular heatbeat event instead of the runqueue solves two problems: - When there is just one long running task which fills up the disk, the previous approach did not notice that until after the completion of the task because _execute_runqueue() only gets called on task state changes. As a result, aborting a build did not work in this case. - When there are many short-lived tasks, disk space was getting checked very frequently. When the storage that is getting checked is on an NFS server, that can lead to noticable traffic to the server. Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07cooker process: fire heartbeat event at regular time intervalsPatrick Ohly
The intended usage is for recording current system statistics from /proc in buildstats.bbclass during a build and for improving the BB_DISKMON_DIRS implementation. All other existing hooks are less suitable because they trigger at unpredictable rates: too often can be handled by doing rate-limiting in the event handler, but not often enough (for example, when there is only one long-running task) cannot because the handler does not get called at all. The implementation of the new heartbeat event hooks into the cooker process event queue. The process already wakes up every 0.1s, which is often enough for the intentionally coarse 1s delay between heartbeats. That value was chosen to keep the overhead low while still being frequent enough for the intended usage. If necessary, BB_HEARTBEAT_EVENT can be set to a float specifying the delay in seconds between these heartbeat events. Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30ast: remove BBVERSIONS supportRoss Burton
BBVERSIONS is moderately horrible and it doesn't appear to be actually used by anyone, so remove it to simplify the finalise codepaths. Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30codeparser.py: support deeply nested tokensPatrick Ohly
For shell constructs like echo hello & wait $! the process_tokens() method ended up with a situation where "token" in the "name, value = token" assignment was a list of tuples and not the expected tuple, causing the assignment to fail. There were already two for loops (one in _parse_shell(), one in process_tokens()) which iterated over token lists. Apparently the actual nesting can also be deeper. Now there is just one such loop in process_token_list() which calls itself recursively when it detects that a list entry is another list. As a side effect (improvement?!) of the loop removal in _parse_shell(), the local function definitions in process_tokens() get executed less often. Fixes: [YOCTO #10668] Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30fetch2: obey BB_ALLOWED_NETWORKS when checking network accessMarkus Lehtonen
[YOCTO #10508] Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30toaster: buildinfohelper Clarify log message for build historyMichael Wood
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30toaster: buildinfohelper fix _get_layer_version_for_dependencyMichael Wood
This function is simplified by not trying to handle replacing the regex and just compiling and using it for matching. - Fix typo in logger output with undefined variable - Fix pyflake errors Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30toaster: buildinfohelper Simplify layer event to toaster layer functionMichael Wood
Simplify the layer event information to layer version object in toaster function. Previously this attempted many different methods of trying to obtain the correct layer from toaster by manipulating the data from the event or the data from the known layers to try and match them together. We speed up and simplify this process by making better use of django's orm methods and by working down the most likely matching methods in order of accuracy. [YOCTO #10220] Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30toaster: bldcontrol Move CustomImageRecipe file creation into own functionMichael Wood
Move the custom image file creation (i.e. create the layer file structure, conf and recipe file) into it's own function and remove the creation of the BRLayer as this is done at schedule_build just like all the other layers. Fix a bug where the toaster-custom-images layer was always being appened to the layer list if the directory exists. Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30toaster: orm models Handle CustomImageRecipe BRLayer hereMichael Wood
The schedule_build function on the project object is where the BRLayers are created for the build. Instead of creating the BRLayer for the CustomImageRecipe in the localhostbbcontroller create it here so that all that mechanism is in one place. Also fix a number of pyflake errors. Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30toaster: orm models Project class Fix pyflake errorsMichael Wood
Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30toaster: buildinfohelper toaster-custom-images layerMichael Wood
This fixes the unidentified layers issue by making the toaster-custom-images layer a local layer. By doing this we also fix the git assumptions made for the local layers which stop recipes and other meta data being associated with them. This also removed some of the special casing previously needed when we didn't have the concept of a local (non git) layer. Also rename created flag var to a have a different var for each returned value so that the same value isn't used multiple times. [YOCTO #10220] Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30toaster: localhostbecontroller write toaster layers for project to ↵Sujith H
toaster-bblayers.conf Instead of updating conf/bblayers, here we update toaster-bblayers.conf file. So extra effort to update bblayers.conf can be removed safely. Signed-off-by: Sujith H <sujith.h@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30toaster: localhostbecontroller accept custom init script for buildSujith H
When passed variable CUSTOM_BUILD_INIT_SCRIPT to toaster setting, it would be nice to use it. Else toaster can use oe-init script. This gives an oppurtunity to use customized build init scritps. Signed-off-by: Sujith H <sujith.h@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30toaster: tablejs Fix missing close square bracketReyna, David
There is a missing close square bracket. [YOCTO #10631] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30toaster: orm gen_layerdeps Protect against circular Layer dependenciesReyna, David
Limit the recursion (to say 20 levels) when processing layer dependencies so that circular dependecies do not cause infinite decent and an out-of-memory failure. The duplicate found layers are already immediately filtered in the code. [YOCTO #10630] Signed-off-by: David Reyna <David.Reyna@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30toaster: customrecipejs Consume click event on 'a' link if disabledMichael Wood
Consume the click event on the download recipe link if it's disabled. To prevent the link from sending user to an error page. See http://getbootstrap.com/css/#forms-disabled-fieldsets and a link caveat. [YOCTO #10151] Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30toaster: Add an example production settings fileMichael Wood
Add an example settings that can be used for the basis of the production instance of Toaster. [YOCTO #10581] Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30toaster: tests Add management command testsMichael Wood
Add some simple sanity tests for the management commands that we use for Toaster. Can be executed with ./manage.py test tests.commands For faster execution use the test settings and keepdb flag: DJANGO_SETTINGS_MODULE=toastermain.settings_test ./manage.py test tests.commands --keepdb Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-30toaster: runbuilds Write the pidfile in python rather than shell scriptMichael Wood
Write the pid file out in the start up of this management command. This ensures this has happened instead of relying on the shell command having been run which may or may not be the case. This also makes it simpler for testing. Couple of clean ups of runbuilds as identified by pyflake Signed-off-by: Michael Wood <michael.g.wood@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit 1.2.3-korg (git 2.39.0) at 2024-04-19 11:46:01 +0000