summaryrefslogtreecommitdiffstats
path: root/meta/classes/buildstats.bbclass
AgeCommit message (Collapse)Author
2021-07-10buildstats.bbclass: log host data on failure to task specific fileSakib Sajal
host data, for both interval and failure, was previously logged into the same file which was difficult to read as the files file were usually large. host data is now logged into separate files, for each type of logging (failure and interval) and also for each failed task making it easier to read/parse. (From OE-Core rev: 1a0fb3c0794f4e66086e567a297b4d9379c6b8f3) Signed-off-by: Sakib Sajal <sakib.sajal@windriver.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-04-27buildstats.bbclass: collect data in the same file.Sakib Sajal
Previously "at interval" and "on failure" logs were collected in separate files. Collect both types of logging in the same file for better analysis. Introduced new variable which allows different set of commands to be run by the different logging, interval or failure. The variables are BB_LOG_HOST_STAT_CMDS_INTERVAL and BB_LOG_HOST_STAT_CMDS_FAILURE respecteviely. (From OE-Core rev: 4fbf422351668f755a14811ac39161c889087e81) Signed-off-by: Sakib Sajal <sakib.sajal@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-03-11buildstats.bbclass: improve timeout handlingSakib Sajal
The subprocess that runs the commands periodically times out prematurely even when there is sufficient time available within the given interval for the commands to run. This change improves timeout handling and give all the commands equal time to run. (From OE-Core rev: 74e5c438025ed42cfacb993261dc664c3b7959a8) Signed-off-by: Sakib Sajal <sakib.sajal@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2021-01-17buildstats.bbclass: add functionality to collect build system statsSakib Sajal
There are a number of timeout and hang defects where it would be useful to collect statistics about what is running on a build host when that condition occurs. This adds functionality to collect build system stats on a regular interval and/or on task failure. Both features are disabled by default. To enable logging on a regular interval, set: BB_HEARTBEAT_EVENT = "<interval>" BB_LOG_HOST_STAT_ON_INTERVAL = <boolean> Logs are stored in ${BUILDSTATS_BASE}/<build_name>/host_stats To enable logging on a task failure, set: BB_LOG_HOST_STAT_ON_FAILURE = "<boolean>" Logs are stored in ${BUILDSTATS_BASE}/<build_name>/build_stats The list of commands, along with the desired options, need to be specified in the BB_LOG_HOST_STAT_CMDS variable delimited by ; as such: BB_LOG_HOST_STAT_CMDS = "command1 ; command2 ;... ;" (From OE-Core rev: edb7098e9e0a8978568a45057c1c3ad2c6cacd67) Signed-off-by: Sakib Sajal <sakib.sajal@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-04buildstats.bbclass: Do not recalculate build start timeJunling Zheng
The build start time is recorded in __timedata_build, which we should use rather than recalculating it. (From OE-Core rev: b723e9a4de67f33f61f2e7fe36ea451a46840b3e) Signed-off-by: Junling Zheng <zhengjunling@huawei.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2020-06-04buildstats.bbclass: Remove useless variablesJunling Zheng
Remove useless variables bn and bsdir in write_task_data(). (From OE-Core rev: de26d6ce9d2154b5bb51491d9f4bd590cb5a0fc7) Signed-off-by: Junling Zheng <zhengjunling@huawei.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-12-15buildstats: Replace 'is' with '==' in python condition check expressionKhem Raj
python 3.8 has started to throw this as a warning but usage is wrong see [1], it seems to be working by accident and not design. Fixes SyntaxWarning: "is" with a literal. Did you mean "=="? [1]: https://bugs.python.org/issue34850 (From OE-Core rev: c3cb6bc21f3f84e2ed103b6438516ab85222faa9) Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-01buildstats.bbclass: Avoid index exception in /proc/PID/io parsingPavel Modilaynen
There is some probability (depends on system load) to get empty or line containing "0" as the last line while reading /proc/PID/io. Avoid build failure by checking if line contains separator ":" before split. (From OE-Core rev: b26feaf51af55f17fad79dbd53dd3ec0a37c38ff) Signed-off-by: Pavel Modilaynen <pavelmn@axis.com> Signed-off-by: Daniel Lublin <daniel@lublin.se> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-09classes/buildstats: don't expand variable pointing to SystemStatsPaul Eggleton
We're placing an object into the datastore - it's very definitely not something we want to be expanding. (From OE-Core rev: adfee41d2b1edbd2abd09f9101c359919e21e5d6) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-16meta: remove True option to getVar callsJoshua Lock
getVar() now defaults to expanding by default, thus remove the True option from getVar() calls with a regex search and replace. Search made with the following regex: getVar ?\(( ?[^,()]*), True\) (From OE-Core rev: 7c552996597faaee2fbee185b250c0ee30ea3b5f) Signed-off-by: Joshua Lock <joshua.g.lock@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07buildstats: record disk space usagePatrick Ohly
Hooks into the new monitordisk.py event and records the used space for each volume. That is probably the only relevant value when it comes to visualizing the build and recording more would only increase disk usage. (From OE-Core rev: 21a5b569370f47cc02291e1d8b76fe43faa04ea6) Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07buildstats: add system state samplingPatrick Ohly
/proc/[diskstats|meminfo|stat] get sampled and written to the same proc_<filename>.log files as during normal bootchat logging. This will allow rendering the CPU, disk and memory usage charts. Right now sampling happens once a second, triggered by the heartbeat event.That produces quite a bit of data for long builds, which will be addressed in a separate commit by storing the data in a more compact form. (From OE-Core rev: 6f4e8180b5b4857eaf6caf410fd3a4a41ed85930) Signed-off-by: Patrick Ohly <patrick.ohly@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-11-23buildstats: Place 'Elapsed Time' stat into a single lineLeonardo Sandoval
All lines except one (the one containing the 'Elapsed Time') follows the format 'stat: value'. Fix that so post parsing the stats is simpler. (From OE-Core rev: 47dfd172754440a7ab28eb1b59d9bd484cdad22a) Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-25buildstats: check IMAGE_ROOTFS exists before checking its sizeMikko Ylinen
After 0d6b7276003f1afabc6de683f663540327d52bdc, the exceptions are correctly checked if the rootfs size check fails. In case of a failure a build error is triggered. However, there are cases where this is known to fail (e.g., with meta-swupd the rootfs for swupd images is other than IMAGE_ROOTFS). Because of that, check IMAGE_ROOTFS exists before trying to get the size of it. Also, in case of any error catched as err, simply print out a warning. (From OE-Core rev: b4929542ff01a24bea5edd1c40e3174f55e213ff) Signed-off-by: Mikko Ylinen <mikko.ylinen@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-01subprocess: remove Popen in favor of check_outputStephano Cetola
This begins moving away from the deprecated subprocess calls in an effort to eventually move to some more global abstraction using the run convenience method provided in python 3.5. [ YOCTO #9342 ] (From OE-Core rev: 0d6b7276003f1afabc6de683f663540327d52bdc) Signed-off-by: Stephano Cetola <stephano.cetola@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-15buildstats: Fix tracebacks for early task failuresRichard Purdie
If a failure occurs early in the task, its possible we can have a TaskFailed before the TaskStarted event can be triggered. This in turn causes another traceback as the directory buildstats writes files into doesn't exist. Ensure the directory exists so we can see the original error. (From OE-Core rev: bd0a65ec47ebf55c549c9ef276b201b72396ce2b) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-28buildstats.bbclass: Don't assume /proc/<pid>/io presentJuro Bystricky
It is not guaranteed Linux kernel was configured with process I/O statistics enabled. If process I/O statistcs are not present, issue a one time warning and do not attempt to read the non-existing stats counters. [YOCTO#9025] (From OE-Core rev: b39e84edb02d03102b9a571c21e5328c159c4378) Signed-off-by: Juro Bystricky <juro.bystricky@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-11buildstats.bbclass: remove dead URL from commentRandy MacLeod
(From OE-Core rev: ec4f113bf1779f7df054c635bd9bed7e72157a94) Signed-off-by: Randy MacLeod <Randy.MacLeod@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18buildstats: Improve to add getrusage data and corrected IO statsRichard Purdie
Add IO stats and getrusage() data to the task statistics. We also drop the CPU percentage calculation since its pretty arbitrary and not very accurate/useful. In particular we can now see the user and sys times as well as the wall clock times. (From OE-Core rev: b849130f71d3ba32a6fa94c291ca6ce7c7c3b3d1) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18buildstats: Separate out the build and task data to allow improvementsRichard Purdie
The combined build and task data code makes changing things hard, separate out the functions so that changes can be made to the task data whilst the build data remains unchanged. (From OE-Core rev: c79cfce4f820f20346d0565df8df626832976e28) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18buildstats: Clean up e.data and bb.data referencesRichard Purdie
Rather than bb.data and e.data, cleanup to use 'd' and match the standard coding style. (From OE-Core rev: dbce2c6122d492ba86873db4d02322f0df7a6752) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18buildstats: Drop get_bn/set_pn and just use BUILDNAMERichard Purdie
The current setting and getting of the "name" to use for buildstats is convoluted and not particularly interesting. We only need this for the e.getPkgs()[0] component of the path which is the first target listed on the commandline. This is pretty arbitrary. If we drop that piece, we can assume BUILDNAME is common for all events and simply use this and query it. If BUILDNAME did change, that would be a bug and it should be fixed elsewhere. Also take the opportunity to share some common code since the function now has the eventmask. (From OE-Core rev: 918d83460639df273f38ae079ffeebd6a79b3373) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18buildstats: Drop disk data from buildstatsRichard Purdie
The existing diskstats data from buildstats simply isn't useful. It gives stats on the total IO counts on some random disk within the system. This means that the count includes data from all other tasks running at the same time and from any other process running on the system. I've been unable to find any use for the data so as a start at reworking the class, remove the related code. (From OE-Core rev: f9fb02909f332365cad329352956a29cff6eba77) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-05-07classes/buildstats: misc cleanupsPaul Eggleton
* Rename BNFILE and DEVFILE so that they are clearly specific to this class, since they end up in the global scope * Use "with open" when opening files consistently * Use getVar('PF', True) instead of expand('${PF}') * Drop some unnecessary assignments (From OE-Core rev: 2392c9bae9fc7732ebafb8c2cd42e49ab281afc9) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-05-07classes/buildstats: don't catch all errors from mkdirhierPaul Eggleton
mkdirhier() doesn't raise an exception if the directory exists, so if it does raise one we should just be failing, otherwise we're just going to hit errors later anyway. (From OE-Core rev: 76daab82185851608e5e225487e411504ecb6569) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-05-07classes/buildstats: whitespace cleanupPaul Eggleton
* Drop trailing whitespace * Use spaces around equals in assignments * Replace an errant tab with spaces (From OE-Core rev: e6d2b407979869219da1f15ed4b5c1c804548fce) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-03-16buildstats.bbclass: typecast argument to os.minor/os.major to longKhem Raj
Started to run into this error on ubuntu 15.04 ERROR: Execution of event handler 'run_buildstats' failed Traceback (most recent call last): File "run_buildstats(e)", line 18, in run_buildstats(e=<bb.event.BuildStarted object at 0x7f93341c1b50>) File "buildstats.bbclass", line 17, in set_device(e=<bb.event.BuildStarted object at 0x7f93341c1b50>) SystemError: ../Objects/longobject.c:998: bad argument to internal function ERROR: Command execution failed: Traceback (most recent call last): File "/home/ubuntu/work/bleeding/openembedded-core/bitbake/lib/bb/command.py", line 102, in runAsyncCommand commandmethod(self.cmds_async, self, options) File "/home/ubuntu/work/bleeding/openembedded-core/bitbake/lib/bb/command.py", line 303, in buildTargets command.cooker.buildTargets(pkgs_to_build, task) File "/home/ubuntu/work/bleeding/openembedded-core/bitbake/lib/bb/cooker.py", line 1336, in buildTargets bb.event.fire(bb.event.BuildStarted(buildname, fulltargetlist), self.data) File "/home/ubuntu/work/bleeding/openembedded-core/bitbake/lib/bb/event.py", line 163, in fire fire_class_handlers(event, d) File "/home/ubuntu/work/bleeding/openembedded-core/bitbake/lib/bb/event.py", line 102, in fire_class_handlers execute_handler(name, handler, event, d) File "/home/ubuntu/work/bleeding/openembedded-core/bitbake/lib/bb/event.py", line 76, in execute_handler ret = handler(event) File "run_buildstats(e)", line 18, in run_buildstats File "buildstats.bbclass", line 17, in set_device SystemError: ../Objects/longobject.c:998: bad argument to internal function similar to here http://pycuda.2962900.n2.nabble.com/bad-argument-to-internal-function-td4063274.html If it is right fix .. Dont know Change-Id: I84f3a4043ad5246e080dfd7e2f066e5292d4af91 (From OE-Core rev: 5f85a7549d119177c5d39f965b949e957e07867e) Signed-off-by: Khem Raj <raj.khem@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-03-25buildstats: use TaskBase time for elapsed timeMarius Avram
To avoid any further inconsistencies between buildstats and buildinfohelper from toaster, buildstats will measure task duration using the time field from within the TaskBase events: TaskStarted and TaskSucceeded/TaskFailed. (From OE-Core rev: 406acd647a288694c2f776a9faa1f5607f3e8e7a) Signed-off-by: Marius Avram <marius.avram@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-09-01meta: Don't use deprecated bitbake APIRichard Purdie
These have been deprecated for a long time, convert the remaining references to the correct modules and prepare for removal of the compatibility support from bitbake. (From OE-Core rev: 6a39835af2b2b3c7797fe05479341d71a3f3aaf6) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-06-14classes/conf: Add eventmasks for event handlersRichard Purdie
Now that bitbake supports masking events for event handlers, lets use this so event handlers are only called for events they care about. This lets us simplify the code indentation a bit at least as well as mildly improving the event handling performance. (From OE-Core rev: bff73743280f9eafebe4591f7368ead91a4eb74d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-05-09buildstats: Drop spurious open() callRichard Purdie
This file is already opened a few lines previously, so drop the duplicated call. (From OE-Core rev: 4808629faa1222b31f92a3e410e06adb8e081293) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-05-09buildstats: Update to ensure files are closedRichard Purdie
python3 doesn't like files being left open. This updates the code style to ensure file are closed. (From OE-Core rev: ec74285bd2108f12f33fc8ac0dc1d124ab48be21) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-08-21meta/classes: Various python whitespace fixesRichard Purdie
It was pointed out we have a number of weird indentations in the python functions. This patch cleans up 3, 7 and other weird indentations for the core bbclass files. It also fixes some wierd (odd) shell function indentation which my searches picked up. (From OE-Core rev: 8385d6d74624000d68814f4e3266d47bc8885942) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-03-05meta: Replace bb.data.expand(xxx, d) -> d.expand(xxx)Richard Purdie
sed \ -e 's:bb.data.\(expand([^,()]*\), *\([^) ]*\) *):\2.\1):g' \ -i `grep -ril bb.data.expand *` Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-02-28More quoting fixesRichard Purdie
(From OE-Core rev: 749d252475df090d51313cfbbe3f159db9f0566d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-01-16buildstats: tolerate absence of /proc/diskstatsJean-François Dagenais
In OpenVZ containers (and probably lx containers as well), the diskstats entry is not even present. Use the "NoLogicalDrive" introduced by Elizabeth Flanagan in such case. This allows the bitbaking to occure within such containers. (From OE-Core rev: 16e09b850dcb44cb1afe411439e40a4bae7e8002) Signed-off-by: Jean-François Dagenais <jeff.dagenais@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-11-10Convert to use direct access to the data store (instead of bb.data.*Var*())Richard Purdie
This is the result of running the following over the metadata: sed \ -e 's:bb.data.\(setVar([^,()]*,[^,()]*\), *\([^ )]*\) *):\2.\1):g' \ -e 's:bb.data.\(setVarFlag([^,()]*,[^,()]*,[^,()]*\), *\([^) ]*\) *):\2.\1):g' \ -e 's:bb.data.\(getVar([^,()]*\), *\([^(), ]*\) *,\([^)]*\)):\2.\1,\3):g' \ -e 's:bb.data.\(getVarFlag([^,()]*,[^,()]*\), *\([^(), ]*\) *,\([^)]*\)):\2.\1,\3):g' \ -e 's:bb.data.\(getVarFlag([^,()]*,[^,()]*\), *\([^() ]*\) *):\2.\1):g' \ -e 's:bb.data.\(getVar([^,()]*\), *\([^) ]*\) *):\2.\1):g' \ -i `grep -ril bb.data *` (From OE-Core rev: b22831fd63164c4db9c0b72934d7d734a6585251) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-11-07buildstats: Fix for buildstats on tmpfsElizabeth Flanagan
tmpfs/encryptfs/(and most likely, but not confirmed)ramfs TMPDIRs cause diskstats to choke. No device entry ends up in /proc/diskstats for these fs types, which ends up causing the failure. The short term solution is to exclude these fs types from diskstat collection. Longer term we will want to see if we can collect meaningful diskio for each of these, and other, use cases, but for this cleans up Bug 1700. [YOCTO #1700] (From OE-Core rev: 2b14046c12855b6f484ba5bd6bc0a8022de6873e) Signed-off-by: Elizabeth Flanagan <elizabeth.flanagan@intel.com> Corrected YOCTO bug location and format Signed-off-by: Saul Wold <sgw@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-08-17buildstats.bbclass: Fixing task endtimeBeth Flanagan
Task endtime got left out of recent changes. Adding it back in. (From OE-Core rev: f1e4b049438218347922f13de810782be0205c3d) Signed-off-by: Beth Flanagan <elizabeth.flanagan@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-08-05buildstats.bbclass: Adding disk io measurementBeth Flanagan
This commit adds disk-io statistics functionality on a build and per-task basis. It pulls measurements for whatever partition TMPDIR exists on. This data could be off if SSTATE_DIR and DL_DIR exist on a different partition/volume. Notes on what this pulls: ReadsComp: Total number of reads complete ReadsMerged: Total number of adjacent reads merged SectRead: Total number of sectors read TimeReads: Total number of m/s spent reading WritesComp: Total number of writes completed SectWrite: Total number of sectors written TimeWrite: Total number of m/s spent writing IOinProgress: Total amount of IO in progress at the time of we look at /proc/diskstats. TimeIO: Total number of m/s spent doing IO WTimeIO: Weighted time doing I/O. From iostats.txt: "This field is incremented at each I/O start, I/O completion, I/O merge, or read of these stats by the number of I/Os in progress (field 9) times the number of milliseconds spent doing I/O since the last update of this field. This can provide an easy measure of both I/O completion time and the backlog that may be accumulating." (From OE-Core rev: eeebcebf0d695358e72f3aed753f66cddd5e0e61) Signed-off-by: Beth Flanagan <elizabeth.flanagan@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-02-16buildstats.bbclass: Remove unneeded debug outputRichard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-02-16Doing uname without subprocess: buildstats.bbclassBeth Flanagan
Instead of using subprocess which is problematic, we should use platform instead. Signed-off-by: Beth Flanagan <elizabeth.flanagan@intel.com>
2011-02-15buildstats.bbclass: Fix expand call syntaxRichard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-02-15buildstats.bbclass: Save ended time in TaskFinished eventsRichard Purdie
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-02-15buildstats: Disable uname subprocess call for now since it appears to be ↵Richard Purdie
breaking runqueue somehow Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2011-02-15Buildstats commit: buildstats.bbclassBeth Flanagan
Used to track some basic build metrics by build and task/event level. Signed-off-by: Beth Flanagan <elizabeth.flanagan@intel.com>
generated by cgit 1.2.3-korg (git 2.39.0) at 2024-04-27 09:05:11 +0000