summaryrefslogtreecommitdiffstats
path: root/meta/lib
AgeCommit message (Collapse)Author
2019-06-30oeqa/logparser: ignore test failure commentaryRoss Burton
The output format for Python and GLib both can be of this form: FAIL: foobar (Segmentation fault) In this case the test is called foobar not foobar_segmentation_fault. (From OE-Core rev: 95031da4f08295ad81efac1c082c48dd5c330fb0) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-30rootfs: Fix dependency for every dpkg runRicardo Ribalda Delgado
Avoid getting a warning on do_rootfs. Fixes: NOTE: Installing the following packages: libgdk-pixbuf-2.0-locale-en-gb glibc-locale-en-gb libatspi-locale-en-gb gstreamer1.0-locale-en-gb gtk+3-locale-en libatk-1.0-locale-en-gb gtk+3-locale-en-gb gstreamer1.0-plugins-good-locale-en-gb gstreamer1.0-plugins-base-locale-en-gb gstreamer1.0-plugins-bad-locale-en-gb libglib-2.0-locale-en-gb avahi-locale-en-gb vte-locale-en-gb xkeyboard-config-locale-en-gb WARNING: Unable to install packages. Command '/workdir/build/tmp/work/qt5122-poky-linux/bottlecam-image/1.0-r0/recipe-sysroot-native/usr/bin/apt-get install --force-yes --allow-unauthenticated libgdk-pixbuf-2.0-locale-en-gb glibc-locale-en-gb libatspi-locale-en-gb gstreamer1.0-locale-en-gb gtk+3-locale-en libatk-1.0-locale-en-gb gtk+3-locale-en-gb gstreamer1.0-plugins-good-locale-en-gb gstreamer1.0-plugins-base-locale-en-gb gstreamer1.0-plugins-bad-locale-en-gb libglib-2.0-locale-en-gb avahi-locale-en-gb vte-locale-en-gb xkeyboard-config-locale-en-gb' returned 100: Reading package lists... Building dependency tree... Reading state information... (From OE-Core rev: 747d80a496741090f9925c007b03cd68501ef131) Signed-off-by: Ricardo Ribalda Delgado <ricardo@ribalda.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-30manifest.py: fix test_SDK_manifest_entriesChen Qi
TOOLCHAIN_OUTPUTNAME could be overridden. So use this variable directly instead of its default value ${SDK_NAME}-toolchain-${SDK_VERSION}. (From OE-Core rev: eb32dd2956da99813136842acdb010c8471f5e3c) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-30context.py: fix skipping functionChen Qi
The current codes to skip test cases are logically correct, but they do not work correctly in reality. It does skip the tests as the command line argument specifies, but the related information is wrong. e.g. $ oe-selftest -R archiver bblayers runtime_test 2019-06-17 09:24:53,764 - oe-selftest - WARNING - meta-selftest layer not found in BBLAYERS, adding it 2019-06-17 09:25:06,309 - oe-selftest - INFO - Adding layer libraries: 2019-06-17 09:25:06,310 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta/lib 2019-06-17 09:25:06,310 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta-yocto-bsp/lib 2019-06-17 09:25:06,310 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta-selftest/lib 2019-06-17 09:25:06,312 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable 2019-06-17 09:25:10,521 - oe-selftest - INFO - Adding: "include selftest.inc" in /buildarea5/chenqi/SWAT/poky/build-selftest/conf/local.conf 2019-06-17 09:25:10,521 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2019-06-17 09:25:10,522 - oe-selftest - INFO - test_archiver_allows_to_filter_on_recipe_name (archiver.Archiver) 2019-06-17 09:25:10,522 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "runtime_test"' 2019-06-17 09:25:10,522 - oe-selftest - INFO - Skip by the command line argument "runtime_test" 2019-06-17 09:25:10,523 - oe-selftest - INFO - test_archiver_filters_by_type (archiver.Archiver) 2019-06-17 09:25:10,523 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "runtime_test"' 2019-06-17 09:25:10,523 - oe-selftest - INFO - Skip by the command line argument "runtime_test" The archiver.Archiver.xxx tests should be skipped by 'archiver' command line argument, not 'runtime_test'. Change to use a function generator to achieve the desired effect. After the change, the effect is as follows. $ oe-selftest -R archiver bblayers runtime_test 2019-06-17 09:19:06,223 - oe-selftest - WARNING - meta-selftest layer not found in BBLAYERS, adding it 2019-06-17 09:19:19,598 - oe-selftest - INFO - Adding layer libraries: 2019-06-17 09:19:19,599 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta/lib 2019-06-17 09:19:19,599 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta-yocto-bsp/lib 2019-06-17 09:19:19,599 - oe-selftest - INFO - /buildarea5/chenqi/SWAT/poky/meta-selftest/lib 2019-06-17 09:19:19,602 - oe-selftest - INFO - Running bitbake -e to test the configuration is valid/parsable 2019-06-17 09:19:24,368 - oe-selftest - INFO - Adding: "include selftest.inc" in /buildarea5/chenqi/SWAT/poky/build-selftest/conf/local.conf 2019-06-17 09:19:24,368 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2019-06-17 09:19:24,369 - oe-selftest - INFO - test_archiver_allows_to_filter_on_recipe_name (archiver.Archiver) 2019-06-17 09:19:24,369 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "archiver"' 2019-06-17 09:19:24,369 - oe-selftest - INFO - Skip by the command line argument "archiver" 2019-06-17 09:19:24,369 - oe-selftest - INFO - test_archiver_filters_by_type (archiver.Archiver) 2019-06-17 09:19:24,370 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "archiver"' 2019-06-17 09:19:24,370 - oe-selftest - INFO - Skip by the command line argument "archiver" 2019-06-17 09:19:24,370 - oe-selftest - INFO - test_archiver_filters_by_type_and_name (archiver.Archiver) 2019-06-17 09:19:24,370 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "archiver"' 2019-06-17 09:19:24,371 - oe-selftest - INFO - Skip by the command line argument "archiver" 2019-06-17 09:19:24,371 - oe-selftest - INFO - test_archiver_srpm_mode (archiver.Archiver) 2019-06-17 09:19:24,371 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "archiver"' 2019-06-17 09:19:24,372 - oe-selftest - INFO - Skip by the command line argument "archiver" 2019-06-17 09:19:24,372 - oe-selftest - INFO - test_bitbakelayers_add_remove (bblayers.BitbakeLayers) 2019-06-17 09:19:24,373 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "bblayers"' 2019-06-17 09:19:24,373 - oe-selftest - INFO - Skip by the command line argument "bblayers" 2019-06-17 09:19:24,373 - oe-selftest - INFO - test_bitbakelayers_createlayer (bblayers.BitbakeLayers) 2019-06-17 09:19:24,373 - oe-selftest - INFO - ... skipped 'Skip by the command line argument "bblayers"' 2019-06-17 09:19:24,374 - oe-selftest - INFO - Skip by the command line argument "bblayers" [snip] (From OE-Core rev: efdde9ee80f17a302e2b435b6a1103b001c612dc) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-30selftest/devtool: fix URI to MarkupSafe packageOleksandr Kravchuk
PyPi packages are now hosted at files.pythonhosted.org. [YOCTO #13243] (From OE-Core rev: d498a41d89adcead8995668f54c01a175c081bca) Signed-off-by: Oleksandr Kravchuk <open.source@oleksandr-kravchuk.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-18qemurunner: fix undefined variableAdrian Freihofer
While hacking on this I got an Exception. It's better to define variables also in python. Signe:-off-by: Adrian Freihofer <adrian.freihofer@siemens.com> (From OE-Core rev: 7c604a62356538de660835bd2d3ba26beccd810d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-18logparser: Add LTP compliance sectionArmin Kuster
(From OE-Core rev: 386795d14c23d2e4084563c6234779b90e4d400f) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-18logparser: Add decoding ltp logsArmin Kuster
(From OE-Core rev: 8c0a03c24248bf69e9c8c0f2d949ff79efc9c71e) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-18oeqa/utils/qemurunner: Set both the threadport&serverport with tcpserial ↵Kevin Hao
parameter After the commit ad522ea6a64e ("runqemu: Let qemuparams override default settings"), the order of the two "-serial" parameters when running the qemu have been switched. The effect of this is that the logging thread will use ttyS1 (of course can't capture the kernel boot message anymore), and the test command will run on the ttyS0. So the output of the test command may be mangled by the kernel message (such as call trace), and let the test command produce a fake timeout error message. We can't fix it by just adjusting the order of the threadport and serverport, since it will break some machines such as qemuarm64 which use the virtio serial. So using the tcpserial to setup both the threadport and serverport. [YOCTO Bug 13309] (From OE-Core rev: 1ffea0c0fb3688f3cee0a5fa741e969581d01bd3) Signed-off-by: Kevin Hao <kexin.hao@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-18resulttool: modify to be multi-machineJon Mason
Currently, the code will sum all of the different machine results into a single report of the tests results. This can lead to confusion as to which machine may be experiencing issues. Modify the code to store the results in a per machine basis and report them accordingly. (From OE-Core rev: 7ba5a0af31605eaf91eb75b5f673f4caf81361dc) Signed-off-by: Jon Mason <jdmason@kudzu.us> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-18meta/lib+scripts: Convert to SPDX license headersRichard Purdie
This adds SPDX license headers in place of the wide assortment of things currently in our script headers. We default to GPL-2.0-only except for the oeqa code where it was clearly submitted and marked as MIT on the most part or some scripts which had the "or later" GPL versioning. The patch also drops other obsolete bits of file headers where they were encoountered such as editor modelines, obsolete maintainer information or the phrase "All rights reserved" which is now obsolete and not required in copyright headers (in this case its actually confusing for licensing as all rights were not reserved). More work is needed for OE-Core but this takes care of the bulk of the scripts and meta/lib directories. The top level LICENSE files are tweaked to match the new structure and the SPDX naming. (From OE-Core rev: 3248a9e3c5a197321b1c4417509b9309cc3bae97) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-18ltp_compliance: add new runtimeArmin Kuster
test runtimes in sec. AIO: 14 MEM: 94 MSG: 89 SEM: 30 SIG: 194 THR: 399 TMR: 867 TPS: 23 (From OE-Core rev: 2ad81e503afdde2711ccc54942e83fce02f4cbe5) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-18ltp: add runtime testArmin Kuster
This adds the framework for running ltp tests. Here are some times: math: 61 syscalls: 3957 dio: 18472 io: 29 mm: 551 ipc: 48 sched: 165 nptl: 46 pty: 37 containers: 52 controllers: 9625 filecaps: 27 cap_bounds: 27 fcntl-locktests: 29 connectors: 27 timers: 37 commands: 165 net.ipv6_lib: 30 input: 29 fs_perms_simple: 31 fs: 3476 fsx: 30 fs_bind: 28 fs_ext4: 28 cve: 675 (From OE-Core rev: 2a5767f584c159f4711afc5a01ee740d44c94d02) Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-18lib/oe/reciputils.py: support character `+' in git pvHongxu Jia
While pv contains char `+' (such as ncurses 6.1+20181013), it was incorrectly converted to `6.1' In this commit: - Convert [^\+]* to .*? Support pv to contain char `+' and not greedy match (Such as ncurses 6.1+20181013) - Add [^\+] to sfx match Support sfx contains extra chars between `+' and `git' (such as asciidoc "8.6.9+py3-gitAUTOINC+618f6e6f6b") - Make sfx and rev greedy match Run `devtool check-upgrade-status --all' in poky and compare results, only one difference on ncurses version: Without the commit: INFO: ncurses 6.1 6.1+20181013 Hongxu Jia <hongxu.jia@windriver.com> 7a97a7f937762ba342d5b2fd7cd090885a809835 With the commit: INFO: ncurses 6.1+20181013 MATCH Hongxu Jia <hongxu.jia@windriver.com> 7a97a7f937762ba342d5b2fd7cd090885a809835 (From OE-Core rev: 65e2b1a2333a2268f4ee7de6a20cb13ec697b50e) Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-07oeqa/targetcontrol.py: fix qemuparams not work in runqemu with launch_cmdHongxu Jia
As runqemu with launch_cmd means directly run the command, don't need set rootfs or env vars. Since commit [a847dd7202 runqemu: Let qemuparams override default settings] applied in oe-core, if launch_cmd contains "qemuparams='***'", it does not work, which is overridden by latter qemuparams="-serial tcp:127.0.0.1" in QemuRunner.launch(); So we set qemuparams as a parameter in runqemu, the fix makes it work (From OE-Core rev: 00c58ae552d7414116f13f43d4df2f6e57c0d1ba) Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-07oeqa/sdk: use bash to execute SDK test commandsMikko Rapeli
The commands only work with with bash. If /bin/sh is dash like in Debian, the command execution fails with errors like: Standard Output: /bin/sh: 5: export: --sysroot: bad variable name and all SDK tests fail. (From OE-Core rev: 0423d5049492516a6df6caf3404ddaa1734ff1d9) Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-07oeqa/concurrenttest: Patch subunit module to handle classSetup failuresRichard Purdie
Currently setupClass errors were not being mapped back to the failing tests and they were hence being marked as UNKNOWN and the test statistics were inaccurate. This is because whilst the errors were being encoded into the test results stream, the decoder doesn't cope with an error outside a testStart event. We patch in an addError handler to the outsideTest parser so that this does get handled in a way similar to the non-concurrent case. It would be nice if we didn't have to do this but there doesn't seem to be any other way to fix this other than forking subunit. We also make a minor change so another of our changes can cope with tests without a start time. (From OE-Core rev: 741cf82d0e7f9e797ba35157a1466fadbf74dc64) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-06-07oeqa/runner: Fix subunit setupClass/setupModule failure handlingRichard Purdie
The string format for subunit setupClass/setupModule failures is slightly different, tweak the regex to correctly handle both cases. (From OE-Core rev: 3c645ecdb884ecafa7c492bdd8a952cce96d3906) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-20oeqa/target/ssh: Replace suggogatepass with ignoring errorsRichard Purdie
We continued to see encoding problems with ssh commands run in oeqa. After much research the conclusion was we should use ignore the errors since some occasional bad locale encoding is better than the unicode decoding issues we were seeing which crashed large parts of tests. (From OE-Core rev: e4037c6911592700799d207ca8c3976afba18e25) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-20oeqa/core/runner: Handle unexpectedSucessesRichard Purdie
Instead of showing: RESULTS - ptest.PtestRunnerTest.test_ptestrunner - Testcase 1600: UNKNOWN (32.30s) map unexpectedSuccesses to PASSED and improve the way they're displayed. We expect/allow ptest runner to fail but if it passes we should handle it correctly. (From OE-Core rev: 21079473e0cbd0e43682dc0d15a651e7e64db747) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-20oeqa: Drop OETestIDRichard Purdie
These IDs refer to testopia which we're no longer using. We would now use the test names to definitively reference tests and the IDs can be dropped, along with their supporting code. (From OE-Core rev: 551153b0bd1ebbc05582f6014e3d88b9ce4a46d0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> [Backported to keep in-sync with future qa changes] Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-20oeqa/ssh: Avoid unicode decode exceptionsRichard Purdie
This code really needs to be rewritten to not split potential multibyte characters, for now work around it to avoid exceptions like: File "/home/pokybuild/yocto-worker/qa-extras2/build/meta/lib/oeqa/core/target/ssh.py", line 211, in run data = reader.read(1024, 4096) File "/usr/lib64/python3.6/codecs.py", line 503, in read newchars, decodedbytes = self.decode(data, self.errors) UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 0: invalid start byte (From OE-Core rev: 918e15e22c797d52d33c3399324e480aa1a2cc7f) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-12utils/multiprocess_launch: Improve failing subprocess outputRichard Purdie
Output before this patch: ERROR: bash-4.4.18-r0 do_package_write_ipk: Fatal errors occurred in subprocesses: Command 'PATH="X" opkg-build -Z xz -a "--memlimit=50% --threads=88" Foobar /media/build1/poky/build/nodistro-glibc/work/core2-64-oe-linux/bash/4.4.18-r0/deploy-ipks/core2-64' returned non-zero exit status 1.: Traceback (most recent call last): File "/media/build1/poky/meta/lib/oe/utils.py", line 272, in run ret = self._target(*self._args, **self._kwargs) File "/media/build1/poky/meta/classes/package_ipk.bbclass", line 230, in ipk_write_pkg shell=True) File "/usr/lib/python3.6/subprocess.py", line 336, in check_output **kwargs).stdout File "/usr/lib/python3.6/subprocess.py", line 418, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command 'PATH="X" opkg-build -Z xz -a "--memlimit=50% --threads=88" Foobar /media/build1/poky/build/nodistro-glibc/work/core2-64-oe-linux/bash/4.4.18-r0/deploy-ipks/core2-64' returned non-zero exit status 1. Note how stdout/stderr from the failing command isn't shown. After this patch: ERROR: bash-4.4.18-r0 do_package_write_ipk: Fatal errors occurred in subprocesses: Command 'PATH="X" opkg-build -Z xz -a "--memlimit=50% --threads=88" Foobar /media/build1/poky/build/nodistro-glibc/work/core2-64-oe-linux/bash/4.4.18-r0/deploy-ipks/core2-64' returned non-zero exit status 1. Subprocess output:Foobar *** Error: Package name Foobar contains illegal characters, (other than [a-z0-9.+-]) opkg-build: Please fix the above errors and try again. We suddenly get a much more usable error message. The traceback is supressed as its distracting from the real problem in this case. Ideally python itself would handle this but it doesn't so we have to wrap the exception. We already do this in bitbake itself for the same reason. (From OE-Core rev: c27cc2355d331e063f170b45728671988a967deb) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-12oeqa/core/runner: dump stdout and stderr of each test caseMardegan, Alberto
Some CI pipelines might perform further processing of the test output (for instance, to plot some metrics into a chart). However, Since `thud` we switched away from the XML-based jUnit reporting, and at the same time we lost the ability of collecting the stdout and stderr of the various tests. We now restore this functionality by adding `stdout` and `stderr` keys to the JSON reports. This behavior is off by default; in order to enable it, one must set the `TESTREPORT_FULLLOGS` variable in the bitbake configuration. (From OE-Core rev: fd57b34d7c8a120273d65cd361be208fbdaeff50) Signed-off-by: Alberto Mardegan <amardegan@luxoft.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-05-12buildhistory: call a dependency parser only on actual dependency listsAlexander Kanavin
Previously it was also called on filelists and possibly other items which broke the parser. (From OE-Core rev: cf525cbbc4277ba08415963cfa7dca384ad501dd) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> Signed-off-by: Armin Kuster <akuster808@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-09oeqa/utils/qemurunner: Fix typo in previous commitRichard Purdie
(From OE-Core rev: e2b4b0580cd719cf9d48576aa7d0b88e4a286921) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-09patch/insane: Rework patch fuzz handlingAndreas Müller
Currently there are three issues which can be enhanced: 1. Fuzz warnings cannot be configured as errors for hardening. It happened often to me that these warnings were overseen and detected after commits were already out. 2. The output is too verbose - particularly when more than one file is affected. Meanwhile all users should know why patch fuzz check is performed. So move links with background information to insane.bbclass. 3. Reduce copy & paste effort slightly by printing PN (nit: <recipe> was not a correct suggestion e.g for native extended recipe - see example below) To achieve patch.py drops patch-fuzz info encapsulated by a header- and footer- string into log.do_patch. With this insane.bbclass can drop warnings/errors depending on 'patch-fuzz' in ERROR_QA or WARN_QA. Default remains unchanged: Spit out warnings only. A message for two fuzzed patches and 'pact-fuzz' in ERROR_QA now looks like: | ERROR: autoconf-native-2.69-r11 do_patch: Fuzz detected: | | Applying patch autoreconf-exclude.patch | patching file bin/autoreconf.in | Hunk #1 succeeded at 73 with fuzz 1 (offset -3 lines). | Hunk #2 succeeded at 143 (offset 6 lines). | Hunk #3 succeeded at 167 (offset 6 lines). | Hunk #4 succeeded at 177 (offset 6 lines). | Hunk #5 succeeded at 281 (offset 15 lines). | Hunk #6 succeeded at 399 (offset 15 lines). | Hunk #7 succeeded at 571 (offset 20 lines). | Hunk #8 succeeded at 612 (offset 20 lines). | Hunk #9 succeeded at 636 (offset 20 lines). | Hunk #10 succeeded at 656 (offset 20 lines). | Hunk #11 succeeded at 683 (offset 20 lines). | | Applying patch autoreconf-gnuconfigize.patch | patching file bin/autoreconf.in | Hunk #1 succeeded at 55 with fuzz 1 (offset -3 lines). | Hunk #3 succeeded at 663 (offset 18 lines). | | The context lines in the patches can be updated with devtool: | | devtool modify autoconf-native | devtool finish --force-patch-refresh autoconf-native <layer_path> | | Don't forget to review changes done by devtool! | | ERROR: autoconf-native-2.69-r11 do_patch: QA Issue: Patch log indicates that patches do not apply cleanly. [patch-fuzz] (From OE-Core rev: c762c0be43a3854a43cb4b9db559b03126d50706) Signed-off-by: Andreas Müller <schnitzeltony@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-09lib/oe/utils: Make prune_suffix prune a suffixAndre Rosa
... instead of replacing a substring that could happen more than once and not only when it ends with it. Do the same for the prefix. See related https://github.com/openembedded/bitbake/pull/24 . There it stops replacing sufixes once first one is matched but not here. (From OE-Core rev: 610ac84170f8a91cc3321edfc336a9e39f24ebe3) Signed-off-by: Andre Rosa <andre.rosa@lge.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-04-04utils.py: added sh_quote() functionEnrico Scholz via Openembedded-core
This function is a wrapper around "shlex.quote()" and can be used in "${@...}" context where shlex (or pipes, which provides similar functionality) is unavailable. (From OE-Core rev: 127141f5023a7e3fc3963dc7d76cfce9067a9e8a) Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-25oeqa/qemurunner: Dont mix binary and non-binary stringsRichard Purdie
self.msg is a str(), bootlog is b'' so this code clearly doesn't work. Add in a decode since its being used as a string. (From OE-Core rev: e7546759d3102b576974bbab52594796f084721e) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-19oeqa/manual/toaster: updated test id naminguninative-2.4Yeoh Ee Peng
All test id (eg. @alias) inside manual testcase file shall follow the same test id naming convention from oeqa automated tests (eg. selftest, runtime, sdk, etc), where the test id consists of <test_module>.<test_suite>.<test_function>. Furthermore, there shall be only 1 unique test_module per each manual testcases file, where test_module match the file name itself. This file was using test_module name that does not match the file name itself. Fixed test_module name as well as the test_suite name. (From OE-Core rev: f24178993e7b0830510900e80e879c7f9f8e28f0) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-18oeqa/manual/toaster: transfer manual toaster test cases to oeqaYeoh Ee Peng
As part of the solution to replace Testopia, manual test cases need to be migrated to OEQA. These manual test case json files will serve two use cases. Use case#1: as input to the future commandline-based manual test runner script, where this script will display actions and expected result information in user friendly text, then it will capture user input for test result and log, finally it will write test result and log into existing standardize test result json format from OEQA framework for automated tests. Use case#2: QA will open and read these json file manually for planning manual test execution. Any reader interested in understanding manual test cases will open and read these files. (From OE-Core rev: fbeecb059d9c006f50a2a11bed5a42740722e3e8) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-18oeqa/manual/kernel-dev.json: test id updatedsangeeta jain
All test id (eg. @alias) inside manual testcase file shall follow the same test id naming convention from oeqa automated tests (eg. selftest, runtime, sdk, etc), where the test id consists of <test_module>.<test_suite>.<test_function>. Furthermore, there shall be only 1 unique test_module per each manual testcases file. This file was using more than 1 unique test_module for testcases. Furthermore, some of the testcases were defined using different test_suite where it was not needed. This patch fix the manual testcases file to have only 1 unique test_module as well as test_suite to simplify test id naming. (From OE-Core rev: be7cbe86f046bf29082c3da574377f9828aff5f4) Signed-off-by: sangeeta jain <sangeeta.jain@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-18oeqa/manual/compliance-test.json: test id updated and missing test cases addedsangeeta jain
Two changes made in oeqa/manual/compliance-test.json: 1. All test id (eg. @alias) inside manual testcase file shall follow the same test id naming convention from oeqa automated tests (eg. selftest, runtime, sdk, etc), where the test id consists of <test_module>.<test_suite>.<test_function>. Furthermore, there shall be only 1 unique test_module per each manual testcases file. This file was using more than 1 unique test_module for testcases. Furthermore, some of the testcases were defined using different test_suite where it was not needed. This patch fix the manual testcases file to have only 1 unique test_module as well as test_suite to simplify test id naming. 2. Added 2 missing test cases for Beaglebone Stress Test. (From OE-Core rev: 5c5402bc6e6505da0203ca9245a3599bd4940dad) Signed-off-by: sangeeta jain <sangeeta.jain@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-18oeqa/manual/bsp-hw.json: test id updated and obsolete test cases removedsangeeta jain
Two changes made in oeqa/manual/bsp-hw.json: 1. All test id (eg. @alias) inside manual testcase file shall follow the same test id naming convention from oeqa automated tests (eg. selftest, runtime, sdk, etc), where the test id consists of <test_module>.<test_suite>.<test_function>. Furthermore, there shall be only 1 unique test_module per each manual testcases file. This file was using more than 1 unique test_module for testcases. Furthermore, some of the testcases were defined using different test_suite where it was not needed. This patch fix the manual testcases file to have only 1 unique test_module as well as test_suite to simplify test id naming. 2. As per review by Intel and Windriver team, 7 testcases were found obsolete. Removed 7 testcases. (From OE-Core rev: e1136bae81672894a277512d7084d27f4e3416b9) Signed-off-by: sangeeta jain <sangeeta.jain@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-07metadata.py: return info after fallback methodsChen Qi
If python3 git module is not installed, we fall back to some git command to get info. A previous commit (b8d22ed6) accidently deleted the return statement, causing errors like below. Exception: UnboundLocalError: local variable 'InvalidGitRepositoryError' referenced before assignment (From OE-Core rev: cd8aedc408afa4f8589f79e4d85befc7cac76ac6) Signed-off-by: Chen Qi <Qi.Chen@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-06imagefeatures: Add test to blacklist building busyboxTom Rini
Now that we have a packagegroup that can be used to replace the overall functionality of busybox as base-utils, add a test that we can continue to build a fairly complex image without busybox being available. This also doubles as documentation-by-test of how to do this. (From OE-Core rev: 6319a59c1d30eeb8ad4871d43641e3469fb543ba) Signed-off-by: Tom Rini <trini@konsulko.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-06devtool: Support kmeta directory usage with devtool modify/finishJaewon Lee
When using Kmeta directories, devtool finish will add every single file in the directory to the bbappend. This is because in the current implementation, the get_recipe_local_files function treats the kmeta directory like a file. Modifying the function to loop through the provided directories and return all included files instead of just the top level directory. This will enable correct file to file comparison when determing which files are new/changed and need to be added to the bbappend. Adding an extra check in devtool-source.bbclass to not copy the cfg file if its already included somewhere in the kmeta directory Also during 'modify', when moving necessary files in the kmeta directory from the workdir to oe-local-files, the dangling parent directories are left behind. This in itself is not an issue as the temporary devtool workspace is automatically deleted, but this causes an incorrect include directory to be added in kernel-yocto.bbclass. Changing the order of the if statements to catch the correct conditional. This is safe to do as when not in the devtool context, there will be no oe-local-files directory. (From OE-Core rev: 4671011b5b02d0989aa0fdcb50e18cb385a0e95e) Signed-off-by: Jaewon Lee <jaewon.lee@xilinx.com> Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com> Acked-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-04masterimage: remove code duplicationAndré Draszik
Two identical classes exist for SystemdbootTarget, introduced in commit f9a61d3400ad ("gummiboot: Remove/change gummiboot references with systemd-boot") poky commit 2dce2648e3d8 Remove the duplicated copy. (From OE-Core rev: a59562c7ddbda1a266f0ea22ab78fb86de4861a3) Signed-off-by: André Draszik <andre.draszik@jci.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-04selftest: add tests for virgl GL accelerationAlexander Kanavin
Note that the tests require that the host machine has a X display, has mesa development files installed and is able to create OpenGL contexts. (From OE-Core rev: 2868e8dfb9e62b49cd06f6c2d010405079d3a71c) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-04selftest: Add multiconfig testRichard Purdie
Add a test for a multiconfig build which mixes tiny and musl builds along with using the mcextend class to combine and package multiple images into another image. This gives the multiconfig a decent test in a scenario users may use. (From OE-Core rev: 0c7fa15a7350808242754944243f01155bc6784c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-03-03parselogs.py: ignore regulatory.db load failure messages for x86*Naveen Saini
[YOCTO #13098] (From OE-Core rev: 9c20f8c6e4bb5157550e347ffd6ef790eb6a1200) Signed-off-by: Naveen Saini <naveen.kumar.saini@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-28oeqa/utils/metadata: Add commit_count to fallback logicRichard Purdie
Currently if python3-git isn't installed we can get odd behaviours when the commit_count is absent. Avoid this set of bugs by adding a fallback here. (From OE-Core rev: b8d22ed681141ce360d742a96cec5f2925a20222) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-28oeqa/oetest: Drop unused importRichard Purdie
(From OE-Core rev: ac4eba5415f39cd797a08071c0efae296ae61a70) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25lib/oe/package_manager.py: turn nativesdk postinst warnings into notesAlexander Kanavin
The warnings deal with two specific cases of failure: 1) unable to execute nativesdk postinsts for mingw SDKs because they need to be run under wine 2) unable to execute target postinsts when there is no qemu usermode support for the target Neither of these should be a big problem as mingw issue was never found to problematic, and target postinsts deal with things that are needed at runtime and not at build time which is the purpose of SDKs. The specific reason to do this is to reduce the amount of warnings shown by the Yocto autobuilder, to zero eventually. (From OE-Core rev: cbc32fcd9b52e750600cce9dd84b33e3ce612eae) Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-25oeqa/utils/gitarchive: Handle case where parent is only on originRichard Purdie
The parent code currently assumed that any parent branch is locally checked out which may not be the case. Use the local branch by default but fall back to the origin. This also means removing the later saftey check as the branch may not exist locally. This fixes the autobuilder resulttool test pushing code. (From OE-Core rev: 0ee636bc89d93282cb3d3cd747b3dd046f6e5a9c) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-21resulttool: Improvements to allow integration to the autobuilderRichard Purdie
This is a combined patch of the various tweaks and improvements I made to resulttool: * Avoid subprocess.run() as its a python 3.6 feature and we have autobuilder workers with 3.5. * Avoid python keywords as variable names * Simplify dict accesses using .get() * Rename resultsutils -> resultutils to match the resultstool -> resulttool rename * Formalised the handling of "file_name" to "TESTSERIES" which the code will now add into the json configuration data if its not present, based on the directory name. * When we don't have failed test cases, print something saying so instead of an empty table * Tweak the table headers in the report to be more readable (reference "Test Series" instead if file_id and ID instead of results_id) * Improve/simplify the max string length handling * Merge the counts and percentage data into one table in the report since printing two reports of the same data confuses the user * Removed the confusing header in the regression report * Show matches, then regressions, then unmatched runs in the regression report, also remove chatting unneeded output * Try harder to "pair" up matching configurations to reduce noise in the regressions report * Abstracted the "mapping" table concept used to pairing in the regression code to general code in resultutils * Created multiple mappings for results analysis, results storage and 'flattening' results data in a merge * Simplify the merge command to take a source and a destination, letting the destination be a directory or a file, removing the need for an output directory parameter * Add the 'IMAGE_PKGTYPE' and 'DISTRO' config options to the regression mappings * Have the store command place the testresults files in a layout from the mapping, making commits into the git repo for results storage more useful for simple comparison purposes * Set the oe-git-archive tag format appropriately for oeqa results storage (and simplify the commit messages closer to their defaults) * Fix oe-git-archive to use the commit/branch data from the results file * Cleaned up the command option help to match other changes * Follow the model of git branch/tag processing used by oe-build-perf-report and use that to read the data using git show to avoid branch change * Add ptest summary to the report command * Update the tests to match the above changes (From OE-Core rev: ff2c029b568f70aa9960dde04ddd207829812ea0) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-21resulttool: enable merge, store, report and regression analysisYeoh Ee Peng
OEQA outputs test results into json files and these files were archived by Autobuilder during QA releases. Example: each oe-selftest run by Autobuilder for different host distro generate a testresults.json file. These scripts were developed as a test result tools to manage these testresults.json file. Using the "store" operation, user can store multiple testresults.json files as well as the pre-configured directories used to hold those files. Using the "merge" operation, user can merge multiple testresults.json files to a target file. Using the "report" operation, user can view the test result summary for all available testresults.json files inside a ordinary directory or a git repository. Using the "regression-file" operation, user can perform regression analysis on testresults.json files specified. Using the "regression-dir" and "regression-git" operations, user can perform regression analysis on directory and git accordingly. These resulttool operations expect the testresults.json file to use the json format below. { "<testresult_1>": { "configuration": { "<config_name_1>": "<config_value_1>", "<config_name_2>": "<config_value_2>", ... "<config_name_n>": "<config_value_n>", }, "result": { "<testcase_namespace_1>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, "<testcase_namespace_2>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, ... "<testcase_namespace_n>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, } }, ... "<testresult_n>": { "configuration": { "<config_name_1>": "<config_value_1>", "<config_name_2>": "<config_value_2>", ... "<config_name_n>": "<config_value_n>", }, "result": { "<testcase_namespace_1>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, "<testcase_namespace_2>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, ... "<testcase_namespace_n>": { "status": "<PASSED or FAILED or ERROR or SKIPPED>", "log": "<failure or error logging>" }, } }, } To use these scripts, first source oe environment, then run the entry point script to look for help. $ resulttool To store test result from oeqa automated tests, execute the below $ resulttool store <source_dir> <git_branch> To merge multiple testresults.json files, execute the below $ resulttool merge <base_result_file> <target_result_file> To report test report, execute the below $ resulttool report <source_dir> To perform regression file analysis, execute the below $ resulttool regression-file <base_result_file> <target_result_file> To perform regression dir analysis, execute the below $ resulttool regression-dir <base_result_dir> <target_result_dir> To perform regression git analysis, execute the below $ resulttool regression-git <source_dir> <base_branch> <target_branch> [YOCTO# 13012] [YOCTO# 12654] (From OE-Core rev: 78a322d7be402a5b9b5abf26ad35670a8535408a) Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-21oe-build-perf-report/gitarchive: Move common useful functions to libraryRichard Purdie
These functions can be reused by the resulttool code so move to the common function library for this purpose. (From OE-Core rev: c66f848938c04e133259c5b6903dc592866ab385) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2019-02-20logrotate.py: restore /etc/logrotate.d/wtmpMingli Yu
During the test logrotate.LogrotateTest.test_1_logrotate_setup, there is below logic: # mkdir $HOME/logrotate_dir # sed -i "s#wtmp {#wtmp {\n olddir $HOME/logrotate_dir#" /etc/logrotate.d/wtmp After all logrotate.LogrotateTest finished, only cleanup $HOME/logrotate_dir as below, but don't restore the config file /etc/logrotate.d/wtmp. [snip] def tearDownClass(cls): cls.tc.target.run('rm -rf $HOME/logrotate_dir') [snip] That's to say, there is one additional line added to /etc/logrotate.d/wtmp and will make the logrotate service start failed when run systemd.SystemdBasicTests.test_systemd_failed Take an example as below when run test as root: # cat /etc/logrotate.d/wtmp # no packages own wtmp -- we'll rotate it here /var/log/wtmp { olddir /root/logrotate_dir missingok monthly create 0664 root utmp minsize 1M rotate 1 } # ls /root/logrotate_dir ls: cannot access '/root/logrotate_dir': No such file or directory # systemctl start logrotate Job for logrotate.service failed because the control process exited with error code. See "systemctl status logrotate.service" and "journalctl -xe" for details. # systemctl status logrotate logrotate.service - Rotate log files Loaded: loaded (/lib/systemd/system/logrotate.service; static; vendor preset> Active: failed (Result: exit-code) since Wed 2019-02-13 03:35:19 UTC; 7s ago Docs: man:logrotate(8) man:logrotate.conf(5) Process: 540 ExecStart=/usr/sbin/logrotate /etc/logrotate.conf (code=exited, status=1/FAILURE) Main PID: 540 (code=exited, status=1/FAILURE) Feb 13 03:35:18 qemumips systemd[1]: Starting Rotate log files... Feb 13 03:35:19 qemumips logrotate[540]: error: wtmp:9 error verifying olddir path /root/logrotate_dir: No such file or directory Feb 13 03:35:19 qemumips logrotate[540]: error: found error in file wtmp, skipping Feb 13 03:35:19 qemumips systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE Feb 13 03:35:19 qemumips systemd[1]: logrotate.service: Failed with result 'exit-code'. Feb 13 03:35:19 qemumips systemd[1]: Failed to start Rotate log files. Add the logic to restore /etc/logrotate.d/wtmp to make the cleanup complete to fix the above issue. (From OE-Core rev: a2db9320d97d12d87524ff16a329f9c38a8da33f) Signed-off-by: Mingli Yu <Mingli.Yu@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit 1.2.3-korg (git 2.39.0) at 2024-05-07 01:23:42 +0000