| Age | Commit message (Collapse) | Author |
|---|
|
The parent code currently assumed that any parent branch is locally
checked out which may not be the case.
Use the local branch by default but fall back to the origin. This
also means removing the later saftey check as the branch may not exist
locally.
This fixes the autobuilder resulttool test pushing code.
(From OE-Core rev: 633e024f5d492ba38debf745b59813f6a2f4a3f0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
These functions can be reused by the resulttool code so move to the common
function library for this purpose.
(From OE-Core rev: c66f848938c04e133259c5b6903dc592866ab385)
(From OE-Core rev: 94a3ca85d30fc957f8f01a216a75342be49f9143)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This turns the core of the script into a library function. Ultimately this
will let us call that code with custom 'keywords' rather than relying
on the data parsed from bitbake metadata which can't be used when archiving
historical results.
(From OE-Core rev: 4820ca2b0850e29b04a4fd5659a6e9837d6714d0)
(From OE-Core rev: ab5d49ba8bb02410fb9f3024f98ff19a04acb3ca)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This is a combined patch of the various tweaks and improvements I
made to resulttool:
* Avoid subprocess.run() as its a python 3.6 feature and we
have autobuilder workers with 3.5.
* Avoid python keywords as variable names
* Simplify dict accesses using .get()
* Rename resultsutils -> resultutils to match the resultstool ->
resulttool rename
* Formalised the handling of "file_name" to "TESTSERIES" which the code
will now add into the json configuration data if its not present, based
on the directory name.
* When we don't have failed test cases, print something saying so
instead of an empty table
* Tweak the table headers in the report to be more readable (reference
"Test Series" instead if file_id and ID instead of results_id)
* Improve/simplify the max string length handling
* Merge the counts and percentage data into one table in the report
since printing two reports of the same data confuses the user
* Removed the confusing header in the regression report
* Show matches, then regressions, then unmatched runs in the regression
report, also remove chatting unneeded output
* Try harder to "pair" up matching configurations to reduce noise in
the regressions report
* Abstracted the "mapping" table concept used to pairing in the
regression code to general code in resultutils
* Created multiple mappings for results analysis, results storage and
'flattening' results data in a merge
* Simplify the merge command to take a source and a destination,
letting the destination be a directory or a file, removing the need for
an output directory parameter
* Add the 'IMAGE_PKGTYPE' and 'DISTRO' config options to the regression
mappings
* Have the store command place the testresults files in a layout from
the mapping, making commits into the git repo for results storage more
useful for simple comparison purposes
* Set the oe-git-archive tag format appropriately for oeqa results
storage (and simplify the commit messages closer to their defaults)
* Fix oe-git-archive to use the commit/branch data from the results file
* Cleaned up the command option help to match other changes
* Follow the model of git branch/tag processing used by oe-build-perf-report
and use that to read the data using git show to avoid branch change
* Add ptest summary to the report command
* Update the tests to match the above changes
(From OE-Core rev: e4195565d2a50046d4378c97f7a593c41bed51bd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
OEQA outputs test results into json files and these files were
archived by Autobuilder during QA releases. Example: each oe-selftest
run by Autobuilder for different host distro generate a
testresults.json file.
These scripts were developed as a test result tools to manage
these testresults.json file.
Using the "store" operation, user can store multiple testresults.json
files as well as the pre-configured directories used to hold those files.
Using the "merge" operation, user can merge multiple testresults.json
files to a target file.
Using the "report" operation, user can view the test result summary
for all available testresults.json files inside a ordinary directory
or a git repository.
Using the "regression-file" operation, user can perform regression
analysis on testresults.json files specified. Using the "regression-dir"
and "regression-git" operations, user can perform regression analysis
on directory and git accordingly.
These resulttool operations expect the testresults.json file to use
the json format below.
{
"<testresult_1>": {
"configuration": {
"<config_name_1>": "<config_value_1>",
"<config_name_2>": "<config_value_2>",
...
"<config_name_n>": "<config_value_n>",
},
"result": {
"<testcase_namespace_1>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
"<testcase_namespace_2>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
...
"<testcase_namespace_n>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
}
},
...
"<testresult_n>": {
"configuration": {
"<config_name_1>": "<config_value_1>",
"<config_name_2>": "<config_value_2>",
...
"<config_name_n>": "<config_value_n>",
},
"result": {
"<testcase_namespace_1>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
"<testcase_namespace_2>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
...
"<testcase_namespace_n>": {
"status": "<PASSED or FAILED or ERROR or SKIPPED>",
"log": "<failure or error logging>"
},
}
},
}
To use these scripts, first source oe environment, then run the
entry point script to look for help.
$ resulttool
To store test result from oeqa automated tests, execute the below
$ resulttool store <source_dir> <git_branch>
To merge multiple testresults.json files, execute the below
$ resulttool merge <base_result_file> <target_result_file>
To report test report, execute the below
$ resulttool report <source_dir>
To perform regression file analysis, execute the below
$ resulttool regression-file <base_result_file> <target_result_file>
To perform regression dir analysis, execute the below
$ resulttool regression-dir <base_result_dir> <target_result_dir>
To perform regression git analysis, execute the below
$ resulttool regression-git <source_dir> <base_branch> <target_branch>
[YOCTO# 13012]
[YOCTO# 12654]
(From OE-Core rev: bb0bc6368bb51ac0be77d13fe931601d493951ea)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
During the test logrotate.LogrotateTest.test_1_logrotate_setup,
there is below logic:
# mkdir $HOME/logrotate_dir
# sed -i "s#wtmp {#wtmp {\n olddir $HOME/logrotate_dir#" /etc/logrotate.d/wtmp
After all logrotate.LogrotateTest finished, only cleanup
$HOME/logrotate_dir as below, but don't restore
the config file /etc/logrotate.d/wtmp.
[snip]
def tearDownClass(cls):
cls.tc.target.run('rm -rf $HOME/logrotate_dir')
[snip]
That's to say, there is one additional line added
to /etc/logrotate.d/wtmp and will make the logrotate
service start failed when run systemd.SystemdBasicTests.test_systemd_failed
Take an example as below when run test as root:
# cat /etc/logrotate.d/wtmp
# no packages own wtmp -- we'll rotate it here
/var/log/wtmp {
olddir /root/logrotate_dir
missingok
monthly
create 0664 root utmp
minsize 1M
rotate 1
}
# ls /root/logrotate_dir
ls: cannot access '/root/logrotate_dir': No such file or directory
# systemctl start logrotate
Job for logrotate.service failed because the control process exited with error code.
See "systemctl status logrotate.service" and "journalctl -xe" for details.
# systemctl status logrotate
logrotate.service - Rotate log files
Loaded: loaded (/lib/systemd/system/logrotate.service; static; vendor preset>
Active: failed (Result: exit-code) since Wed 2019-02-13 03:35:19 UTC; 7s ago
Docs: man:logrotate(8)
man:logrotate.conf(5)
Process: 540 ExecStart=/usr/sbin/logrotate /etc/logrotate.conf (code=exited, status=1/FAILURE)
Main PID: 540 (code=exited, status=1/FAILURE)
Feb 13 03:35:18 qemumips systemd[1]: Starting Rotate log files...
Feb 13 03:35:19 qemumips logrotate[540]: error: wtmp:9 error verifying olddir path /root/logrotate_dir: No such file or directory
Feb 13 03:35:19 qemumips logrotate[540]: error: found error in file wtmp, skipping
Feb 13 03:35:19 qemumips systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Feb 13 03:35:19 qemumips systemd[1]: logrotate.service: Failed with result 'exit-code'.
Feb 13 03:35:19 qemumips systemd[1]: Failed to start Rotate log files.
Add the logic to restore /etc/logrotate.d/wtmp to
make the cleanup complete to fix the above issue.
(From OE-Core rev: a2db9320d97d12d87524ff16a329f9c38a8da33f)
(From OE-Core rev: a5dcf0c4bd9a0be2f2484b0b1b3e77f1f2128dfc)
Signed-off-by: Mingli Yu <Mingli.Yu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The test works by excluding curl-dev which curl-staticdev depends upon.
When static libraries aren't disabled, this leads to an odd looking test
failure.
Simply exclude curl-staticdev as well in case its enabled to make sure
the test always works.
(From OE-Core rev: dfded083d9456c8d2bb168dbe8e08b2ffb3e1a26)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
oe-selftest's devtool tests have been broken since commit 2457cd57
(oe-selftest: devtool: avoid parallel races by using temporary copy of
core) if meta is a symbolic link.
(From OE-Core rev: daba6c5a991b370709d17e51305334f55a3858ec)
(From OE-Core rev: 3eb59559ecd2e93fb590a330b47de1db0750fc0b)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Some of the devtool tests make changes to files under meta/ -
legitimately since we want these tests to be working with real recipes
and associated files. Unfortunately with the new oe-selftest
parallelisation this can break other tests if files go missing at the
wrong time (among other scenarios). To avoid this issue, simply take a
copy of the core repository and use that for these tests. (We copy the
entire repository since changing the path of meta/ influences COREBASE
and thus we need to have things like scripts/ alongside as well).
(From OE-Core rev: 2457cd57b4195924ef127f497efa2f34f411e660)
(From OE-Core rev: 9cb8353a4f0137823d6ed3e467db9dd7ead7b3de)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
testimage.bbclass installs a SIGTERM handler which conflicts with the
use of multiprocessing here. This is paritcularly problematic if the http
service is terminated before its started and hence before its had a chance
to reset the default signal handler (as the code was written).
Instead, temporarily remove testimage's handler whilst forking the http process
which means the correct handler is installed and won't deadlock.
Also take the opportunity to add in some log messages about the server start
and shutdown so that future debugging is easier and its clearer what the code
is doing.
(From OE-Core rev: cc0471439aa0085ca87deccf061c5b676ef12388)
(From OE-Core rev: 4eac9a5337d93b6cbd3916af97f62bb04881c9cd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
During debugging dnf issue, we found that the test_dnf_install
PASSED the testing even though the environment does not allow
dnf install to run successfully. Further debugging had identified
that current test_dnf_install will execute dnf install even when
the package to be installed already exist, thus dnf install
will just skipped and this test will PASSED even though it was
not.
To solve this, added additional logic to check if the package
to be installed already exist, if yes then remove the package
before actually run dnf install. This will make sure dnf install
was tested as expected.
(From OE-Core rev: 4f662b253f7313c4e02bfafb527cdac076b6309a)
(From OE-Core rev: a80498e62898110e2ed6b01cbb9f5dd85995d13b)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
fixes:
DEBUG: [Running]$ ssh -l root -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o LogLevel=ERROR 192.168.7.4 export PATH=/usr/sbin:/sbin:/usr/bin:/bin; perl -e '$_="Uryyb, jbeyq"; tr/a-zA-Z/n-za-mN-ZA-M/;print'
| DEBUG: time: 1548816904.4024463, endtime: 1548817204.397057
| DEBUG: Partial data from SSH call: ssh: connect to host 192.168.7.4 port 22: Connection refused
for master/thud/sumo
(From OE-Core rev: 76c66e061cdcdcbad73ed503668115120feb0ea3)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
As with the Python test, this can be both better and faster. No need to copy a
file, just run a one-liner.
(From OE-Core rev: c6eef46747fe58bb2310be4f06d2fa9b67901d72)
(From OE-Core rev: 9188ef8d1edbba8041a73d3bb8a9bfd194db0e92)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We should only be wiping out things on target if the tests have
actually run.
(From OE-Core rev: d38c3eac0a5a1a9b0eb98385832e92f48145655e)
(From OE-Core rev: dab22dc58eabaeb421afa3c7de1cc08c5ec34c61)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
(From OE-Core rev: 564de3681353fe8e203425388e8be9703a89d2da)
(From OE-Core rev: 1207949c986cc9f6b3940a99e87bc4d8cfb86d5b)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Qemu do not use the ip input from external. It will
retrieve ip from QemuRunner instance and assign
ip value.
(From OE-Core rev: 14d99dc6c39c963ba3e0d9a30274846bd5369210)
(From OE-Core rev: e4990ae01c9f3f486b4b745a2602795e9d496109)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Mark up these tests as needing a compiler, make and kernel source code
as appropriate, the image feature requirements can then be retired.
(From OE-Core rev: 9f64e6c25abdf494fb511e9cd401f8dcaa08be2a)
(From OE-Core rev: 544c56b6950f0598d5eaf62d9ae9d9de5fcb7eaf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add the OEHasPackage decorator to a variety of tests so they determine
automatically if they should run against a given image.
To ensure tests can do this we need to move target operations such
as scp commands into the tests and out of the class startup/teardown.
(From OE-Core rev: 60d6580b85714b8960a964e775d76a7f937f5e5a)
(From OE-Core rev: 03b7658369bb7c1c8fbbaac7d9e281617cc16135)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently this is three test cases:
1) test_python_exists. Fail if python3 isn't in PATH.
2) test_python_stdout. Run a Python script and check the output is as expected
3) test_python_testfile. Check that a file test_python_stdout wrote to exists.
(1) should be a setup and skip the test module if it isn't present.
(2) and (3) should be merged, there's no point copying over a two line Python
file, and the test doesn't verify that the file doesn't exist in the first
place.
Rewrite the test to check that Python is present in a class setup so the entire
test is skipped if it isn't and do some simple rot13 to verify that bytecode is
being executed correctly.
(From OE-Core rev: a35be5f32b4fe70b18ac1e2eccfd94558cecfbba)
(From OE-Core rev: babdd2f6addb06c23e4882107be07034ca49bcb8)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
For example, core-image-sato skipped the test alltogether, as it
no longer pulls in Python 2.x at all.
(From OE-Core rev: 5ad0fe9ac6b6362011a17afaa7bee8e788093915)
(From OE-Core rev: bde50fc78ae75fd585f8914e458d65c328857fb2)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The test cases assume that rpm-doc is built out, as it says it its
comment. This is not always true. And it sometimes results in
following error.
| cls.tc.target.copyTo(test_file, dst)
| UnboundLocalError: local variable 'test_file' referenced before assignment
Change to use base-passwd-doc, as this package is more likely to be
built out than rpm-doc.
(From OE-Core rev: 2e23543b48921182307065c1fa9e8b9d7fbb3cdc)
(From OE-Core rev: 1ce378ce07d2c49f40054893a623456c8471e177)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
As I will be leaving Intel, this address will no longer be valid,
so swap it for my personal one for now.
(From OE-Core rev: ea58ff101e28dfda3410de66d775df3d8a1e5a96)
(From OE-Core rev: 61bca5a7f6b3fc03cd88cbf8867708d28e0a3522)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There are failures when multiple users run oe-selftest on the same
host:
PermissionError: [Errno 13] Permission denied: '/tmp/oe-saved-
tests/201812250324_qemu'
This is because /tmp/oe-saved-tests was created by user A, while user B tries
to write data in it, then the error will happen. This patch can fix the
problem.
Move the dumped data to ${LOG_DIR}/runtime-hostdump/ rather than
/tmp/oe-saved-tests/ to fix the problem.
(From OE-Core rev: e219fe5329599cd6c3682f521eaee3852a2c8980)
(From OE-Core rev: 872c6e5101f4f27dcfc63d141d1b6568f46d2b5f)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add test cases to test “exclude” and “installroot“ options, also modify
the logic of filtering packages on the feed to have all the packages
needed by the tests.
[YOCTO #10744]
(From OE-Core rev: 1121806603c6f621d084b692216f3f616a0768dc)
(From OE-Core rev: e1b050f53ece2a31cd6866d2d737d7c67a44cea4)
Signed-off-by: Jose Perez Carranza <jose.perez.carranza@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fixed:
MACHINE = "qemux86"
QEMU_USE_KVM = "qemux86"
IMAGE_CLASSES += "testimage"
$ oe-selftest -r runqemu.RunqemuTests.test_boot_rootfs
[snip]
File "/buildarea1/lyang1/poky/meta/lib/oe/types.py", line 122, in boolean
raise ValueError("Invalid boolean value '%s'" % value)
ValueError: Invalid boolean value 'qemux86'
Now QEMU_USE_KVM can only be boolean, can not contain MACHINE any more, kvm
will be enabled if target_arch == build_arch or both of them are x86 archs.
(From OE-Core rev: 7c1a8a624cad8d967635c6cb5f99cf655bde3d44)
(From OE-Core rev: de1b80f7f7b787f6b5b62c576ca6c62d2440031c)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
warnings
Use our own lsb function instead as used elsewhere by the codebase.
(From OE-Core rev: acac45a6fd604d28ef7c23d67482af3d7e8bcfe3)
(From OE-Core rev: 570256a64af5a3fa994a20a5cc4c74d59ffc361f)
(From OE-Core rev: d58fe9d352ae7de857e7f55b88f6e7d35b2cd706)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The 5s timeout for non-kvm is too short, especially when the load is high,
which leads to unexpected errors, so set timeout to 60s by default.
(From OE-Core rev: 8197d0f638a760fc03062c7a9009117d083d7ead)
(From OE-Core rev: 6c930a8f5c11947c3b916efe23f33af462701bee)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This is useful for debugging.
(From OE-Core rev: da527d73f5eeae0f29b5f99aab757491d3f87ec7)
(From OE-Core rev: 7fb953573635afb1b6482dbd9f51eb0f05731eae)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
2018-12-06 23:19:24,564 - oe-selftest - INFO - Traceback (most recent call last):
File "/media/build1/poky-sumo/meta/lib/oeqa/core/case.py", line 32, in _oeSetUpClass
clss.setUpClassMethod()
File "/media/build1/poky-sumo/meta/lib/oeqa/selftest/cases/eSDK.py", line 78, in setUpClass
cls.tmpdir_eSDKQA = cls.tempdirobj.name
AttributeError: type object 'oeSDKExtSelfTest' has no attribute 'tempdirobj'
(From OE-Core rev: 2c60908a2039d333a9fe2651622750ff6ed4cce1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
INFO - ======================================================================
INFO - ERROR: setUpClass (eSDK.oeSDKExtSelfTest)
INFO - ----------------------------------------------------------------------
INFO - Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/core/case.py", line 32, in _oeSetUpClass
clss.setUpClassMethod()
File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/eSDK.py", line 76, in setUpClass
cls.tmpdirobj = tempfile.TemporaryDirectory(prefix="selftest-esdk-", dir=bb_vars["WORKDIR"])
File "/usr/lib/python3.5/tempfile.py", line 929, in __init__
self.name = mkdtemp(suffix, prefix, dir)
File "/usr/lib/python3.5/tempfile.py", line 507, in mkdtemp
_os.mkdir(file, 0o700)
FileNotFoundError: [Errno 2] No such file or directory: '/home/pokybuild/yocto-worker/oe-selftest-debian/build/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/selftest-esdk-q7ln84gc'
(From OE-Core rev: 170a601a99836d13b69e5287bee0d3e71983dd46)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
break tests
Having image-buildinfo enabled causes containerimage.ContainerImageTests.test_expected_files
to fail due to the presence of an unexpected file:
['./',
'./etc/',
- './etc/build',
'./etc/default/',
'./etc/default/postinst',
Tweak the class to allow it to be disabled and disable it from the test just in
case it was enabled.
(From OE-Core rev: af67bf422a4df5b7e07894512ff73a5f493682ab)
(From OE-Core rev: f49ab8b1610c045acaed7b964d12f07f969df856)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The python git module may or may not be enabled, allow this code to
function without it, falling back to the same method as metadata_scm.bbclass
uses. This will be cleaned up in the next round of feature development.
(From OE-Core rev: 6350586ba9f4a4107a2d457590824cd4d662d5b9)
(From OE-Core rev: 32c9169b76e13e53b6a9ab4a59932cea7863d992)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently sdk & sdkext will output json file to LOG_DIR, while
selftest will output json file to TOPDIR/log.
Standardize selftest json output file to LOG_DIR.
(From OE-Core rev: 9f2e39684cbbe9f87eeef6a81961e6db783439e3)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Tweak the preceeding commit to:
* Add STARTTIME to the identifier to make it unique
* Use LOG_DIR
* Store the layer config in a more natural json format
* Drop '_' function prefixes
(From OE-Core rev: 173f59acf9722e2ef27fdd49c20f7d3d664917eb)
(From OE-Core rev: 3b69099edc7db99c11bfb41eab2af50bd0e3d4f2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
As part of the solution to replace Testopia to store testresult,
OEQA selftest need to output testresult into json files, where
these json testresult files will be stored into git repository
by the future test-case-management tools.
By default, json testresult file will be written to "oeqa"
directory under the oe-selftest log directory.
To configure multiple instances of bitbake to write json testresult
to a single testresult file at custom directory, user will define
the variable "OEQA_JSON_RESULT_DIR" with the custom directory for
json testresult.
(From OE-Core rev: a95218525a4c8228fff9908ffbda85c6b85e101c)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We want to see failures/errors listed last since this is the most easily
visible part of the log on consoles or autobuilder output and makes
human processing easier rather than having to scroll up and scan for
a single failure.
(From OE-Core rev: 2cc07ab253f1ba6a1f07a66051c9ba6d98cd2357)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Its useful to have the counts of success/failure/error/skipped at the end of the
results to allow for easier human reading of what happened.
(From OE-Core rev: 5942318a261ce7a885f351e214669068ff9d8931)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This allows the ptest results from ptest-runner, run in an image to be
transferred over to the resulting json results output.
Each test is given a pass/skip/fail so individual results can be monitored
and the raw log output from the ptest-runner is also dumped into the
results json file as this means after the fact debugging becomes much easier.
Currently the log output is not split up per test but that would make a good
future enhancement.
I attempted to implement this as python subTests however it failed as the
output was too confusing, subTests don't support any kind of log
output handling, subTest successes aren't logged and it was making things
far more complex than they needed to be.
We mark ptest-runner as "EXPECTEDFAILURE" since its unlikely every ptest
will pass currently and we don't want that to fail the whole image test run.
Its assumed there would be later analysis of the json output to determine
regressions. We do have to change the test runner code so that
'unexpectedsuccess' is not a failure.
Also, the test names are manipuated to remove spaces and brackets with
"_" used as a replacement and any duplicate occurrences truncated.
(From OE-Core rev: a13e088942e2a3c3521e98954a394e61a15234e8)
(From OE-Core rev: 526ceab9d0e43f73635bb92e8dd7763ef75ad33b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
As part of the solution to replace Testopia to store testresult,
OEQA need to output testresult into single json file, where json
testresult file will be stored in git repository by the future
test-case-management tools.
The json testresult file will store more than one set of results,
where each set of results was uniquely identified by the result_id.
The result_id would be like "runtime-qemux86-core-image-sato", where
it was a runtime test with target machine equal to qemux86 and running
on core-image-sato image. The json testresult file will only store
the latest test content for a given result_id. The json testresult
file contains the configuration (eg. COMMIT, BRANCH, MACHINE, IMAGE),
result (eg. PASSED, FAILED, ERROR), test log, and result_id.
Based on the destination json testresult file directory provided,
it could have multiple instances of bitbake trying to write json
testresult to a single testresult file, using locking a lockfile
alongside the results file directory to prevent races.
Also the library class inside this patch will be reused by the future
test-case-management tools to write json testresult for manual test
case executed.
(From OE-Core rev: 00e03b5004f1eb6d59295544b3a8620504278f51)
(From OE-Core rev: 045511425577ccbe89d8eb91e2a87e385390cabf)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Refactor the original _getDetailsNotPassed method to return
testresult details (test status and log), which will be reused
by future OEQA code to write json testresult.
Take the opportunity to consolidate and simplify the logic used
to gather test status and log within the TestResult instance.
(From OE-Core rev: 79ee7d1c371a86edeb61c99679985118da657e5d)
(From OE-Core rev: bc444181f9658423856621b2f2c60364642ae5b1)
Signed-off-by: Yeoh Ee Peng <ee.peng.yeoh@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This isn't present on modern distros by default and doesn't work with
testtools, needing multiple code paths in the code. Remove it in favour
of finding a better replacement for results collection/analysis.
(From OE-Core rev: 8001d933a8dc86004db014777f094d718086687d)
(From OE-Core rev: 02449e89d62a7714d30ab6e2e58c476dc9441f7a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There doesn't appear to be any reason we need this _results indirection
any more so remove it.
(From OE-Core rev: b618261811c48ff3b98eab1b340a8cd09ef183c6)
(From OE-Core rev: ab271b49d9b55ea271d519c3a4da0b639a07f0bb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We have several options for parallel processing in oeqa, parallel
execution of modules, threading and mulitple processes for the runners.
After much experimentation is appears the most scalable and least
invasive approach is multiple processes using concurrenttestsuite
from testtools. This means we can drop the current threading code
which is only used by the sdk test execution.
oeqa/decorator/depends: Remove threading code
Revert "oeqa/sdk: Enable usage of OEQA thread mode"
This reverts commit adc434c0636b7dea2ef70c8d2c8e61cdb5c703b1.
Revert "oeqa/core/tests: Add tests of OEQA Threaded mode"
This reverts commit a4eef558c9933eb32413b61ff80a11b999951b40.
Revert "oeqa/core/decorator/oetimeout: Add support for OEQA threaded mode"
This reverts commit d3d4ba902dee8b19fa1054330cffdf73f9b81fe7.
(From OE-Core rev: a98ab5e560e73b6988512fbae5cefe9e42ceed53)
(From OE-Core rev: bb9a85e157e669d7a91c3bbefc8d5138e7b8b6ae)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The current code assumes if something isn't a failure of some
kind, it was a pass. When test case IDs weren't matching, this lead
to very confusing output where things would fail, then be listed as
passing.
This adds code to track successes, ensuring we don't end up in this
position again with unmatched entries being listed as UNKNOWN.
(From OE-Core rev: 4374c296d8963e4f6a1aa7bef7983ad0a1c2fcff)
(From OE-Core rev: bcb2948773d76befef2be787be6d25cf544e49a9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We can directly compare the test case IDs rather than representations,
then if we're using subunit to split the tests, the comparisions still
work as intended.
(From OE-Core rev: 72e5f46f75454ba4c445c65c1cbc616a9e72fc6e)
(From OE-Core rev: 4c53aac5315f9d0a0ed95cbeb48b7704e274e3a9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The existing logfile is simply placed in the current directory. Since the test
changes cwd to BUILDDIR, the symlink to the log can be placed in an invalid
directory. We also see trackbacks if the symlink is invalid.
Improve things by:
* Placing logs in LOG_DIR (or BUILDDIR if unset).
* Using a full path to the log meaning the log and link are placed in the same directory.
* Using lexists instead of exists so invalid symlinks are handled correctly.
(From OE-Core rev: 750ece11bed0e62a11e0003d1d16a81f7c219761)
(From OE-Core rev: 9c4c3c876dd5d224133571fcad1095af1098ae1d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fix the warnings:
meta/lib/oeqa/utils/qemurunner.py:250: DeprecationWarning: invalid escape sequence \.
ips = re.findall("((?:[0-9]{1,3}\.){3}[0-9]{1,3})", cmdline.split("ip=")[1])
meta/lib/oeqa/utils/qemurunner.py:343: DeprecationWarning: invalid escape sequence \-
if re.search("root@[a-zA-Z0-9\-]+:~#", output):
poky/meta/lib/oeqa/utils/qemurunner.py:350: DeprecationWarning: invalid escape sequence \-
if re.search("root@[a-zA-Z0-9\-]+:~#", output):
meta/lib/oeqa/utils/qemurunner.py:448: DeprecationWarning: invalid escape sequence \-
if re.search("[a-zA-Z0-9]+@[a-zA-Z0-9\-]+:~#", data):
by correctly marking the regexs.
(From OE-Core rev: 8e6987735002560fca714f77ea8ece9d4b28f7fa)
(From OE-Core rev: 7eb1f0be82d7ee12f893cdd40384da306fa597a7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Avoid the warning:
meta/lib/oeqa/selftest/context.py:8: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import imp
In this case importlib is a direct replacement.
(From OE-Core rev: db7a60c36a2d3eefc61ae6e1ede01680dc932035)
(From OE-Core rev: 351a7b75959593922909d7e1929a6429a2bf94a7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Avoid warnings such as:
meta/lib/oeqa/utils/commands.py:213: ResourceWarning: unclosed file <_io.BufferedReader name=4>
return runCmd(cmd, ignore_status, timeout, output_log=output_log, **options)
(From OE-Core rev: 6a68c42de08cffbadb59ebda63fa5e19f6e5acef)
(From OE-Core rev: 682d7b2810b235e86a28a8afe034e3853dbe8c45)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Clean up the warning:
meta/lib/oeqa/core/loader.py:27: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
_failed_test_args = inspect.getargspec(unittest.loader._make_failed_test).args
(From OE-Core rev: d2deb66830be2d44532fea3d5db763b57778252a)
(From OE-Core rev: f9ab10bb08446052fd6af2a21f38d8454e466d51)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
Richard Purdie
Yeoh Ee Peng
Mingli Yu
Peter Kjellerstedt
Paul Eggleton
Armin Kuster
Ross Burton
Robert Yang
Alexander Kanavin
Chen Qi
Jose Perez Carranza