aboutsummaryrefslogtreecommitdiffstats

PLEASE DO NOT COMMIT TEST RESULTS TO MASTER.

The Purpose of this Repo

This repo contains QA testresults for Yocto releases. Each release has its own branch on which test results reside. This collateral is used to generate a merged test report (testreport.txt) that is published as part of the release artefacts.

Pleae note that this repo is intended for release engineering automation and is not intended to be a public archive for QA test results. The repo is intended to provide a place to stage test reaults from QA teams in a predictable format for use with release engineering scripts. Hence it is not a version repo intended for public consumption. Official testresults are published on http://downloads.yoctoproject.org/releases/yocto/ in the respective release directories, and in machine readible format in https://git.yoctoproject.org/git/yocto-testresults.

While we do tag this staging repo at release to flag the results used for each release, it is not meant to be the defacto source for official test results for community consumption.

Maintainers and Contact Info

This repo is owned and managed by The Yocto Project Release Engineering team.

Maintainers

Tracy Graydon tracy.graydon@intel.com Vineela Tummalapalli vineela.tummalapalli@intel.com Michael Halstead aka Deus Ex Machina mhalstead@linuxfoundation.org

If you are part of a Yocto Project-related QA team and need access to this repo, please contact Michael.

For questions about this repo, the release process, etc., please contact Vineela or Tracy.

How To Use This Repo

For each release line (warrior, thud, etc) there will be a corresponding branch used to archive the related QA testresults. Nothing should be committed on master. It is intentionally "empty," aside from this README.

Each successive release on a given release line (i.e. 2.5.x, 2.6.x, 2.7.x) will be committed on the same branch. We follow the branching stragegy used for the Poky repo: https://git.yoctoproject.org/cgit/cgit.cgi/poky/.

At release, the branch will be tagged with the release name so that it is trackable and easy to pull only the collateral for a given release. This is the same tagging approach used for poky and other Yocto Project repos.

i.e. tag format: yocto-2.7, yocto-2.6.2, etc.

Branching

A new release line branch is created by branching off the "empty" master branch. Only this README.md file should exist on master. All test results are to be commited on their respective branch.

QA teams may create a new release branch if the one they need does not already exist. If issues are encountered or help is needed, or you just don't wanna', release engineering will happily do it. Please see contact information above. We permit QA teams to create a branch as needed so as to prevent RE from being gating to teams wanting to just go ahead and get things in. Hence, QA teams are free to do so, if they need/want to.

As noted above, we tag the branch at the time of release so we always have a quick way to see what was in the previous release. Again, this repo is basically just to facilitate RE automation/scripts, and is not intended to be a comprehensive, versioned archive of QA team results. Given the sequential nature and timing of releases on a given branch, there is virtually no danger of different release testresults colliding.

Create a Branch

First checkout the master branch to branch from. Please do not branch off of other release branches.

$ git checkout master $ git branch $ git checkout

i.e. $ git checkout master $ git branch warrior $ git checkout warrior

Alternatively you can also use:

$ git checkout master $ git checkout -b

Branch Exists

If the release branch you require already exists, you only need to check it out and add your files "on top" and push them. We "clobber" existing testresults.

Add Test Results

After getting the correct branch, you can then add your files to the repo. We are going to assume from here that someone has a basic understanding of how to commit changes to a git repo. If you get stuck or run into issues, the RE team is happy to help.

The actual test results should be in a directory named in the general format of testresults-.

Example:

For the Intel QA team, the results directory would be:

testresults-intel

This directory should contain the testresults in json format. To see an example if what this looks like, take a peek here at the 2.7 release:

http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults-intel/

Along with the actual test results, a testreport "header" file with a report summary should be included to summarize findings.

This file should be named in the format of:

header-.txt.

Example, using Intel QA:

header-intel.txt

This header file should include anything relevant that would be sent out in the QA email to the Yocto mailing list, but in a very generalized format. See http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testreport.txt for an example.

The very top portion of that testreport.txt file is the "header."

Example from the testreport.txt file for the 2.7 release:


QA cycle report for 2.7 RC2:

No high milestone defects. No new defects are found in this cycle. No existing issues observed in this release No ptests are failing in this release which were passing in previous release. No timeout issues.

QA Hint: One failure was observed on public AB:

testseries | result_id : qemuppc | oeselftest_centos-7_qemuppc_20190414221329 runqemu.QemuTest.test_qemu_can_shutdown No bug filed for this issue, as it appears to be an intermittent failure and worked on rc1 test run.


The report should not reference this repo, nor any links on the Yocto Autobuilder, such as https://autobuilder.yocto.io/pub/releases/. Again, this repo is not intended for community consumption, and links like the above vanish at release time and will not exist once the release is announced. Links to Yocto bugs and the like are fine.

What RE Does

Release engineering takes the collateral in the repo and does some basic sanity checking to ensure the naming/formatting is consistent, the content looks reasonable, the testreports contain the correct poky hash for the release, and there are no other glaring issues.

If everything passes muster, the testresults are placed in the appropriately named subdirectory within the release directory. A tool is run to take the various testresults subdirectory contents, and turn everything into a "master" test report (testrport.txt). All of this is published with the release.

After that is done, RE pushes the testresults, in machine readable format, to the yocto-testresults repo: https://git.yoctoproject.org/git/yocto-testresults.

The yocto-testresults-contrib repo is tagged with the release tag, should we have a need to refer back, etc.

This workflow is being automated with scripts as part of the release process.

Please see the contact list above and let us know if you have questions or run into any issues.