path: root/README.md

PLEASE DO NOT COMMIT TEST RESULTS TO MASTER.
============================================

The Purpose of this Repo
========================

This repo contains QA testresults for Yocto releases. Each release has its own
branch on which test results reside. This collateral is used to generate a
merged test report (testreport.txt) that is published as part of the release
artefacts.

Pleae note that this repo is intended for release engineering automation and is
not intended to be a public archive for QA test results. The repo is intended
to provide a place to stage test reaults from QA teams in a predictable format
for use with release engineering scripts. Hence it is not a version repo
intended for public consumption. Official testresults are published on
http://downloads.yoctoproject.org/releases/yocto/ in the respective release
directories, and in machine readible format in
https://git.yoctoproject.org/git/yocto-testresults.

While we do tag this staging repo at release to flag the results used for each
release, it is not meant to be the defacto source for official test results for 
community consumption. 

Maintainers and Contact Info
============================

This repo is owned and managed by The Yocto Project Release Engineering team.

Maintainers
-----------
Tracy Graydon <tracy.graydon@intel.com>
Vineela Tummalapalli <vineela.tummalapalli@intel.com>
Michael Halstead aka Deus Ex Machina <mhalstead@linuxfoundation.org>

If you are part of a Yocto Project-related QA team and need access to this
repo, please contact Michael.

For questions about this repo, the release process, etc., please contact
Vineela or Tracy.


How To Use This Repo
====================

For each release line (warrior, thud, etc) there will be a corresponding branch
used to archive the related QA testresults. Nothing should be committed on
master. It is intentionally "empty," aside from this README. 

Each successive release on a given release line (i.e. 2.5.x, 2.6.x, 2.7.x) will
be committed on the same branch. We follow the branching stragegy used for the
Poky repo: https://git.yoctoproject.org/cgit/cgit.cgi/poky/. 

At release, the branch will be tagged with the release name so that it is
trackable and easy to pull only the collateral for a given release. This is the
same tagging approach used for poky and other Yocto Project repos.

i.e. tag format: yocto-2.7, yocto-2.6.2, etc. 

Branching
=========

A new release line branch is created by branching off the "empty" master
branch. Only this README.md file should exist on master. All test results are
to be commited on their respective branch.

QA teams may create a new release branch if the one they need does not already exist.
If issues are encountered or help is needed, or you just don't wanna', release engineering
will happily do it. Please see contact information above. We permit QA teams to
create a branch as needed so as to prevent RE from being gating to teams
wanting to just go ahead and get things in. Hence, QA teams are free to do so,
if they need/want to.

As noted above, we tag the branch at the time of release so we always have a 
quick way to see what was in the previous release. Again, this repo is 
basically just to facilitate RE automation/scripts, and is not intended to
be a comprehensive, versioned archive of QA team results. Given the sequential 
nature and timing of releases on a given branch, there is virtually no
danger of different release testresults colliding.


Create a Branch
---------------

First checkout the master branch to branch from. Please do not branch off of other release branches.

$ git checkout master
$ git branch <release_codename>
$ git checkout <release_codename>

i.e.
$ git checkout master
$ git branch warrior
$ git checkout warrior

Alternatively you can also use:

$ git checkout master
$ git checkout -b <release_codename>


Branch Exists
-------------

If the release branch you require already exists, you only need to check it out
and add your files "on top" and push them. We "clobber" existing testresults.


Add Test Results
----------------

After getting the correct branch, you can then add your files to the repo. We
are going to assume from here that someone has a basic understanding of how to
commit changes to a git repo. If you get stuck or run into issues, the RE team
is happy to help. 

The actual test results should be in a directory named in the general format
of testresults-<QA_team>.

Example: 

For the Intel QA team, the results directory would be:

testresults-intel

This directory should contain the testresults in json format. To see an example
if what this looks like, take a peek here at the 2.7 release:

http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testresults-intel/

Along with the actual test results, a testreport "header" file with a report summary should be included to
summarize findings. 

This file should be named in the format of:

header-<QA_team>.txt. 

Example, using Intel QA:

header-intel.txt

This header file should include anything relevant that would be sent out in the
QA email to the Yocto mailing list, but in a very generalized format. See
http://downloads.yoctoproject.org/releases/yocto/yocto-2.7/testreport.txt for
an example.

The very top portion of that testreport.txt file is the "header."

Example from the testreport.txt file for the 2.7 release:

------------
QA cycle report for 2.7 RC2:

No high milestone defects.
No new defects are found in this cycle.
No existing issues observed in this release
No ptests are failing in this release which were passing in previous release.
No timeout issues.

QA Hint: One failure was observed on public AB:

testseries | result_id : qemuppc | oeselftest_centos-7_qemuppc_20190414221329
runqemu.QemuTest.test_qemu_can_shutdown
No bug filed for this issue, as it appears to be an intermittent failure and
worked on rc1 test run.

------------

The report should not reference this repo, nor any links on the Yocto
Autobuilder, such as https://autobuilder.yocto.io/pub/releases/<release>.
Again, this repo is not intended for community consumption, and links like the
above vanish at release time and will not exist once the release is announced. 
Links to Yocto bugs and the like are fine. 

What RE Does
============

Release engineering takes the collateral in the repo and does some basic sanity
checking to ensure the naming/formatting is consistent, the content looks
reasonable, the testreports contain the correct poky hash for the release, and
there are no other glaring issues. 

If everything passes muster, the testresults are placed in the appropriately
named subdirectory within the release directory. A tool is run to take the
various testresults subdirectory contents, and turn everything into a "master"
test report (testrport.txt). All of this is published with the release.

After that is done, RE pushes the testresults, in machine readable format, to
the yocto-testresults repo: https://git.yoctoproject.org/git/yocto-testresults.

The yocto-testresults-contrib repo is tagged with the release tag, should we 
have a need to refer back, etc.

This workflow is being automated with scripts as part of the release process.

Please see the contact list above and let us know if you have questions or run
into any issues.