diff options
Diffstat (limited to 'documentation/test-manual/intro.rst')
| -rw-r--r-- | documentation/test-manual/intro.rst | 528 |
1 files changed, 528 insertions, 0 deletions
diff --git a/documentation/test-manual/intro.rst b/documentation/test-manual/intro.rst new file mode 100644 index 0000000000..6168ad7700 --- /dev/null +++ b/documentation/test-manual/intro.rst @@ -0,0 +1,528 @@ +.. SPDX-License-Identifier: CC-BY-SA-2.0-UK + +***************************************** +The Yocto Project Test Environment Manual +***************************************** + +Welcome +======= + +Welcome to the Yocto Project Test Environment Manual! This manual is a +work in progress. The manual contains information about the testing +environment used by the Yocto Project to make sure each major and minor +release works as intended. All the project's testing infrastructure and +processes are publicly visible and available so that the community can +see what testing is being performed, how it's being done and the current +status of the tests and the project at any given time. It is intended +that Other organizations can leverage off the process and testing +environment used by the Yocto Project to create their own automated, +production test environment, building upon the foundations from the +project core. + +Currently, the Yocto Project Test Environment Manual has no projected +release date. This manual is a work-in-progress and is being initially +loaded with information from the README files and notes from key +engineers: + +- *yocto-autobuilder2:* This + :yocto_git:`README.md </yocto-autobuilder2/tree/README.md>` + is the main README which detials how to set up the Yocto Project + Autobuilder. The ``yocto-autobuilder2`` repository represents the + Yocto Project's console UI plugin to Buildbot and the configuration + necessary to configure Buildbot to perform the testing the project + requires. + +- *yocto-autobuilder-helper:* This :yocto_git:`README </yocto-autobuilder-helper/tree/README/>` + and repository contains Yocto Project Autobuilder Helper scripts and + configuration. The ``yocto-autobuilder-helper`` repository contains + the "glue" logic that defines which tests to run and how to run them. + As a result, it can be used by any Continuous Improvement (CI) system + to run builds, support getting the correct code revisions, configure + builds and layers, run builds, and collect results. The code is + independent of any CI system, which means the code can work `Buildbot <https://docs.buildbot.net/0.9.15.post1/>`__, + Jenkins, or others. This repository has a branch per release of the + project defining the tests to run on a per release basis. + +Yocto Project Autobuilder Overview +================================== + +The Yocto Project Autobuilder collectively refers to the software, +tools, scripts, and procedures used by the Yocto Project to test +released software across supported hardware in an automated and regular +fashion. Basically, during the development of a Yocto Project release, +the Autobuilder tests if things work. The Autobuilder builds all test +targets and runs all the tests. + +The Yocto Project uses now uses standard upstream +`Buildbot <https://docs.buildbot.net/0.9.15.post1/>`__ (version 9) to +drive its integration and testing. Buildbot Nine has a plug-in interface +that the Yocto Project customizes using code from the +``yocto-autobuilder2`` repository, adding its own console UI plugin. The +resulting UI plug-in allows you to visualize builds in a way suited to +the project's needs. + +A ``helper`` layer provides configuration and job management through +scripts found in the ``yocto-autobuilder-helper`` repository. The +``helper`` layer contains the bulk of the build configuration +information and is release-specific, which makes it highly customizable +on a per-project basis. The layer is CI system-agnostic and contains a +number of Helper scripts that can generate build configurations from +simple JSON files. + +.. note:: + + The project uses Buildbot for historical reasons but also because + many of the project developers have knowledge of python. It is + possible to use the outer layers from another Continuous Integration + (CI) system such as + `Jenkins <https://en.wikipedia.org/wiki/Jenkins_(software)>`__ + instead of Buildbot. + +The following figure shows the Yocto Project Autobuilder stack with a +topology that includes a controller and a cluster of workers: + +.. image:: figures/ab-test-cluster.png + :align: center + +Yocto Project Tests - Types of Testing Overview +=============================================== + +The Autobuilder tests different elements of the project by using +thefollowing types of tests: + +- *Build Testing:* Tests whether specific configurations build by + varying :term:`MACHINE`, + :term:`DISTRO`, other configuration + options, and the specific target images being built (or world). Used + to trigger builds of all the different test configurations on the + Autobuilder. Builds usually cover many different targets for + different architectures, machines, and distributions, as well as + different configurations, such as different init systems. The + Autobuilder tests literally hundreds of configurations and targets. + + - *Sanity Checks During the Build Process:* Tests initiated through + the :ref:`insane <ref-classes-insane>` + class. These checks ensure the output of the builds are correct. + For example, does the ELF architecture in the generated binaries + match the target system? ARM binaries would not work in a MIPS + system! + +- *Build Performance Testing:* Tests whether or not commonly used steps + during builds work efficiently and avoid regressions. Tests to time + commonly used usage scenarios are run through ``oe-build-perf-test``. + These tests are run on isolated machines so that the time + measurements of the tests are accurate and no other processes + interfere with the timing results. The project currently tests + performance on two different distributions, Fedora and Ubuntu, to + ensure we have no single point of failure and can ensure the + different distros work effectively. + +- *eSDK Testing:* Image tests initiated through the following command:: + + $ bitbake image -c testsdkext + + The tests utilize the ``testsdkext`` class and the ``do_testsdkext`` task. + +- *Feature Testing:* Various scenario-based tests are run through the + :ref:`OpenEmbedded Self test (oe-selftest) <ref-manual/ref-release-process:Testing and Quality Assurance>`. We test oe-selftest on each of the main distrubutions + we support. + +- *Image Testing:* Image tests initiated through the following command:: + + $ bitbake image -c testimage + + The tests utilize the :ref:`testimage* <ref-classes-testimage*>` + classes and the :ref:`ref-tasks-testimage` task. + +- *Layer Testing:* The Autobuilder has the possibility to test whether + specific layers work with the test of the system. The layers tested + may be selected by members of the project. Some key community layers + are also tested periodically. + +- *Package Testing:* A Package Test (ptest) runs tests against packages + built by the OpenEmbedded build system on the target machine. See the + :ref:`Testing Packages With + ptest <dev-manual/dev-manual-common-tasks:Testing Packages With ptest>` section + in the Yocto Project Development Tasks Manual and the + ":yocto_wiki:`Ptest </Ptest>`" Wiki page for more + information on Ptest. + +- *SDK Testing:* Image tests initiated through the following command:: + + $ bitbake image -c testsdk + + The tests utilize the :ref:`testsdk <ref-classes-testsdk>` class and + the ``do_testsdk`` task. + +- *Unit Testing:* Unit tests on various components of the system run + through :ref:`bitbake-selftest <ref-manual/ref-release-process:Testing and Quality Assurance>` and + :ref:`oe-selftest <ref-manual/ref-release-process:Testing and Quality Assurance>`. + +- *Automatic Upgrade Helper:* This target tests whether new versions of + software are available and whether we can automatically upgrade to + those new versions. If so, this target emails the maintainers with a + patch to let them know this is possible. + +How Tests Map to Areas of Code +============================== + +Tests map into the codebase as follows: + +- *bitbake-selftest:* + + These tests are self-contained and test BitBake as well as its APIs, + which include the fetchers. The tests are located in + ``bitbake/lib/*/tests``. + + From within the BitBake repository, run the following:: + + $ bitbake-selftest + + To skip tests that access the Internet, use the ``BB_SKIP_NETTEST`` + variable when running "bitbake-selftest" as follows:: + + $ BB_SKIP_NETTEST=yes bitbake-selftest + + The default output is quiet and just prints a summary of what was + run. To see more information, there is a verbose option:: + + $ bitbake-selftest -v + + Use this option when you wish to skip tests that access the network, + which are mostly necessary to test the fetcher modules. To specify + individual test modules to run, append the test module name to the + "bitbake-selftest" command. For example, to specify the tests for the + bb.data.module, run:: + + $ bitbake-selftest bb.test.data.module + + You can also specify individual tests by defining the full name and module + plus the class path of the test, for example:: + + $ bitbake-selftest bb.tests.data.TestOverrides.test_one_override + + The tests are based on `Python + unittest <https://docs.python.org/3/library/unittest.html>`__. + +- *oe-selftest:* + + - These tests use OE to test the workflows, which include testing + specific features, behaviors of tasks, and API unit tests. + + - The tests can take advantage of parallelism through the "-j" + option, which can specify a number of threads to spread the tests + across. Note that all tests from a given class of tests will run + in the same thread. To parallelize large numbers of tests you can + split the class into multiple units. + + - The tests are based on Python unittest. + + - The code for the tests resides in + ``meta/lib/oeqa/selftest/cases/``. + + - To run all the tests, enter the following command:: + + $ oe-selftest -a + + - To run a specific test, use the following command form where + testname is the name of the specific test:: + + $ oe-selftest -r <testname> + + For example, the following command would run the tinfoil + getVar API test:: + + $ oe-selftest -r tinfoil.TinfoilTests.test_getvar + + It is also possible to run a set + of tests. For example the following command will run all of the + tinfoil tests:: + + $ oe-selftest -r tinfoil + +- *testimage:* + + - These tests build an image, boot it, and run tests against the + image's content. + + - The code for these tests resides in ``meta/lib/oeqa/runtime/cases/``. + + - You need to set the :term:`IMAGE_CLASSES` variable as follows:: + + IMAGE_CLASSES += "testimage" + + - Run the tests using the following command form:: + + $ bitbake image -c testimage + +- *testsdk:* + + - These tests build an SDK, install it, and then run tests against + that SDK. + + - The code for these tests resides in ``meta/lib/oeqa/sdk/cases/``. + + - Run the test using the following command form:: + + $ bitbake image -c testsdk + +- *testsdk_ext:* + + - These tests build an extended SDK (eSDK), install that eSDK, and + run tests against the eSDK. + + - The code for these tests resides in ``meta/lib/oeqa/esdk``. + + - To run the tests, use the following command form:: + + $ bitbake image -c testsdkext + +- *oe-build-perf-test:* + + - These tests run through commonly used usage scenarios and measure + the performance times. + + - The code for these tests resides in ``meta/lib/oeqa/buildperf``. + + - To run the tests, use the following command form:: + + $ oe-build-perf-test <options> + + The command takes a number of options, + such as where to place the test results. The Autobuilder Helper + Scripts include the ``build-perf-test-wrapper`` script with + examples of how to use the oe-build-perf-test from the command + line. + + Use the ``oe-git-archive`` command to store test results into a + Git repository. + + Use the ``oe-build-perf-report`` command to generate text reports + and HTML reports with graphs of the performance data. For + examples, see + :yocto_dl:`/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.html` + and + :yocto_dl:`/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.txt`. + + - The tests are contained in ``lib/oeqa/buildperf/test_basic.py``. + +Test Examples +============= + +This section provides example tests for each of the tests listed in the +:ref:`test-manual/intro:How Tests Map to Areas of Code` section. + +For oeqa tests, testcases for each area reside in the main test +directory at ``meta/lib/oeqa/selftest/cases`` directory. + +For oe-selftest. bitbake testcases reside in the ``lib/bb/tests/`` +directory. + +``bitbake-selftest`` +-------------------- + +A simple test example from ``lib/bb/tests/data.py`` is:: + + class DataExpansions(unittest.TestCase): + def setUp(self): + self.d = bb.data.init() + self.d["foo"] = "value_of_foo" + self.d["bar"] = "value_of_bar" + self.d["value_of_foo"] = "value_of_'value_of_foo'" + + def test_one_var(self): + val = self.d.expand("${foo}") + self.assertEqual(str(val), "value_of_foo") + +In this example, a ``DataExpansions`` class of tests is created, +derived from standard python unittest. The class has a common ``setUp`` +function which is shared by all the tests in the class. A simple test is +then added to test that when a variable is expanded, the correct value +is found. + +Bitbake selftests are straightforward python unittest. Refer to the +Python unittest documentation for additional information on writing +these tests at: https://docs.python.org/3/library/unittest.html. + +``oe-selftest`` +--------------- + +These tests are more complex due to the setup required behind the scenes +for full builds. Rather than directly using Python's unittest, the code +wraps most of the standard objects. The tests can be simple, such as +testing a command from within the OE build environment using the +following example:: + + class BitbakeLayers(OESelftestTestCase): + def test_bitbakelayers_showcrossdepends(self): + result = runCmd('bitbake-layers show-cross-depends') + self.assertTrue('aspell' in result.output, msg = "No dependencies were shown. bitbake-layers show-cross-depends output: %s"% result.output) + +This example, taken from ``meta/lib/oeqa/selftest/cases/bblayers.py``, +creates a testcase from the ``OESelftestTestCase`` class, derived +from ``unittest.TestCase``, which runs the ``bitbake-layers`` command +and checks the output to ensure it contains something we know should be +here. + +The ``oeqa.utils.commands`` module contains Helpers which can assist +with common tasks, including: + +- *Obtaining the value of a bitbake variable:* Use + ``oeqa.utils.commands.get_bb_var()`` or use + ``oeqa.utils.commands.get_bb_vars()`` for more than one variable + +- *Running a bitbake invocation for a build:* Use + ``oeqa.utils.commands.bitbake()`` + +- *Running a command:* Use ``oeqa.utils.commandsrunCmd()`` + +There is also a ``oeqa.utils.commands.runqemu()`` function for launching +the ``runqemu`` command for testing things within a running, virtualized +image. + +You can run these tests in parallel. Parallelism works per test class, +so tests within a given test class should always run in the same build, +while tests in different classes or modules may be split into different +builds. There is no data store available for these tests since the tests +launch the ``bitbake`` command and exist outside of its context. As a +result, common bitbake library functions (bb.\*) are also unavailable. + +``testimage`` +------------- + +These tests are run once an image is up and running, either on target +hardware or under QEMU. As a result, they are assumed to be running in a +target image environment, as opposed to a host build environment. A +simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains +the following:: + + class PythonTest(OERuntimeTestCase): + @OETestDepends(['ssh.SSHTest.test_ssh']) + @OEHasPackage(['python3-core']) + def test_python3(self): + cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" + status, output = self.target.run(cmd) + msg = 'Exit status was not 0. Output: %s' % output + self.assertEqual(status, 0, msg=msg) + +In this example, the ``OERuntimeTestCase`` class wraps +``unittest.TestCase``. Within the test, ``self.target`` represents the +target system, where commands can be run on it using the ``run()`` +method. + +To ensure certain test or package dependencies are met, you can use the +``OETestDepends`` and ``OEHasPackage`` decorators. For example, the test +in this example would only make sense if python3-core is installed in +the image. + +``testsdk_ext`` +--------------- + +These tests are run against built extensible SDKs (eSDKs). The tests can +assume that the eSDK environment has already been setup. An example from +``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following:: + + class DevtoolTest(OESDKExtTestCase): + @classmethod def setUpClass(cls): + myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp") + cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp") + shutil.copytree(myapp_src, cls.myapp_dst) + subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst) + subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst) + subprocess.check_output(['git', 'commit', '-m', "'test commit'"], cwd=cls.myapp_dst) + + @classmethod + def tearDownClass(cls): + shutil.rmtree(cls.myapp_dst) + def _test_devtool_build(self, directory): + self._run('devtool add myapp %s' % directory) + try: + self._run('devtool build myapp') + finally: + self._run('devtool reset myapp') + def test_devtool_build_make(self): + self._test_devtool_build(self.myapp_dst) + +In this example, the ``devtool`` +command is tested to see whether a sample application can be built with +the ``devtool build`` command within the eSDK. + +``testsdk`` +----------- + +These tests are run against built SDKs. The tests can assume that an SDK +has already been extracted and its environment file has been sourced. A +simple example from ``meta/lib/oeqa/sdk/cases/python2.py`` contains the +following:: + + class Python3Test(OESDKTestCase): + def setUp(self): + if not (self.tc.hasHostPackage("nativesdk-python3-core") or + self.tc.hasHostPackage("python3-core-native")): + raise unittest.SkipTest("No python3 package in the SDK") + + def test_python3(self): + cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" + output = self._run(cmd) + self.assertEqual(output, "Hello, world\n") + +In this example, if nativesdk-python3-core has been installed into the SDK, the code runs +the python3 interpreter with a basic command to check it is working +correctly. The test would only run if python3 is installed in the SDK. + +``oe-build-perf-test`` +---------------------- + +The performance tests usually measure how long operations take and the +resource utilisation as that happens. An example from +``meta/lib/oeqa/buildperf/test_basic.py`` contains the following:: + + class Test3(BuildPerfTestCase): + def test3(self): + """Bitbake parsing (bitbake -p)""" + # Drop all caches and parse + self.rm_cache() + oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True) + self.measure_cmd_resources(['bitbake', '-p'], 'parse_1', + 'bitbake -p (no caches)') + # Drop tmp/cache + oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True) + self.measure_cmd_resources(['bitbake', '-p'], 'parse_2', + 'bitbake -p (no tmp/cache)') + # Parse with fully cached data + self.measure_cmd_resources(['bitbake', '-p'], 'parse_3', + 'bitbake -p (cached)') + +This example shows how three specific parsing timings are +measured, with and without various caches, to show how BitBake's parsing +performance trends over time. + +Considerations When Writing Tests +================================= + +When writing good tests, there are several things to keep in mind. Since +things running on the Autobuilder are accessed concurrently by multiple +workers, consider the following: + +**Running "cleanall" is not permitted.** + +This can delete files from DL_DIR which would potentially break other +builds running in parallel. If this is required, DL_DIR must be set to +an isolated directory. + +**Running "cleansstate" is not permitted.** + +This can delete files from SSTATE_DIR which would potentially break +other builds running in parallel. If this is required, SSTATE_DIR must +be set to an isolated directory. Alternatively, you can use the "-f" +option with the ``bitbake`` command to "taint" tasks by changing the +sstate checksums to ensure sstate cache items will not be reused. + +**Tests should not change the metadata.** + +This is particularly true for oe-selftests since these can run in +parallel and changing metadata leads to changing checksums, which +confuses BitBake while running in parallel. If this is necessary, copy +layers to a temporary location and modify them. Some tests need to +change metadata, such as the devtool tests. To prevent the metadate from +changes, set up temporary copies of that data first. |