diff options
Diffstat (limited to 'documentation/test-manual/test-manual-intro.rst')
| -rw-r--r-- | documentation/test-manual/test-manual-intro.rst | 550 |
1 files changed, 0 insertions, 550 deletions
diff --git a/documentation/test-manual/test-manual-intro.rst b/documentation/test-manual/test-manual-intro.rst deleted file mode 100644 index 53ad650b35..0000000000 --- a/documentation/test-manual/test-manual-intro.rst +++ /dev/null @@ -1,550 +0,0 @@ -.. SPDX-License-Identifier: CC-BY-2.0-UK - -***************************************** -The Yocto Project Test Environment Manual -***************************************** - -.. _test-welcome: - -Welcome -======= - -Welcome to the Yocto Project Test Environment Manual! This manual is a -work in progress. The manual contains information about the testing -environment used by the Yocto Project to make sure each major and minor -release works as intended. All the project's testing infrastructure and -processes are publicly visible and available so that the community can -see what testing is being performed, how it's being done and the current -status of the tests and the project at any given time. It is intended -that Other organizations can leverage off the process and testing -environment used by the Yocto Project to create their own automated, -production test environment, building upon the foundations from the -project core. - -Currently, the Yocto Project Test Environment Manual has no projected -release date. This manual is a work-in-progress and is being initially -loaded with information from the README files and notes from key -engineers: - -- *yocto-autobuilder2:* This - :yocto_git:`README.md </cgit.cgi/yocto-autobuilder2/tree/README.md>` - is the main README which detials how to set up the Yocto Project - Autobuilder. The ``yocto-autobuilder2`` repository represents the - Yocto Project's console UI plugin to Buildbot and the configuration - necessary to configure Buildbot to perform the testing the project - requires. - -- *yocto-autobuilder-helper:* This :yocto_git:`README </cgit.cgi/yocto-autobuilder-helper/tree/README/>` - and repository contains Yocto Project Autobuilder Helper scripts and - configuration. The ``yocto-autobuilder-helper`` repository contains - the "glue" logic that defines which tests to run and how to run them. - As a result, it can be used by any Continuous Improvement (CI) system - to run builds, support getting the correct code revisions, configure - builds and layers, run builds, and collect results. The code is - independent of any CI system, which means the code can work `Buildbot <https://docs.buildbot.net/0.9.15.post1/>`__, - Jenkins, or others. This repository has a branch per release of the - project defining the tests to run on a per release basis. - -.. _test-yocto-project-autobuilder-overview: - -Yocto Project Autobuilder Overview -================================== - -The Yocto Project Autobuilder collectively refers to the software, -tools, scripts, and procedures used by the Yocto Project to test -released software across supported hardware in an automated and regular -fashion. Basically, during the development of a Yocto Project release, -the Autobuilder tests if things work. The Autobuilder builds all test -targets and runs all the tests. - -The Yocto Project uses now uses standard upstream -`Buildbot <https://docs.buildbot.net/0.9.15.post1/>`__ (version 9) to -drive its integration and testing. Buildbot Nine has a plug-in interface -that the Yocto Project customizes using code from the -``yocto-autobuilder2`` repository, adding its own console UI plugin. The -resulting UI plug-in allows you to visualize builds in a way suited to -the project's needs. - -A ``helper`` layer provides configuration and job management through -scripts found in the ``yocto-autobuilder-helper`` repository. The -``helper`` layer contains the bulk of the build configuration -information and is release-specific, which makes it highly customizable -on a per-project basis. The layer is CI system-agnostic and contains a -number of Helper scripts that can generate build configurations from -simple JSON files. - -.. note:: - - The project uses Buildbot for historical reasons but also because - many of the project developers have knowledge of python. It is - possible to use the outer layers from another Continuous Integration - (CI) system such as - `Jenkins <https://en.wikipedia.org/wiki/Jenkins_(software)>`__ - instead of Buildbot. - -The following figure shows the Yocto Project Autobuilder stack with a -topology that includes a controller and a cluster of workers: - -.. image:: figures/ab-test-cluster.png - :align: center - -.. _test-project-tests: - -Yocto Project Tests - Types of Testing Overview -=============================================== - -The Autobuilder tests different elements of the project by using -thefollowing types of tests: - -- *Build Testing:* Tests whether specific configurations build by - varying :term:`MACHINE`, - :term:`DISTRO`, other configuration - options, and the specific target images being built (or world). Used - to trigger builds of all the different test configurations on the - Autobuilder. Builds usually cover many different targets for - different architectures, machines, and distributions, as well as - different configurations, such as different init systems. The - Autobuilder tests literally hundreds of configurations and targets. - - - *Sanity Checks During the Build Process:* Tests initiated through - the :ref:`insane <ref-classes-insane>` - class. These checks ensure the output of the builds are correct. - For example, does the ELF architecture in the generated binaries - match the target system? ARM binaries would not work in a MIPS - system! - -- *Build Performance Testing:* Tests whether or not commonly used steps - during builds work efficiently and avoid regressions. Tests to time - commonly used usage scenarios are run through ``oe-build-perf-test``. - These tests are run on isolated machines so that the time - measurements of the tests are accurate and no other processes - interfere with the timing results. The project currently tests - performance on two different distributions, Fedora and Ubuntu, to - ensure we have no single point of failure and can ensure the - different distros work effectively. - -- *eSDK Testing:* Image tests initiated through the following command:: - - $ bitbake image -c testsdkext - - The tests utilize the ``testsdkext`` class and the ``do_testsdkext`` task. - -- *Feature Testing:* Various scenario-based tests are run through the - :ref:`OpenEmbedded Self test (oe-selftest) <ref-manual/ref-release-process:Testing and Quality Assurance>`. We test oe-selftest on each of the main distrubutions - we support. - -- *Image Testing:* Image tests initiated through the following command:: - - $ bitbake image -c testimage - - The tests utilize the :ref:`testimage* <ref-classes-testimage*>` - classes and the :ref:`ref-tasks-testimage` task. - -- *Layer Testing:* The Autobuilder has the possibility to test whether - specific layers work with the test of the system. The layers tested - may be selected by members of the project. Some key community layers - are also tested periodically. - -- *Package Testing:* A Package Test (ptest) runs tests against packages - built by the OpenEmbedded build system on the target machine. See the - :ref:`Testing Packages With - ptest <dev-manual/dev-manual-common-tasks:Testing Packages With ptest>` section - in the Yocto Project Development Tasks Manual and the - ":yocto_wiki:`Ptest </wiki/Ptest>`" Wiki page for more - information on Ptest. - -- *SDK Testing:* Image tests initiated through the following command:: - - $ bitbake image -c testsdk - - The tests utilize the :ref:`testsdk <ref-classes-testsdk>` class and - the ``do_testsdk`` task. - -- *Unit Testing:* Unit tests on various components of the system run - through :ref:`bitbake-selftest <ref-manual/ref-release-process:Testing and Quality Assurance>` and - :ref:`oe-selftest <ref-manual/ref-release-process:Testing and Quality Assurance>`. - -- *Automatic Upgrade Helper:* This target tests whether new versions of - software are available and whether we can automatically upgrade to - those new versions. If so, this target emails the maintainers with a - patch to let them know this is possible. - -.. _test-test-mapping: - -How Tests Map to Areas of Code -============================== - -Tests map into the codebase as follows: - -- *bitbake-selftest:* - - These tests are self-contained and test BitBake as well as its APIs, - which include the fetchers. The tests are located in - ``bitbake/lib/*/tests``. - - From within the BitBake repository, run the following:: - - $ bitbake-selftest - - To skip tests that access the Internet, use the ``BB_SKIP_NETTEST`` - variable when running "bitbake-selftest" as follows:: - - $ BB_SKIP_NETTEST=yes bitbake-selftest - - The default output is quiet and just prints a summary of what was - run. To see more information, there is a verbose option:: - - $ bitbake-selftest -v - - Use this option when you wish to skip tests that access the network, - which are mostly necessary to test the fetcher modules. To specify - individual test modules to run, append the test module name to the - "bitbake-selftest" command. For example, to specify the tests for the - bb.data.module, run:: - - $ bitbake-selftest bb.test.data.module - - You can also specify individual tests by defining the full name and module - plus the class path of the test, for example:: - - $ bitbake-selftest bb.tests.data.TestOverrides.test_one_override - - The tests are based on `Python - unittest <https://docs.python.org/3/library/unittest.html>`__. - -- *oe-selftest:* - - - These tests use OE to test the workflows, which include testing - specific features, behaviors of tasks, and API unit tests. - - - The tests can take advantage of parallelism through the "-j" - option, which can specify a number of threads to spread the tests - across. Note that all tests from a given class of tests will run - in the same thread. To parallelize large numbers of tests you can - split the class into multiple units. - - - The tests are based on Python unittest. - - - The code for the tests resides in - ``meta/lib/oeqa/selftest/cases/``. - - - To run all the tests, enter the following command:: - - $ oe-selftest -a - - - To run a specific test, use the following command form where - testname is the name of the specific test:: - - $ oe-selftest -r <testname> - - For example, the following command would run the tinfoil - getVar API test:: - - $ oe-selftest -r tinfoil.TinfoilTests.test_getvar - - It is also possible to run a set - of tests. For example the following command will run all of the - tinfoil tests:: - - $ oe-selftest -r tinfoil - -- *testimage:* - - - These tests build an image, boot it, and run tests against the - image's content. - - - The code for these tests resides in ``meta/lib/oeqa/runtime/cases/``. - - - You need to set the :term:`IMAGE_CLASSES` variable as follows:: - - IMAGE_CLASSES += "testimage" - - - Run the tests using the following command form:: - - $ bitbake image -c testimage - -- *testsdk:* - - - These tests build an SDK, install it, and then run tests against - that SDK. - - - The code for these tests resides in ``meta/lib/oeqa/sdk/cases/``. - - - Run the test using the following command form:: - - $ bitbake image -c testsdk - -- *testsdk_ext:* - - - These tests build an extended SDK (eSDK), install that eSDK, and - run tests against the eSDK. - - - The code for these tests resides in ``meta/lib/oeqa/esdk``. - - - To run the tests, use the following command form:: - - $ bitbake image -c testsdkext - -- *oe-build-perf-test:* - - - These tests run through commonly used usage scenarios and measure - the performance times. - - - The code for these tests resides in ``meta/lib/oeqa/buildperf``. - - - To run the tests, use the following command form:: - - $ oe-build-perf-test <options> - - The command takes a number of options, - such as where to place the test results. The Autobuilder Helper - Scripts include the ``build-perf-test-wrapper`` script with - examples of how to use the oe-build-perf-test from the command - line. - - Use the ``oe-git-archive`` command to store test results into a - Git repository. - - Use the ``oe-build-perf-report`` command to generate text reports - and HTML reports with graphs of the performance data. For - examples, see - :yocto_dl:`/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.html` - and - :yocto_dl:`/releases/yocto/yocto-2.7/testresults/buildperf-centos7/perf-centos7.yoctoproject.org_warrior_20190414204758_0e39202.txt`. - - - The tests are contained in ``lib/oeqa/buildperf/test_basic.py``. - -Test Examples -============= - -This section provides example tests for each of the tests listed in the -:ref:`test-manual/test-manual-intro:How Tests Map to Areas of Code` section. - -For oeqa tests, testcases for each area reside in the main test -directory at ``meta/lib/oeqa/selftest/cases`` directory. - -For oe-selftest. bitbake testcases reside in the ``lib/bb/tests/`` -directory. - -.. _bitbake-selftest-example: - -``bitbake-selftest`` --------------------- - -A simple test example from ``lib/bb/tests/data.py`` is:: - - class DataExpansions(unittest.TestCase): - def setUp(self): - self.d = bb.data.init() - self.d["foo"] = "value_of_foo" - self.d["bar"] = "value_of_bar" - self.d["value_of_foo"] = "value_of_'value_of_foo'" - - def test_one_var(self): - val = self.d.expand("${foo}") - self.assertEqual(str(val), "value_of_foo") - -In this example, a ``DataExpansions`` class of tests is created, -derived from standard python unittest. The class has a common ``setUp`` -function which is shared by all the tests in the class. A simple test is -then added to test that when a variable is expanded, the correct value -is found. - -Bitbake selftests are straightforward python unittest. Refer to the -Python unittest documentation for additional information on writing -these tests at: https://docs.python.org/3/library/unittest.html. - -.. _oe-selftest-example: - -``oe-selftest`` ---------------- - -These tests are more complex due to the setup required behind the scenes -for full builds. Rather than directly using Python's unittest, the code -wraps most of the standard objects. The tests can be simple, such as -testing a command from within the OE build environment using the -following example:: - - class BitbakeLayers(OESelftestTestCase): - def test_bitbakelayers_showcrossdepends(self): - result = runCmd('bitbake-layers show-cross-depends') - self.assertTrue('aspell' in result.output, msg = "No dependencies were shown. bitbake-layers show-cross-depends output: %s"% result.output) - -This example, taken from ``meta/lib/oeqa/selftest/cases/bblayers.py``, -creates a testcase from the ``OESelftestTestCase`` class, derived -from ``unittest.TestCase``, which runs the ``bitbake-layers`` command -and checks the output to ensure it contains something we know should be -here. - -The ``oeqa.utils.commands`` module contains Helpers which can assist -with common tasks, including: - -- *Obtaining the value of a bitbake variable:* Use - ``oeqa.utils.commands.get_bb_var()`` or use - ``oeqa.utils.commands.get_bb_vars()`` for more than one variable - -- *Running a bitbake invocation for a build:* Use - ``oeqa.utils.commands.bitbake()`` - -- *Running a command:* Use ``oeqa.utils.commandsrunCmd()`` - -There is also a ``oeqa.utils.commands.runqemu()`` function for launching -the ``runqemu`` command for testing things within a running, virtualized -image. - -You can run these tests in parallel. Parallelism works per test class, -so tests within a given test class should always run in the same build, -while tests in different classes or modules may be split into different -builds. There is no data store available for these tests since the tests -launch the ``bitbake`` command and exist outside of its context. As a -result, common bitbake library functions (bb.\*) are also unavailable. - -.. _testimage-example: - -``testimage`` -------------- - -These tests are run once an image is up and running, either on target -hardware or under QEMU. As a result, they are assumed to be running in a -target image environment, as opposed to a host build environment. A -simple example from ``meta/lib/oeqa/runtime/cases/python.py`` contains -the following:: - - class PythonTest(OERuntimeTestCase): - @OETestDepends(['ssh.SSHTest.test_ssh']) - @OEHasPackage(['python3-core']) - def test_python3(self): - cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" - status, output = self.target.run(cmd) - msg = 'Exit status was not 0. Output: %s' % output - self.assertEqual(status, 0, msg=msg) - -In this example, the ``OERuntimeTestCase`` class wraps -``unittest.TestCase``. Within the test, ``self.target`` represents the -target system, where commands can be run on it using the ``run()`` -method. - -To ensure certain test or package dependencies are met, you can use the -``OETestDepends`` and ``OEHasPackage`` decorators. For example, the test -in this example would only make sense if python3-core is installed in -the image. - -.. _testsdk_ext-example: - -``testsdk_ext`` ---------------- - -These tests are run against built extensible SDKs (eSDKs). The tests can -assume that the eSDK environment has already been setup. An example from -``meta/lib/oeqa/sdk/cases/devtool.py`` contains the following:: - - class DevtoolTest(OESDKExtTestCase): - @classmethod def setUpClass(cls): - myapp_src = os.path.join(cls.tc.esdk_files_dir, "myapp") - cls.myapp_dst = os.path.join(cls.tc.sdk_dir, "myapp") - shutil.copytree(myapp_src, cls.myapp_dst) - subprocess.check_output(['git', 'init', '.'], cwd=cls.myapp_dst) - subprocess.check_output(['git', 'add', '.'], cwd=cls.myapp_dst) - subprocess.check_output(['git', 'commit', '-m', "'test commit'"], cwd=cls.myapp_dst) - - @classmethod - def tearDownClass(cls): - shutil.rmtree(cls.myapp_dst) - def _test_devtool_build(self, directory): - self._run('devtool add myapp %s' % directory) - try: - self._run('devtool build myapp') - finally: - self._run('devtool reset myapp') - def test_devtool_build_make(self): - self._test_devtool_build(self.myapp_dst) - -In this example, the ``devtool`` -command is tested to see whether a sample application can be built with -the ``devtool build`` command within the eSDK. - -.. _testsdk-example: - -``testsdk`` ------------ - -These tests are run against built SDKs. The tests can assume that an SDK -has already been extracted and its environment file has been sourced. A -simple example from ``meta/lib/oeqa/sdk/cases/python2.py`` contains the -following:: - - class Python3Test(OESDKTestCase): - def setUp(self): - if not (self.tc.hasHostPackage("nativesdk-python3-core") or - self.tc.hasHostPackage("python3-core-native")): - raise unittest.SkipTest("No python3 package in the SDK") - - def test_python3(self): - cmd = "python3 -c \\"import codecs; print(codecs.encode('Uryyb, jbeyq', 'rot13'))\"" - output = self._run(cmd) - self.assertEqual(output, "Hello, world\n") - -In this example, if nativesdk-python3-core has been installed into the SDK, the code runs -the python3 interpreter with a basic command to check it is working -correctly. The test would only run if python3 is installed in the SDK. - -.. _oe-build-perf-test-example: - -``oe-build-perf-test`` ----------------------- - -The performance tests usually measure how long operations take and the -resource utilisation as that happens. An example from -``meta/lib/oeqa/buildperf/test_basic.py`` contains the following:: - - class Test3(BuildPerfTestCase): - def test3(self): - """Bitbake parsing (bitbake -p)""" - # Drop all caches and parse - self.rm_cache() - oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True) - self.measure_cmd_resources(['bitbake', '-p'], 'parse_1', - 'bitbake -p (no caches)') - # Drop tmp/cache - oe.path.remove(os.path.join(self.bb_vars['TMPDIR'], 'cache'), True) - self.measure_cmd_resources(['bitbake', '-p'], 'parse_2', - 'bitbake -p (no tmp/cache)') - # Parse with fully cached data - self.measure_cmd_resources(['bitbake', '-p'], 'parse_3', - 'bitbake -p (cached)') - -This example shows how three specific parsing timings are -measured, with and without various caches, to show how BitBake's parsing -performance trends over time. - -.. _test-writing-considerations: - -Considerations When Writing Tests -================================= - -When writing good tests, there are several things to keep in mind. Since -things running on the Autobuilder are accessed concurrently by multiple -workers, consider the following: - -**Running "cleanall" is not permitted.** - -This can delete files from DL_DIR which would potentially break other -builds running in parallel. If this is required, DL_DIR must be set to -an isolated directory. - -**Running "cleansstate" is not permitted.** - -This can delete files from SSTATE_DIR which would potentially break -other builds running in parallel. If this is required, SSTATE_DIR must -be set to an isolated directory. Alternatively, you can use the "-f" -option with the ``bitbake`` command to "taint" tasks by changing the -sstate checksums to ensure sstate cache items will not be reused. - -**Tests should not change the metadata.** - -This is particularly true for oe-selftests since these can run in -parallel and changing metadata leads to changing checksums, which -confuses BitBake while running in parallel. If this is necessary, copy -layers to a temporary location and modify them. Some tests need to -change metadata, such as the devtool tests. To prevent the metadate from -changes, set up temporary copies of that data first. |