Matthew Allum matthew@openedhand.com 2003 2004 OpenedHand Ltd 0.3 2004-04-19

Testing is important. Matchbox is likely to be a key part of the system - even a minor bug can cause serious user problems. The asynchronous nature of X11, interdependencies of code mechanisms and the complexities of window management all can lead to the introduction of bugs when even minor code is added or modified. Therefore thorough testing is important to avoid such problems. The best way to test Matchbox is 'in the wild' from real user usage and feedback, however this is not always practical - it's extremely useful to perform a standard set of tests after changes to the source code and before making a release. This document discusses: how to run Matchbox in a testing environment, how to use the test scripts included in the distribution and what to do if tests fail. The documentation here is relevant for the 0.9 matchbox release.

The matchbox-test package uses autotools and is built and installed with the usual; % ./configure % make % sudo make install This will build the various binary test utilities and install a special 'mbtest' theme.

Xnest basically provides a X server in an X window - it being both an X Server and an X Client at the same time. This makes it useful for both testing Matchbox on a desktop machine and 'emulating' PDA sized displays. The procedure for running Matchbox in an Xnest follows: % Xnest :1 -ac -geometry 240x320 % export DISPLAY=:1 % matchbox A similar technique can be used to run Matchbox locally on a development machine but have it manage windows on another display (such as a handheld). On the remote machine, make sure there is no window manager already running and then give access for the X Server from the development machine (e.g. with xhost +dev machine IP or by using Xnest with -ac argument). On the development machine set the DISPLAY environment variable to the remote machine's IP and then launch Matchbox. Matchbox can be run under Valgrind on x86 platforms to debug memory allocations and management. It is recommended, though not essential, that the Matchbox build is configured with the --enable-debug switch. The resulting binaries will spew verbose messages to standard out which contain information that's useful for fixing any failed tests.

All tests are in the Matchbox distribution's tests/ directory. You should run tests from within that directory.

Tests Matchbox's handling of various window types by means of the winspew application.

Creates some various window types and then replays some Xnee data ( test-interaction.xnee ) automating common window operations on these created windows. A screenshot is then taken at completion and compared with a reference one ( test-interaction.xwd ) to indicate success or failure. If installed, Image Magick's 'display' program will be used to visually present failed screenshots. To work correctly this test also requires matchbox-desktop to be installed. This test will only function correctly with a 240x320 or 640x480 sized display. Its highly recommended you avoid any system interaction while this test is running as this can lead to false results.

Tests mbdock by running it under Valgrind in different orientations, adding and removing panel applets.

test-gconf.sh currently just edits the 'cycle_window' shortcut and then 'tests' it via sendkey.

This is a binary based test for libmb. Currently it only basically tests font rendering and clipping functionality. For more libmatchbox tests, see the section on unit tests.

The testrun.sh script aims to run all tests automated in a standardised environment before producing a report summary. It performs the following: Starts Xnest and sets $DISPLAY to this server. Starts Matchbox optionally running under Valgrind. Sets the Matchbox theme via mbcontrol Runs each test, non-interactively Finally produces a simple report, commenting on how each test went. The script reads parameters set in test.conf and runs all listed tests. You can specify particular tests to run and and specify an alternate configuration file to use. Run testrun.sh with '-h' to see how to set this. To add a new test to be run, just name the new script test-XXXX.sh (where XXXX is anything you like) and add to tests directory.

There are unit test for libmatchbox in libmatchbox/tests/. These are written using the Check unit testing framework (), and currently test the .desktop file parser and the pixmap library. To execute the tests, simply run make check in libmatchbox. $ make check ... Running suite(s): DotDesktop 100%: Checks: 5, Failures: 0, Errors: 0 PASS: dot-desktop Running suite(s): MbPixbuf 100%: Checks: 13, Failures: 0, Errors: 0 PASS: pixbuf ================== All 2 tests passed ================== As can be seen, two test suites were executed (DotDesktop and MbPixbuf), and these both passed. At the time of writing there is a bug in MbPixbuf which results in a single test failing with the message Composite image incorrect. Also, unless you have support for the fr_FR locale, the DotDesktop test dotdesktop_l10n_present will fail. The tests can also be run with gcov coverage reports. If libmatchbox is built with --enable-gcov, then executing tests/gcov.sh will run all of the unit tests and then display coverage information. To extend the test suite, create a new C program which uses the Check library and provides a main entry point, then add it to check_PROGRAMS in libmatchbox/tests/Makefile.am. Once the Makefile has been remade, running make check will execute the new tests.

Please report failed tests to bugs.openedhand.com, giving as much information as possible.

Both the matchbox-window-manager and libmatchbox packages have support for the GNU Code coverage tool Gcov. To enable, at build time configure the packages with the --enable-gcov option. The built binaries will then produce gcov statistics, which can further be processed by running gcov on source files. Within the test package exists gcov-wrapper.sh. This script will run all or any of the tests and report the percentage of code per source file the test run covered. Usage is; ./gcov-wrapper.sh <matchbox wm src path> <libmatchbox src path> [test] 'test' defaults to testrun.sh Set GCOV_FLAGS to any required gcov switches It is advised you make clean and then remake matchbox-window-manager and libmatchbox before each gcov-wrapper.sh run to avoid statistics accumulating. More information on using Gcov is available at http://gcc.gnu.org/onlinedocs/gcc-3.0/gcc_8.html

An open-source memory debugger for x86-GNU/Linux, available from . By default the tests run Valgrind with the following arguments. valgrind --error-limit=no --num-callers=32 --leak-check=yes

Xnee allows the recording and replaying of X protocol data. To record data for Xnee, I'm currently using something like; xnee -ae -o test-interaction.xnee -l -1 Which basically translates to capture all events ( -ae ) , output ( -o ) to test-interaction.xnee and keep capturing for ever ( -l -1 ). I stop capture with ctrl-c. To then replay the data ( as in test-interaction.sh ), use; xnee -rep -f test-interaction.xnee I'm using xnee 1.07.

Xnest is an X server that is itself an X client. This allows you to run a server within a server - useful for testing new window managers and other X clients. Included in XFree86 distribution from

Creates dummy windows of arbitrary types and properties. Included with Matchbox in tests/, pass '-help' to get usage info.

Uses XTest extension to send arbitrary keycodes and modifiers to the X Server from the command line. Included with Matchbox in tests/, pass '-help' to get usage info.

Simple remote control of Matchbox, such as changing the theme. Included with Matchbox in utils/.

Please refer the the man pages for useful tools such xprop, xclientlist, xwd Also see and for examples of X-Resource extension usage. 'XRes' is a new XFree86 server extension (> 4.2) which can used to monitor client resource usage and detect server side pixmap leaks.