One complaint occasionally encountered is that package test suites take too long to run. One of the packages for which that complaint is sometimes encountered is Paludis. The complainers rarely mention the difficulty of recovering a system from an error that should have been caught before installation, but still, running test suites faster cannot be a bad thing. One way to go about that is to make better use of parallelism.
There are a number of issues involved here. The first of these doesn’t apply to Paludis, since we use our own src_test in ebuilds and exhereses (for unrelated reasons), but other package maintainers may find it interesting: the default src_test in Gentoo EAPIs calls emake check -j1.
The reasons for this are historical: when Nick and I worked out the original src_test, a good number of the packages upon which we tested it hated parallel tests. Admittedly, it didn’t help that nearly all of the packages in question were using hand-rolled test suite runners… Also, those were the bad old days when nearly everyone was using a single core x86 CPU, and very few people cared enough to make sure their packages built in parallel.
Even now, the -j1 isn’t something we could just remove arbitrarily on Gentoo. When looking at it for Exherbo’s Exheres format, Ingmar found a non-trivial number of packages that hated not having -j1. For Gentoo, removing the -j1 would certainly require EAPI control.
But here’s the problem: the -j1 doesn’t just affect running the tests. It also affects building the tests. For Paludis, over half the compile time is spent building tests. Were that done with -j1 on a typical quad core box, it would double the compile time. In contrast to running the tests, building usually is parallel-safe, so the -j1 has considerable impact. Alas, most build systems don’t provide a target for building tests without running them.
Even without the -j1, the second issue looms large. The Automake test runner doesn’t parallelise test execution. Tests are run one after another in strict sequence regardless of the number of jobs make is allowed to use.
Fortunately, Automake 1.11 includes a new test runner that does support parallel execution. Alas, this test runner is only used by packages that explicitly request it, and it isn’t something that can be shoved into packages externally by the package manager.
Since every Paludis test is already safely parallelisable (no test does any work outside its individual temporary test directory), I’ve switched us over to using the new test runner. Doing so consisted of the following:
- Adding
AUTOMAKE_OPTIONS = parallel-teststo everyMakefile.am. Note that this option cannot simply be set in the top level makefile. - Switching from using
TESTS_ENVIRONMENTtoLOG_COMPILER. The former still works, but is marked for end user use. - Working around an annoying ‘feature’ that prevents rules being generated for running tests for
TESTSthat include a variable set byconfigure.ac. - Doing some ungodly hacks with file descriptors to be able to output to stdout from the test script runner.
- Splitting up some of the larger tests into multiple smaller tests, to avoid having no output for several minutes, and to increase parallelisability.
The LOG_COMPILER deserves further comment. To avoid massive confusion, when running tests in parallel, output is automatically redirected to a .log file. That’s all very well, but Automake is excessively quiet on this, and it looks a lot like nothing is happening whilst tests run. To work around this, the following ungodly hack appears to work:
LOG_COMPILER = \
test "x$$BASH_VERSION" == x || \
eval "exec 3<&1 ; export PALUDIS_TESTS_REAL_STDOUT_FD=3" ; \
env \
VARS_NEEDED_FOR_TESTS="whatever" \
sh $(top_srcdir)/test/run_test.sh
Then, the test runner can output progress messages to stdout by using the $PALUDIS_TESTS_REAL_STDOUT_FD file descriptor (if the environment variable is set; unfortunately, some shells won’t let you do this), while leaving more verbose information to be logged as normal. Since POSIX guarantees writes to pipes of no more than a certain size to be atomic, we don’t have to worry about intermingled output so long as we keep our lines short.
As for the annoying ‘feature’, things like this work fine with the old test runner, but not the new one:
TESTS = $(variable_set_by_configure) LOG_COMPILER = blah
For Paludis, we use this kind of construct to run tidy on HTML files, where the list of HTML files is taken from some configure.ac voodoo. Fortunately, we can work around it, so long as all of the tests have a common file extension:
TESTS = $(variable_set_by_configure) TEST_EXTENSIONS = .html HTML_LOG_COMPILER = blah
Note the arbitrary and annoying case change for HTML.
The end result of this tinkering is that Paludis tests on a quad core box now take around two minutes to run rather than six.
I understand it is traditional when writing long, rambling and largely pointless blog posts about parallel builds to end with shameless whoring of an Amazon wishlist.
