UIs for Parallelism

Getting the UI right for parallel execution is tricky.

Make, for example, doesn’t even try. It just displays output from all the children all mixed together, and relies upon commands prefixing their output with a filename or similar identifier. This generally works for what make needs to do, but it’s a bit too crude for some things.

For the secret “not supposed to use it” parallelism in Paludis, we’re currently using something slightly more sophisticated: each line of output from each child job is prefixed by the job’s name. So you’ll see things like:

sync unavailable> wget -T 30 -t 1 -O /var/tmp/paludis-tarsync-cvh9gX/exherbo_repositories.tar.bz2 http://git.exherbo.org/exherbo_repositories.tar.bz2
sync unavailable-unofficial> wget -T 30 -t 1 -O /var/tmp/paludis-tarsync-kvEzRZ/exherbo_unofficial_repositories.tar.bz2
sync unavailable-unofficial> --14:24:06--  http://git.exherbo.org/exherbo_unofficial_repositories.tar.bz2
sync unavailable-unofficial>            => `/var/tmp/paludis-tarsync-kvEzRZ/exherbo_unofficial_repositories.tar.bz2'
sync unavailable> --14:24:06--  http://git.exherbo.org/exherbo_repositories.tar.bz2

Something like this is necessary because the commands involved don’t do their own prefixing. But whilst it’s an improvement, the output’s rather cluttered and it’s hard to work out what’s going on. I’ve been experimenting with something more like this (slowdown induced artificially to make the point):

Starting sync

Repository                    Status                Pending Active  Done
-> alsa                       starting              16      1       0
-> ciaranm                    starting              15      2       0
-> ferdy                      starting              14      3       0
-> compnerd                   starting              13      4       0
-> arbor                      starting              12      5       0
-> compnerd                   success               12      4       1
-> gnome                      starting              11      5       1
-> alsa                       failed                11      4       2
    ... fatal: The remote end hung up unexpectedly
-> hardware                   starting              10      5       2
-> ferdy                      success               10      4       3
-> media                      starting              9       5       3
-> gnome                      success               9       4       4
-> mozilla                    starting              8       5       4
-> hardware                   success               8       4       5
-> pioto-exheres              starting              7       5       5
-> media                      success               7       4       6
-> python                     starting              6       5       6
-> pioto-exheres              success               6       4       7
-> rbrown                     starting              5       5       7
-> mozilla                    success               5       4       8
-> texlive                    starting              4       5       8
-> ciaranm                    active                4       5       8
-> python                     success               4       4       9
-> unavailable                starting              3       5       9
-> texlive                    success               3       4       10
-> unavailable-unofficial     starting              2       5       10
-> rbrown                     success               2       4       11
-> unwritten                  starting              1       5       11
-> arbor                      active                1       5       11
-> unwritten                  success               1       4       12
-> x11                        starting              0       5       12
-> ciaranm                    active                0       5       12
    ... Initialized empty Git repository in /var/db/paludis/repositories/ciaranm/.git/
-> unavailable                active                0       5       12
    ... Literal data: 0 bytes
    ... Matched data: 0 bytes
    ... File list size: 749
    ... File list generation time: 0.001 seconds
    ... File list transfer time: 0.000 seconds
    ... Total bytes sent: 771
    ... Total bytes received: 26
    ...
    ... sent 771 bytes  received 26 bytes  1594.00 bytes/sec
    ... total size is 57557  speedup is 72.22
-> unavailable-unofficial     active                0       5       12
    ... Literal data: 0 bytes
    ... Matched data: 0 bytes
    ... File list size: 110
    ... File list generation time: 0.001 seconds
    ... File list transfer time: 0.000 seconds
    ... Total bytes sent: 132
    ... Total bytes received: 26
    ...
    ... sent 132 bytes  received 26 bytes  316.00 bytes/sec
    ... total size is 4013  speedup is 25.40
-> arbor                      active                0       5       12
-> unavailable                success               0       4       13
-> unavailable-unofficial     success               0       3       14
-> x11                        active                0       3       14
-> ciaranm                    active                0       3       14
-> ciaranm                    success               0       2       15
-> x11                        success               0       1       16
-> arbor                      success               0       0       17

 * Cleaning write cache for ebuild format repositories...
 * Done cleaning write cache for ebuild format repositories

 * No unread news items found

Sync results

* alsa:                       sync of '/var/db/paludis/repositories/alsa' from 'git://git.exherbo.org/demonstrate/failure/alsa.git' failed (paludis::SyncFailedError)
    Log file:                 /var/log/sync-alsa.1225118654.log
* arbor:                      success
* ciaranm:                    success
* compnerd:                   success
* ferdy:                      success
* gnome:                      success
* hardware:                   success
* media:                      success
* mozilla:                    success
* pioto-exheres:              success
* python:                     success
* rbrown:                     success
* texlive:                    success
* unavailable:                success
* unavailable-unofficial:     success
* unwritten:                  success
* x11:                        success

In particular:

• Output is automatically logged to a file, rather than dumping everything to the screen. That file can be automatically removed if the job succeeds.
• A summary of what’s going on is displayed every time a job starts or finishes.
• Every ten seconds (for some value of ten), if a job hasn’t finished, we automatically display the last ten (for some other value of ten) lines of its output (but not any output we’ve already displayed) to the screen, along with a note that it’s still active. We also tail the log if a job fails.

This seems to be a comfortable balance between not showing anything except job statuses (which for jobs that can take an hour to run might well leave you wondering whether it’s hung) and just dumping everything to the screen. Now it’s just a case of finding appropriate values of ten…

Paludis is about Choices

Paludis 0.32 will include a new mechanism called ‘choices’. Examples of choices on Gentoo include use flags, the various USE_EXPAND settings (linguas, video_cards and so on) and a shiny new setting called build_options. For use flags and the USE_EXPAND settings, there’s no change from a user perspective, and you can carry on using use.conf as normal — choices are merely a more general mechanism for handling package settings.

On Exherbo, things do change a bit, and if you’re one of those naughty people who ignored the “Exherbo has no users” warning you’d better have been paying attention.

So what’s this build_options thing? You’ll see it cropping up for most packages:

* net-print/cups::gentoo [U 1.3.8-r1 -> 1.3.8-r2] <target>
    Reasons: app-text/ghostscript-gpl-8.62:0::installed, net-print/foomatic-filters-3.0.20080507:0::installed, 2 more
    X acl -avahi -dbus -gnutls -java jpeg -kerberos -ldap pam perl -php png ppds python -samba
    -slp ssl -static -tiff -xinetd -zeroconf LINGUAS: -de en -es -et -fr -he -id -it -ja -pl -sv
    -zh_TW build_options: -optional_tests split strip

Packages using 0-based EAPIs will usually have three build options:

• optional_tests. This handles what --checks used to do. If this choice is set, src_test will be run.
• split. This used to be --debug-build split.
• strip. This used to be --debug-build internal.

Packages using kdebuild-1 or exheres-0 will have recommended_tests instead of optional_tests, which defaults to being enabled rather than disabled.

Despite what you might expect from older Paludis versions, these are not use flags. You can set them in use.conf, though — this means you can select whether to run tests or whether to keep debug information on a per-package basis, which was tricky to do with the old commandline options.

xorg-x11 hackery for a Logitech RX1000 Mouse

The Logitech RX1000 mouse has a scroll wheel that can tilt left or right. This generates HWHEEL events by default, which are pretty much useless. I find it more useful to have the tilt generate WHEEL events (scroll up and down), so I can press and hold left or right rather than repeatedly spinning the mouse wheel.

The wheel can also be pressed to get a middle click, but if I don’t concentrate I end up scrolling as well as middle clicking. There’s another button with a magnifying glass icon on it just below the scroll wheel which is more useful, which usually shows up as button 8. By making this button 2 we can use that for X11 paste, open-in-new-tab in Firefox and so on.

So I don’t forget how to get this working:

Section "InputDevice"
    Identifier  "Mouse1"
    Driver      "evdev"
    Option      "Name" "Logitech USB Optical Mouse"
    Option      "HWHEELRelativeAxisButtons" "4 5"
EndSection

Unfortunately, evdev doesn’t seem to have a ButtonMapping equivalent, so we have to fall back to xmodmap to do the button remapping. Doubly unfortunately, this makes pressing the middle mouse button useless.

pointer = 1 6 3 4 5 2 7 8 9 10

Thanks Nicolas.

Path and installable-to dependencies

A while ago, Paludis got from-repository dependencies. 0.32 will extend this further with path and installable-to dependencies.

Path dependencies are fairly simple: cat/pkg::/ will match cat/pkg installed at root /. A full path can be used too, so cat/pkg::/var/mychroot will work (as well as anything involving non-/ roots works, anyway…).

Installable-to dependencies take a bit more explaining. There are two kinds of installable-to dependency: repository and path. An installable-to-repository dependency looks like cat/pkg::repo? and is matched by any cat/pkg that could be installed to the repository named repo. An installable-to-path dependency looks like cat/pkg::/? and is matched by any cat/pkg that could be installed to any repository with root /.

Installable-to dependencies will not match masked packages. To specify “packages that would be installable, ignoring masks”, use a double question mark, like cat/pkg::/??. At present the double question mark ignores all masks, even those that cannot be overridden (unsupported EAPI, unavailable or unwritten). This may change to only include overridable masks; I’m not sure which is better behaviour yet.

Paludis 0.30.3 Released

Paludis 0.30.3 has been released:

• Tests for 0.30.2 would fail on some systems. This is now fixed.

Paludis 0.30.2 Released

Paludis 0.30.2 has been released:

• Work around ebuilds that treat cwd as secure.

Dealing with Unwritten Packages

A while ago Exherbo came up with a solution for the “large number of repositories” problem. It turns out this solution works rather well, and I’m now convinced that anyone rushing off and assuming that Git on its own will magically solve anything other than CVS sucking is going in the wrong direction.

Paludis also has ways of tracking packages installed by hand. Whilst useful on Gentoo, this really comes into play on Exherbo, where we don’t want to end up with a massive unmaintainable tree full of packages used by only two people.

The logical next step is tracking packages that don’t exist yet.

An unimaginative person might think that the way to solve this would be to have a bug for each requested package. But that gets messy — it’s not integrated with the package manager, and requires considerable extra effort from the user. A while ago Bernd suggested something more interesting: tracking unwritten packages in a repository in a similar way to how we do unavailable packages.

The implications:

• Most obviously, we can keep track of things we want that haven’t been written yet in a way that doesn’t involve leaving the package manager to look things up.
• We can also use it to track version bumps that we know will take a while. Doing paludis --query '>=hugescarypackage-4', say, would show that it’s being worked upon.
• We can also depend upon things that don’t exist, rather than leaving incomplete dependency strings around. This is fine in at least two cases — if a dependency is conditional upon an option that should probably be implemented but isn’t yet, we can add the option and make it unusable. And we can handle obscure suggested dependencies (e.g. git has lots of optional dependencies upon weird perl modules, so we can say “if you want support for git-send-email with SSL, you need to write such and such a package”).
• Bored developers could simply paludis --query '*/*::unwritten' and get ideas for what to do.

Adding support to Paludis for this only took a couple of hours. So now Exherbo developers can use this:

format = unwritten
location = /var/db/paludis/repositories/unwritten
sync = git://git.exherbo.org/unwritten.git
importance = -100

And see this:

$ paludis -q genesis
* sys-apps/genesis
    unwritten:               (1.0)X* {:0}
    Description:             The daddy of all init systems
    Homepage:                http://www.exherbo.org/
    Comment:                 We need an init system.
    Masked by unwritten:     Package has not been written yet

As with unavailable, installing behaves sensibly:

$ paludis -pi genesis
Building target list...
Building dependency list...
Query error:
  * In program paludis -pi genesis:
  * When performing install action from command line:
  * When executing install task:
  * When building dependency list:
  * When adding PackageDepSpec 'sys-apps/genesis':
  * All versions of 'sys-apps/genesis' are masked. Candidates are:
    * sys-apps/genesis-1.0:0::unwritten: Masked by unwritten (Package has not been written yet)

Those interested in the repository format can browse the tree. The observant might notice that the file format is quite similar to the one used by unavailable repositories, and wonder whether I was feeling lazy and swiped a load of code rather than implementing a new repository from scratch.

In the spirit of silly buzzwords, one could argue that this increases distributivity and democratisation of the repository, since there’s nothing to stop motivated users from creating their own wishlist repositories. It wouldn’t be hard to make a simple web interface that lets users request and vote on packages and version bumps, automatically generating a repository that anyone can use. But fortunately Exherbo is a dictatorship and has no users, so we don’t have to put up with that kind of nonsense.

EAPI 2: doman language support

This is the final post in a series on EAPI 2.

The doman helper is one of those pesky little beasts that makes specifying EAPI behaviour formally such a nuisance (although it is nowhere near as horrible as dohtml). EAPI 2 makes it even peskier.

I’ll try that again.

The doman helper makes writing ebuilds substantially easier by automagically doing the right thing when installing manual pages, freeing the developer from having to care about manual sections. EAPI 2 makes doman even more useful by making it aware of language codes as well as sections.

The painful details are available in PMS, but basically this will now ‘do the right thing’:

doman foo.1 foo.en.1 foo.en_GB.1

Previously only the first of the items would go to the right place.

This one’s a Gentoo innovation; see bug 222439 for its history. It was shamelessly stolen for exheres-0, but was too late for kdebuild-1.