Paludis 0.62.1 Released

Paludis 0.62.1 has been released:

• ‘cave resolve –hide’ now affects resolvent selection too.
• Various minor bug fixes.

Configuring Repositories Automatically via RepositoryRepository

Paludis is aware of packages that are in repositories you don’t have configured thanks to the unavailable repository. However, once Paludis has shown you that the package you want is in a repository you don’t have configured, you need to set up a configuration file for that repository (and any repositories it requires) and then sync. This is more work than is really necessary.

Enter RepositoryRepository, also known as r^2. Conceptually, it works as follows:

As well as providing special packages for packages in unavailable repositories, the unavailable repository also now provides packages named ‘repository/blah’ for repositories you don’t have configured. The metadata for these packages includes dependency information etc, along with useful things like the repository’s sync URI.

A new repository, using format = repository, provides special packages for repositories you do have configured.

Repository packages in unavailable repositories can be ‘installed’ to repository repositories. ‘Installing’ a repository creates a configuration file for it, and then syncs the newly created repository.

The configuration file it creates is controlled by a simple template, so it can contain anything you want it to contain.

Exherbo users can follow the setup instructions to start using this. On Gentoo this functionality is not yet available, since we won’t be switching the generated unavailable data to the new format until we’re reasonably sure that everyone is using a Paludis release that supports it.

Paludis is Going Into a Cave

The following applies to Gentoo, not Exherbo. Exherbo developers (Exherbo has no users) already know what’s going on there. I figure it’s worth having a clear source of information on this for Gentoo users, though, rather than making people rely upon rumours and third hand transcriptions of what’s been said on IRC.

The original Paludis client was more or less designed to be a less perverse version of emerge, with additional options for things like querying. On the plus side, it made things slightly easier for Gentoo users to pick up. However:

• Users would be confused as to why things like --show-use-descriptions, which is an option for the --install action, would not also do something with --query. Funnily enough, users don’t seem to wonder why emerge --sync --changelog --pretend doesn’t work.
• We had to be extremely careful allocating short options. Thus, we only allocated short options when we were sure an option would regularly be used both enabled and disabled, since otherwise it could just go in PALUDIS_OPTIONS. This worked well enough for users who were paying attention, but seemed to scare off certain people who jumped in without understanding what they were doing.
• It meant we regularly had to decide between breaking familiarity or doing something dumb just to make Portage converts comfortable.

I’ve never especially liked the result. It’s not close enough to emerge to make learning zero cost, and it’s too close to emerge to be a pleasant user interface. Fortunately, the UI and the library are mostly nicely separated, so we can fix this. The plan is as follows:

• A new client, named cave, will be provided. All pretence of feeling anything like emerge has been abandoned for cave. Instead, the interface is roughly based upon git and similar tools.
• Depending upon whether or not we can be bothered, a second new client named egress may or may not be produced. If it is, it will be command-line-compatible with emerge, and with a reasonably similar output format (within reason). I’m not convinced something like this would be worth the effort to write, although there are some Gentoo developers who insist that something that behaves almost identically to emerge is a critical requirement for any official Gentoo package manager…
• The paludis client will be deprecated and phased out.

To make things simpler, we’ve also decided to use the introduction of cave to switch to the shiny new resolver. The shiny new resolver is a lot more flexible, a lot more powerful and a heck of a lot easier to maintain and modify than the old resolver. However, making the paludis client use it would be quite a bit of work, so we’re not going to backport it.

There is no specific timeframe for any of this, and no estimates will be provided. There are, however, lists of things that need to be done.

• The Basic Functionality milestone contains all of the things that have to be done before we consider cave to be ‘more or less usable’. Until this milestone is complete, we won’t be enabling cave in any of the ebuilds on Gentoo, and we don’t recommend users sneakily enabling it themselves. Note in particular that some of the things not yet implemented include “displaying a user-readable error message rather than a big fat “UI not implemented!” when attempting an unsafe package uninstall, and “handle virtual blockers”.
• The Useful Functionality milestone contains all of the things that have to be done before we remove the paludis client. We will be making cave easily available before it includes every bit of functionality present in paludis, but we won’t be forcing a switch until we’ve either implemented equivalents for every feature or decided we’re not going to support a particular feature at all.
• The Long Term Extras milestone contains things we’ll be doing at some point.

These milestones are probably incomplete, and are definitely open to arbitrary additions, removals, changes and being ignored based upon developer whims, available development time, bribery, patches, lack of interest, features looking fun or boring to code and the phase of the moon.

I realise this is probably futile, but I should stress that users should not attempt to enable or use cave on Gentoo until we pass the Basic Functionality milestone. This isn’t taking away your fun — some of the not yet completed Basic Functionality items are necessary for basic usability and correct operation on Gentoo.

Paludis kdebuild-1 Removal

kdebuild-1 was an EAPI used by the Gentoo KDE team and the Genkdesvn project for various live ebuilds in overlays. It was created following a request from the Gentoo KDE project, who needed certain features that Portage had not yet implemented. It was a documented, public standard, in the same way that EAPIs 0, 1 and 2 are.

The Gentoo Council has decided that all mention of the kdebuild-1 EAPI is to be removed from the Package Manager Specification immediately. A request that this be done only after a deprecation period to give users more time to catch up was rejected. As such, support for this EAPI, including support for uninstalling packages with that EAPI, will be removed from Paludis in version 0.44.

Users should verify that they have no kdebuild-1 packages installed. If Paludis was built with the ‘inquisitio’ use flag enabled, you can use:

$ inquisitio -K installed -k EAPI kdebuild-1

Otherwise, use:

$ paludis -q '*/*::/[.EAPI=kdebuild-1]'

and, if any such packages are installed, they must be uninstalled before Paludis is upgraded.

Changes to the Paludis Git Syncer

I’ve just committed two changes to the Paludis Git syncer.

The first allows you to do sync_options = --branch=foo to specify a particular branch. I don’t expect this to be widely used, since branching for repositories usually means you’re not taking advantage of the better facilities available for that kind of thing. Still, it has its uses.

The second is sync_options = --reset. Unlike with, say, rsync, syncing via Git will merge any changes you’ve made to the repository with new upstream changes (it uses git pull). With --reset, it will instead discard any local changes and just become equal to whatever you’re syncing against (using instead git fetch and git reset --hard).

It’s a matter of considerable debate as to whether the reset behaviour is the right thing to do.

On the one hand, some people like working directly on a checkout to which Paludis is pointed, but still want cave sync to bring in updates.

On the other hand, doing that is horrible and evil, and a much better workflow is this:

• Have a local checkout for development work. Commit your changes to it.
• Have a separate checkout that is synced against your local checkout for Paludis use. After making changes, commit them to your local checkout and sync.
• When happy with your changes, rebase and squash them in your local checkout, and then push them upstream.

Doing that requires --reset, since otherwise the checkout Paludis sees will end up as some horrible auto-merged mess that includes changes you thought you’d discarded ages ago.

I’m strongly considering making --reset by default sometime in the future. However, this will make syncing a destructive operation for anyone who hates puppies enough to be doing work on a Paludis-synced checkout (in the same way that it’s already destructive for rsync).

Managing Accounts with the Package Manager

Paludis is a multi-format package manager. One beneficial side effect of this is that the core code is sufficiently flexible to make handling things that aren’t really ‘packages’ in the conventional sense very easy; in the past, this has been used to deliver unavailable, unwritten and unpackaged repositories.

One of the things Exherbo inherited from Gentoo without modification was user and group management. In Gentoo, this is done by functions called enewuser and enewgroup from eutils.eclass; a package that needs a user or group ID must call these functions from pkg_setup. Although usable, this is moderately icky; Exherbo can do better than that.

Really, user and group accounts are just resources. A package that needs a particular user ID can be thought of as depending upon that ID — the only disconnect is that currently dependencies are for packages, not resources. Can we find a way of representing resources as something like packages, in a way that makes sense?

Fortunately, the obvious solution works. Having user/paludisbuild and group/paludisbuild as packages makes sense; adding the user or group is equivalent to installing the appropriate package, and if the user or group is present on the system, it shows up as installed. Then, instead of calling functions, the exheres can just do:

DEPENDENCIES="
        build+run:
            user/paludisbuild
            group/paludisbuild
    "

What about defaults? Different users need different shells, home directories, groups and so on. We could represent these a bit like options, but there’s a better way.

If two or more ebuilds need the same user, they all have to do the useradd call. This means duplicating things like home directory information and preferred uid over lots of different ebuilds, which is bad. It would be better to place the users somewhere else. For Exherbo, we’ve gone with metadata/accounts/{users,groups}/*.conf. A user’s settings look something like this (the username is taken from a filename, so this would be metadata/accounts/users/paludisbuild.conf):

shell = /bin/bash
gecos = Used by Paludis for operations that require an unprivileged user
home = /var/tmp/paludis
primary_group = paludisbuild
extra_groups =
preferred_uid =

And a group, metadata/accounts/groups/paludisbuild.conf:

preferred_gid =

We only specify ’empty’ keys for demonstration purposes; ordinarily they would be omitted.

We automatically make users depend upon the groups they use. The existing dependency abstractions are sufficient for this. There’s a bit of trickery in Paludis to allow supplemental repositories to override user defaults found in their masters; details are in the source for those who care.

One more thing to note: with accounts specified this way, we can be sure that the package manager only manages relevant accounts. There’s no danger of having the package manager accidentally start messing with your user accounts.

So what are the implications?

• We’re no longer tied to a particular method of adding users. If a user doesn’t want to use useradd and groupadd, they can write their own handler for the package manager to update users via LDAP or whatever. Paludis supports multiple handlers here.
• Users who would rather manage a particular account manually can add it themselves, and the package manager will treat it as being already installed and won’t try to mess with it.
• User and group defaults are in one place, not everywhere that uses them.
• It’s much more obvious when an account is going to be added.
• Accounts that are no longer required can be purged using the usual uninstall-unused mechanism.

And what does it look like?

$ paludis -pi test-pkg
Building target list...
Building dependency list...   

These packages will be installed:

* group/alsogroupdemo [N 0]
    Reasons: *user/accountsdemo-0:0::accounts
    "alsogroupdemo"
* group/groupdemo [N 0]
    Reasons: *user/accountsdemo-0:0::accounts
    "groupdemo"
* group/thirdgroupdemo [N 0]
    Reasons: *user/accountsdemo-0:0::accounts
    "thirdgroupdemo"
* user/accountsdemo [N 0]
    Reasons: *test-cat/test-pkg-2:2::ciaranm_exheres_test
    "A demo account"
* test-cat/test-pkg::ciaranm_exheres_test :2 [N 2] <target>
    -foo build_options: recommended_tests split strip
    "Dummy test package"

We can have a look at the accounts before they’re installed:

$ paludis -q accountsdemo groupdemo
* user/accountsdemo
    accounts:                0* {:0} 
    Username:                accountsdemo
    Description:             A demo account
    Default Group:           groupdemo
    Extra Groups:            alsogroupdemo thirdgroupdemo
    Shell:                   /sbin/nologin
    Home Directory:          /dev/null

* group/groupdemo
    accounts:                0* {:0} 
    Groupname:               groupdemo
    Preferred GID:           123

Note the dependencies:

$ paludis -qDM accountsdemo
* user/accountsdemo
    accounts:                0* {:0} 
    username:                accountsdemo
    gecos:                   A demo account
    default_group:           groupdemo
    extra_groups:            alsogroupdemo thirdgroupdemo
    shell:                   /sbin/nologin
    home:                    /dev/null
    dependencies:            group/alsogroupdemo, group/groupdemo, group/thirdgroupdemo
    location:                /var/db/paludis/repositories/ciaranm_exheres_test/metadata/accounts/users/accountsdemo.conf
    defined_by:              ciaranm_exheres_test

The install is fairly boring:

(4 of 5) Installing user/accountsdemo-0:0::accounts

* Executing phase 'merge' as instructed
>>> Installing user/accountsdemo-0:0::accounts using passwd handler
useradd -r accountsdemo -c 'A demo account' -G 'alsogroupdemo,thirdgroupdemo' -s '/sbin/nologin' -d '/dev/null'
>>> Finished installing user/accountsdemo-0:0::accounts

And once they’re installed:

$ paludis -q accountsdemo groupdemo
* user/accountsdemo
    installed-accounts:      0* {:0} 

* group/groupdemo
    installed-accounts:      0* {:0} 

Exherbo will be migrating to this new mechanism shortly — package manager support is already there (it was only a few hours’ work), so it’s just a case of gradually hunting down and killing those enew* function calls.

UIs for Parallelism

Getting the UI right for parallel execution is tricky.

Make, for example, doesn’t even try. It just displays output from all the children all mixed together, and relies upon commands prefixing their output with a filename or similar identifier. This generally works for what make needs to do, but it’s a bit too crude for some things.

For the secret “not supposed to use it” parallelism in Paludis, we’re currently using something slightly more sophisticated: each line of output from each child job is prefixed by the job’s name. So you’ll see things like:

sync unavailable> wget -T 30 -t 1 -O /var/tmp/paludis-tarsync-cvh9gX/exherbo_repositories.tar.bz2 http://git.exherbo.org/exherbo_repositories.tar.bz2
sync unavailable-unofficial> wget -T 30 -t 1 -O /var/tmp/paludis-tarsync-kvEzRZ/exherbo_unofficial_repositories.tar.bz2
sync unavailable-unofficial> --14:24:06--  http://git.exherbo.org/exherbo_unofficial_repositories.tar.bz2
sync unavailable-unofficial>            => `/var/tmp/paludis-tarsync-kvEzRZ/exherbo_unofficial_repositories.tar.bz2'
sync unavailable> --14:24:06--  http://git.exherbo.org/exherbo_repositories.tar.bz2

Something like this is necessary because the commands involved don’t do their own prefixing. But whilst it’s an improvement, the output’s rather cluttered and it’s hard to work out what’s going on. I’ve been experimenting with something more like this (slowdown induced artificially to make the point):

Starting sync

Repository                    Status                Pending Active  Done
-> alsa                       starting              16      1       0
-> ciaranm                    starting              15      2       0
-> ferdy                      starting              14      3       0
-> compnerd                   starting              13      4       0
-> arbor                      starting              12      5       0
-> compnerd                   success               12      4       1
-> gnome                      starting              11      5       1
-> alsa                       failed                11      4       2
    ... fatal: The remote end hung up unexpectedly
-> hardware                   starting              10      5       2
-> ferdy                      success               10      4       3
-> media                      starting              9       5       3
-> gnome                      success               9       4       4
-> mozilla                    starting              8       5       4
-> hardware                   success               8       4       5
-> pioto-exheres              starting              7       5       5
-> media                      success               7       4       6
-> python                     starting              6       5       6
-> pioto-exheres              success               6       4       7
-> rbrown                     starting              5       5       7
-> mozilla                    success               5       4       8
-> texlive                    starting              4       5       8
-> ciaranm                    active                4       5       8
-> python                     success               4       4       9
-> unavailable                starting              3       5       9
-> texlive                    success               3       4       10
-> unavailable-unofficial     starting              2       5       10
-> rbrown                     success               2       4       11
-> unwritten                  starting              1       5       11
-> arbor                      active                1       5       11
-> unwritten                  success               1       4       12
-> x11                        starting              0       5       12
-> ciaranm                    active                0       5       12
    ... Initialized empty Git repository in /var/db/paludis/repositories/ciaranm/.git/
-> unavailable                active                0       5       12
    ... Literal data: 0 bytes
    ... Matched data: 0 bytes
    ... File list size: 749
    ... File list generation time: 0.001 seconds
    ... File list transfer time: 0.000 seconds
    ... Total bytes sent: 771
    ... Total bytes received: 26
    ...
    ... sent 771 bytes  received 26 bytes  1594.00 bytes/sec
    ... total size is 57557  speedup is 72.22
-> unavailable-unofficial     active                0       5       12
    ... Literal data: 0 bytes
    ... Matched data: 0 bytes
    ... File list size: 110
    ... File list generation time: 0.001 seconds
    ... File list transfer time: 0.000 seconds
    ... Total bytes sent: 132
    ... Total bytes received: 26
    ...
    ... sent 132 bytes  received 26 bytes  316.00 bytes/sec
    ... total size is 4013  speedup is 25.40
-> arbor                      active                0       5       12
-> unavailable                success               0       4       13
-> unavailable-unofficial     success               0       3       14
-> x11                        active                0       3       14
-> ciaranm                    active                0       3       14
-> ciaranm                    success               0       2       15
-> x11                        success               0       1       16
-> arbor                      success               0       0       17

 * Cleaning write cache for ebuild format repositories...
 * Done cleaning write cache for ebuild format repositories

 * No unread news items found

Sync results

* alsa:                       sync of '/var/db/paludis/repositories/alsa' from 'git://git.exherbo.org/demonstrate/failure/alsa.git' failed (paludis::SyncFailedError)
    Log file:                 /var/log/sync-alsa.1225118654.log
* arbor:                      success
* ciaranm:                    success
* compnerd:                   success
* ferdy:                      success
* gnome:                      success
* hardware:                   success
* media:                      success
* mozilla:                    success
* pioto-exheres:              success
* python:                     success
* rbrown:                     success
* texlive:                    success
* unavailable:                success
* unavailable-unofficial:     success
* unwritten:                  success
* x11:                        success

In particular:

• Output is automatically logged to a file, rather than dumping everything to the screen. That file can be automatically removed if the job succeeds.
• A summary of what’s going on is displayed every time a job starts or finishes.
• Every ten seconds (for some value of ten), if a job hasn’t finished, we automatically display the last ten (for some other value of ten) lines of its output (but not any output we’ve already displayed) to the screen, along with a note that it’s still active. We also tail the log if a job fails.

This seems to be a comfortable balance between not showing anything except job statuses (which for jobs that can take an hour to run might well leave you wondering whether it’s hung) and just dumping everything to the screen. Now it’s just a case of finding appropriate values of ten…

Paludis is about Choices

Paludis 0.32 will include a new mechanism called ‘choices’. Examples of choices on Gentoo include use flags, the various USE_EXPAND settings (linguas, video_cards and so on) and a shiny new setting called build_options. For use flags and the USE_EXPAND settings, there’s no change from a user perspective, and you can carry on using use.conf as normal — choices are merely a more general mechanism for handling package settings.

On Exherbo, things do change a bit, and if you’re one of those naughty people who ignored the “Exherbo has no users” warning you’d better have been paying attention.

So what’s this build_options thing? You’ll see it cropping up for most packages:

* net-print/cups::gentoo [U 1.3.8-r1 -> 1.3.8-r2] <target>
    Reasons: app-text/ghostscript-gpl-8.62:0::installed, net-print/foomatic-filters-3.0.20080507:0::installed, 2 more
    X acl -avahi -dbus -gnutls -java jpeg -kerberos -ldap pam perl -php png ppds python -samba
    -slp ssl -static -tiff -xinetd -zeroconf LINGUAS: -de en -es -et -fr -he -id -it -ja -pl -sv
    -zh_TW build_options: -optional_tests split strip

Packages using 0-based EAPIs will usually have three build options:

• optional_tests. This handles what --checks used to do. If this choice is set, src_test will be run.
• split. This used to be --debug-build split.
• strip. This used to be --debug-build internal.

Packages using kdebuild-1 or exheres-0 will have recommended_tests instead of optional_tests, which defaults to being enabled rather than disabled.

Despite what you might expect from older Paludis versions, these are not use flags. You can set them in use.conf, though — this means you can select whether to run tests or whether to keep debug information on a per-package basis, which was tricky to do with the old commandline options.

Path and installable-to dependencies

A while ago, Paludis got from-repository dependencies. 0.32 will extend this further with path and installable-to dependencies.

Path dependencies are fairly simple: cat/pkg::/ will match cat/pkg installed at root /. A full path can be used too, so cat/pkg::/var/mychroot will work (as well as anything involving non-/ roots works, anyway…).

Installable-to dependencies take a bit more explaining. There are two kinds of installable-to dependency: repository and path. An installable-to-repository dependency looks like cat/pkg::repo? and is matched by any cat/pkg that could be installed to the repository named repo. An installable-to-path dependency looks like cat/pkg::/? and is matched by any cat/pkg that could be installed to any repository with root /.

Installable-to dependencies will not match masked packages. To specify “packages that would be installable, ignoring masks”, use a double question mark, like cat/pkg::/??. At present the double question mark ignores all masks, even those that cannot be overridden (unsupported EAPI, unavailable or unwritten). This may change to only include overridable masks; I’m not sure which is better behaviour yet.

Dealing with Unwritten Packages

A while ago Exherbo came up with a solution for the “large number of repositories” problem. It turns out this solution works rather well, and I’m now convinced that anyone rushing off and assuming that Git on its own will magically solve anything other than CVS sucking is going in the wrong direction.

Paludis also has ways of tracking packages installed by hand. Whilst useful on Gentoo, this really comes into play on Exherbo, where we don’t want to end up with a massive unmaintainable tree full of packages used by only two people.

The logical next step is tracking packages that don’t exist yet.

An unimaginative person might think that the way to solve this would be to have a bug for each requested package. But that gets messy — it’s not integrated with the package manager, and requires considerable extra effort from the user. A while ago Bernd suggested something more interesting: tracking unwritten packages in a repository in a similar way to how we do unavailable packages.

The implications:

• Most obviously, we can keep track of things we want that haven’t been written yet in a way that doesn’t involve leaving the package manager to look things up.
• We can also use it to track version bumps that we know will take a while. Doing paludis --query '>=hugescarypackage-4', say, would show that it’s being worked upon.
• We can also depend upon things that don’t exist, rather than leaving incomplete dependency strings around. This is fine in at least two cases — if a dependency is conditional upon an option that should probably be implemented but isn’t yet, we can add the option and make it unusable. And we can handle obscure suggested dependencies (e.g. git has lots of optional dependencies upon weird perl modules, so we can say “if you want support for git-send-email with SSL, you need to write such and such a package”).
• Bored developers could simply paludis --query '*/*::unwritten' and get ideas for what to do.

Adding support to Paludis for this only took a couple of hours. So now Exherbo developers can use this:

format = unwritten
location = /var/db/paludis/repositories/unwritten
sync = git://git.exherbo.org/unwritten.git
importance = -100

And see this:

$ paludis -q genesis
* sys-apps/genesis
    unwritten:               (1.0)X* {:0} 
    Description:             The daddy of all init systems
    Homepage:                http://www.exherbo.org/
    Comment:                 We need an init system.
    Masked by unwritten:     Package has not been written yet

As with unavailable, installing behaves sensibly:

$ paludis -pi genesis
Building target list... 
Building dependency list...
Query error:
  * In program paludis -pi genesis:
  * When performing install action from command line:
  * When executing install task:
  * When building dependency list:
  * When adding PackageDepSpec 'sys-apps/genesis':
  * All versions of 'sys-apps/genesis' are masked. Candidates are:
    * sys-apps/genesis-1.0:0::unwritten: Masked by unwritten (Package has not been written yet)

Those interested in the repository format can browse the tree. The observant might notice that the file format is quite similar to the one used by unavailable repositories, and wonder whether I was feeling lazy and swiped a load of code rather than implementing a new repository from scratch.

In the spirit of silly buzzwords, one could argue that this increases distributivity and democratisation of the repository, since there’s nothing to stop motivated users from creating their own wishlist repositories. It wouldn’t be hard to make a simple web interface that lets users request and vote on packages and version bumps, automatically generating a repository that anyone can use. But fortunately Exherbo is a dictatorship and has no users, so we don’t have to put up with that kind of nonsense.