Managing Accounts with the Package Manager

Paludis is a multi-format package manager. One beneficial side effect of this is that the core code is sufficiently flexible to make handling things that aren’t really ‘packages’ in the conventional sense very easy; in the past, this has been used to deliver unavailable, unwritten and unpackaged repositories.

One of the things Exherbo inherited from Gentoo without modification was user and group management. In Gentoo, this is done by functions called enewuser and enewgroup from eutils.eclass; a package that needs a user or group ID must call these functions from pkg_setup. Although usable, this is moderately icky; Exherbo can do better than that.

Really, user and group accounts are just resources. A package that needs a particular user ID can be thought of as depending upon that ID — the only disconnect is that currently dependencies are for packages, not resources. Can we find a way of representing resources as something like packages, in a way that makes sense?

Fortunately, the obvious solution works. Having user/paludisbuild and group/paludisbuild as packages makes sense; adding the user or group is equivalent to installing the appropriate package, and if the user or group is present on the system, it shows up as installed. Then, instead of calling functions, the exheres can just do:

DEPENDENCIES="
        build+run:
            user/paludisbuild
            group/paludisbuild
    "

What about defaults? Different users need different shells, home directories, groups and so on. We could represent these a bit like options, but there’s a better way.

If two or more ebuilds need the same user, they all have to do the useradd call. This means duplicating things like home directory information and preferred uid over lots of different ebuilds, which is bad. It would be better to place the users somewhere else. For Exherbo, we’ve gone with metadata/accounts/{users,groups}/*.conf. A user’s settings look something like this (the username is taken from a filename, so this would be metadata/accounts/users/paludisbuild.conf):

shell = /bin/bash
gecos = Used by Paludis for operations that require an unprivileged user
home = /var/tmp/paludis
primary_group = paludisbuild
extra_groups =
preferred_uid =

And a group, metadata/accounts/groups/paludisbuild.conf:

preferred_gid =

We only specify ’empty’ keys for demonstration purposes; ordinarily they would be omitted.

We automatically make users depend upon the groups they use. The existing dependency abstractions are sufficient for this. There’s a bit of trickery in Paludis to allow supplemental repositories to override user defaults found in their masters; details are in the source for those who care.

One more thing to note: with accounts specified this way, we can be sure that the package manager only manages relevant accounts. There’s no danger of having the package manager accidentally start messing with your user accounts.

So what are the implications?

• We’re no longer tied to a particular method of adding users. If a user doesn’t want to use useradd and groupadd, they can write their own handler for the package manager to update users via LDAP or whatever. Paludis supports multiple handlers here.
• Users who would rather manage a particular account manually can add it themselves, and the package manager will treat it as being already installed and won’t try to mess with it.
• User and group defaults are in one place, not everywhere that uses them.
• It’s much more obvious when an account is going to be added.
• Accounts that are no longer required can be purged using the usual uninstall-unused mechanism.

And what does it look like?

$ paludis -pi test-pkg
Building target list...
Building dependency list...   

These packages will be installed:

* group/alsogroupdemo [N 0]
    Reasons: *user/accountsdemo-0:0::accounts
    "alsogroupdemo"
* group/groupdemo [N 0]
    Reasons: *user/accountsdemo-0:0::accounts
    "groupdemo"
* group/thirdgroupdemo [N 0]
    Reasons: *user/accountsdemo-0:0::accounts
    "thirdgroupdemo"
* user/accountsdemo [N 0]
    Reasons: *test-cat/test-pkg-2:2::ciaranm_exheres_test
    "A demo account"
* test-cat/test-pkg::ciaranm_exheres_test :2 [N 2] <target>
    -foo build_options: recommended_tests split strip
    "Dummy test package"

We can have a look at the accounts before they’re installed:

$ paludis -q accountsdemo groupdemo
* user/accountsdemo
    accounts:                0* {:0} 
    Username:                accountsdemo
    Description:             A demo account
    Default Group:           groupdemo
    Extra Groups:            alsogroupdemo thirdgroupdemo
    Shell:                   /sbin/nologin
    Home Directory:          /dev/null

* group/groupdemo
    accounts:                0* {:0} 
    Groupname:               groupdemo
    Preferred GID:           123

Note the dependencies:

$ paludis -qDM accountsdemo
* user/accountsdemo
    accounts:                0* {:0} 
    username:                accountsdemo
    gecos:                   A demo account
    default_group:           groupdemo
    extra_groups:            alsogroupdemo thirdgroupdemo
    shell:                   /sbin/nologin
    home:                    /dev/null
    dependencies:            group/alsogroupdemo, group/groupdemo, group/thirdgroupdemo
    location:                /var/db/paludis/repositories/ciaranm_exheres_test/metadata/accounts/users/accountsdemo.conf
    defined_by:              ciaranm_exheres_test

The install is fairly boring:

(4 of 5) Installing user/accountsdemo-0:0::accounts

* Executing phase 'merge' as instructed
>>> Installing user/accountsdemo-0:0::accounts using passwd handler
useradd -r accountsdemo -c 'A demo account' -G 'alsogroupdemo,thirdgroupdemo' -s '/sbin/nologin' -d '/dev/null'
>>> Finished installing user/accountsdemo-0:0::accounts

And once they’re installed:

$ paludis -q accountsdemo groupdemo
* user/accountsdemo
    installed-accounts:      0* {:0} 

* group/groupdemo
    installed-accounts:      0* {:0} 

Exherbo will be migrating to this new mechanism shortly — package manager support is already there (it was only a few hours’ work), so it’s just a case of gradually hunting down and killing those enew* function calls.

EAPI 2: doman language support

This is the final post in a series on EAPI 2.

The doman helper is one of those pesky little beasts that makes specifying EAPI behaviour formally such a nuisance (although it is nowhere near as horrible as dohtml). EAPI 2 makes it even peskier.

I’ll try that again.

The doman helper makes writing ebuilds substantially easier by automagically doing the right thing when installing manual pages, freeing the developer from having to care about manual sections. EAPI 2 makes doman even more useful by making it aware of language codes as well as sections.

The painful details are available in PMS, but basically this will now ‘do the right thing’:

doman foo.1 foo.en.1 foo.en_GB.1

Previously only the first of the items would go to the right place.

This one’s a Gentoo innovation; see bug 222439 for its history. It was shamelessly stolen for exheres-0, but was too late for kdebuild-1.

EAPI 2: default_ phase functions and the default function

This post is part of a series on EAPI 2.

With EAPIs 0 and 1, if you want to add something to, say, src_unpack, you have to manually write out the default implementation and then add your code. This is easy to screw up — developers are highly prone to getting the quoting wrong and forgetting which functions do and do not want a || die on the end.

EAPI 2 makes the default implementation of phase functions available as functions themselves. These functions are named default_src_unpack, default_src_configure and so on.

Typing out default_src_compile in full is pointless, though (especially since it’s illegal to call phase functions or default phase functions from other phase functions). So we also introduce the special default function, which calls whichever default_ phase function is appropriate for the phase we’re in. Thus:

src_compile() {
    default
    if useq extras ; then
        emake extras || die "splat"
    fi
}

Both features first appeared in exheres-0.

An alternative proposal (I think it came from the Pkgcore camp) was to make all EAPI default implementations available through functions named like eapi0_src_compile, eapi1_src_compile and eapi2_src_compile. This proposal was rejected because various Paludis people moaned about it not making sense or having any legitimate use cases (the ‘obvious’ use cases don’t work if you think them through), and no-one stood up to defend it.

EAPI 2: src_configure and src_compile

This is post five in a series describing EAPI 2.

EAPI 2 splits src_compile into src_configure and src_compile. Like src_prepare, it’s mostly a convenience thing to reduce copying default implementations, although in this case it also makes it easier to hook in code in between configure and make being run.

The default src_configure implementation behaves like this:

src_configure() {
    if [[ -x ${ECONF_SOURCE:-.}/configure ]]; then
        econf
    fi
}

This is the first half of EAPI 1’s src_compile, not the non-ECONF_SOURCE-aware EAPI 0 version.

The default src_compile implementation is reduced accordingly:

src_compile() {
    if [[ -f Makefile ]] || [[ -f GNUmakefile ]] || [[ -f makefile ]]; then
        emake || die "emake failed"
    fi
}

The split configure / compile setup was first used in exheres-0, which uses more elaborate default implementations. Like src_prepare, it was considered but rejected for kdebuild-1 because of eclass difficulties.

EAPI 2: src_prepare

This is post four in a series describing EAPI 2.

EAPI 2 has a new phase function called src_prepare. It is called after src_unpack, and can be used to apply patches, do sed voodoo and so on. The default implementation does nothing.

This function is purely for convenience. It gets rather tedious copying out the default implementation of src_unpack just to add a patch in somewhere.

src_prepare was first introduced in exheres-0 (which has a more elaborate default implementation). It was considered but rejected for kdebuild-1 because making best use of it requires eclass awareness, and the packages using kdebuild-1 had to share eclasses with the main Gentoo tree.

EAPI 2: !! Blockers

This is part three of a series of posts describing EAPI 2.

Blockers are a nuisance for end users. It’s rarely obvious how to fix them or what they mean, and getting it wrong can leave systems unusable.

There have been various proposals on how to fix this. For exheres-0, we’re going to go with something like this:

DEPENDENCIES="
    !app-misc/superfrozbinator [[
        description = [ Can only have one frozbinator installed at once ]
        resolution = uninstall-blocked-after
        url = [ http://explain.example.org/?only-one-frozbinator ]
    ]]
    !dev-libs/icky [[
        description = [ Having icky installed breaks the build process ]
        resolution = [ manual ]
        url = [ http://explain.example.org/?myfroz-hates-icky ]
    ]]"

The user can then be presented with a list of things that would need to be uninstalled to resolve blockers, along with clear descriptions of why they need to do so. Once the user has explicitly accepted the uninstalls, the package manager could then safely perform the installs.

Unfortunately, annotations aren’t something that can be implemented for Portage any time soon. Instead, Portage has gone with a fairly horrible and dangerous semi-automatic block resolution system that sometimes removes blocked packages automatically (often screwing up the user’s system in the process). Whilst doing so, Portage changed the meaning of EAPI 0 / 1 blockers from “this must not be installed when we do the build” to “this must be uninstalled after we do the build”.

EAPI 2 introduces a new kind of blocker using double exclamation marks, like !!app-misc/other. This goes back to the old meaning of “this must not be installed when we do the build”, keeping !app-misc/other for “this must be uninstalled after we do the build”.

This does not, unfortunately, make the user any safer, but it does allow packages that really can’t have something installed at build time to say so.

EAPI 2: Use Dependencies

This is the second post in a series of posts describing EAPI 2.

Use dependencies have been needed for a very long time. They eliminate most of the built_with_use errors you see during pkg_setup, replacing them with an error that is seen at pretend-install time.

The first two real world trials of use dependencies were with Exherbo‘s exheres-0 and Gentoo‘s kdebuild-1. It became apparent that an awful lot of packages would end up with dependencies like:

blah? ( app-misc/foo[blah] ) !blah? ( app-misc/foo )
monkey? ( app-misc/foo[monkey] ) !monkey? ( app-misc/foo[-monkey] )
fnord? ( app-misc/foo ) !fnord? ( app-misc/foo[-fnord] )

Syntactically, that’s rather inconvenient. For exheres-0 and kdebuild-1, we added the following syntax:

[opt]

The flag must be enabled.

[opt=]

The flag must be enabled if the flag is enabled for the package with the dependency, or disabled otherwise.

[opt!=]

The flag must be disabled if the flag is enabled for the package with the dependency, or enabled otherwise.

[opt?]

The flag must be enabled if the flag is enabled for the package with the dependency.

[opt!?]

The flag must be enabled if the use flag is disabled for the package with the dependency.

[-opt]

The flag must be disabled.

[-opt?]

The flag must be disabled if the flag is enabled for the package with the dependency.

[-opt!?]

The flag must be disabled if the flag is disabled for the package with the dependency.

Dependencies could be combined by specifying multiple blocks, as in foo/bar[baz][monkey?].

For EAPI 2, Zac decided to go with an arbitrarily different syntax:

[opt]

The flag must be enabled.

[opt=]

The flag must be enabled if the flag is enabled for the package with the dependency, or disabled otherwise.

[!opt=]

The flag must be disabled if the flag is enabled for the package with the dependency, or enabled otherwise.

[opt?]

The flag must be enabled if the flag is enabled for the package with the dependency.

[!opt?]

The flag must be disabled if the use flag is disabled for the package with the dependency.

[-opt]

The flag must be disabled.

And to combine use dependencies, one uses a comma, as in foo/bar[baz,monkey?].

In both cases, the slot dependency must go before the dependency, so foo/bar:1[baz], not foo/bar[baz]:1. The use dependency goes after any version restrictions, so >=foo/bar-2.1:2[baz].

In both cases, it is illegal to reference a use flag that does not exist (including USE_EXPAND flags that are not explicitly listed in IUSE). So foo/bar[opt] when any version of foo/bar does not have opt in IUSE is illegal and has undefined behaviour, as is foo/baz[opt?] if either the owning package or foo/baz has no opt. For cases where only some versions of a package have a flag, use dependencies can be combined with version or slot restrictions.

From an implementation perspective: the package manager should not try to automatically solve unmet use dependencies. The package manager doesn’t know the impact of changing a use flag (changing some flags makes a system unbootable), so it can’t simply override the user’s choice. (Paludis will suggest an automatic reinstall if and only if the user has already modified their use.conf, so you don’t need to manually reinstall a dependency if you’re ok with altering the flags with which it is built.)

EAPI 2: SRC_URI Arrows

This is the first item in a series of posts describing EAPI 2.

Some upstreams use annoyingly named tarballs. Most commonly, they don’t include either the package name or the version in the filename. Because DISTDIR is a flat directory, this causes problems — the tree must not use two different tarballs with the same name. Previously, the solution to horrible upstream naming was to manually mirror the tarball with a new filename; this was considered excessively icky.

There have been two sane solutions proposed for this over time. The one we didn’t use was to define a DISTDIR_SUBDIR variable, and do all downloads into there. This would have made the A variable quite a bit messier, and complicated sharing certain tarballs between packages.

The arrows solution was something I came up with for early Paludis experimental EAPIs, and was adopted for kdebuild-1 and from there into 2; it’s also always been present in exheres-0. It works like this:

SRC_URI="http://example.com/stupid-named/1.23/stupid.tar.bz2 -> stupid-1.23.tar.bz2"

or using variables:

SRC_URI="http://example.com/stupid-named/${PV}/${PN}.tar.bz2 -> ${P}.tar.bz2"

This tells the package manager to look at the URL on the left of the arrow, but save to the filename on the right.

Mirroring effects are slightly subtle. Consider:

SRC_URI="mirror://foo/${PN}/${PV}.tar.bz2 -> ${P}.tar.bz2"

The package manager will look both on mirror://foo/ and mirror://gentoo/ for the download. When looking on foo, the raw filename must be used, but when looking on gentoo, the rewritten filename is used.

Anyone using arrows on mirror://gentoo/ URIs gets stabbed.

Arrows make another proposed but rejected EAPI feature irrelevant: there was a proposal floating around (I think it originated with drobbins, but I can’t find an original source) to make unpack ignore ;sf=tbz2 and ;sf=tgz suffixes on filenames, for interoperability with gitweb. Arrows are a more general solution.

Implementation-wise, anyone still using a lexer-based parser will need a single token of lookahead for this. Apparently this causes minor inconveniences in some broken programming languages that only support what C++ calls input iterators; I consider this a good thing, because it might make people either use a better iterator model or stop using lexers.

What’s in EAPI 2?

EAPI 2 has been approved by the Gentoo Council and so can now be used in ebuilds. The first package manager with support was Paludis 0.30.1; Portage support came along with 2.2_rc11.

EAPI 2 consists purely of extensions to EAPI 1. The new features are:

• SRC_URI arrows
• Use dependencies
• !! blockers
• src_prepare
• src_configure, and a different default src_compile
• default_ phase functions and the default function
• doman language support

Formal definitions can be found in PMS; an overview of each feature will follow in subsequent posts.