Intuitive Packaging is Doing It Wrong

Donnie has taken time out of his busy schedule of managing Gentoo to comment on some possible design issues for EAPI 3. He believes that adding support for exheres-0 style DEFAULT_ parameters to ebuilds would result in a less intuitive packaging system, which he considers bad.

Unfortunately, both the term ‘intuitive’ and the conclusion are nonsense. Ebuilds are not intuitive, intuitiveness would not be a useful property for them to have, and allowing parametrisation of default_ functions would not alter any of this.

The only truly intuitive interface is the nipple.
– Jay Vollmer

Let’s look at what intuitive means:

in⋅tu⋅i⋅tive /ɪnˈtuɪtɪv, -ˈtyu-/ [in-too-i-tiv, -tyoo-]
-adjective

1. perceiving by intuition, as a person or the mind.
2. perceived by, resulting from, or involving intuition: intuitive knowledge.
3. having or possessing intuition: an intuitive person.
4. capable of being perceived or known by intuition.

Ok, let’s look at intuition:

in⋅tu⋅i⋅tion /ˌɪntuˈɪʃən, -tyu-/ [in-too-ish-uhn, -tyoo-]
–noun

1. direct perception of truth, fact, etc., independent of any reasoning process; immediate apprehension.
2. a fact, truth, etc., perceived in this way.
3. a keen and quick insight.
4. the quality or ability of having such direct perception or quick insight.

So apparently Donnie wants people to be able to write ebuilds without requiring rational thought. Whilst that would go some way towards explaining the state of the tree, it’s evident that ebuilds are not currently intuitive and should not be made intuitive.

What qualities, then, should ebuild design aspire to? Let’s start with these:

1. Ebuilds should be as obvious as reasonably possible, given the complications of the underlying packaging system and the overall design requirements, to a person with an appropriate level of skill and access to the documentation.
2. Ebuilds should work to reduce the amount of boilerplate and cut-and-paste duplication required.
3. Ebuilds should take steps to catch and prevent common errors.

Looking at the first point, one may think it is too weak a requirement — why not “ebuilds should be accessible to your average user”? But then, why should it be?

If you think the average user should have to write ebuilds to be able to get their package manager to track a package they can build by hand — why? Why not simply improve the package manager to be able to track hand-built packages without ebuilds?

If you think the average user should be able to modify ebuilds to add in patches — why? Why not simply improve the package manager to make it easy for the user to add in patches to existing packages?

If you think it will help solve the developer shortage problem — why? There’s no shortage of badly written ebuilds sitting around in bugzilla; making it easier to create more badly written ebuilds won’t fix this. The problem Gentoo faces is how to get more high quality ebuilds, and doing that requires skilled developers who have read and understood the documentation.

Introducing DEFAULT_ parameters has no major effect either way on the first point.

The second and third points are where DEFAULT_ parameters kick in. The reason the default src_configure does something as opposed to nothing is that the something it does is enough for many ebuilds. If instead it were a no-op, a typical simple ebuild would be considerably longer.

Except, these days a lot of ebuilds have a few simple configure options controlled by use flags, so the default src_configure in EAPI 2 (or src_compile in EAPIs 0 and 1) is no good. DEFAULT_ parameters bring this proportion way down.

This brings us to why the default src_install is a no-op. For most packages, something along the lines of “if there’s a Makefile, make DESTDIR="${D}" install” is not enough. For a good proportion of packages, though, that plus an ebuild-supplied list of doc files would suffice.

Donnie claims that specifying things in variables this way is a major change in how ebuilds work. But there are already plenty of examples of things done in this style:

• The S variable is a declarative parameter to the package manager’s “before we run a phase” functions.
• Lots of eclasses make use of a DOCS variable.
• Indeed, nearly all parameterisation of eclasses is done through variables. It could just as easily be done by callback or overridable functions, but developers haven’t opted to do so.

A perfect example of that last point: Donnie’s own x-modular eclass has a variable named PATCHES, which ebuilds set in global scope. If x-modular were using EAPI 3 with a src_prepare and exheres-0 style declarative patches lists, the package manager would already be providing exactly what he’s gone out of his way to implement.

So what gives, Donnie? Do you think your use of PATCHES was a design mistake that you will be correcting? And do you think all those other developers who have been doing the same kind of thing for years are fundamentally wrong?

EAPI 2: doman language support

This is the final post in a series on EAPI 2.

The doman helper is one of those pesky little beasts that makes specifying EAPI behaviour formally such a nuisance (although it is nowhere near as horrible as dohtml). EAPI 2 makes it even peskier.

I’ll try that again.

The doman helper makes writing ebuilds substantially easier by automagically doing the right thing when installing manual pages, freeing the developer from having to care about manual sections. EAPI 2 makes doman even more useful by making it aware of language codes as well as sections.

The painful details are available in PMS, but basically this will now ‘do the right thing’:

doman foo.1 foo.en.1 foo.en_GB.1

Previously only the first of the items would go to the right place.

This one’s a Gentoo innovation; see bug 222439 for its history. It was shamelessly stolen for exheres-0, but was too late for kdebuild-1.

EAPI 2: default_ phase functions and the default function

This post is part of a series on EAPI 2.

With EAPIs 0 and 1, if you want to add something to, say, src_unpack, you have to manually write out the default implementation and then add your code. This is easy to screw up — developers are highly prone to getting the quoting wrong and forgetting which functions do and do not want a || die on the end.

EAPI 2 makes the default implementation of phase functions available as functions themselves. These functions are named default_src_unpack, default_src_configure and so on.

Typing out default_src_compile in full is pointless, though (especially since it’s illegal to call phase functions or default phase functions from other phase functions). So we also introduce the special default function, which calls whichever default_ phase function is appropriate for the phase we’re in. Thus:

src_compile() {
    default
    if useq extras ; then
        emake extras || die "splat"
    fi
}

Both features first appeared in exheres-0.

An alternative proposal (I think it came from the Pkgcore camp) was to make all EAPI default implementations available through functions named like eapi0_src_compile, eapi1_src_compile and eapi2_src_compile. This proposal was rejected because various Paludis people moaned about it not making sense or having any legitimate use cases (the ‘obvious’ use cases don’t work if you think them through), and no-one stood up to defend it.

EAPI 2: src_configure and src_compile

This is post five in a series describing EAPI 2.

EAPI 2 splits src_compile into src_configure and src_compile. Like src_prepare, it’s mostly a convenience thing to reduce copying default implementations, although in this case it also makes it easier to hook in code in between configure and make being run.

The default src_configure implementation behaves like this:

src_configure() {
    if [[ -x ${ECONF_SOURCE:-.}/configure ]]; then
        econf
    fi
}

This is the first half of EAPI 1’s src_compile, not the non-ECONF_SOURCE-aware EAPI 0 version.

The default src_compile implementation is reduced accordingly:

src_compile() {
    if [[ -f Makefile ]] || [[ -f GNUmakefile ]] || [[ -f makefile ]]; then
        emake || die "emake failed"
    fi
}

The split configure / compile setup was first used in exheres-0, which uses more elaborate default implementations. Like src_prepare, it was considered but rejected for kdebuild-1 because of eclass difficulties.

EAPI 2: src_prepare

This is post four in a series describing EAPI 2.

EAPI 2 has a new phase function called src_prepare. It is called after src_unpack, and can be used to apply patches, do sed voodoo and so on. The default implementation does nothing.

This function is purely for convenience. It gets rather tedious copying out the default implementation of src_unpack just to add a patch in somewhere.

src_prepare was first introduced in exheres-0 (which has a more elaborate default implementation). It was considered but rejected for kdebuild-1 because making best use of it requires eclass awareness, and the packages using kdebuild-1 had to share eclasses with the main Gentoo tree.

EAPI 2: !! Blockers

This is part three of a series of posts describing EAPI 2.

Blockers are a nuisance for end users. It’s rarely obvious how to fix them or what they mean, and getting it wrong can leave systems unusable.

There have been various proposals on how to fix this. For exheres-0, we’re going to go with something like this:

DEPENDENCIES="
    !app-misc/superfrozbinator [[
        description = [ Can only have one frozbinator installed at once ]
        resolution = uninstall-blocked-after
        url = [ http://explain.example.org/?only-one-frozbinator ]
    ]]
    !dev-libs/icky [[
        description = [ Having icky installed breaks the build process ]
        resolution = [ manual ]
        url = [ http://explain.example.org/?myfroz-hates-icky ]
    ]]"

The user can then be presented with a list of things that would need to be uninstalled to resolve blockers, along with clear descriptions of why they need to do so. Once the user has explicitly accepted the uninstalls, the package manager could then safely perform the installs.

Unfortunately, annotations aren’t something that can be implemented for Portage any time soon. Instead, Portage has gone with a fairly horrible and dangerous semi-automatic block resolution system that sometimes removes blocked packages automatically (often screwing up the user’s system in the process). Whilst doing so, Portage changed the meaning of EAPI 0 / 1 blockers from “this must not be installed when we do the build” to “this must be uninstalled after we do the build”.

EAPI 2 introduces a new kind of blocker using double exclamation marks, like !!app-misc/other. This goes back to the old meaning of “this must not be installed when we do the build”, keeping !app-misc/other for “this must be uninstalled after we do the build”.

This does not, unfortunately, make the user any safer, but it does allow packages that really can’t have something installed at build time to say so.

EAPI 2: Use Dependencies

This is the second post in a series of posts describing EAPI 2.

Use dependencies have been needed for a very long time. They eliminate most of the built_with_use errors you see during pkg_setup, replacing them with an error that is seen at pretend-install time.

The first two real world trials of use dependencies were with Exherbo’s exheres-0 and Gentoo’s kdebuild-1. It became apparent that an awful lot of packages would end up with dependencies like:

blah? ( app-misc/foo[blah] ) !blah? ( app-misc/foo )
monkey? ( app-misc/foo[monkey] ) !monkey? ( app-misc/foo[-monkey] )
fnord? ( app-misc/foo ) !fnord? ( app-misc/foo[-fnord] )

Syntactically, that’s rather inconvenient. For exheres-0 and kdebuild-1, we added the following syntax:

[opt]

The flag must be enabled.

[opt=]

The flag must be enabled if the flag is enabled for the package with the dependency, or disabled otherwise.

[opt!=]

The flag must be disabled if the flag is enabled for the package with the dependency, or enabled otherwise.

[opt?]

The flag must be enabled if the flag is enabled for the package with the dependency.

[opt!?]

The flag must be enabled if the use flag is disabled for the package with the dependency.

[-opt]

The flag must be disabled.

[-opt?]

The flag must be disabled if the flag is enabled for the package with the dependency.

[-opt!?]

The flag must be disabled if the flag is disabled for the package with the dependency.

Dependencies could be combined by specifying multiple blocks, as in foo/bar[baz][monkey?].

For EAPI 2, Zac decided to go with an arbitrarily different syntax:

[opt]

The flag must be enabled.

[opt=]

The flag must be enabled if the flag is enabled for the package with the dependency, or disabled otherwise.

[!opt=]

The flag must be disabled if the flag is enabled for the package with the dependency, or enabled otherwise.

[opt?]

The flag must be enabled if the flag is enabled for the package with the dependency.

[!opt?]

The flag must be disabled if the use flag is disabled for the package with the dependency.

[-opt]

The flag must be disabled.

And to combine use dependencies, one uses a comma, as in foo/bar[baz,monkey?].

In both cases, the slot dependency must go before the dependency, so foo/bar:1[baz], not foo/bar[baz]:1. The use dependency goes after any version restrictions, so >=foo/bar-2.1:2[baz].

In both cases, it is illegal to reference a use flag that does not exist (including USE_EXPAND flags that are not explicitly listed in IUSE). So foo/bar[opt] when any version of foo/bar does not have opt in IUSE is illegal and has undefined behaviour, as is foo/baz[opt?] if either the owning package or foo/baz has no opt. For cases where only some versions of a package have a flag, use dependencies can be combined with version or slot restrictions.

From an implementation perspective: the package manager should not try to automatically solve unmet use dependencies. The package manager doesn’t know the impact of changing a use flag (changing some flags makes a system unbootable), so it can’t simply override the user’s choice. (Paludis will suggest an automatic reinstall if and only if the user has already modified their use.conf, so you don’t need to manually reinstall a dependency if you’re ok with altering the flags with which it is built.)

EAPI 2: SRC_URI Arrows

This is the first item in a series of posts describing EAPI 2.

Some upstreams use annoyingly named tarballs. Most commonly, they don’t include either the package name or the version in the filename. Because DISTDIR is a flat directory, this causes problems — the tree must not use two different tarballs with the same name. Previously, the solution to horrible upstream naming was to manually mirror the tarball with a new filename; this was considered excessively icky.

There have been two sane solutions proposed for this over time. The one we didn’t use was to define a DISTDIR_SUBDIR variable, and do all downloads into there. This would have made the A variable quite a bit messier, and complicated sharing certain tarballs between packages.

The arrows solution was something I came up with for early Paludis experimental EAPIs, and was adopted for kdebuild-1 and from there into 2; it’s also always been present in exheres-0. It works like this:

SRC_URI="http://example.com/stupid-named/1.23/stupid.tar.bz2 -> stupid-1.23.tar.bz2"

or using variables:

SRC_URI="http://example.com/stupid-named/${PV}/${PN}.tar.bz2 -> ${P}.tar.bz2"

This tells the package manager to look at the URL on the left of the arrow, but save to the filename on the right.

Mirroring effects are slightly subtle. Consider:

SRC_URI="mirror://foo/${PN}/${PV}.tar.bz2 -> ${P}.tar.bz2"

The package manager will look both on mirror://foo/ and mirror://gentoo/ for the download. When looking on foo, the raw filename must be used, but when looking on gentoo, the rewritten filename is used.

Anyone using arrows on mirror://gentoo/ URIs gets stabbed.

Arrows make another proposed but rejected EAPI feature irrelevant: there was a proposal floating around (I think it originated with drobbins, but I can’t find an original source) to make unpack ignore ;sf=tbz2 and ;sf=tgz suffixes on filenames, for interoperability with gitweb. Arrows are a more general solution.

Implementation-wise, anyone still using a lexer-based parser will need a single token of lookahead for this. Apparently this causes minor inconveniences in some broken programming languages that only support what C++ calls input iterators; I consider this a good thing, because it might make people either use a better iterator model or stop using lexers.

What’s in EAPI 2?

EAPI 2 has been approved by the Gentoo Council and so can now be used in ebuilds. The first package manager with support was Paludis 0.30.1; Portage support came along with 2.2_rc11.

EAPI 2 consists purely of extensions to EAPI 1. The new features are:

• SRC_URI arrows
• Use dependencies
• !! blockers
• src_prepare
• src_configure, and a different default src_compile
• default_ phase functions and the default function
• doman language support

Formal definitions can be found in PMS; an overview of each feature will follow in subsequent posts.