EAPI 3: Profile Controlled IUSE Injection

This is part of a series on EAPI 3.

EAPI 3 makes some fairly substantial changes to how use flags are handled inside the package manager. For developers, this requires a certain amount of attention; for users, there’s probably not much visible difference.

Historically, core Portage didn’t really do much with USE flags. They were used by conditional dependency specifications, but that was it — things like the use function were just bits of shell script that looked at the USE variable, which mostly just came directly from the user’s configuration. Similarly, IUSE was originally only used for cosmetic purposes — Portage used it to decide which flags to display in output, but didn’t use it to restrict USE in any way. Thus, packages could query use flags that weren’t listed in IUSE and still expect things to work.

USE_EXPAND gets even more complicated. Developers often didn’t bother even trying to list such flags in IUSE, leading to much confusion and strange behaviour.

This is a bit of a nuisance. It makes later user-facing functionality like “has this USE flag changed?” queries much harder to define, and it severely cripples use dependency defaults (was a flag not present for a given version, and thus should the default be used, or was it present but just not listed?). EAPI 3 fixes this.

In EAPI 3, IUSE is strictly enforced. If a flag isn’t listed in IUSE (and isn’t covered by the exceptions detailed later), the package manager will filter it from USE and any relevant USE_EXPAND variable, and make the use functions die if the flag is used in a query.

The problem is, Gentoo developers don’t tend to like the idea of having to explicitly list ARCH flags like x86 or certain USE_EXPAND flags like userland_GNU in IUSE. There are two reasons given for this:

• The flags shouldn’t be user visible.
• It’s lots of effort.

The first argument is mostly invalid; we already have USE_EXPAND_HIDDEN, and EAPI 3 is tweaking the definition of USE_EXPAND to let us treat ARCH as a USE_EXPAND value that isn’t prefixed with arch_ like it otherwise would be, allowing us to hide those too.

The second argument may or may not have merit; Gentoo’s gone for the “at least keep the possibility that we’ll avoid having to make people explicitly list things open” approach (at the expense of use dependency defaults not being usable on anything that’s treated that way), whilst Exherbo’s gone for “list everything”.

Here’s (roughly — see PMS for the gory details) how the compromise works:

• The package manager works out something we’ll call IUSE_EFFECTIVE (which may or may not be an ebuild-visible variable, is not an ebuild-modifiable variable and which will end up in VDB). It contains everything in IUSE, along with profile-injected extra values.
• Any value in the profile IUSE_IMPLICIT variable is treated as being in IUSE_EFFECTIVE. This would let the much-abused build flag carry on working; it has not yet been decided whether any values will be set this way in the profile.
• Any USE_EXPAND or USE_EXPAND_UNPREFIXED variable whose name appears in the profile USE_EXPAND_IMPLICIT variable is treated as having its values implicitly listed. This will probably contain USERLAND, KERNEL, ARCH etc, but not LINGUAS et al. For any USE_EXPAND_IMPLICIT variable FOO, the profile USE_EXPAND_VALUES_FOO variable lists its (unprefixed) implicitly injected values.
• IUSE_EFFECTIVE is thus a fixed, well defined set rather than an infinite set as it is in older EAPIs. Any value that doesn’t end up in IUSE_EFFECTIVE cannot be queried in any way, and is treated as ‘missing’ for use dependency defaults. Any value in IUSE_EFFECTIVE, whether or not it is explicitly in IUSE, is ‘present’ for use dependency defaults.

For developers, all you really need to know is that you need to list things in IUSE if you want to use them, except if they’re in a small group of special values that will likely contain ARCH, USERLAND and KERNEL flags and possibly things like build.

EAPI 3: pkg_info Changes

This is part of a series on EAPI 3.

With EAPI 3, the requirement that a package be installed before the package manager calls its pkg_info function has been dropped. If developers find they usually need to ask for certain information when a package build fails, they can make a pkg_info that will try to display that information, rather than having to ask the user to run commands by hand.

Some points to note:

• You can’t rely upon anything in particular being installed. We don’t force the package manager to ensure that a package’s dependencies are met. Thus, write code that only tries to display information, and doesn’t mind if it can’t.
• You could use has_version to determine whether something’s installed, but it’s better to just try to do whatever you want.
• There is no variable telling you whether your package is installed. This is because it’s not a simple boolean choice — do you care whether a package is installed, whether our exact version is installed, whether our slot is installed or something else? Again, has_version is available, but using it is probably a sign you’re writing code that’s trying to be too clever.

This feature was originally in kdebuild-1.

EAPI 3: New econf Arguments

This is part of a series on EAPI 3.

In EAPI 3, econf will pass --disable-dependency-tracking. This might marginally speed up a few packages’ builds, and breaks a few others that have funky half-hand-written makefiles.

What it won’t do, despite what you may of heard from certain less well informed Council members, is break packages that are using custom configure scripts. Those packages, if they have configure scripts that die on unrecognised options, aren’t using econf anyway since econf already passes all kinds of weird nonsense through.

In summary, pretty much no-one has any reason to care about this particular feature, and it’s only being discussed here for completeness.

EAPI 3: unpack xz Support

This is part of a series on EAPI 3.

In another disappointing defeat for the shadowy cabal controlling Gentoo’s every move (no, not that cabal, the other one), EAPI 3 will see unpack support xz format archives.

This is icky.

Support for unpacking lzma files was retroactively shoved into existing EAPIs because Mike Frysinger decided to just commit support to Portage without an EAPI change and start relying upon it for coreutils, making it impossible for everyone else to do anything except copy the change. This, it turns out, was a waste of time, since the lzma format is effectively deprecated in favour of xz.

Now, despite it not having any unmasked extractor on Gentoo, unpack in EAPI 3 is required to know how to unpack xz files. Whilst this is a step better than doing the change without an EAPI bump (no thanks to Mike, who tried to put it in without one), it’s still silly.

Of course, soon we’ll have to add in support for whatever the cool kids are using next week into EAPI 4.

EAPI 3: dohard and dosed Banned

This is part of a series on EAPI 3.

The dohard utility is used to create a hard link between two files. Except it can’t necessarily do that, and even when it does it probably won’t end up being merged as a hard link; up until recently, Portage only supported merging hard links by fluke if it happened to take a particular fast-path in the code, and few people even noticed. Even in ideal circumstances, dealing with hard links is fraught with mess, so we’ve decided to ban dohard in EAPI 3 as a way of discouraging their use.

As for dosed… It doesn’t really do what you think, it’s been considered deprecated for ages and there are better alternatives. EAPI 3 bans it too.

Both of these were originally banned in kdebuild-1. Unfortunately, kdebuild-1’s banning of the abomination that is dohtml hasn’t made it into EAPI 3.

EAPI 3: dodoc -r and doins Symlink Support

This is part of a series on EAPI 3.

Currently, to install a directory full of documentation, you have to either copy the directory yourself (after making the parent directory), or play around with into and doins -r. EAPI 3 adds a -r switch to dodoc for recursively installing a directory.

On the subject of doins, currently doins -r’s behaviour is undefined for anything except files or directories. With EAPI 3, doins -r will also be guaranteed to install symlinks correctly.

Incidentally, there’re probably a whole bunch more small improvements like these that we could make to the utilities. Suggestions (along with demonstrations of use cases) for EAPI 4 will be being taken in the not too distant future; something like the above is easy to get in.

EAPI 3: Controllable Compression

This is part of a series on EAPI 3.

Gentoo is, as everyone knows, about choice. This means it is considered vitally important that users get to choose such earth-shatteringly important matters as which program is used to compress their documentation files. It doesn’t matter that blockers repeatedly confuse people, that the tree is an unmanageable mess or that there’s no way of handling USE flag requirements — ensuring that the user’s choice of compression is applied to a small number of documentation files in every case is critical to Gentoo’s future.

As such, EAPI 3 introduces controllable compression. It works like this:

• The package manager maintains a list of things that may be compressed in some arbitrary way. By default, this list includes /usr/share/doc/, /usr/share/info and /usr/share/man.
• The package manager also maintains a list of things to be excluded from the above compression. By default, this list includes /usr/share/doc/${PF}/html.
• Inside src_install, packages can add things to the inclusion list using docompress blah, and to the exclusion list using docompress -x blah.
• After src_install has finished, the package manager may apply some kind of compression to anything in the inclusion list that is not also in the exclusion list.

PMS is carefully worded to allow all of this nonsense to be a no-op.

Controllable compression is a Gentoo-original feature, and clear proof that there is no magic external cabal dictating Gentoo’s future, since they’d never let people get away with this if there were.

EAPI 3: default src_install

This is part of a series on EAPI 3.

EAPI 3 will feature a default src_install that isn’t just a no-op. While it won’t eliminate the need for some packages to write their own implementation, it will help in a good number of cases.

The exact default is still subject to bikeshedding and could be changed at any point right up until PMS is tagged for EAPI 3. Right now we’re going with the following, and it’s very likely the final choice will look more or less like this:

src_install() {
    if [[ -f Makefile ]] || [[ -f GNUmakefile ]] || [[ -f makefile ]]; then
        emake DESTDIR="${D}" install
    fi

    if ! declare -p DOCS >/dev/null 2>&1 ; then
        local d
        for d in README* ChangeLog AUTHORS NEWS TODO CHANGES \
                THANKS BUGS FAQ CREDITS CHANGELOG ; do
            [[ -s "${d}" ]] && dodoc "${d}"
        done
    elif declare -p DOCS | grep -q '^declare -a ' ; then
        dodoc "${DOCS[@]}"
    else
        dodoc ${DOCS}
    fi
}

Note the following:

• There will be a DOCS variable. This was decided by a Council vote, since not everyone sees the obvious sense of the idea.
• If something is explicitly listed in DOCS, it will be an error for it not to exist. But if we’re using the default values, things are only installed as documentation if they’re there and not empty.
• We don’t use -j1 for the install. This will break some packages built using older autotools. This is in line with Gentoo policy.

The first practical non-empty src_install was in exheres-0. The exheres-0 version remains considerably more flexible and parametrisable than the above.

EAPI 3: DEFINED_PHASES and PROPERTIES Mandatory

This is part of a series on EAPI 3.

DEFINED_PHASES and PROPERTIES were retroactively introduced into earlier EAPIs. We could get away with this because it’s legal for a package manager to do absolutely nothing with them and pretend they don’t exist.

PROPERTIES isn’t very interesting; it’s only mandatory in EAPI 3 because we like to tidy things up as we go along. DEFINED_PHASES, however, is now mandatory for a very good reason: it’s necessary to make pkg_pretend usable.

Invoking an ebuild process to run pkg_pretend, even if it does nothing, takes something like a tenth of a second. If you’re doing a world update, that could mean thirty seconds of sitting around waiting, and if you’re doing a full reinstall, several minutes. In the grand scheme of things, it’s not a huge penalty, but it’s one that occurs when the user is sitting waiting rather than something added to build time — and if we add too much to pretend time, users will just stop waiting for it, which defeats the whole point of it. By requiring DEFINED_PHASES support, a package manager only has to pay the penalty for packages where it will be useful.

This one’s more something that would be annoying if it were not there than a revolutionary new feature; users and developers shouldn’t have to care about it, except if they’re interested in the thought and planning that the shadowy cabal directing Gentoo’s future from their secret moon base put into all of this.

DEFINED_PHASES first appeared in exheres-0, where it was always mandatory.

EAPI 3: Use Dependency Defaults

This is part of a series on EAPI 3.

In EAPI 2, use dependencies are only allowed to reference USE flags that are listed in IUSE for every package that matches the associated specification. Unfortunately, this isn’t enforced, and developers often forget to pay special attention to it.

Use dependency defaults provide a way of saying “if this flag isn’t listed in IUSE, pretend that it had this value instead”. The syntax is to append (+) to the use flag name (the parentheses aren’t entirely pretty, but they’re necessary to avoid ambiguity) if the package manager should pretend that it is on if not present, and (-) if the package manager should pretend that it is off. So, one can do things like foo/bar[baz(+)] or foo/bar[!baz(+)?].

Because of EAPIs before 3 not being strict about what ends up in IUSE, there are still limitations. This can only be used on flags that would be listed in IUSE were they supported; it is not usable for USE_EXPAND values.

Use dependency defaults still require a degree of care from the developer. In particular, it’s not possible to write ‘pre-emptive’ defaults to handle future possible use flag removals; one does not know whether a use flag removal will be because it is always on or because a feature will be removed.

Use dependency defaults first appeared in exheres-0.