What’s Happened to EAPI 3?

EAPI 3 was effectively finished three months ago; some of you are no doubt wondering what happened to it and why there hasn’t been any news.

EAPI 3’s approval was conditional upon Portage support. This isn’t unreasonable, at least for main tree usage, although it’s arguably less relevant for overlays. Unfortunately, Portage is suffering from a severe lack of maintainers.

You can track Portage’s EAPI 3 implementation progress via Gentoo’s bugzilla. As you can see, some progress has been made, but it’s slow going, and hasn’t met the “we’ll be ready within a month” goal given at the Council meeting a month ago.

In the mean time, Paludis 0.38.0 has EAPI 3 support present but not enabled at install time (unlike Portage, we don’t hardcode EAPI numbers into the source, so supporting a feature doesn’t involve any kind of treatment for the EAPI that provides it). Anyone wishing to play with EAPI 3 support for personal use can just copy the EAPI 3 definition file from the source tree.

It’s a shame that Gentoo has to hold back on delivering a better user experience and making good ebuilds easier to write because of Portage. I’d like to encourage anyone who can handle Portage’s codebase to give Gentoo the help it so badly needs.

As for future EAPIs… The current Council has expressed an interest in handing off the EAPI approval process to a separate group. This is either good news if that group is going to put in the work necessary to get things done and deliver practical results, or terrible news if that group is going to perpetuate the worst habits of the previous Council.

What’s Not in EAPI 3?

I’ve already covered everything that made it into EAPI 3; for those interested, I’ll briefly discuss some of the things that were voted out or didn’t make it.

First, things that made it into the draft proposal, but were voted out:

• A ban on || ( use? ( foo ) ). This construct is impossible to use correctly (despite misinformation to the contrary from certain people who keep trying very hard to come up with a perverse justification for it), is used incorrectly everywhere it is currently used and has lots of special wording for it in PMS. Unfortunately, the Council weaselled out of outright banning it in favour of maybe introducing a repoman check and maybe banning it later.
• doexample and doinclude. Nobody cared enough to make a convincing defence of their necessity, so they were voted out.
• More econf option changes. --enable-fast-install was snuck into the --disable-dependency-tracking feature request with no explanation. The Council voted against it.
• unpack --if-compressed, which would make it otherwise fatal for unpack to be called with an unsupported extension, was voted out. This means that when you explicitly call unpack doesnotexist or unpack foo.tar.bz3, unpack will continue to do nothing silently and not indicate any error.

Then, things that were getting ready before the Council decided that in the interests of getting things done, anything not already on the list would be postponed for consideration for a future EAPI:

• Merger mtime preservation. This one, as worded, received much objection from the secret cabal directing Gentoo’s future from the shadows since it would mandate emulating Portage’s broken behaviour that results in files being merged with ridiculous timestamps.
• Prefix variables. The Gentoo prefix people wanted to sneak a few prefix variables into PMS by the back door, presumably in a way of getting prefix accepted without them having to discuss or address design, goals or implementation — this discussion getting postponed is probably good for Gentoo, since it’s well known that anyone who dares suggest that prefix is anything less than perfect ends up mysteriously no longer being a Gentoo developer.

EAPI 3: Utilities Die, and New nonfatal Utility

This is part of a series on EAPI 3.

With older EAPIs, econf (being a shell function) would die on failure, as do most eclass-provided functions, but emake (being an external, since people call it via xargs) and most package-manager-provided utilities (external or not) merely return non-zero.

In EAPI 3, most utilities (consult PMS for exact details) will now die when an error is encountered. This in turn means the package manager has to be able to die from an external utility.

Occasionally, this isn’t desired behaviour. You might need to call a function, but do something else if it fails rather than aborting. In this case, use the nonfatal utility with the command to run as its arguments. Mostly this is just used to give better error messages, but there are sometimes other cases:

nonfatal emake plugins || die "Making plugins failed. Do you need to frozbinate your zorgons?"

# these usually exist, but sometimes don't, and we're too lazy to check manually
nonfatal dodoc foo bar baz

This feature originated with exheres-0.

EAPI 3: No More RDEPEND=DEPEND

This is part of a series on EAPI 3.

With EAPI 3, the much-hated and overly complicated RDEPEND from DEPEND automagic assignment (along with its highly weird rules for eclasses, which no-one ever remembers) is dead. Developers should set DEPEND and RDEPEND explicitly, and if any sharing is required, a (non-package-manager-handled) COMMON_DEPEND variable of some kind can be used.

EAPI 3: S to WORKDIR Fallback Reduced

This is part of a series on EAPI 3.

The introduction of a default src_install has one weird side effect that probably wouldn’t have been noticed before EAPI 3’s release without extensive use in Exherbo: if S is incorrect, developers will often end up accidentally installing an empty package rather than seeing the expected error. This is because the S to WORKDIR fallback happens even when it probably shouldn’t; with developers often no longer coding a src_install that will die if the expected Makefile isn’t found, this gets through unnoticed until it’s too late.

EAPI 3 therefore removes the S to WORKDIR fallback, except under certain carefully chosen circumstances: if the ebuild has empty A, and it defines none of src_unpack, src_prepare, src_configure, src_compile and src_install, the fallback will occur as normal.

Thus, ebuilds that install nothing don’t have worry; other ebuilds must set S=${WORKDIR} in global scope if necessary.

EAPI 3: AA, KV Gone; New REPLACING_VERSIONS and REPLACED_BY_VERSION Variables

This is part of a series on EAPI 3.

First, two removals: AA and KV. AA has been deprecated for ages, and isn’t useful for ebuilds — it’s like A, but contains every component, even those in unselected conditional branches. KV, meanwhile, contains “the kernel version”. Ebuilds that need that kind of thing should use one of the eclasses that provides similar but more useful functionality instead.

Now the more interesting part: two additions. In the good old days, you could use convoluted has_version magic in pkg_ phases to work out whether we’re in an upgrade, downgrade, reinstall or fresh install. The devmanual used to contain examples. Unfortunately, Zac went and broke all this when he changed phase ordering without an EAPI bump, leaving all the widely used and documented techniques for detecting upgrades unusable. Thus, EAPI 3 adds two new variables to fix this mess.

The REPLACING_VERSIONS variable contains a space separated list of all the versions of the current package name that will be replaced. It is only valid in pkg_preinst and pkg_postinst; if set during pkg_pretend and pkg_setup, it will contain valid values, but it may be misleading in the case of binary package building.

Note that REPLACING_VERSIONS is a list, not a single value. This is to deal with weird cases like having foo-1:2 and foo-2:1 installed, and installing foo-1:1.

The REPLACED_BY_VERSION variable contains the version of the current package name that has been replaced. It is only valid in pkg_prerm and pkg_postrm. If it is unset, it indicates a simple uninstall.

EAPI 3: Profile Controlled IUSE Injection

This is part of a series on EAPI 3.

EAPI 3 makes some fairly substantial changes to how use flags are handled inside the package manager. For developers, this requires a certain amount of attention; for users, there’s probably not much visible difference.

Historically, core Portage didn’t really do much with USE flags. They were used by conditional dependency specifications, but that was it — things like the use function were just bits of shell script that looked at the USE variable, which mostly just came directly from the user’s configuration. Similarly, IUSE was originally only used for cosmetic purposes — Portage used it to decide which flags to display in output, but didn’t use it to restrict USE in any way. Thus, packages could query use flags that weren’t listed in IUSE and still expect things to work.

USE_EXPAND gets even more complicated. Developers often didn’t bother even trying to list such flags in IUSE, leading to much confusion and strange behaviour.

This is a bit of a nuisance. It makes later user-facing functionality like “has this USE flag changed?” queries much harder to define, and it severely cripples use dependency defaults (was a flag not present for a given version, and thus should the default be used, or was it present but just not listed?). EAPI 3 fixes this.

In EAPI 3, IUSE is strictly enforced. If a flag isn’t listed in IUSE (and isn’t covered by the exceptions detailed later), the package manager will filter it from USE and any relevant USE_EXPAND variable, and make the use functions die if the flag is used in a query.

The problem is, Gentoo developers don’t tend to like the idea of having to explicitly list ARCH flags like x86 or certain USE_EXPAND flags like userland_GNU in IUSE. There are two reasons given for this:

• The flags shouldn’t be user visible.
• It’s lots of effort.

The first argument is mostly invalid; we already have USE_EXPAND_HIDDEN, and EAPI 3 is tweaking the definition of USE_EXPAND to let us treat ARCH as a USE_EXPAND value that isn’t prefixed with arch_ like it otherwise would be, allowing us to hide those too.

The second argument may or may not have merit; Gentoo’s gone for the “at least keep the possibility that we’ll avoid having to make people explicitly list things open” approach (at the expense of use dependency defaults not being usable on anything that’s treated that way), whilst Exherbo’s gone for “list everything”.

Here’s (roughly — see PMS for the gory details) how the compromise works:

• The package manager works out something we’ll call IUSE_EFFECTIVE (which may or may not be an ebuild-visible variable, is not an ebuild-modifiable variable and which will end up in VDB). It contains everything in IUSE, along with profile-injected extra values.
• Any value in the profile IUSE_IMPLICIT variable is treated as being in IUSE_EFFECTIVE. This would let the much-abused build flag carry on working; it has not yet been decided whether any values will be set this way in the profile.
• Any USE_EXPAND or USE_EXPAND_UNPREFIXED variable whose name appears in the profile USE_EXPAND_IMPLICIT variable is treated as having its values implicitly listed. This will probably contain USERLAND, KERNEL, ARCH etc, but not LINGUAS et al. For any USE_EXPAND_IMPLICIT variable FOO, the profile USE_EXPAND_VALUES_FOO variable lists its (unprefixed) implicitly injected values.
• IUSE_EFFECTIVE is thus a fixed, well defined set rather than an infinite set as it is in older EAPIs. Any value that doesn’t end up in IUSE_EFFECTIVE cannot be queried in any way, and is treated as ‘missing’ for use dependency defaults. Any value in IUSE_EFFECTIVE, whether or not it is explicitly in IUSE, is ‘present’ for use dependency defaults.

For developers, all you really need to know is that you need to list things in IUSE if you want to use them, except if they’re in a small group of special values that will likely contain ARCH, USERLAND and KERNEL flags and possibly things like build.

EAPI 3: pkg_info Changes

This is part of a series on EAPI 3.

With EAPI 3, the requirement that a package be installed before the package manager calls its pkg_info function has been dropped. If developers find they usually need to ask for certain information when a package build fails, they can make a pkg_info that will try to display that information, rather than having to ask the user to run commands by hand.

Some points to note:

• You can’t rely upon anything in particular being installed. We don’t force the package manager to ensure that a package’s dependencies are met. Thus, write code that only tries to display information, and doesn’t mind if it can’t.
• You could use has_version to determine whether something’s installed, but it’s better to just try to do whatever you want.
• There is no variable telling you whether your package is installed. This is because it’s not a simple boolean choice — do you care whether a package is installed, whether our exact version is installed, whether our slot is installed or something else? Again, has_version is available, but using it is probably a sign you’re writing code that’s trying to be too clever.

This feature was originally in kdebuild-1.

EAPI 3: New econf Arguments

This is part of a series on EAPI 3.

In EAPI 3, econf will pass --disable-dependency-tracking. This might marginally speed up a few packages’ builds, and breaks a few others that have funky half-hand-written makefiles.

What it won’t do, despite what you may of heard from certain less well informed Council members, is break packages that are using custom configure scripts. Those packages, if they have configure scripts that die on unrecognised options, aren’t using econf anyway since econf already passes all kinds of weird nonsense through.

In summary, pretty much no-one has any reason to care about this particular feature, and it’s only being discussed here for completeness.

EAPI 3: unpack xz Support

This is part of a series on EAPI 3.

In another disappointing defeat for the shadowy cabal controlling Gentoo’s every move (no, not that cabal, the other one), EAPI 3 will see unpack support xz format archives.

This is icky.

Support for unpacking lzma files was retroactively shoved into existing EAPIs because Mike Frysinger decided to just commit support to Portage without an EAPI change and start relying upon it for coreutils, making it impossible for everyone else to do anything except copy the change. This, it turns out, was a waste of time, since the lzma format is effectively deprecated in favour of xz.

Now, despite it not having any unmasked extractor on Gentoo, unpack in EAPI 3 is required to know how to unpack xz files. Whilst this is a step better than doing the change without an EAPI bump (no thanks to Mike, who tried to put it in without one), it’s still silly.

Of course, soon we’ll have to add in support for whatever the cool kids are using next week into EAPI 4.