Why GLEP 55 is a Good Idea and thus a Bad Thing

As it seems that ill-thought-out GLEP 55 bashing is back in fashion again, I thought I’d explain why GLEP 55 is a good idea, and thus a bad thing for Gentoo.

In short, GLEP 55 proposes moving the EAPI from being ordinary ebuild metadata to being encoded as part of the filename (e.g. .ebuild-4). The advantages of this are:

• It allows global scope changes that affect metadata generation to be made. Right now, it is impossible for EAPIs to add new global scope functions, and it is impossible for EAPIs to change the behaviour of existing functions, since all ebuilds must be able to be sourced and have at least the EAPI part of their metadata generated by a package manager that does not support newer EAPIs.
• It allows changes to the versioning rules. New EAPIs cannot introduce new package version formats or fix arbitrary limitations with existing formats, since as soon as an older package manager encounters what it thinks is an ill-formatted version, it will produce a noisy, user-visible warning. With GLEP 55, instead these new versions will be invisible.
• It allows ebuilds to use newer bash features in ebuilds without breaking older package managers.
• It makes a package’s EAPI consistently defined throughout the ebuild. Right now it’s legal to set EAPI deep inside a nested chain of eclasses (so long as doing so doesn’t break metadata invariance rules), which means any code encountered before EAPI is set will have a false impression of what the eventual EAPI will be.

Because it’s currently impossible to add per-package eclasses (due to the first item), ebuilds are doing all sorts of stupid things to get the same effect. Because the second item means it’s impossible to add sane versioning for SCM packages, maintainers are making do with lots of 1.2.9999 versions that don’t quite work with dependencies properly. Due to a combination of the first two, I had to write the hideous abomination that is the versionator eclass because it’s impossible to let Portage support versions like 1.23-alpha1 directly, and impossible to add package manager provided version parsing functions that can be used in global scope.

The third is a continual source of problems, as developers frequently accidentally use newer bash features, thus screwing up the upgrade path for anyone who hasn’t updated their box for a few months.

As for the fourth, it’s highly discouraged from a QA perspective these days, but people have set EAPI via an eclass in the past. This one’s more an illustration of icky design than anything else.

Because GLEP 55’s author once tried using a package manager that isn’t Portage, and is thus irrevocably tainted, some alternative ways of handling these deficiencies being proposed.

Alternative One: Repository Capabilities

First, it is suggested that repositories specify, at the repository level, their requirements (presumably via layout.conf). The implications of this are:

• We can’t start using anything new until we’re sure that everyone is using a compliant package manager. That means yet another two year wait before any of this goes anywhere.
• New capabilities can’t be used in the main tree for at least a year after their introduction, because users must be able to upgrade their package manager on a box they haven’t touched for a while.

This means that the cost of introducing a change is extremely high, and so developers will be encouraged to continue using bad solutions rather than fixing things properly.

Also, if repository capabilities replace rather than supplement EAPIs, as some have suggested, then it becomes impossible to change a repository’s capabilities without rewriting every single ebuild to the new standard. For example, one could not turn on a strict-s-checking feature without first checking every single existing package and fixing any that rely upon the older “S doesn’t have to exist” behaviour. Certain developers have proposed branching the tree once a year and rewriting everything to do this; quite how they plan to find the manpower to pull this off has yet to be answered.

Thus, assuming a mass rewrite is out of the question, repository capabilities must be combined with another solution to be of practical use, which brings us to:

Alternative Two: Magic Markers

Second, it is suggested that ebuilds include some kind of magic marker to allow their EAPI to be determined without going through the usual sourcing process. This does absolutely nothing to fix the version formats problem, and so must be combined with repository capabilities anyway. On top of that, none of the proposed magic marker methods are particularly pleasant.

Magic via Fixed EAPI Strings and Parsing Not Using Bash

The first kind of magic marker proposal is to require that ebuilds specify the EAPI string in a particular, fixed format. Proposals are typically along the lines of “it must be specified within the first N lines, and it must be exactly in the form EAPI=X”. So, depending upon the exact wording chosen, none of the following would be legal:

# Copyright 1999-2010 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

EAPI='4'

(Single quotes might not be legal)

# Copyright 1999-2010 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

EAPI="4"

(Nor double quotes)

# Copyright 2010 Joe Bloggs
# Derived from foo-1.23.ebuild from Gentoo, which is:
#     Copyright 1999-2010 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

EAPI=4

(Too many comment lines at the start)

# Copyright 1999-2010 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

    EAPI=4

(Indenting isn’t allowed)

# Copyright 1999-2010 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

inherit eclass-that-sets-eapi

(Eclasses can set every metadata variable except EAPI)

# Copyright 1999-2010 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

SLOT=1
EAPI=4

(EAPI must be first)

To make matters worse, this exception would only apply to the EAPI variable, even though in every other way EAPI would remain a normal metadata variable. This restriction would not necessarily be enforceable by the package manager either, so sometimes screwups would lead to weird results rather than errors.

Also, existing ebuilds don’t follow this format — all of the above examples have occurred in real code. Thus, before switching to this, the entire tree and every single overlay everywhere would have to be fixed.

Magic via Special Comment Strings

The second magic marker variant is to shove a special comment line at the start of every single ebuild containing EAPI information. Again, this would require updating every single existing ebuild, and would also require yet another huge wait before it becomes usable.

This sort of resembles what’s done in certain fixed binary formats. However, ebuilds aren’t fixed binary formats; ebuilds are scripts. In addition, fixed binary formats that do this were designed from the start with different versions in mind; the ebuild format has instead been gradually grown over time, often by people who actively oppose any kind of standardisation.

Magic via Extended Attributes

The third magic marker variant is to store the marker in extended attributes, rather than in the file itself. This means:

• It becomes impossible to store ebuilds anywhere that doesn’t support extended attributes, such as in most version control systems, on pastebins, on bugzilla and on pastebins.
• The EAPI is effectively invisible except to people using obscure tools.

And, again, a huge wait.

Alternative Three: eapi Function

Third, make eapi a function. This will allow early exit for unsupported EAPIs, so global scope changes can be made so long as they occur after the eapi function is called. Again, this means a huge wait, and again, this requires an additional proposal to fix version formats.

Why none of this matters anyway

In other words, the choices come down to:

• Put the EAPI in the filename. Be able to use new features straight away, including in the main repository.
• Introduce repository capability support to package managers. Wait a year or more. Every time you want to use a new capability, release package manager support for it, then wait a year before you can use the new feature. In addition, either rewrite the entire tree every time you change capabilities, or also introduce a second solution.
• Introduce some kind of magic marker. Rewrite every existing ebuild, including the ones in overlays, to use this marker. Wait a year. Still be unable to change version format rules, unless a second solution is also introduced.
• Move eapi to be a function. Wait a year. Still be unable to change version format rules, etc.

Or, of course, the usual Gentoo solution can be applied: do nothing. Continue to be unable to support per-package eclasses, and continue adding extensive convoluted hacks into ebuilds to try sourcing things in FILESDIR to get the same result. Continue to use silly 9999 version numbers, which can’t be used in dependencies correctly, to support version branches of scm packages. Continue to arbitrarily not support certain upstream version formats just because they use -alpha instead of _alpha, and require ebuilds to carry on doing transformations to deal with this. Continue to reject proposals based upon the author rather than technical merit.

It’s obvious that doing nothing is the way best suited to Gentoo’s needs. Gentoo’s main priority is to avoid making any change that is in any way controversial or that could in any way be associated with the wrong people. Delivering a better product to users or making things easier for developers is irrelevant. Thus, maintaining the status quo whilst vaguely mentioning alternatives that can’t and won’t ever happen is the way forward. If things get desperate, and it starts to become impossible to fire any developer who asks for better tools, then at that point one of the “wait several years” solutions can be pulled out as proof that “something is being done”.

At this point, all that switching to a solution that would enable changes to be made would do is illustrate that Gentoo can’t deliver even the simplest of the changes developers need. By repeatedly hovering around proposals that will require long waits before their introduction, Gentoo has a plausible-looking excuse to mask an almost complete lack of progress in Portage; if GLEP 55 were to be introduced, developers might start asking why they still don’t have per-package eclasses, less arbitrarily restricted version numbers, package manager provided version parsing helpers or the ability to use newer bash features in ebuilds.