Blag

He's not dead, he's resting

Why GLEP 55 is a Good Idea and thus a Bad Thing

As it seems that ill-thought-out GLEP 55 bashing is back in fashion again, I thought I’d explain why GLEP 55 is a good idea, and thus a bad thing for Gentoo.

In short, GLEP 55 proposes moving the EAPI from being ordinary ebuild metadata to being encoded as part of the filename (e.g. .ebuild-4). The advantages of this are:

  • It allows global scope changes that affect metadata generation to be made. Right now, it is impossible for EAPIs to add new global scope functions, and it is impossible for EAPIs to change the behaviour of existing functions, since all ebuilds must be able to be sourced and have at least the EAPI part of their metadata generated by a package manager that does not support newer EAPIs.
  • It allows changes to the versioning rules. New EAPIs cannot introduce new package version formats or fix arbitrary limitations with existing formats, since as soon as an older package manager encounters what it thinks is an ill-formatted version, it will produce a noisy, user-visible warning. With GLEP 55, instead these new versions will be invisible.
  • It allows ebuilds to use newer bash features in ebuilds without breaking older package managers.
  • It makes a package’s EAPI consistently defined throughout the ebuild. Right now it’s legal to set EAPI deep inside a nested chain of eclasses (so long as doing so doesn’t break metadata invariance rules), which means any code encountered before EAPI is set will have a false impression of what the eventual EAPI will be.

Because it’s currently impossible to add per-package eclasses (due to the first item), ebuilds are doing all sorts of stupid things to get the same effect. Because the second item means it’s impossible to add sane versioning for SCM packages, maintainers are making do with lots of 1.2.9999 versions that don’t quite work with dependencies properly. Due to a combination of the first two, I had to write the hideous abomination that is the versionator eclass because it’s impossible to let Portage support versions like 1.23-alpha1 directly, and impossible to add package manager provided version parsing functions that can be used in global scope.

The third is a continual source of problems, as developers frequently accidentally use newer bash features, thus screwing up the upgrade path for anyone who hasn’t updated their box for a few months.

As for the fourth, it’s highly discouraged from a QA perspective these days, but people have set EAPI via an eclass in the past. This one’s more an illustration of icky design than anything else.

Because GLEP 55’s author once tried using a package manager that isn’t Portage, and is thus irrevocably tainted, some alternative ways of handling these deficiencies being proposed.

Alternative One: Repository Capabilities

First, it is suggested that repositories specify, at the repository level, their requirements (presumably via layout.conf). The implications of this are:

  • We can’t start using anything new until we’re sure that everyone is using a compliant package manager. That means yet another two year wait before any of this goes anywhere.
  • New capabilities can’t be used in the main tree for at least a year after their introduction, because users must be able to upgrade their package manager on a box they haven’t touched for a while.

This means that the cost of introducing a change is extremely high, and so developers will be encouraged to continue using bad solutions rather than fixing things properly.

Also, if repository capabilities replace rather than supplement EAPIs, as some have suggested, then it becomes impossible to change a repository’s capabilities without rewriting every single ebuild to the new standard. For example, one could not turn on a strict-s-checking feature without first checking every single existing package and fixing any that rely upon the older “S doesn’t have to exist” behaviour. Certain developers have proposed branching the tree once a year and rewriting everything to do this; quite how they plan to find the manpower to pull this off has yet to be answered.

Thus, assuming a mass rewrite is out of the question, repository capabilities must be combined with another solution to be of practical use, which brings us to:

Alternative Two: Magic Markers

Second, it is suggested that ebuilds include some kind of magic marker to allow their EAPI to be determined without going through the usual sourcing process. This does absolutely nothing to fix the version formats problem, and so must be combined with repository capabilities anyway. On top of that, none of the proposed magic marker methods are particularly pleasant.

Magic via Fixed EAPI Strings and Parsing Not Using Bash

The first kind of magic marker proposal is to require that ebuilds specify the EAPI string in a particular, fixed format. Proposals are typically along the lines of “it must be specified within the first N lines, and it must be exactly in the form EAPI=X”. So, depending upon the exact wording chosen, none of the following would be legal:

# Copyright 1999-2010 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

EAPI='4'

(Single quotes might not be legal)

# Copyright 1999-2010 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

EAPI="4"

(Nor double quotes)

# Copyright 2010 Joe Bloggs
# Derived from foo-1.23.ebuild from Gentoo, which is:
#     Copyright 1999-2010 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

EAPI=4

(Too many comment lines at the start)

# Copyright 1999-2010 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

    EAPI=4

(Indenting isn’t allowed)

# Copyright 1999-2010 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

inherit eclass-that-sets-eapi

(Eclasses can set every metadata variable except EAPI)

# Copyright 1999-2010 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

SLOT=1
EAPI=4

(EAPI must be first)

To make matters worse, this exception would only apply to the EAPI variable, even though in every other way EAPI would remain a normal metadata variable. This restriction would not necessarily be enforceable by the package manager either, so sometimes screwups would lead to weird results rather than errors.

Also, existing ebuilds don’t follow this format — all of the above examples have occurred in real code. Thus, before switching to this, the entire tree and every single overlay everywhere would have to be fixed.

Magic via Special Comment Strings

The second magic marker variant is to shove a special comment line at the start of every single ebuild containing EAPI information. Again, this would require updating every single existing ebuild, and would also require yet another huge wait before it becomes usable.

This sort of resembles what’s done in certain fixed binary formats. However, ebuilds aren’t fixed binary formats; ebuilds are scripts. In addition, fixed binary formats that do this were designed from the start with different versions in mind; the ebuild format has instead been gradually grown over time, often by people who actively oppose any kind of standardisation.

Magic via Extended Attributes

The third magic marker variant is to store the marker in extended attributes, rather than in the file itself. This means:

  • It becomes impossible to store ebuilds anywhere that doesn’t support extended attributes, such as in most version control systems, on pastebins, on bugzilla and on pastebins.
  • The EAPI is effectively invisible except to people using obscure tools.

And, again, a huge wait.

Alternative Three: eapi Function

Third, make eapi a function. This will allow early exit for unsupported EAPIs, so global scope changes can be made so long as they occur after the eapi function is called. Again, this means a huge wait, and again, this requires an additional proposal to fix version formats.

Why none of this matters anyway

In other words, the choices come down to:

  • Put the EAPI in the filename. Be able to use new features straight away, including in the main repository.
  • Introduce repository capability support to package managers. Wait a year or more. Every time you want to use a new capability, release package manager support for it, then wait a year before you can use the new feature. In addition, either rewrite the entire tree every time you change capabilities, or also introduce a second solution.
  • Introduce some kind of magic marker. Rewrite every existing ebuild, including the ones in overlays, to use this marker. Wait a year. Still be unable to change version format rules, unless a second solution is also introduced.
  • Move eapi to be a function. Wait a year. Still be unable to change version format rules, etc.

Or, of course, the usual Gentoo solution can be applied: do nothing. Continue to be unable to support per-package eclasses, and continue adding extensive convoluted hacks into ebuilds to try sourcing things in FILESDIR to get the same result. Continue to use silly 9999 version numbers, which can’t be used in dependencies correctly, to support version branches of scm packages. Continue to arbitrarily not support certain upstream version formats just because they use -alpha instead of _alpha, and require ebuilds to carry on doing transformations to deal with this. Continue to reject proposals based upon the author rather than technical merit.

It’s obvious that doing nothing is the way best suited to Gentoo’s needs. Gentoo’s main priority is to avoid making any change that is in any way controversial or that could in any way be associated with the wrong people. Delivering a better product to users or making things easier for developers is irrelevant. Thus, maintaining the status quo whilst vaguely mentioning alternatives that can’t and won’t ever happen is the way forward. If things get desperate, and it starts to become impossible to fire any developer who asks for better tools, then at that point one of the “wait several years” solutions can be pulled out as proof that “something is being done”.

At this point, all that switching to a solution that would enable changes to be made would do is illustrate that Gentoo can’t deliver even the simplest of the changes developers need. By repeatedly hovering around proposals that will require long waits before their introduction, Gentoo has a plausible-looking excuse to mask an almost complete lack of progress in Portage; if GLEP 55 were to be introduced, developers might start asking why they still don’t have per-package eclasses, less arbitrarily restricted version numbers, package manager provided version parsing helpers or the ability to use newer bash features in ebuilds.

Advertisements

9 responses to “Why GLEP 55 is a Good Idea and thus a Bad Thing

  1. Ewoud Kohl van Wijngaarden July 8, 2010 at 3:19 pm

    Although currently I don’t like the ebuild-1 or ebuild-2 because of cosmetic reasons, I do see the easy migration path which far outweighs the cosmetic reasons. In fact, using a different extension makes even more sense since they can in fact be completely different file types. Perhaps by EAPI 10 ebuilds are written in python or lua (who knows) and using ebuild-10 as an extension would make this clear to everyone. Either the package manager supports ebuild-10 in which case it’s parsed or it doesn’t and it’s ignored.

    You could even do neat tricks like writing portage-x.y.z.ebuild (and all its dependencies) to allow older package managers to install x.y.z which in turn is able to handle EAPI 10. From that point on you can continue to upgrade your ancient system. And portage-x.y.z.ebuild could even be as simple as installing a very limited tarball with bundled libraries (thus reducing dependencies) which can only install portage-x.y.z.ebuild-10.

    Imho, in the long run there’s hardly a choice.

    • Ciaran McCreesh July 8, 2010 at 3:28 pm

      I’d note that GLEP 55 explicitly bans having two ebuilds that differ only by EAPI, partly for sanity’s sake and partly because we don’t define an ‘order’ for EAPIs (it’s just a fluke that EAPIs so far mostly look like integers). The upgrade path is done by keeping core system packages at low EAPIs for as long as possible.

      Incidentally, this isn’t a new problem with GLEP 55 — it’s already illegal to have foo-1.ebuild and foo-01.ebuild, so also banning having both foo-1.ebuild-4 and foo-1.ebuild-5 is no big deal.

      • ferringb July 8, 2010 at 8:58 pm

        Actually, having version XYZ available in multiple EAPI’s is useful/good for an upgrade path- if used *only* for upgrade paths. General rule, single CPV, single EAPI should be the case- that said the implementations should be able to handle multiple. Actual ordering of which to choose, which is ‘better’… well, an ordering is needed. Personally I’d be inclined to default to chronological EAPI creation order, potentially overridable at the repo level (although I’d be damned if I can think of a valid scenario for it, beyond working around broken upgrade pathways).

        • Ciaran McCreesh July 9, 2010 at 7:18 am

          In theory, yeah, it has a slight use. In practice, I suspect it would bring only severe pain…

  2. tuXXX July 8, 2010 at 7:42 pm

    Putting EAPI in the filename would allow XML ebuilds, which is not possible with the last two propositions.
    And using repository capabilities seems to take ages to do anything.

    (XML maybe not be the best format for ebuild, but it’s something different enough from the current format, so I think that it’s a good example because if this could work, almost anything could work)

  3. nico July 9, 2010 at 5:32 pm

    Is there someone with the authority to decide which solution should be applied? Same with –as-needed, baselayout migration, website redesign or any other significant change.

    • Ciaran McCreesh July 9, 2010 at 5:36 pm

      The Council has the authority. However, past Councils have made up of people who got elected by having a reputation for not rocking the boat. Any actual decision would upset a few people, so they repeatedly postpone decisions, talk about setting up committees that never happen, stop meetings after exactly one hour has passed even if no decision is reached, etc.

  4. Andrew D Kirch July 12, 2010 at 4:57 am

    Ciaran,

    The problem is that your base premise in GLEP 55 is wrong, in that BASH must be used to parse the ebuilds, and that this will magically break if a number of QA issues are introduced to the ebuild.
    Let me address each:
    1. single quote, it might not be legal, unless we decide it _IS_ and standardize on it. It doesn’t break anything.
    2. double quote, same as single quote
    3. too many comments, err, don’t put so many comment lines at the start?
    4. indent, then don’t @#&@%&@$%&@$%@#’ing indent.
    5. eclass, well no, it can’t do that, so why propose it?
    6. stuff exists before EAPI. Well, move the stuff to _UNDER_ the EAPI- line then.

    GLEP-55 doesn’t actually FIX any of the above by the way, it simply relocates the problem to the filename. Either 1 or 2 is a valid solution to the EAPI problem.

    Let me explain a few problems with GLEP-55.
    Under GLEP-55, the following will not work
    4-ebuild-version-rN.ebuild
    ebuild-4-version-rN.ebuild
    ebuild-version-4-rN.ebuild
    ebuild-version-rN-4.ebuild

    Pay no attention to the emperor, he wears no clothes. Nae, not even a kilt.

    I’ve said it once, and it bears repeating. PMS is the most accurately acronym’d document perhaps in all of history. However I think they misunderstood the true meaning of the acronym.

    • Ciaran McCreesh July 12, 2010 at 7:42 am

      The filename is already structured data. You already have to specify the filename in an extremely tightly controlled way, and GLEP 55 does nothing to change that. Ebuilds, however, are unstructured bash files that can be and are formatted with all kinds of changes in order, whitespace and quoting. Who’s going to update every single ebuild everywhere, plus in every single overlay, to the new format?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s