Ten Ways PMS Raped your Baby

Since hating PMS seems to be back in fashion again this week, I thought I’d list ten of the stupidest claims that I’ve seen of late in the hope that some of the FUD might die down:

1. PMS slows down new features and prevents innovation

Actually, once new ebuild-usable features end up in Portage, they very quickly end up in a published EAPI. The reason you can’t use all the fancy new features in EAPI 3 that you’ve all been waiting for for so long is that Portage still hasn’t implemented them. In addition, we’ve gone from “EAPI 3 will be ready for Portage within a month” three months ago to “there’s an 80% chance EAPI 3 support will be ready in Portage by the end of the year“.

Time-wise, EAPI 3 has been waiting for Portage for six months and before that, for the Council to come to decisions for two months. The total overhead imposed by PMS was around four days, and those four days weren’t holding up anything else anyway.

Still, at least there’s a long way to go before EAPI 3 takes as long as it took Portage to get use dependencies…

Similarly, PMS isn’t to blame for profiles not being able to make use of new features. People who are telling you this are probably thinking about an undocumented Portage feature that isn’t in PMS and that isn’t supported by other package managers. This feature could very easily be in PMS, but there’s been no interest from the Council in retroactively adding it to EAPI 3 or in doing an EAPI 2.1 just to include that feature. The feature almost certainly will be in EAPI 4, but work on EAPI 4 isn’t going to start until Portage is done with EAPI 3. So again, PMS isn’t the reason you can’t use it.

2. PMS or EAPI is about ebuilds, not profiles

This one’s from people who haven’t bothered to read the opening of PMS, or who haven’t been paying attention to the Council.

PMS covers ebuilds and the tree format, including things like profiles. The aim is to cover everything necessary to produce a package manager that can use ebuilds (except possibly VDB, which probably shouldn’t be necessary for ebuild support but currently is…).

EAPI is used to indicate the rules used to handle ebuilds, and also profiles following the Council accepting Zac’s proposal last year.

3. PMS imposes an Exherbo agenda upon Gentoo

Exherbo doesn’t use PMS or Gentoo EAPIs.

4. PMS imposes a Paludis agenda upon Gentoo

Again, no. There’s no feature in any EAPI that’s there because of Paludis. Every feature in EAPIs 1, 2 and 3 was either requested by a Gentoo developer or included to make things easier for Portage. To get into an EAPI, features merely have to be vetted by the Portage team and the Council.

5. PMS is only used by Paludis anyway

Nope. PMS is used by both Portage and Paludis, as well as a number of third party libraries and utilities which don’t support full package management operations (things that need to compare two versions, for example, need to use PMS), and it was also used by Pkgcore.

Saying that PMS is only of use for third party package managers is like saying that the HTTP specification is irrelevant for Internet Explorer.

6. PMS stops other distributions from doing things

Again, no. Other distributions can ignore PMS entirely. Doing so would of course be a horrible idea, as all the people who wrote websites to work on Internet Explorer 5 found out, but that’s their decision to make. A much better option would be for those distributions to roll their own derived EAPIs, which, as happened for the Gentoo KDE project’s kdebuild-1, could be added to PMS. That way they are guaranteed that any undocumented features they rely upon won’t break with the next release, as well as avoiding complaints from users who want to use a different package manager, thus avoiding the problems people who wrote Internet Explorer 5 specific code rather than following the HTML specification encountered.

7. PMS stops Gentoo from deciding its own features

PMS is run by a Gentoo developer and approval of new EAPI features is handled by the Portage team and the Gentoo council. For that matter, the PMS team has never rejected a single feature for inclusion in a future EAPI.

8. PMS introduces red tape

No, the previous Council introduced red tape, primarily because a couple of Council members refused to read any submissions more than five minutes before a meeting. Had the Council used the two weeks between meetings to read over and state their opinions on the EAPI 3 feature list, EAPI 3 would have been approved within two meetings rather than dragging on for months.

Unfortunately the current Council doesn’t seem to have improved, with at least one member showing up to a meeting having not read the agenda beforehand.

9. PMS imposed stupid ordering on metadata files

There’s a tendency amongst certain people to blame PMS for stupid or arbitrary rules. A typical example is to moan about PMS because it says that EAPI has to be on line 15 of metadata cache files. Quite how PMS is to blame for a decision that was made before PMS existed, and that was made because line 15 was the first available cache line when EAPI was introduced as a metadata key, is completely beyond me. Similarly, the rules for handling leading 0s in version numbers is Portage’s fault (although ultimately it’s Perl’s fault), as is any other format gripe you care to name.

10. PMS will stop me from using my favourite feature in configuration files

PMS doesn’t discuss user configuration at all. Handling of user configuration is left entirely to the package manager.

Funtoo’s Website is Designed for Internet Explorer

Taken from the Funtoo FAQ:

“Is Mozilla Firefox compatible with funtoo.org?”

Mozilla Firefox does not appear to be compatible with the Funtoo.org website, unfortunately. It does not support the insertAdjacentHTML function, and also does not support XML data islands. These features are supported by Internet Explorer and we use both of these features. I think that insertAdjacentHTML is a good idea, and I also think it’s a good idea for the browser to support XML data islands, which are particularly useful for formatting text. Funtoo.org uses its own documentation management method, and these documents are easier to maintain if insertAdjacentHTML is correctly supported by the browser, and XML data islands are a great help to us as well.

We could maintain patched versions of Mozilla Firefox that would be usable with the Funtoo.org website, but I don’t want to get into the business of supporting a non-standard browser as the upstream project does not seem very supportive of getting these compatibility issues resolved, and thus this could turn into a maintenance burden for Funtoo. So I’d rather just not support Mozilla Firefox for now.

I guess I expected open source projects to be more widely aware of the importance of having open standards, coding to a standard rather than an implementation and driving innovation through new, public standards rather than abusing undocumented proprietary extensions.

Update: This post seems to have been exposed to a wider audience than my usual readership. For those who don’t feel like reading the comments, and who don’t understand what’s being discussed here, you can compare the quote above to the Funtoo FAQ entry introduced by this commit (unfortunately there’s no way to link to individual Funtoo FAQ entries).

EAPI 3: A Preview

Gentoo is shuffling its way towards EAPI 3. The details haven’t been worked out yet, but there’s a provisional list of things likely to show up that’s mostly been agreed upon. This post will provide a summary; when EAPI 3’s finalised, I’ll do a series of posts with full descriptions as I did for EAPI 2. PMS will remain the definitive definition; I’ve put together a a draft branch (I’ll be rebasing this, so don’t base work off it if you don’t know how to deal with that).

Everything on this list is subject to removal, arbitrary change or nuking from orbit. We’re looking for a finalisation reasonably soon, so if it turns out Portage is unable to support any of these, they’ll be dropped rather than holding the EAPI up.

EAPI 3 will be defined in terms of differences to EAPI 2. These differences may include:

• pkg_pretend support. This will let ebuilds signal a lot more errors at pretend-time, rather than midway through an install of a hundred packages that you’ve left running overnight. This feature is already in exheres-0.
• Slot operator dependencies. This will let ebuilds specify what to do when they depend upon a package that has multiple slots available — using :* deps will mean “I can use any slot, and it can change at runtime”, whilst := means “I need the best slot that was there at compile time”. This feature is already in exheres-0 and kdebuild-1.
• Use dependency defaults. With EAPI 2 use dependencies, it’s illegal to reference a flag in another package unless that package has that flag in IUSE. With use dependency defaults, you’ll be able to use foo/bar[flag(+)] and foo/bar[flag(-)] to mean “pretend it’s enabled (disabled) if it’s not present”. This feature is already in exheres-0.
• DEFINED_PHASES and PROPERTIES will become mandatory (they’re currently optional). This won’t have any effect for users (although without the former, pkg_pretend would be slooooow).
• There’s going to be a default src_install of some kind. Details are yet to be entirely worked out.
• Ebuilds will be able to tell the package manager that it’s ok or not ok to compress certain documentation things using the new docompress function.
• dodoc will have a -r, for recursively installing directories.
• doins will support symlinks properly.
• || ( use? ( ... ) ) will be banned.
• dohard and dosed will be banned. (Maybe. This one’s still under discussion.)
• New doexample and doinclude functions. (Again, maybe. Quite a few people think these’re icky and unnecessary.)
• unpack will support a few new extensions, probably xz, tar.xz and maybe xpi.
• econf will pass --disable-dependency-tracking --enable-fast-install. This is already done for exheres-0.
• pkg_info will be usable on uninstalled packages too. This is already in exheres-0 and kdebuild-1.
• USE and friends will no longer contain arbitrary extra values. (Possibly. Not sure Portage will have this one done in time.)
• AA and KV will be removed.
• New REPLACED_BY_VERSION and REPLACING_VERSIONS variables, to let packages work out whether they’re upgrading / downgrading / reinstalling. exheres-0 has a more sophisticated version.
• The automatic S to WORKDIR fallback will no longer happen under certain conditions. exheres-0 already has this.
• unpack will consider unrecognised suffixes an error unless --if-compressed is specified, and the default src_unpack will pass this. exheres-0 already has this. (Maybe. Not everyone’s seen the light on this one yet.)
• The automagic RDEPEND=DEPEND ick will be gone.
• Utilities will die on failure unless prefixed by nonfatal. exheres-0 already has this.

Unless, of course, something completely different happens.

New Gentoo EAPI Things: Secure Working Directories

This is the third post in a series about new Gentoo EAPI things.

Consider an ebuild that does something like this:

pkg_setup() {
    python -m foo -c bar
}

Here, python will first look in the current working directory for the foo module. If the superuser calling the package manager is doing so from a directory writeable by a malicious user, that user can create a foo.py in that directory that does something naughty, and it will be executed with the privileges of the superuser.

Historically, it has been the ebuild’s responsibility to work around this kind of behaviour. This is tricky, however, and most ebuilds have never bothered — most developers aren’t aware of this kind of attack, and may not even realise that their code eventually calls some program that has this behaviour (Python isn’t the only culprit). Thus, it was decided in Gentoo bug 239560 to make it the package manager’s responsibility to ensure a trusted working directory for any ebuild calls.

New Gentoo EAPI Things: Profile eapi Files

This is the second post in a series about new Gentoo EAPI things.

EAPIs as originally conceived applied only to individual package versions. However, EAPI dependent things like package dependency specifications show up in other places too, such as in various profiles files. PMS previously restricted such specifications to EAPI 0 rules, since that was all that could safely be used, but this means no slot dependencies.

A better option would be to let profile files specify which EAPI’s rules to use when parsing the files. This is what the newly-defined profile eapi files are for. Any directory under profiles/, including profiles/ itself, may now contain a file named eapi, consisting of a single line with an EAPI name in it. If present, rather than using EAPI 0 rules for files in that directory the package manager shall use that EAPI. If a package manager doesn’t support that EAPI, it must refuse to use the profile in question — or, for the top level profiles/ directory, the entire repository.

Since this is a new introduction, people must still wait until compliant package managers are ‘universally’ available (that is to say, in release stages) before starting to use EAPI 1 features elsewhere. But this is now a future possibility, rather than something that can’t happen at all.

Incidentally, you might think you’ve thought of something clever you could do with profiles if only you could use use dependencies there. Your clever idea has already occurred to others, and it doesn’t work. There is no legitimate use for any of EAPI 2’s new features in profiles.

Paludis has profile eapi file support in the next release (0.32.4 or 0.34.0). Portage doesn’t yet support this.

New Gentoo EAPI Things: DEFINED_PHASES

This is the first post in a series about new Gentoo EAPI things.

DEFINED_PHASES is a new magic metadata variable that we’ve retroactively added to all existing EAPIs. As such, package managers can’t assume that it’s supported, but that’s not really a problem. DEFINED_PHASES is purely for package manager use; it may or may not be exported in the environment to ebuilds, and ebuilds mustn’t touch it.

If a package manager supports it, at metadata generation time, the package manager will generate a list of phase names whose phase functions are defined by an ebuild (or an eclass inherited by that ebuild). The DEFINED_PHASES metadata cache key contains this list, space separated and in no particular order. For example, the current value for app-editors/vim-7.2.021 would be (possibly not in this order):

DEFINED_PHASES="setup unpack compile install test postinst postrm"

In this case, all of these functions come from eclasses.

As a special rule, if an ebuild defines no phase functions (for Gentoo, this is mostly just metapackages and new style virtuals, since the default src_install is useless), the value is a single hyphen rather than an empty string. This lets package managers tell the difference between “cache generated by an old package manager” and “no phases defined”.

Note that even the name DEFINED_PHASES is largely meaningless. The current cache format is a flat list, rather than a key=value format, so DEFINED_PHASES doesn’t show up somewhere. It has a name because at some point the switch to a key=value format cache might be made, and to avoid any future name collisions in ebuilds which might otherwise use DEFINED_PHASES as a variable.

So what’s the point?

With current 0-based EAPIs, DEFINED_PHASES has a few uses:

• It can be used to save a bit of time during the build process by ignoring phases that have an empty default implementation and no ebuild implementation, but this isn’t really a huge deal. Package managers could get this benefit anyway by regenerating metadata locally up-front.
• For interactive packages, it lets the package manager avoid having to go into ‘interactive’ mode for most phases.
• It lets the package manager parallelise pkg_ phases if there’s no implementation. Normally pkg_ phases have to be run exclusively because they can modify the live filesystem.

DEFINED_PHASES really comes into its own with various things that will probably be in future EAPIs (some of which are already in exheres-0):

• The proposed pkg_pretend function, which will be run at pretend-install time, would require about 0.1s per package to be changed. When doing a large upgrade, this can result in the user having to sit there waiting for tens of seconds whilst the package manager runs lots of mostly useless ebuild phase functions.
• Similarly, src_fetch_extra (to handle fetching scm things, rather than using src_unpack) shouldn’t be run except where necessary.

It might be possible to use this to get pkg_nofetch information upfront at pretend-install time too. I’m still thinking about that.

Paludis has had DEFINED_PHASES support since 0.32.2, although no cache line had been standardised so it was only used for self-generated caches. The next release (0.32.4 or 0.34.0) will use the agreed-upon line. Portage doesn’t yet support this, so Gentoo-generated caches won’t include a value yet anyway.

What’s New in Gentoo EAPIs?

Three moderately interesting new things have been approved by the Gentoo Council:

• DEFINED_PHASES
• Profile eapi files
• Secure working directories

Full details can be found in a series of patches which will be merged to PMS shortly; an overview of each will appear in subsequent posts.

MYOPTIONS: It’s like IUSE with Candy

The exheres-0 package format, used primarily by Exherbo, calls what Gentoo calls ‘USE flags’ ‘options’. What PMS EAPIs call IUSE, exheres-0 calls MYOPTIONS.

Up until recently, this has just been a differently named variable, minus support for IUSE defaults because we hateses them. But now that Paludis has the choices API we’re not stuck using that format. The first extension is fairly simple:

MYOPTIONS="foo bar baz linguas: en en_GB fr"

This is much nicer than having to write out linguas:en linguas:en_GB etc., and is especially important for exheres-0 because SUBOPTIONS (USE_EXPAND) values must be explicitly listed.

The next step is flag descriptions. use.local.desc is rather crude, and XML is horrible, so we thought about re-using annotations:

MYOPTIONS="
    X [[ description = [ Build a graphical user interface ] ]]
    python [[ description = [ Build Python language bindings ] ]]
    nls
    linguas: en en_GB fr"

Any undescribed flag falls back to the global description — general consensus is to keep those, because they make it easier for a user to set up a fresh options.conf.

Whilst we’re at it, we might as well solve the conflicting options problem. In the good old days, use flags were used only when something was optional — that is, if support for foo also needed support for bar, USE="-foo bar" would just compile without bar. Unfortunately, a few people didn’t really like that, and even more unfortunately developers started doing horrible die calls in pkg_setup rather than coming up with a proper solution.

With half of the pkg_setup die calls eliminated by use dependencies, it seems a shame not to fix the other not-quite-half-because-of-a-few-obscure-things. There’s already pkg_pretend for exheres-0, which is a big improvement, but moving handling of the common cases into the package manager is cleaner.

So, we start with the simplest case: flags requiring other flags.

MYOPTIONS="
        X
        python
        gtk [[ requires = [ X python ] ]]
        qt [[ requires = X ]]
        motif [[ requires = X ]]
        "

We might like SUBOPTIONS and negatives too:

MYOPTIONS="
        minimal
        python [[ requires = [ -minimal ] ]]
        linguas:
            en
            en_GB [[ requires = [ linguas: en ] ]]
        "

There might be a case for “if blah is not enabled then …” requirements:

MYOPTIONS="
        X
        -ncurses [[ requires = [ X ] ]]
        "

although we have a nicer solution this particular case. Note that it’s ok to list the same flag multiple times, so the above can be written as:

MYOPTIONS="
        X
        ncurses
        -ncurses [[ requires = [ X ] ]]
        "

For convenience, we’d like to be able to apply the same requires annotation to multiple items:

MYOPTIONS="
        X
        python
        (
            gtk [[ requires = python ]]
            qt
            motif
        ) [[ requires = X ]]
        "

Here, gtk requires both X and python (although excessive mixing of things is discouraged for style reasons).

Sometimes, you have to select one of a number of flags:

MYOPTIONS="
        (
            gtk
            qt
            motif
        ) [[ number-selected = exactly-one ]]
        "

Also allowed are at-least-one and at-most-one.

Sometimes requirements are conditional, too:

MYOPTIONS="
        X
        python
        X? (
            (
                gtk [[ requires = python ]]
                qt
                motif
            ) [[
                number-selected = exactly-one
                requires = X
            ]]
        )
        "

Although, for style reasons, this would end up looking more like:

MYOPTIONS="
        X [[ description = [ Include a GUI ] ]]
        python [[ description = [ Build Python language bindings ] ]]
        gtk
        qt
        motif

        gtk [[ requires = python ]]
        ( gtk qt motif ) [[ requires = X ]]
        X? ( ( gtk qt motif ) [[ number-selected = exactly-one ]] )
        "

As for how these are verified… They’re checked at --pretend --install time, right before pkg_pretend is run. Even if a requirement fails, though, pkg_pretend is still run, allowing us to show as many notices as necessary at the same time.

The failure is indicated to the user by the pkg_bad_options function. This probably won’t be overridden by very many packages, and those that do will almost certainly call default. The default output looks like this:

These packages will be installed:

* test-cat/test-pkg::ciaranm :1 [N 1] <target>
    X gtk -motif -python qt build_options: recommended_tests split strip
    "Dummy test package"

Total: 1 package (1 new)

 * The following option requirements are unmet for test-cat/test-pkg-1:
 *     Enabling option 'gtk' requires option 'python' to be enabled
 *     Exactly one of options ( gtk, qt, motif ) must be met

* Cannot continue with install due to the errors indicated above

And just think, all that without resorting to convoluted and incomprehensible set theory.

EAPI 2: doman language support

This is the final post in a series on EAPI 2.

The doman helper is one of those pesky little beasts that makes specifying EAPI behaviour formally such a nuisance (although it is nowhere near as horrible as dohtml). EAPI 2 makes it even peskier.

I’ll try that again.

The doman helper makes writing ebuilds substantially easier by automagically doing the right thing when installing manual pages, freeing the developer from having to care about manual sections. EAPI 2 makes doman even more useful by making it aware of language codes as well as sections.

The painful details are available in PMS, but basically this will now ‘do the right thing’:

doman foo.1 foo.en.1 foo.en_GB.1

Previously only the first of the items would go to the right place.

This one’s a Gentoo innovation; see bug 222439 for its history. It was shamelessly stolen for exheres-0, but was too late for kdebuild-1.

EAPI 2: default_ phase functions and the default function

This post is part of a series on EAPI 2.

With EAPIs 0 and 1, if you want to add something to, say, src_unpack, you have to manually write out the default implementation and then add your code. This is easy to screw up — developers are highly prone to getting the quoting wrong and forgetting which functions do and do not want a || die on the end.

EAPI 2 makes the default implementation of phase functions available as functions themselves. These functions are named default_src_unpack, default_src_configure and so on.

Typing out default_src_compile in full is pointless, though (especially since it’s illegal to call phase functions or default phase functions from other phase functions). So we also introduce the special default function, which calls whichever default_ phase function is appropriate for the phase we’re in. Thus:

src_compile() {
    default
    if useq extras ; then
        emake extras || die "splat"
    fi
}

Both features first appeared in exheres-0.

An alternative proposal (I think it came from the Pkgcore camp) was to make all EAPI default implementations available through functions named like eapi0_src_compile, eapi1_src_compile and eapi2_src_compile. This proposal was rejected because various Paludis people moaned about it not making sense or having any legitimate use cases (the ‘obvious’ use cases don’t work if you think them through), and no-one stood up to defend it.