Paludis 0.32.4 Released

Paludis 0.32.4 has been released by spb.

C++ Explicit Template Instantiation Hate

In part two of a never ending series of why I hate C++ but have to use it anyway because there’s nothing else, we come to explicit template instantiation.

Explicit template instantiation is a nuisance that only exists because all build systems suck. Unfortunately, doing everything implicitly makes compiles take way too long, so explicit instantiation is a pragmatic nuisance. So let’s have a header that doesn’t slow down the compiler too much:

#include <string>

template <typename Item_>
struct ItemMaker
{
    Item_ make_me_an_item() const;
};

typedef ItemMaker<int> IntMaker;
typedef ItemMaker<std::string> StringMaker;

And then an implementation:

template <typename Item_>
Item_
ItemMaker<Item_>::make_me_an_item() const
{
    return Item_();
}

But then we need explicit instantiation. Should be no problem, right?

template class IntMaker;
template class StringMaker;

Wrong! Explicit instantiation only works for declarations, not typedefs, so we have to copy things out all over again:

template class ItemMaker<int>;
template class ItemMaker<std::string>;

Yes, thanks for that, Standard guys.

C++ Overload Resolution Hate

Sometimes, C++’s overload resolution rules are a pain in the ass.

Let’s say we have the following:

#include <tr1/memory>

struct Bar
{
    Bar()
    {
    }
};

struct Baz
{
    Baz()
    {
    }
};

struct Foo
{
    explicit Foo(const std::tr1::shared_ptr<const Bar> &)
    {
    }

    explicit Foo(const std::tr1::shared_ptr<const Baz> &)
    {
    }
};

Then the following is ambiguous, and won’t compile:

Foo foo(std::tr1::shared_ptr<Bar>(new Bar));

Here’s why: Neither constructor exactly matches the argument given, so the compiler falls back to construction and type conversions. std::tr1::shared_ptr<T_> has an implicit constructor template <typename U_> shared_ptr(const shared_ptr<U_> &), which is good because it lets you use a shared pointer to a derived class when a shared pointer to a base class is expected. But that conversion can take place for all U_, which means the compiler doesn’t know whether you want to convert to a shared pointer to const Bar or const Baz — it isn’t until the constructor body is instantiated that the compiler finds that only one of the two conversions will compile successfully.

So, one has to be explicit when creating the shared pointer:

Foo foo(std::tr1::shared_ptr<const Bar>(new Bar));

Except, usually we create shared pointers using a helper function, to avoid specifying the type name twice:

template <typename T_>
std::tr1::shared_ptr<T_> make_shared_ptr(T_ * const t)
{
    return std::tr1::shared_ptr<T_>(t);
}

So we’re stuck having to use a slightly weird looking allocation:

Foo foo(make_shared_ptr(new const Bar));

Incidentally, C++0x has a std::make_shared which is a lot better than this, but it requires rvalue references and std::forward to work. It would look like this:

Foo foo(std::make_shared<Bar>());

Or, if we still need the const:

Foo foo(std::make_shared<const Bar>());

Why might we not need the const? The current C++0x draft standard includes the wording “[the template] constructor shall not participate in the overload resolution unless U_ * is implicitly convertible to T_ *“, which presumably means implementations have to solve the problem using concepts to restrict the template constructor.

And, of course, there’s one final gotcha. The new const Foo form is only legal if Foo has a user defined constructor. I have no idea why, but C++03 explicitly says so.

New Gentoo EAPI Things: Secure Working Directories

This is the third post in a series about new Gentoo EAPI things.

Consider an ebuild that does something like this:

pkg_setup() {
    python -m foo -c bar
}

Here, python will first look in the current working directory for the foo module. If the superuser calling the package manager is doing so from a directory writeable by a malicious user, that user can create a foo.py in that directory that does something naughty, and it will be executed with the privileges of the superuser.

Historically, it has been the ebuild’s responsibility to work around this kind of behaviour. This is tricky, however, and most ebuilds have never bothered — most developers aren’t aware of this kind of attack, and may not even realise that their code eventually calls some program that has this behaviour (Python isn’t the only culprit). Thus, it was decided in Gentoo bug 239560 to make it the package manager’s responsibility to ensure a trusted working directory for any ebuild calls.

New Gentoo EAPI Things: Profile eapi Files

This is the second post in a series about new Gentoo EAPI things.

EAPIs as originally conceived applied only to individual package versions. However, EAPI dependent things like package dependency specifications show up in other places too, such as in various profiles files. PMS previously restricted such specifications to EAPI 0 rules, since that was all that could safely be used, but this means no slot dependencies.

A better option would be to let profile files specify which EAPI’s rules to use when parsing the files. This is what the newly-defined profile eapi files are for. Any directory under profiles/, including profiles/ itself, may now contain a file named eapi, consisting of a single line with an EAPI name in it. If present, rather than using EAPI 0 rules for files in that directory the package manager shall use that EAPI. If a package manager doesn’t support that EAPI, it must refuse to use the profile in question — or, for the top level profiles/ directory, the entire repository.

Since this is a new introduction, people must still wait until compliant package managers are ‘universally’ available (that is to say, in release stages) before starting to use EAPI 1 features elsewhere. But this is now a future possibility, rather than something that can’t happen at all.

Incidentally, you might think you’ve thought of something clever you could do with profiles if only you could use use dependencies there. Your clever idea has already occurred to others, and it doesn’t work. There is no legitimate use for any of EAPI 2’s new features in profiles.

Paludis has profile eapi file support in the next release (0.32.4 or 0.34.0). Portage doesn’t yet support this.

New Gentoo EAPI Things: DEFINED_PHASES

This is the first post in a series about new Gentoo EAPI things.

DEFINED_PHASES is a new magic metadata variable that we’ve retroactively added to all existing EAPIs. As such, package managers can’t assume that it’s supported, but that’s not really a problem. DEFINED_PHASES is purely for package manager use; it may or may not be exported in the environment to ebuilds, and ebuilds mustn’t touch it.

If a package manager supports it, at metadata generation time, the package manager will generate a list of phase names whose phase functions are defined by an ebuild (or an eclass inherited by that ebuild). The DEFINED_PHASES metadata cache key contains this list, space separated and in no particular order. For example, the current value for app-editors/vim-7.2.021 would be (possibly not in this order):

DEFINED_PHASES="setup unpack compile install test postinst postrm"

In this case, all of these functions come from eclasses.

As a special rule, if an ebuild defines no phase functions (for Gentoo, this is mostly just metapackages and new style virtuals, since the default src_install is useless), the value is a single hyphen rather than an empty string. This lets package managers tell the difference between “cache generated by an old package manager” and “no phases defined”.

Note that even the name DEFINED_PHASES is largely meaningless. The current cache format is a flat list, rather than a key=value format, so DEFINED_PHASES doesn’t show up somewhere. It has a name because at some point the switch to a key=value format cache might be made, and to avoid any future name collisions in ebuilds which might otherwise use DEFINED_PHASES as a variable.

So what’s the point?

With current 0-based EAPIs, DEFINED_PHASES has a few uses:

• It can be used to save a bit of time during the build process by ignoring phases that have an empty default implementation and no ebuild implementation, but this isn’t really a huge deal. Package managers could get this benefit anyway by regenerating metadata locally up-front.
• For interactive packages, it lets the package manager avoid having to go into ‘interactive’ mode for most phases.
• It lets the package manager parallelise pkg_ phases if there’s no implementation. Normally pkg_ phases have to be run exclusively because they can modify the live filesystem.

DEFINED_PHASES really comes into its own with various things that will probably be in future EAPIs (some of which are already in exheres-0):

• The proposed pkg_pretend function, which will be run at pretend-install time, would require about 0.1s per package to be changed. When doing a large upgrade, this can result in the user having to sit there waiting for tens of seconds whilst the package manager runs lots of mostly useless ebuild phase functions.
• Similarly, src_fetch_extra (to handle fetching scm things, rather than using src_unpack) shouldn’t be run except where necessary.

It might be possible to use this to get pkg_nofetch information upfront at pretend-install time too. I’m still thinking about that.

Paludis has had DEFINED_PHASES support since 0.32.2, although no cache line had been standardised so it was only used for self-generated caches. The next release (0.32.4 or 0.34.0) will use the agreed-upon line. Portage doesn’t yet support this, so Gentoo-generated caches won’t include a value yet anyway.

What’s New in Gentoo EAPIs?

Three moderately interesting new things have been approved by the Gentoo Council:

• DEFINED_PHASES
• Profile eapi files
• Secure working directories

Full details can be found in a series of patches which will be merged to PMS shortly; an overview of each will appear in subsequent posts.

Paludis 0.32.3 Released

Paludis 0.32.3 has been released:

• Fixed handling of VIDEO_CARDS=”foo” in make.conf, ** in package.keywords with PortageEnvironment.
• Fixed new/changed options display.
• Fixed the stripper.
• RESTRICTed build options are no longer shown.

Paludis 0.32.2 Released

Paludis 0.32.2 has been released:

• Fixed various compiler issues.
• ERepository::package_names (and by extension, paludis --list-packages) are now quite a bit faster, but may list package names whose package directories include no versions.
• DEFINED_PHASES is now used and supported (but not used for 0-based EAPIs with pre-generated cache, since a cache line has yet to be standardised). As a result, ebuild hooks for phases that aren’t used will not necessarily be triggered.