Stopping Pythons from eating your Rams

As various people have found out the hard way, and much to my annoyance because my laptop is memory starved, building Paludis can sometimes take lots and lots of RAM.

Originally, we didn’t do anything about this. But unfortunately lots of users have silly things like MAKEOPTS="-j9", which can result in the build process wanting somewhere in the region of eight gigs of RAM, which in turn leads to users whining about gcc internal errors or random processes being OOMed. So we stuck a nasty hack in the ebuild that would reduce MAKEOPTS based upon how much free memory you had — all very well, but it screws over distcc users and isn’t even necessary for many combinations of USE flags and CXXFLAGS.

It’s worth working out exactly what makes the compiler memory usage so high. It’s fairly easy to figure out that it’s only an issue when building the Python bindings. We use Boost.Python for these, and unfortunately Boost will quite happily use horrible preprocessor hacks that result in huge generated source files and all sorts of nasty workarounds just to get code to work on ancient unsupported Microsoft compilers. It’s enough of a problem for enough people that there’s a tutorial section on reducing memory consumption for Boost.Python, but we already do those things.

There’s something else interesting, though. With debugging turned on (which autotools does by default), we need something like 800MBytes to compile one particular Python binding file. With it turned off, we only need 300MBytes, which is much less likely to be a problem (and more to the point it won’t make my laptop start swapping). It turns out that building the Python bindings with -g isn’t even useful — gdb doesn’t give particularly useful backtraces on the Python interpreter, and there are better ways of tracking down problems there.

So it looks like it makes sense to add -g0 (after checking that the compiler supports it) to CXXFLAGS for the Python bindings. Easy enough, right?

Wrong. As with everything else involving autotools, we have to jump through all sorts of convoluted hoops to do this. CXXFLAGS is a user variable (so we aren’t supposed to change it), and it takes precedence over AM_CXXFLAGS (which we can change). There’s no ‘more important than the user variable’ variable, and we can’t sensibly override CXXCOMPILE, so this gets messy. We have to abuse configure.ac to remove the debugging options from the user’s CXXFLAGS and move them into something that ends up in AM_CXXFLAGS, which we can then override. Horrid.

The next Paludis release will include this voodoo, which should improve things considerably and let us avoid the nasty MAKEOPTS mangling. But it’s still not ideal.

Most Gentoo users have USE="python" set, either from profiles or explicitly. Most of these users do not want to build the Python bindings. Some of these users don’t think to look at the dependencies before moaning that Paludis requires Boost, so they don’t even realise it’s only because they’re using a USE flag they probably don’t want enabled. So what can we do about this?

We can’t use IUSE defaults, since we don’t really want to use anything later than EAPI 0 for package manager ebuilds. We could turn off python selectively in package.use, but lots of users still have USE="python" set explicitly. So we use a different USE flag name. We’ve gone for python-bindings, along with a use description that makes it clear that thinking “well I have some things that use Python so I probably want this flag on” is wrong.

For consistency, we’ve also renamed ruby to ruby-bindings. These are a lot more useful than the Python bindings (playman is written in Ruby), and a lot faster to build thanks to Ruby having a reasonably sane API, so we might end up having to mess with profiles to turn them on by default.

I might end up reverting all of this if it turns out it does more harm than good. We’ll see.

Paludis Ruby Bindings and Template Classes

A PackageID in Paludis supports various actions. An Action is represented by a subclass instance, such as InstallAction or ConfigAction, some of which carry member data (for example, an InstallAction carries information about the target repository for the install).

To perform an action, the PackageID::perform_action method is used. But not all IDs support all actions — you can’t, for example, uninstall a package that isn’t installed, and not all EAPIs support the ‘pretend’ action. So there’s a second method, PackageID::supports_action, that returns a bool saying whether an action is supported.

That’s all very well, but it would require constructing an Action subclass instance just for querying purposes. There’s not much point in this. So we make PackageID::supports_action take a SupportsActionTest<SomeAction> parameter rather than an Action. Unlike the base Action subclass, the SupportsActionTest<T_> template class carries no member data, so it doesn’t need fancy construction. This lets us do this:

if (my_package_id->supports_action(SupportsActionTest<FetchAction>()))
{
    FetchAction fetch_action(_imp->fetch_options);
    my_package_id->perform_action(fetch_action);
}

This pattern crops up in two other places. To speed up certain queries, we can ask a Repository whether some of its IDs might support a particular action. The Repository::some_ids_might_support_action method will always return true if any of its IDs support a particular action, and might return false if it’s known for sure that none of them will (this weasel wording is necessary because we might, for example, have a repository full of ebuilds with unsupported EAPIs, and unsupported EAPIs means no actions are possible).

Similarly, the new filter system has filter::SupportsAction<ActionClass>. The implementation of this filter uses the Repository and PackageID methods to be as lazy as possible.

Which is all well and good, in C++, but with bindings things get a bit icky since templates don’t translate naturally. In Ruby, we used to have a bunch of classes. SupportsInstallActionTest.new() would be like SupportsActionTest<InstallAction>(), SupportsConfigActionTest.new() would be like SupportsActionTest<ConfigAction> and so on. This isn’t particularly nice.

It occurred to me that SupportsActionTest.new(InstallAction) is legal syntactically in Ruby. It passes the value InstallAction, which is a variable of class Class, as the parameter. Then we have to screw around a bit in the bindings code, remembering that SomeClass <= OtherClass in Ruby means “SomeClass is or is a subclass of OtherClass“:

/*
 * Document-method: SupportsActionTest.new
 *
 * call-seq:
 *     SupportsActionTest.new(ActionClass) -> SupportsActionTest
 *
 * Create new SupportsActionTest object. The ActionClass should be, e.g. InstallAction.
 */
static VALUE
supports_action_test_new(VALUE self, VALUE action_class)
{
    std::tr1::shared_ptr<const SupportsActionTestBase> * ptr(0);

    try
    {
        if (Qtrue == rb_funcall2(action_class, rb_intern("<="), 1, install_action_value_ptr()))
            ptr = new std::tr1::shared_ptr<const SupportsActionTestBase>(make_shared_ptr(new SupportsActionTest<InstallAction>()));
        else if (Qtrue == rb_funcall2(action_class, rb_intern("<="), 1, installed_action_value_ptr()))
            ptr = new std::tr1::shared_ptr<const SupportsActionTestBase>(make_shared_ptr(new SupportsActionTest<InstalledAction>()));
        else if (Qtrue == rb_funcall2(action_class, rb_intern("<="), 1, uninstall_action_value_ptr()))
            ptr = new std::tr1::shared_ptr<const SupportsActionTestBase>(make_shared_ptr(new SupportsActionTest<UninstallAction>()));
        else if (Qtrue == rb_funcall2(action_class, rb_intern("<="), 1, pretend_action_value_ptr()))
            ptr = new std::tr1::shared_ptr<const SupportsActionTestBase>(make_shared_ptr(new SupportsActionTest<PretendAction>()));
        else if (Qtrue == rb_funcall2(action_class, rb_intern("<="), 1, config_action_value_ptr()))
            ptr = new std::tr1::shared_ptr<const SupportsActionTestBase>(make_shared_ptr(new SupportsActionTest<ConfigAction>()));
        else if (Qtrue == rb_funcall2(action_class, rb_intern("<="), 1, fetch_action_value_ptr()))
            ptr = new std::tr1::shared_ptr<const SupportsActionTestBase>(make_shared_ptr(new SupportsActionTest<FetchAction>()));
        else if (Qtrue == rb_funcall2(action_class, rb_intern("<="), 1, info_action_value_ptr()))
            ptr = new std::tr1::shared_ptr<const SupportsActionTestBase>(make_shared_ptr(new SupportsActionTest<InfoAction>()));
        else if (Qtrue == rb_funcall2(action_class, rb_intern("<="), 1, pretend_fetch_action_value_ptr()))
            ptr = new std::tr1::shared_ptr<const SupportsActionTestBase>(make_shared_ptr(new SupportsActionTest<PretendFetchAction>()));
        else
            rb_raise(rb_eTypeError, "Can't convert %s into an Action subclass", rb_obj_classname(action_class));

        VALUE tdata(Data_Wrap_Struct(self, 0, &Common<std::tr1::shared_ptr<const SupportsActionTestBase> >::free, ptr));
        rb_obj_call_init(tdata, 0, &self);
        return tdata;
    }
    catch (const std::exception & e)
    {
        delete ptr;
        exception_to_ruby_exception(e);
    }
}

/*
 * Document-class: Paludis::SupportsActionTest
 *
 * Tests whether a Paludis::PackageID supports a particular action.
 */
c_supports_action_test = rb_define_class_under(paludis_module(), "SupportsActionTest", rb_cObject);
rb_define_singleton_method(c_supports_action_test, "new", RUBY_FUNC_CAST(&supports_action_test_new), 1);

Not exactly pretty, for now. Hopefully when C++0x comes along we’ll be able to invent some obscene hack involving std::initializer_list, lambdas and std::find_if which will make the whole thing somewhat more elegant.

Unfortunately, the Python bindings are still stuck using package_id.supports_action(SupportsFetchActionTest()) etc. So far as I can see there’s no nice way to get the same effect whilst using boost.python, even though Python lets you pass classes around in a similar manner to Ruby.