He's not dead, he's resting

Stopping Pythons from eating your Rams

As various people have found out the hard way, and much to my annoyance because my laptop is memory starved, building Paludis can sometimes take lots and lots of RAM.

Originally, we didn’t do anything about this. But unfortunately lots of users have silly things like MAKEOPTS="-j9", which can result in the build process wanting somewhere in the region of eight gigs of RAM, which in turn leads to users whining about gcc internal errors or random processes being OOMed. So we stuck a nasty hack in the ebuild that would reduce MAKEOPTS based upon how much free memory you had — all very well, but it screws over distcc users and isn’t even necessary for many combinations of USE flags and CXXFLAGS.

It’s worth working out exactly what makes the compiler memory usage so high. It’s fairly easy to figure out that it’s only an issue when building the Python bindings. We use Boost.Python for these, and unfortunately Boost will quite happily use horrible preprocessor hacks that result in huge generated source files and all sorts of nasty workarounds just to get code to work on ancient unsupported Microsoft compilers. It’s enough of a problem for enough people that there’s a tutorial section on reducing memory consumption for Boost.Python, but we already do those things.

There’s something else interesting, though. With debugging turned on (which autotools does by default), we need something like 800MBytes to compile one particular Python binding file. With it turned off, we only need 300MBytes, which is much less likely to be a problem (and more to the point it won’t make my laptop start swapping). It turns out that building the Python bindings with -g isn’t even useful — gdb doesn’t give particularly useful backtraces on the Python interpreter, and there are better ways of tracking down problems there.

So it looks like it makes sense to add -g0 (after checking that the compiler supports it) to CXXFLAGS for the Python bindings. Easy enough, right?

Wrong. As with everything else involving autotools, we have to jump through all sorts of convoluted hoops to do this. CXXFLAGS is a user variable (so we aren’t supposed to change it), and it takes precedence over AM_CXXFLAGS (which we can change). There’s no ‘more important than the user variable’ variable, and we can’t sensibly override CXXCOMPILE, so this gets messy. We have to abuse to remove the debugging options from the user’s CXXFLAGS and move them into something that ends up in AM_CXXFLAGS, which we can then override. Horrid.

The next Paludis release will include this voodoo, which should improve things considerably and let us avoid the nasty MAKEOPTS mangling. But it’s still not ideal.

Most Gentoo users have USE="python" set, either from profiles or explicitly. Most of these users do not want to build the Python bindings. Some of these users don’t think to look at the dependencies before moaning that Paludis requires Boost, so they don’t even realise it’s only because they’re using a USE flag they probably don’t want enabled. So what can we do about this?

We can’t use IUSE defaults, since we don’t really want to use anything later than EAPI 0 for package manager ebuilds. We could turn off python selectively in package.use, but lots of users still have USE="python" set explicitly. So we use a different USE flag name. We’ve gone for python-bindings, along with a use description that makes it clear that thinking “well I have some things that use Python so I probably want this flag on” is wrong.

For consistency, we’ve also renamed ruby to ruby-bindings. These are a lot more useful than the Python bindings (playman is written in Ruby), and a lot faster to build thanks to Ruby having a reasonably sane API, so we might end up having to mess with profiles to turn them on by default.

I might end up reverting all of this if it turns out it does more harm than good. We’ll see.


6 responses to “Stopping Pythons from eating your Rams

  1. Chris January 13, 2009 at 11:26 am

    I like the descriptive use flag names for Ruby and Python, but wouldn’t bindings_ruby, bindings_python be better names, bindings could then become a new USE_EXPAND entry, which if accepted would help in autodocumenting use flags.

    Oh, thanks for Paludis


    • Ciaran McCreesh January 13, 2009 at 2:29 pm

      Gentoo needs a better unified way of dealing with bindings and so on, and a BINDINGS USE_EXPAND probably isn’t it. If by some miracle anything ever happens on that front, we can quite easily change again.

  2. Steven Oliver January 14, 2009 at 9:28 pm

    Having any sort of USE flag for bindings in profiles shouldn’t be allowed. Especially if this is what I’m going to run into.

    I assume python would be in ones profile for no other reason than portage and just because portage is written in python doesn’t mean I want every python binding out there.

    I don’t know how accurate Gentoo-Portage is, but searching by the python use flag returns 153 packages. Not sexy…

  3. Ciaran McCreesh January 14, 2009 at 9:34 pm

    Python’s in profiles largely because USE dependencies are a recent thing. Same for a lot of the other ‘heavy’ USE flags.

  4. Arne Babenhauserheide January 27, 2009 at 1:11 pm

    If you take a look at portage you’ll see that it doesn’t have the “python” USE flag.

    It would be useless if it had it, since it always needs python, so there is no sense in adding an option to disable the Python dependency.

    And a look at the python USE flag shows

    $ euses python
    python – Adds support/bindings for the Python language

    So you _only_ need the Python USE flag if you either want to use python yourself or want to use some python-using functions of applications.

  5. Ciaran McCreesh January 27, 2009 at 1:18 pm

    Except that Portage means everyone has Python installed anyway, so people generally expect things with optional Python support to have it turned on.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s