Now with 17% more caffeine
Stopping Pythons from eating your Rams
As various people have found out the hard way, and much to my annoyance because my laptop is memory starved, building Paludis can sometimes take lots and lots of RAM.
Originally, we didn’t do anything about this. But unfortunately lots of users have silly things like
MAKEOPTS="-j9", which can result in the build process wanting somewhere in the region of eight gigs of RAM, which in turn leads to users whining about gcc internal errors or random processes being OOMed. So we stuck a nasty hack in the ebuild that would reduce MAKEOPTS based upon how much free memory you had — all very well, but it screws over distcc users and isn’t even necessary for many combinations of
USE flags and
It’s worth working out exactly what makes the compiler memory usage so high. It’s fairly easy to figure out that it’s only an issue when building the Python bindings. We use Boost.Python for these, and unfortunately Boost will quite happily use horrible preprocessor hacks that result in huge generated source files and all sorts of nasty workarounds just to get code to work on ancient unsupported Microsoft compilers. It’s enough of a problem for enough people that there’s a tutorial section on reducing memory consumption for Boost.Python, but we already do those things.
There’s something else interesting, though. With debugging turned on (which autotools does by default), we need something like 800MBytes to compile one particular Python binding file. With it turned off, we only need 300MBytes, which is much less likely to be a problem (and more to the point it won’t make my laptop start swapping). It turns out that building the Python bindings with
-g isn’t even useful — gdb doesn’t give particularly useful backtraces on the Python interpreter, and there are better ways of tracking down problems there.
So it looks like it makes sense to add
-g0 (after checking that the compiler supports it) to
CXXFLAGS for the Python bindings. Easy enough, right?
Wrong. As with everything else involving autotools, we have to jump through all sorts of convoluted hoops to do this.
CXXFLAGS is a user variable (so we aren’t supposed to change it), and it takes precedence over
AM_CXXFLAGS (which we can change). There’s no ‘more important than the user variable’ variable, and we can’t sensibly override
CXXCOMPILE, so this gets messy. We have to abuse
configure.ac to remove the debugging options from the user’s
CXXFLAGS and move them into something that ends up in
AM_CXXFLAGS, which we can then override. Horrid.
The next Paludis release will include this voodoo, which should improve things considerably and let us avoid the nasty
MAKEOPTS mangling. But it’s still not ideal.
Most Gentoo users have
USE="python" set, either from profiles or explicitly. Most of these users do not want to build the Python bindings. Some of these users don’t think to look at the dependencies before moaning that Paludis requires Boost, so they don’t even realise it’s only because they’re using a
USE flag they probably don’t want enabled. So what can we do about this?
We can’t use
IUSE defaults, since we don’t really want to use anything later than EAPI 0 for package manager ebuilds. We could turn off
python selectively in
package.use, but lots of users still have
USE="python" set explicitly. So we use a different
USE flag name. We’ve gone for
python-bindings, along with a use description that makes it clear that thinking “well I have some things that use Python so I probably want this flag on” is wrong.
For consistency, we’ve also renamed
ruby-bindings. These are a lot more useful than the Python bindings (
playman is written in Ruby), and a lot faster to build thanks to Ruby having a reasonably sane API, so we might end up having to mess with profiles to turn them on by default.
I might end up reverting all of this if it turns out it does more harm than good. We’ll see.