Programming – Principles and Practice using C++

Programming — Principles and Practice Using C++ is the new book by Bjarne Stroustrup, the daddy of C++. It’s an introduction to programming rather than an advanced book; I’ve been holding off writing up my impressions of it because I’m not entirely sure what to say.

The overriding theme of this book seems to be “there are lots of complications, special cases and obscure things”. This is of course true, and it’s a refreshing change from most introductory books that go out of their way to construct highly contrived examples that conveniently ignore any obscurity. But I suspect it goes too far — pretty much every example is twice as long as it probably should be. There’s so much focus on dealing with complexities that the underlying “what’s going on?” is lost.

Partly this is down to C++. A language designed to handle real world, large scale problems and provide for maintainability over decades isn’t going to be the most elegant. On the other hand, purely teaching languages that dismiss the real world entirely are of no practical use. The question is whether C++ as a first language is a sensible idea, and I’m not in the least bit convinced that it is.

Partly, though, this is down to the choice of projects. An example: two chapters are devoted to writing a calculator program. These chapters cover lexers, parsers, grammars and error recovery. This isn’t one of those cop-out calculator programs where syntax is carefully selected to hide any kind of mess, either — it almost looks like the book is going to end up implementing a compiler… Unfortunately, there’s nothing in the final program that really needs any of this complexity; a simple “tokenise into a list, then replace all the multiplications with their result, then replace all the divisions with their result and so on” would work just as well for the requirements, and wouldn’t have most of the mess.

The scope of the book is impressive, though. It doesn’t gloss over classes, templates, pointers, exceptions or even dealing with code written in C. It’s extremely comprehensive, even in places where it probably shouldn’t be.

Finally, a note on writing style. The word ‘basically’ appears on average once per page, and sometimes three or four times in a single paragraph. This gets very annoying very quickly. Stroustrup’s other books don’t suffer from this.

I suppose my conclusion is: if you have to learn C++ as a first language, this is the book to use. If you have a choice, though, learn one of the monkey languages first, and then pick up The C++ Programming Language and The C++ Standard Library.

Paludis 0.34.1 Released

Paludis 0.34.1 has been released:

• We can now skip src_ phases where it is safe to do so.
• Documentation updates for repository configuration.
• Support for managing user and group accounts (requires distribution support).

Managing Accounts with the Package Manager

Paludis is a multi-format package manager. One beneficial side effect of this is that the core code is sufficiently flexible to make handling things that aren’t really ‘packages’ in the conventional sense very easy; in the past, this has been used to deliver unavailable, unwritten and unpackaged repositories.

One of the things Exherbo inherited from Gentoo without modification was user and group management. In Gentoo, this is done by functions called enewuser and enewgroup from eutils.eclass; a package that needs a user or group ID must call these functions from pkg_setup. Although usable, this is moderately icky; Exherbo can do better than that.

Really, user and group accounts are just resources. A package that needs a particular user ID can be thought of as depending upon that ID — the only disconnect is that currently dependencies are for packages, not resources. Can we find a way of representing resources as something like packages, in a way that makes sense?

Fortunately, the obvious solution works. Having user/paludisbuild and group/paludisbuild as packages makes sense; adding the user or group is equivalent to installing the appropriate package, and if the user or group is present on the system, it shows up as installed. Then, instead of calling functions, the exheres can just do:

DEPENDENCIES="
        build+run:
            user/paludisbuild
            group/paludisbuild
    "

What about defaults? Different users need different shells, home directories, groups and so on. We could represent these a bit like options, but there’s a better way.

If two or more ebuilds need the same user, they all have to do the useradd call. This means duplicating things like home directory information and preferred uid over lots of different ebuilds, which is bad. It would be better to place the users somewhere else. For Exherbo, we’ve gone with metadata/accounts/{users,groups}/*.conf. A user’s settings look something like this (the username is taken from a filename, so this would be metadata/accounts/users/paludisbuild.conf):

shell = /bin/bash
gecos = Used by Paludis for operations that require an unprivileged user
home = /var/tmp/paludis
primary_group = paludisbuild
extra_groups =
preferred_uid =

And a group, metadata/accounts/groups/paludisbuild.conf:

preferred_gid =

We only specify ‘empty’ keys for demonstration purposes; ordinarily they would be omitted.

We automatically make users depend upon the groups they use. The existing dependency abstractions are sufficient for this. There’s a bit of trickery in Paludis to allow supplemental repositories to override user defaults found in their masters; details are in the source for those who care.

One more thing to note: with accounts specified this way, we can be sure that the package manager only manages relevant accounts. There’s no danger of having the package manager accidentally start messing with your user accounts.

So what are the implications?

• We’re no longer tied to a particular method of adding users. If a user doesn’t want to use useradd and groupadd, they can write their own handler for the package manager to update users via LDAP or whatever. Paludis supports multiple handlers here.
• Users who would rather manage a particular account manually can add it themselves, and the package manager will treat it as being already installed and won’t try to mess with it.
• User and group defaults are in one place, not everywhere that uses them.
• It’s much more obvious when an account is going to be added.
• Accounts that are no longer required can be purged using the usual uninstall-unused mechanism.

And what does it look like?

$ paludis -pi test-pkg
Building target list...
Building dependency list...   

These packages will be installed:

* group/alsogroupdemo [N 0]
    Reasons: *user/accountsdemo-0:0::accounts
    "alsogroupdemo"
* group/groupdemo [N 0]
    Reasons: *user/accountsdemo-0:0::accounts
    "groupdemo"
* group/thirdgroupdemo [N 0]
    Reasons: *user/accountsdemo-0:0::accounts
    "thirdgroupdemo"
* user/accountsdemo [N 0]
    Reasons: *test-cat/test-pkg-2:2::ciaranm_exheres_test
    "A demo account"
* test-cat/test-pkg::ciaranm_exheres_test :2 [N 2] <target>
    -foo build_options: recommended_tests split strip
    "Dummy test package"

We can have a look at the accounts before they’re installed:

$ paludis -q accountsdemo groupdemo
* user/accountsdemo
    accounts:                0* {:0}
    Username:                accountsdemo
    Description:             A demo account
    Default Group:           groupdemo
    Extra Groups:            alsogroupdemo thirdgroupdemo
    Shell:                   /sbin/nologin
    Home Directory:          /dev/null

* group/groupdemo
    accounts:                0* {:0}
    Groupname:               groupdemo
    Preferred GID:           123

Note the dependencies:

$ paludis -qDM accountsdemo
* user/accountsdemo
    accounts:                0* {:0}
    username:                accountsdemo
    gecos:                   A demo account
    default_group:           groupdemo
    extra_groups:            alsogroupdemo thirdgroupdemo
    shell:                   /sbin/nologin
    home:                    /dev/null
    dependencies:            group/alsogroupdemo, group/groupdemo, group/thirdgroupdemo
    location:                /var/db/paludis/repositories/ciaranm_exheres_test/metadata/accounts/users/accountsdemo.conf
    defined_by:              ciaranm_exheres_test

The install is fairly boring:

(4 of 5) Installing user/accountsdemo-0:0::accounts

* Executing phase 'merge' as instructed
>>> Installing user/accountsdemo-0:0::accounts using passwd handler
useradd -r accountsdemo -c 'A demo account' -G 'alsogroupdemo,thirdgroupdemo' -s '/sbin/nologin' -d '/dev/null'
>>> Finished installing user/accountsdemo-0:0::accounts

And once they’re installed:

$ paludis -q accountsdemo groupdemo
* user/accountsdemo
    installed-accounts:      0* {:0} 

* group/groupdemo
    installed-accounts:      0* {:0}

Exherbo will be migrating to this new mechanism shortly — package manager support is already there (it was only a few hours’ work), so it’s just a case of gradually hunting down and killing those enew* function calls.

Paludis 0.34.0 Released

Paludis 0.34.0 has been released:

• instruo is now parallelised
• For NoConfigEnvironment, profiles are only loaded when needed.

Paludis 0.34.0_alpha2 Released

Paludis 0.34.0_alpha2 has been released:

• Unpackaged repository now defines merge and strip phases, making phases useful for importare.
• Fixed build errors on certain systems.
• Stop pythons from eating all your rams when compiling.
• Updates to zsh completions.

Stopping Pythons from eating your Rams

As various people have found out the hard way, and much to my annoyance because my laptop is memory starved, building Paludis can sometimes take lots and lots of RAM.

Originally, we didn’t do anything about this. But unfortunately lots of users have silly things like MAKEOPTS="-j9", which can result in the build process wanting somewhere in the region of eight gigs of RAM, which in turn leads to users whining about gcc internal errors or random processes being OOMed. So we stuck a nasty hack in the ebuild that would reduce MAKEOPTS based upon how much free memory you had — all very well, but it screws over distcc users and isn’t even necessary for many combinations of USE flags and CXXFLAGS.

It’s worth working out exactly what makes the compiler memory usage so high. It’s fairly easy to figure out that it’s only an issue when building the Python bindings. We use Boost.Python for these, and unfortunately Boost will quite happily use horrible preprocessor hacks that result in huge generated source files and all sorts of nasty workarounds just to get code to work on ancient unsupported Microsoft compilers. It’s enough of a problem for enough people that there’s a tutorial section on reducing memory consumption for Boost.Python, but we already do those things.

There’s something else interesting, though. With debugging turned on (which autotools does by default), we need something like 800MBytes to compile one particular Python binding file. With it turned off, we only need 300MBytes, which is much less likely to be a problem (and more to the point it won’t make my laptop start swapping). It turns out that building the Python bindings with -g isn’t even useful — gdb doesn’t give particularly useful backtraces on the Python interpreter, and there are better ways of tracking down problems there.

So it looks like it makes sense to add -g0 (after checking that the compiler supports it) to CXXFLAGS for the Python bindings. Easy enough, right?

Wrong. As with everything else involving autotools, we have to jump through all sorts of convoluted hoops to do this. CXXFLAGS is a user variable (so we aren’t supposed to change it), and it takes precedence over AM_CXXFLAGS (which we can change). There’s no ‘more important than the user variable’ variable, and we can’t sensibly override CXXCOMPILE, so this gets messy. We have to abuse configure.ac to remove the debugging options from the user’s CXXFLAGS and move them into something that ends up in AM_CXXFLAGS, which we can then override. Horrid.

The next Paludis release will include this voodoo, which should improve things considerably and let us avoid the nasty MAKEOPTS mangling. But it’s still not ideal.

Most Gentoo users have USE="python" set, either from profiles or explicitly. Most of these users do not want to build the Python bindings. Some of these users don’t think to look at the dependencies before moaning that Paludis requires Boost, so they don’t even realise it’s only because they’re using a USE flag they probably don’t want enabled. So what can we do about this?

We can’t use IUSE defaults, since we don’t really want to use anything later than EAPI 0 for package manager ebuilds. We could turn off python selectively in package.use, but lots of users still have USE="python" set explicitly. So we use a different USE flag name. We’ve gone for python-bindings, along with a use description that makes it clear that thinking “well I have some things that use Python so I probably want this flag on” is wrong.

For consistency, we’ve also renamed ruby to ruby-bindings. These are a lot more useful than the Python bindings (playman is written in Ruby), and a lot faster to build thanks to Ruby having a reasonably sane API, so we might end up having to mess with profiles to turn them on by default.

I might end up reverting all of this if it turns out it does more harm than good. We’ll see.

Paludis 0.34.0_alpha1 Released

Paludis 0.34.0_alpha1 has been released:

• Masks that have been overridden by the user are now shown.
• Support phases.
• Paludis internals are now threaded using pthreads, and provide the SGI guarantee.

Three Shiny Things

• Flag variables in Bourne shell programs
• Open in Browser Firefox extension, for all those webservers that get MIME types wrong.
• noremap <Leader>v `[v`] to select the thing just pasted in Vim. Inspired by gv, which I always expect to reselect pastes for some reason.