Changes to the Paludis Git Syncer

I’ve just committed two changes to the Paludis Git syncer.

The first allows you to do sync_options = --branch=foo to specify a particular branch. I don’t expect this to be widely used, since branching for repositories usually means you’re not taking advantage of the better facilities available for that kind of thing. Still, it has its uses.

The second is sync_options = --reset. Unlike with, say, rsync, syncing via Git will merge any changes you’ve made to the repository with new upstream changes (it uses git pull). With --reset, it will instead discard any local changes and just become equal to whatever you’re syncing against (using instead git fetch and git reset --hard).

It’s a matter of considerable debate as to whether the reset behaviour is the right thing to do.

On the one hand, some people like working directly on a checkout to which Paludis is pointed, but still want cave sync to bring in updates.

On the other hand, doing that is horrible and evil, and a much better workflow is this:

• Have a local checkout for development work. Commit your changes to it.
• Have a separate checkout that is synced against your local checkout for Paludis use. After making changes, commit them to your local checkout and sync.
• When happy with your changes, rebase and squash them in your local checkout, and then push them upstream.

Doing that requires --reset, since otherwise the checkout Paludis sees will end up as some horrible auto-merged mess that includes changes you thought you’d discarded ages ago.

I’m strongly considering making --reset by default sometime in the future. However, this will make syncing a destructive operation for anyone who hates puppies enough to be doing work on a Paludis-synced checkout (in the same way that it’s already destructive for rsync).

Today In PMS FUD: PMS Shot JFK

Patrick Lauer is complaining that one of his patches to PMS was rejected. What he is not telling you is why it was rejected, so I shall explain here:

The version of bash to be required by ebuilds was decided by the Council, as was the specification defining it. The PMS project quite rightly doesn’t have the authority to override a Council decision. Patrick was told this, and was told that the process for getting EAPI definitions retroactively changed was to go to the gentoo-dev list and from there to the Council. Such changes have been made in the past, and they’re open to being made again in the future.

Unfortunately it looks like he’s not interested in getting his concern fixed, and is instead just trying to cause trouble.

Three Simple Tips for Contributing to Open Source Projects

Three simple tips for contributing to open source projects:

• If a program says “please provide this information when reporting a bug”, provide that information. If it says “please attach these files to your bug report”, attach those files — although you may want to check those files for incriminating data first…
• When reporting a build failure, make sure the output you provide includes the actual error, and remember that the first such message is the actual error, not the last one. Don’t start your paste at the line that says “make: *** [check] Error 2″ or “9 of 18 tests failed”, since the information needed to handle your bug is before that.
• When sending a patch, make sure that your patch applies against a recent master. Receiving patches that don’t apply is annoying, although it’s understandable if your patch used to apply recently but now doesn’t (and often a three way merge will fix that automatically). But if you’re basing your patch off unpublished changes and it doesn’t apply, fixing it has to be done by hand. Spend a little of your time making sure your changes work against upstream (‘git rebase’ and ‘git cherry-pick’ are immensely handy here), rather than wasting the time of everyone who reviews your patch.

Paludis 0.40.1 Released

Paludis 0.40.1 has been released:

• Bugfix: sometimes fetch failures would not register as errors.

Ten Ways PMS Raped your Baby

Since hating PMS seems to be back in fashion again this week, I thought I’d list ten of the stupidest claims that I’ve seen of late in the hope that some of the FUD might die down:

1. PMS slows down new features and prevents innovation

Actually, once new ebuild-usable features end up in Portage, they very quickly end up in a published EAPI. The reason you can’t use all the fancy new features in EAPI 3 that you’ve all been waiting for for so long is that Portage still hasn’t implemented them. In addition, we’ve gone from “EAPI 3 will be ready for Portage within a month” three months ago to “there’s an 80% chance EAPI 3 support will be ready in Portage by the end of the year“.

Time-wise, EAPI 3 has been waiting for Portage for six months and before that, for the Council to come to decisions for two months. The total overhead imposed by PMS was around four days, and those four days weren’t holding up anything else anyway.

Still, at least there’s a long way to go before EAPI 3 takes as long as it took Portage to get use dependencies…

Similarly, PMS isn’t to blame for profiles not being able to make use of new features. People who are telling you this are probably thinking about an undocumented Portage feature that isn’t in PMS and that isn’t supported by other package managers. This feature could very easily be in PMS, but there’s been no interest from the Council in retroactively adding it to EAPI 3 or in doing an EAPI 2.1 just to include that feature. The feature almost certainly will be in EAPI 4, but work on EAPI 4 isn’t going to start until Portage is done with EAPI 3. So again, PMS isn’t the reason you can’t use it.

2. PMS or EAPI is about ebuilds, not profiles

This one’s from people who haven’t bothered to read the opening of PMS, or who haven’t been paying attention to the Council.

PMS covers ebuilds and the tree format, including things like profiles. The aim is to cover everything necessary to produce a package manager that can use ebuilds (except possibly VDB, which probably shouldn’t be necessary for ebuild support but currently is…).

EAPI is used to indicate the rules used to handle ebuilds, and also profiles following the Council accepting Zac’s proposal last year.

3. PMS imposes an Exherbo agenda upon Gentoo

Exherbo doesn’t use PMS or Gentoo EAPIs.

4. PMS imposes a Paludis agenda upon Gentoo

Again, no. There’s no feature in any EAPI that’s there because of Paludis. Every feature in EAPIs 1, 2 and 3 was either requested by a Gentoo developer or included to make things easier for Portage. To get into an EAPI, features merely have to be vetted by the Portage team and the Council.

5. PMS is only used by Paludis anyway

Nope. PMS is used by both Portage and Paludis, as well as a number of third party libraries and utilities which don’t support full package management operations (things that need to compare two versions, for example, need to use PMS), and it was also used by Pkgcore.

Saying that PMS is only of use for third party package managers is like saying that the HTTP specification is irrelevant for Internet Explorer.

6. PMS stops other distributions from doing things

Again, no. Other distributions can ignore PMS entirely. Doing so would of course be a horrible idea, as all the people who wrote websites to work on Internet Explorer 5 found out, but that’s their decision to make. A much better option would be for those distributions to roll their own derived EAPIs, which, as happened for the Gentoo KDE project’s kdebuild-1, could be added to PMS. That way they are guaranteed that any undocumented features they rely upon won’t break with the next release, as well as avoiding complaints from users who want to use a different package manager, thus avoiding the problems people who wrote Internet Explorer 5 specific code rather than following the HTML specification encountered.

7. PMS stops Gentoo from deciding its own features

PMS is run by a Gentoo developer and approval of new EAPI features is handled by the Portage team and the Gentoo council. For that matter, the PMS team has never rejected a single feature for inclusion in a future EAPI.

8. PMS introduces red tape

No, the previous Council introduced red tape, primarily because a couple of Council members refused to read any submissions more than five minutes before a meeting. Had the Council used the two weeks between meetings to read over and state their opinions on the EAPI 3 feature list, EAPI 3 would have been approved within two meetings rather than dragging on for months.

Unfortunately the current Council doesn’t seem to have improved, with at least one member showing up to a meeting having not read the agenda beforehand.

9. PMS imposed stupid ordering on metadata files

There’s a tendency amongst certain people to blame PMS for stupid or arbitrary rules. A typical example is to moan about PMS because it says that EAPI has to be on line 15 of metadata cache files. Quite how PMS is to blame for a decision that was made before PMS existed, and that was made because line 15 was the first available cache line when EAPI was introduced as a metadata key, is completely beyond me. Similarly, the rules for handling leading 0s in version numbers is Portage’s fault (although ultimately it’s Perl’s fault), as is any other format gripe you care to name.

10. PMS will stop me from using my favourite feature in configuration files

PMS doesn’t discuss user configuration at all. Handling of user configuration is left entirely to the package manager.

New Resolver Data Structure Pictures, or, Why I Need Lots of Pens

As some people may or may not have heard, one of the big Paludis projects we’ve been discussing for the past couple of years has been to come up with a super-amazing dependency resolver that can handle ABIs, binaries and chroots perfectly, provide complete customisation so people can do stupid things like “update everything except glibc”, cure cancer, be adapted to support arbitrary new features with no difficulty and explain all of its decisions in an easy to understand manner. Obviously, doing all of that at once is rather ambitious, so in the interests of it ever being finished, I’ve instead been working on a stupid but incrementally expandable resolver designed around:

• Doing only the basics initially, but having a simple design that cleanly splits apart things like ID selection, dependency selection and ordering, even if doing so prevents certain short cuts from being taken. That way, when we add things in later, we don’t have to rely upon lots of subtle interactions between all the different components.
• Making sure that we can explain exactly why we’ve done a particular thing, even if this means not including clever trickery.
• Having easily accessible innards, meaning if people still insist upon having an “upgrade everything except glibc” option, we can easily move a very small amount of code out into a std::tr1::function and let clients handle it that way without having to pollute the resolver.

The basic features all now pretty much work, and cave resolve is usable on Exherbo (although not Gentoo at present, since I haven’t implemented virtuals handling), although there’s no sensible error handling, several obvious optimisations haven’t been made, the UI is highly crude and there are no bells, whistles or cookies. Still. being able to do this is rather fun:

$ cave resolve gnome --explain libbonoboui:2
[snip]

Explaining requested decisions:

For gnome-platform/libbonoboui:2:
    The following constraints were in action:
      * >=gnome-platform/libbonoboui-2.1.1, use installed if possible, installing to /
        because of dependency >=gnome-platform/libbonoboui-2.1.1 from gnome-desktop/gnome-panel-2.26.3:0::gnome
      * >=gnome-platform/libbonoboui-2.1.1, use installed if possible, installing to /
        because of dependency >=gnome-platform/libbonoboui-2.1.1 from gnome-desktop/gnome-panel-2.26.3:0::gnome
      * >=gnome-platform/libbonoboui-2.13.1:2, use installed if possible, installing to /
        because of dependency >=gnome-platform/libbonoboui-2.13.1:2 from gnome-platform/libgnomeui-2.24.0:2::gnome
      * >=gnome-platform/libbonoboui-2.13.1:2, use installed if possible, installing to /
        because of dependency >=gnome-platform/libbonoboui-2.13.1:2 from gnome-platform/libgnomeui-2.24.0:2::gnome
    The decision made was:
        Use gnome-platform/libbonoboui-2.24.0:2::gnome
        Install to / using repository installed

Now to the important part: the pretty pictures!

Regular visitors to #exherbo may have noticed me moaning that I don’t have enough pens to implement their feature of choice. Here’s why:

Resolver Design 7

Since I can’t keep track of more than around five classes at once in my head, I have to have summaries written out on paper. Furthermore, each class summary has to be in a different colour (although my scanner’s done a fairly good job of hiding that in the picture above…), which means I need a pen (a proper fountain pen, or I can’t write with it) for each class. This in turn means that any new feature will likely require one or more additional pens, and I am more or less at my limit.

I also need a couple of colours spare to be able to scribble all over the diagrams, draw lines, change things and generally make a huge mess of things. An earlier design page now looks like this (and note that this is the most readable of the earlier design pages):

Resolver Design 5

On top of that, any problem too complicated to be solved in my head gets its own highly weird picture drawn out. Unfortunately the only example of this that I have handy (working out a circular dependency breaking algorithm) is on A3 paper, which I can’t easily scan…

I’ve found that working on paper for this kind of thing is much faster than working on a computer (writing’s as fast as typing, but the layout’s much quicker on paper, and scribbling over computerised designs doesn’t work). I don’t use a formal design system at this stage because it’s more pain than it’s worth, especially when there’s no need for other people to be able to read the design without being able to ask questions, although in some ways what I do is close to CRC cards with all the bits I don’t need ripped out.

I do not claim that my system is sane; merely that it works.

Assorted C++ Linkage

• A talk by Bjarne Stroustrup on the design of C++0x.
• Range based for loops in C++0x without concepts, good to know this hasn’t been dropped.
• Making std::list::size() O(1) [PDF], a shame to see this hasn’t been dropped.
• And on a less technical note, filtering compiler optimisation flags is not a solution from Diego. Unfortunately, he leaves out some of the worst offenders for flags that users think they can use that do change the meaning of programs: -fvisibility-inlines-hidden and -Wl,--as-needed.

Paludis 0.40.0 Released

Paludis 0.40.0 has been released:

• Notifier callbacks allow clients to tell the user what’s going on when generating metadata, performing resolutions etc.
• Sets now work slightly differently. For sets defined by multiple repositories (e.g. ’system’), ’setname::repo’ can be used to access the part of the set defined by a given repository.
• Bugfix: Upgrading an unpackaged package no longer errors out.
• Bugfix: Combining :slot and ::/path restrictions now works correctly.