He's not dead, he's resting

Distributed Distribution Development, and Why Git and / or Funtoo is Not It

Gentoo is slowly shuffling towards switching from CVS to Git. This is a good thing, because CVS stinks. Using Git will reduce the amount of time developers need to waste to get something committed, make it easier to apply patches from third parties and make tree-wide changes merely a lot of work rather than practically impossible. What it will not do is make Gentoo in any way more ‘distributed’, ‘decentralised’ or ‘democratic’.

Some of the Git work has already been done, in a reduced manner (no history and no mirroring), by Daniel Robbins’ Funtoo, which is purported to be more distributed than Gentoo. The problem is, there’s nothing there to back up the distributed claim.

Distributed development, in the sense for which Git was designed (and ignoring the intervening BitKeeper stage), meant moving away from having a single central repository off of which everyone worked to having everyone work off their own, publishable repositories and providing easy ways of merging changes from one to another. ‘Good’ changes would tend to find their way from the authors up the food chain to the main repository whence official releases are made. Users requiring things that hadn’t made their way to the top would maintain their own repository, and merge in changes from elsewhere that they needed.

Typical Git Workflow Model

Typical Git Workflow Model

For a conventional codebase, this model works. But it’s not particularly nice, and it’s driven by necessity. You’ll note the big red dots in the diagrams. These represent places where people (assisted to some highly variable degree by Git) have to do merges. I chose big red dots rather than soft fluffy clouds because merges can be a lot of work (and because drawing clouds takes effort).

If you’ve got a conventional codebase, you have to do merges to make use of things from multiple sources — the compiler takes a single codebase and produces a program from it. You can do the same thing with a distribution. Funtoo, for example has had the Sunrise repository merged in to the main repository. Such a change would likely not be possible with Gentoo’s current CVS architecture.

It’s not entirely clear whether Funtoo intends to have users who want to use other overlays merge those overlays into their own tree. Doing so would be more Gitish.

Apparent Funtoo Workflow Model

Apparent Funtoo Workflow Model

But why bother? There’s no need to have a single codebase — there’s no compiler that has to take every input at once and turn it into a single monolithic product. Those big red dots are unnecessary.

A lot of fashionable programs are moving away from the big monolithic binary model and towards a plugin-assisted architecture. If you want Firefox to do a few things it doesn’t, you don’t hunt around for people who have already written them and then try to merge their source trees together. You install plugins. Only for more severe changes do you have to dive into the source, and the severity of change requiring a trip to the source is gradually increasing.

There’s a reason for this — whilst the merge model is a lot better than a single authoritative codebase and a bunch of patches, it’s a lot more work than providing limited composable extensibility at a higher level.

What, then, would a plugin-based model look like for a Gentoo-like distribution?

Presumably, one would have a centralised ‘main’ codebase. One could then add additional small extras to that main codebase to obtain new functionality (packages, in this case); these extras would rely upon parts of the main codebase and wouldn’t be able to operate on their own. Sound familiar? Yup, overlays are plugins.

This whole “merging overlays into the main tree” thing is starting to look like a step in the wrong direction. What would be some steps in a better direction?

One thing that comes instantly to mind is improving overlay handling. Portage’s overlay handling currently (at least in stable) looks like this:

Portage Overlay Model

Portage Overlay Model

Portage takes the main Gentoo repository, and then merges it with each overlay in turn, creating one ‘final’ overlay that ends up being used. I’ve used an orange dot here rather than a red one because it’s a different kind of merge. Rather than doing a source-level merge, the orange dot merge more or less (sort of) works like this:

  • If there’s a package with the same name and version in the origin and the overlay we’re merging in, take the overlay version.
  • If there’s an eclass with the same name and version in the origin and the overlay we’re merging in, sometimes take the overlay version.
  • Do some horrid hackery to merge together any colliding profile things in an uncontrolled manner that doesn’t work for more than one merge.
  • Pass everything else through.

Now, to be fair, the orange dot merge usually works. Most overlays don’t try to override eclasses, don’t have eclasses that conflict with each other and don’t mess with profiles. For colliding versions, you end up being stuck with a single selected version, which isn’t always so good.

Unfortunately, some overlays do try to override eclasses and profiles, and the result isn’t pretty. You’re ok so long as you only use a single overlay that does this, and so long as any eclass changes aren’t incompatible, but anything beyond that and weird stuff happens.

A less dangerous model would be to make the package manager support multiple repositories. Presumably most overlays wouldn’t want to have to reimplement all the profile and eclass things in the Gentoo repository, so the model would look like this:

Safer Overlay Model

Safer Overlay Model

Here, repositories, rather than the user, have control over which implementation of eclasses and so on gets used. Paludis uses this model for Gentoo overlays unless told not to.

Sidebar: one might want to go a step further, and allow repositories to use multiple masters. Some Exherbo supplemental repositories do this — the gnome supplemental repository, for example, makes use of both arbor (the ‘main’ repository) and x11:

Exherbo Repository Model

Exherbo Repository Model

Note that we chose not to make a repository use its master’s masters. We could’ve gone either way on this one — it’s slightly easier if masters are inherited, but it can lead to unnecessary inter-repository dependencies.

Unstable Portage, meanwhile, is starting to support controlled masters for eclass merging, but not version handling, which will eventually give:

New Portage Overlay Model

New Portage Overlay Model

A multiple repository model is clearly safer than the Portage model, and does away with the manual merges required by the Funtoo model. This gives us:

Model Multiple Repositories? Manual Merges? Unsafe Automatic Merges?
Portage (Stable) No No Yes
Portage (Unstable) No No Sometimes
Funtoo No Yes No
Safe Yes No No

I consider the multiple repository model to be better for users even ignoring the merge or conflict issues. Here’s why:

  • Users can make selective, rather than all or nothing, use of a repository. It becomes possible to mask the foo-1.2 provided by the dodgy overlay, and use the one in the main tree or a different overlay.
  • Similarly, users can choose not to use anything from a particular overlay except things they explicitly request.
  • It paves the way for handling repositories of different formats.

There aren’t any downsides, either — so long as repositories have user-orderable importance, there’s no loss of functionality.

Finally, I’d like to debunk the myth that the Git model is somehow ‘democratic’. There’s nothing in the least bit democratic about everyone having their own repository. At best, it could be said to be a way of allowing everyone to have their own dictatorship that anyone else can be free to visit — all very well, but when tin pot dictators fall back on old habits it does little to encourage collaboration. A democratic distribution would more likely make use of a special repository which lets people vote on unwritten packages and version bumps — clearly a recipe for disaster, since most people think “I haven’t noticed any bugs” means “stable it instantly”…

The only thing switching Gentoo to Git will solve is the pain of having to use CVS. This alone is enough to make the move worthwhile, but it will do little to nothing to fix Gentoo’s monolithic design and inherently centralised model. Nor does Funtoo’s merge approach solve the problem — on the contrary, it replaces a model where the package manager automatically does unnecessary merging (and sometimes gets things wrong) with a model where people do unnecessary merging (which is a lot of work, and they will still sometimes get things wrong). The future is (or at least should be) in a multi-repository model with good support from the package manager that removes the costs of decentralisation.


19 responses to “Distributed Distribution Development, and Why Git and / or Funtoo is Not It

  1. thewtex April 17, 2009 at 3:04 pm

    It is much easier and there are much more powerful tools to make changes with git than an overlay. If there is a bug in an ebuild and you want an overlay, you have to copy the ebuild, all the patches, maintain an repository … Overlay commit messages + diffs are not as informative if they are again the main tree or another overlay. Incompatible changes in disparate repositories go unnoticed, while they are a MERGE CONFLICT in git. Very important for eclasses, … otherwise breakage…

    “# Users can make selective, rather than all or nothing, use of a repository. It becomes possible to mask the foo-1.2 provided by the dodgy overlay, and use the one in the main tree or a different overlay.
    # Similarly, users can choose not to use anything from a particular overlay except things they explicitly request.”

    Git is a better way of allowing users to make selective use of a repository. You can grab what is explicitly requested ( cherrypicking, other methods… ).

    It is more work to add another repository than to add a git branch. But more importantly, finding a repository that has what you want is very hard versus ‘github network fetch’.

    In sum, git does it all easier and better.

    There may be ways to make funtoo actually distributed in the future, especially if it uses paludis…

    • Ciaran McCreesh April 17, 2009 at 5:02 pm

      It is extremely difficult to make changes with git. You have to explicitly sort out every single change and work out every collision, rather than only fixing things when you have to. People only put up with this for conventional code bases because there’s no other option; where there is (e.g. plugins, modular code or multiple repositories), it’s always a better option than source level merges.

  2. Pingback: Gentoo & Distributed Development « The Weblog of a Gentoo addict

  3. gentoofan23 April 17, 2009 at 6:59 pm

    I disagree that it’s difficult to find a repository that has what you want. For me, All I have to do to search for libfoo is ‘paludis -q libfoo’. If it’s been packaged up and is in a repo, paludis nicely tells me what repo it is in. I then just run playman(layman for paludis) -a repofoo and sync. It’s not that difficult to figure out what repo a package is in when you have package manager support for it.

  4. Stelian Ionescu April 17, 2009 at 7:49 pm

    How does multiple repository inheritance work wrt. eclasses ?
    If an eclass is present in both arbor and x11 and a package in gnome requires it, which one is chosen ?

    • Ciaran McCreesh April 17, 2009 at 8:05 pm

      There are various ways we could do this. The way we’ve gone for is to order masters, so you’d control this via whether you specified masters = arbor x11 or masters = x11 arbor.

      Although, with masters it’s probably unlikely that there will be duplicate eclasses.

      • Stelian Ionescu April 17, 2009 at 8:09 pm

        How about this: select one of the parent repositories as «master» and inherit eclasses only from there, while the rest of parents supply only packages to the final repo ?
        Sort of like single inheritance + interfaces, I suppose.

        • Ciaran McCreesh April 17, 2009 at 8:26 pm

          That defeats the whole point of having multiple masters. Being able to use eclasses from multiple places is extremely useful.

          Also, parents don’t supply packages to the repo. A repo’s packages are just what’s in that repo. There’s no need to merge versions since the package manager is fine with having foo-1.2 in two or more different repositories.

  5. robbat2 April 18, 2009 at 3:23 am

    I’d like to note that I’m mostly in favour of the multiple repo model, but I’m not sure it’s going to be an easy to reach goal, nor that all the issues are worked out.

    I do think you’re overblowing the Git conflict merge issue, noting your complaint of collisions in a comment above. The only place I’ve run into them is concurrent differing changes to ebuilds (which happens in CVS already), plus merge hell in Manifests (which was discussed on the gentoo-scm list).

    For the existing monolithic Gentoo repo, aside from the size issue, will probably continue to exist, even in a port of the Exhberbo repo model, as splitting it still wouldn’t fly (see the overhead discussions on the list).

    P.S. I feel it was misleading in the table to use the label ‘Safe’ for the Exherbo model, rather than just saying Exherbo. The fact it’s safe is secondary, all of the other labels are for their source, not their properties.

    I’d love to discuss more on this, but I’m in the midst of travelling (I just started my trip), and won’t be back for 2-3 weeks.

    • Ciaran McCreesh April 18, 2009 at 1:55 pm

      Sure, it’s not an easy goal. My point was just that it’s still a necessary eventual goal even with the migration to Git. Getting there without doing a single big overhaul is not going to be easy.

      The conflict issue becomes a lot more of a problem if you start regularly merging overlays into a single repository. If it’s just developers simultaneously working on something that’s always been in the same place the merge is usually straight forward (and certainly a lot easier than it is with CVS), but as the Funtoo people found out, anything beyond that is decidedly non-trivial.

      If splitting’s to be done for Gentoo, it has to be done in a much more sophisticated way than has been discussed on the gentoo-scm list. It would first need Portage support for multiple repositories and something along the lines of the ‘unavailable’ format Paludis uses. Then obviously groupable things like KDE and Gnome could be moved purely into their own repositories, and slowly go from there — gradually reducing the size of gentoo-x86 by splitting off parts into repositories with a hundred to a thousand packages over time, rather than a single huge split.

      The Safe model didn’t exactly originate with Exherbo, although various Exherbo people contributed to the design. It was something that was conceived at least in part as an abstract “how would we fix Gentoo if we had lots of manpower and some magic way of getting things done”, rather than specifically “what do we need for Exherbo?”. Were we designing purely for Exherbo, for example, we’d probably have just banned conflicting exlib (eclass) names.

  6. Stuart Longland April 18, 2009 at 12:47 pm

    I suppose the point to be made here is that git is simply a tool that permits distributed development, it does not make it distributed by default.

    The manner in which it’d be used in Gentoo (and from what I understand, is being used in Funtoo) is still very much a centralised development model, and as rightly pointed out, not necessarily democratic.

    Not sure how the separate repository system would go… it’d require a lot of careful thought to plan it out… ultimately there’d still need to be significant parts kept in some kind of central place, even if it were just a list of other repositories. Without planning, you’d wind up with the Microsoft world where users have to go to 1001 different places to obtain the software and drivers they need.

    • Ciaran McCreesh April 18, 2009 at 1:58 pm

      Going to lots of places isn’t really a problem if the tool’s designed to handle it. Having a few lists of repositories (no reason to only have one — Exherbo has a list of official repositories and a list of unofficial repositories that have had a degree of review, but also lets people publish their own lists if they want) that are understood by the package manager makes the decentralisation painless.

  7. Caleb Cushing ( xenoterracide ) April 18, 2009 at 8:48 pm

    I don’t have so many problems… in fact the repo model creates more, case in point. perl-experimental, and inkscape. inkscape needed to be patched for perl-5.10.0 it’s the only bug that was found after funtoo merged 5.10.0 that I’m aware of. The maintainer won’t merge the patch into gentoo, so anyone using inkscape and that overlay has to patch inkscape themselves. This is bad for users. This is what overlays do. Funtoo, or more to the point, Regen2, doesn’t want to merge overlays (funtoo doesn’t do it anymore (that I’ve really noticed)). We have to because gentoo is largely unusable and out of date if we don’t. We have to stabilize newer packages because the only way to run a stable gentoo anymore is by running ~arch.

    kde-testing is one of the only overlays I approve of. because they merge kde stable into gentoo within 0-2 weeks of kde releasing. They are appropriately using it for development and testing, without letting it rot in there tree, like the perl herd does, or haskell, or java, or I’m sure any other number of overlays. In other distributions ‘repos’ cause problems, because a package for one won’t be made compatible with a package in another. Gentoo never had this problem until overlays started being abused. Overlay interdependence doesn’t actually solve this problem either. It would have it’s uses, but in reality, it just endorses existing bad behavior.

    overlays result in low user testing, and should be used for staging and development only.

    when I merged the overlays my number one issue wasn’t/isn’t source code conflicts. git creates markers that I can read. Manifests became a non-issue when I started running repoman manifest on the whole tree. in fact, merging them is easy, and I have very few problems, I’ve heard very few complaints. most of the complaints come from poorly maintained packages in gentoo, not the merged overlays, or any conflicts to do with merged overlays. The biggest problem is global profiles stuff. package.mask is broken in funtoo. I had to split it to make it work in regen2. I imagine that there are a few other files with similar issues like use.local.desc. but it seems to be a mostly non-issue.

    moving from cvs to git will help distributed if devs want it to. I am willing to attach git patches to bugzilla, or mail them to a mailing list. Devs must be willing to accept outside help, or provide good reason for not, and I mean damn good reason, I don’t have time is not a reason not to accept someones patches. make ~arch testing again, be willing to break it. we have something other distro’s don’t. Masks, use them. you should ask a contributer to test his patches, you should take a few minutes to review them before moving them to ~arch. but all that shouldn’t require days if someone else has done it for you. Trust outsiders who may not have time to be full developers, help them learn if they make mistakes.

    between the perl overlay, the java-overlay, and sunrise, there are 3000 packages not in the tree. 3000! and for what? is this really necessary? are all of these really /that/ broken? I doubt it, many may not be stable, but they aren’t /that/ broken. Some of the tree packages are more broken. that’s why overlays were merged, not because it makes it more distributed. you have no clue what you are talking about when trolling funtoo. Yes it creates some issues, but none that couldn’t be resolved by devs discontinuing their poor behavior, and using overlays as ‘fully supported’ (resolved in overlay is not a valid solution).

    • Ciaran McCreesh April 18, 2009 at 9:45 pm

      You appear to think that piss-poor Gentoo QA shows that a multiple repository model is broken. That’s not the case — what it shows is that scaling development to handle hundreds of developers all working in one place isn’t viable. By attempting to do away with overlays and do even more in one place, you’re making the problem far worse.

      Gentoo had all the problems you describe long before layman encouraged people to make heavier use of overlays; all overlays did was make it possible for third parties to fix some of them.

      Your ‘case in point’ shows just the opposite of what you think it does. When you’re at the mercy of a single big central repository, you’re stuffed if the people maintaining that repository decide not to do what you need. With a well defined multiple repository model, it’s up to a combination of repository maintainers and users to decide how to handle things like packages needing patches. It’s entirely possible for a repository to depend upon a package that’s been ‘fixed’.

      You miss the point with conflicts, too. Conflicts occur when you do a merge with Git. There’s absolutely no reason for this to be necessary. The package manager can handle it all in a much safer way, and it can do so entirely automatically and without loss of power to the user.

      You’re also wrong about ~arch. The point of ~arch is that it’s for things that are a candidate for going stable after further testing. It’s not for things that sort of work. Although you can get away with using a few things that only maybe work, when everything only probably works for most people, there comes a point where this adds up to “nothing works for anyone” — something you’ll find out the hard way if you start getting non-trivial numbers of users.

      And you’re wrong about “3000 packages not in the tree” being a bad thing. A better way of looking at it is that there are 10000 packages rotting in gentoo-x86 that needn’t be there. The huge, monolithic nature of the main repository is one of the root causes of Gentoo’s problems, and what you’re doing is the opposite of fixing it.

      • Caleb Cushing ( xenoterracide ) April 19, 2009 at 9:12 am

        I disagree. I think what you are saying is the opposite of what is needed, all it has done is make my life unnecessarily complex, and made gentoo more undesirable to me as a user. If you’d like to use the repo model for exherbo, go ahead. and a package manager has no clue about merging, imho. it’s job is to merge packages into my system, not make sure that the developer did his job right.

        • Ciaran McCreesh April 19, 2009 at 9:26 am

          No, what’s made Gentoo more undesirable to you is Portage’s lousy handling of overlays, combined with Gentoo’s shoddy QA practices. You’re blaming entirely the wrong thing.

          And of course the package manager does merging. That’s what Portage’s view overlay is — it’s something that’s merged into a main repository. It’s just not a source-level merge because it doesn’t need to be.

      • Michael Schmid (Radtoo) April 22, 2009 at 10:34 am

        Short version:
        About your comment on ~arch ‘s purpose and QA inadequacies:

        In terms of QA and keywording and masking, Funtoo has the advantage that they can be usefully consistent for a purpose they proclaim they intend this repository to be used with. I see this as a large advantage over the distributed model where no such purpose can be realistically claimed by the entirety of all repositories.

        As they will never succeed to have such a shared purpose between themselves, a standardized model for keywording and masking will be impossible across distributed repositories. Yes, I can’t see everyone agreeing on a policy even for a few repositories, as long as the real life purpose of the supported software setups is not nearly specified. Even much less will it be possible for each repository to keyword / mask everything properly with regards to every repository. The problem of this is, the resulting inconsistencies in keywording / masking add up to being more work to deal with than their existence prevents.

        This seems true to me as long as keywording / masking is not specified entirely outside of repositories. However, if instead of “repository” maintainers, it were done mainly by user groups with similar ideas pertaining to QA standards, it would be actually solving this problem. Even if this “profile” type of information now were monolithic, it wouldn’t be nearly bad, as long as its contract were clear and would encompassed multiple repositories. That way, it would be possible to get a much more clear-cut contract with regards to keywording / masking and QA, and this would make both the “profile” maintainer’s lifes easier and the users’ adjustments to the contract much more predictable…

        Long version:
        I agree with the article you posted and its conclusions, but this comment about ~arch is interesting. I gave it a bit more thought and think this will be more complicated, regardless what decentralized structure you choose.

        Yes, keywording / masking does NOT work well on existing overlays, and here’s why:
        Maintainers have, and will still only have a small subset of all repositories that are available of them all in their development environment, with only some of all of the available packages installed – simply because they are not really interested in all the packages. And quite a few overlays will not feel too responsible for either asking for too old or patched or too new packages that break other setups in other repositories – its probably the other party’s fault after all and they might have done shoddy work, besides there’s no time to take care of this and yadda yadda… even though they would care if you somehow arranged the fence differently and made it all a single repository (this is another actual benefit I see in Funtoo’s way of doing things, I guess :)). It’s silly, but I think you’re not going to be able to change that.

        Much for the same reason that you’ll not only not get coherent and meaningful keywording / masking. You’ll not even get proper mask lists in repositories against what is objectively broken (or rapid removal of such flags when it’s not broken anymore).

        I’ll assert that the existing keywording / masking will only be in the way with such different standards in QA policies and purposes they aim at. The more repositories you have, the worse the problem will become. You can call it “shoddy QA” or such, but what QA staff is truly interested in making everything working against all “other” combinations of things as well, when “all” they want is fixed in the setup “their” repository envisions?

        My point is, won’t you -at least as much as a more sane merging of repositories- require ways for user-groups to aggregate information and decide on what works and what does not work for them, rather than the individual repository maintainers? And they need to do so separately from the individual repositories entirely, as what packages are “stable” and “unstable” or “broken against important packages” depends a lot on the purpose you’re using them for, and what combination of repositories you’re using.
        Right now we have keywords inside ebuilds and masks / profiles contained in repositories. Whereas the only thing that IMO can work well in a distributed model is no more keywording / masking in ebuilds and repositories but instead monolithic profiles maintained by user groups that can agree with their own policies.

        As a side-note, I do believe that tests might actually set a more tangible standard for keywording/masking that could even be used to feasibly do real verification against _all_ existing repositories, but the sort of clear contracts and high coverage required won’t happen any time soon. I’d not rely on this for now…

        • Ciaran McCreesh April 22, 2009 at 1:34 pm

          I’m not entirely convinced that the problem will necessarily be as large as you think. Keywording, by and large, is (or at least should be — on Gentoo it’s not that way because maintainers throw a hissy fit every time arch teams tell them they disagree on something) the responsibility of arch teams, not of package maintainers. What should happen is that:

          Maintainers commit their new packages keywordless, or as ~arch on archs on which they do proper testing if the arch teams in question agree.
          Arch teams add in ~arch keywords for other archs as they see appropriate.
          Maintainers drop any stable keywords to ~arch when bumping, and preserve any ~arch keywords. On the rare occasions when this ends up being wrong, arch teams jump in to fix it.
          Arch teams move things from ~arch to arch as they see fit.
          Maintainers use a repository mask for anything they don’t consider to be at least “appropriate to go stable after sufficient testing”.
          Maintainers who are also arch people avoid doing both roles for any package, to make sure there’s an extra level of QA in there.

          This really relies upon arch teams being able to get things keyworded across the board. This isn’t necessarily a problem; distributed does not mean “lots of groups who don’t talk to each other”. A degree of coordination is still useful and possible — and ultimately, if it fails, there are ways around that.

          Regarding keywords in ebuilds… It’s something both Gentoo and Exherbo have discussed now and again. For Gentoo, it’s not feasible so long as they carry on using a huge CVS repository. For Exherbo, it’s something that we’ll probably end up doing when we do the profiles overhaul, although even then we’ll likely keep keywording associated with its owning repository. Whilst a greater degree of decoupling might sound nice, it’s also a lot more work, and we’ve yet to establish that it’s necessary.

  8. Pingback: Changes to the Paludis Git Syncer « Ciaran McCreesh’s Blag

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s