Blag

He's not dead, he's resting

Tag Archives: unavailable

Dealing with Unwritten Packages

A while ago Exherbo came up with a solution for the “large number of repositories” problem. It turns out this solution works rather well, and I’m now convinced that anyone rushing off and assuming that Git on its own will magically solve anything other than CVS sucking is going in the wrong direction.

Paludis also has ways of tracking packages installed by hand. Whilst useful on Gentoo, this really comes into play on Exherbo, where we don’t want to end up with a massive unmaintainable tree full of packages used by only two people.

The logical next step is tracking packages that don’t exist yet.

An unimaginative person might think that the way to solve this would be to have a bug for each requested package. But that gets messy — it’s not integrated with the package manager, and requires considerable extra effort from the user. A while ago Bernd suggested something more interesting: tracking unwritten packages in a repository in a similar way to how we do unavailable packages.

The implications:

  • Most obviously, we can keep track of things we want that haven’t been written yet in a way that doesn’t involve leaving the package manager to look things up.
  • We can also use it to track version bumps that we know will take a while. Doing paludis --query '>=hugescarypackage-4', say, would show that it’s being worked upon.
  • We can also depend upon things that don’t exist, rather than leaving incomplete dependency strings around. This is fine in at least two cases — if a dependency is conditional upon an option that should probably be implemented but isn’t yet, we can add the option and make it unusable. And we can handle obscure suggested dependencies (e.g. git has lots of optional dependencies upon weird perl modules, so we can say “if you want support for git-send-email with SSL, you need to write such and such a package”).
  • Bored developers could simply paludis --query '*/*::unwritten' and get ideas for what to do.

Adding support to Paludis for this only took a couple of hours. So now Exherbo developers can use this:

format = unwritten
location = /var/db/paludis/repositories/unwritten
sync = git://git.exherbo.org/unwritten.git
importance = -100

And see this:

$ paludis -q genesis
* sys-apps/genesis
    unwritten:               (1.0)X* {:0} 
    Description:             The daddy of all init systems
    Homepage:                http://www.exherbo.org/
    Comment:                 We need an init system.
    Masked by unwritten:     Package has not been written yet

As with unavailable, installing behaves sensibly:

$ paludis -pi genesis
Building target list... 
Building dependency list...
Query error:
  * In program paludis -pi genesis:
  * When performing install action from command line:
  * When executing install task:
  * When building dependency list:
  * When adding PackageDepSpec 'sys-apps/genesis':
  * All versions of 'sys-apps/genesis' are masked. Candidates are:
    * sys-apps/genesis-1.0:0::unwritten: Masked by unwritten (Package has not been written yet)

Those interested in the repository format can browse the tree. The observant might notice that the file format is quite similar to the one used by unavailable repositories, and wonder whether I was feeling lazy and swiped a load of code rather than implementing a new repository from scratch.

In the spirit of silly buzzwords, one could argue that this increases distributivity and democratisation of the repository, since there’s nothing to stop motivated users from creating their own wishlist repositories. It wouldn’t be hard to make a simple web interface that lets users request and vote on packages and version bumps, automatically generating a repository that anyone can use. But fortunately Exherbo is a dictatorship and has no users, so we don’t have to put up with that kind of nonsense.

Advertisements

Dealing with Lots of Repositories

With the explosion of overlays used by Gentoo, finding a package can get quite messy. Most users won’t want to set up lots and lots of repositories, so they won’t necessarily know when a package (or an ebuild for an scm or beta version of a package whose stable versions are in the main tree) is available.

Exherbo has similar issues. It’s likely that not-widely-used applications will remain permanently in individual developers’ personal repositories, with only reasonably important packages making it into Arbor. This has further implications for dependencies — for example, if X11 remains in its own repository, but core Arbor packages have optional dependencies upon X, how will that work?

Enter yesterday’s Paludis project: UnavailableRepository.

The idea is simple:

  • Have a repository that contains all the packages in all the repositories you don’t have.
  • Make it know enough about those packages to show you that they’re available, but not enough to let you do the install. In Paludis terms, this means making the packages be masked with a non-overridable mask, but still support InstallAction.
  • Make it less important than any ‘proper’ repository.
  • Do something clever to skip doing unavailable IDs for any repository you have configured. (Not strictly speaking necessary, but a lot nicer.)

Configuration is simple. For Exherbo:

format = unavailable
location = /var/db/paludis/repositories/unavailable
sync = tar+http://git.exherbo.org/exherbo_repositories.tar.bz2
importance = -100

And for Gentoo:

format = unavailable
name = layman
location = /var/paludis/repositories/layman
sync = tar+http://git.exherbo.org/layman_repositories.tar.bz2
importance = -100

(Both sync URLs will probably change soon.)

Then you can do this:

$ paludis -q firefox
* net-www/firefox
    unavailable:             (3.0_rc1 (in ::mozilla))X* {:0} 
    Description:             The firefox web browser
    Owning repository:       mozilla
    Repository homepage:     http://git.exherbo.org/?p=mozilla.git
    Masked by unavailable:   In a repository which is unavailable

And this:

$ paludis -pi firefox
Building target list... 
Building dependency list...
Query error:
  * In program paludis -pi firefox:
  * When performing install action from command line:
  * When executing install task:
  * When building dependency list:
  * When adding PackageDepSpec 'net-www/firefox':
  * All versions of 'net-www/firefox' are masked. Candidates are:
    * net-www/firefox-3.0_rc1:0::unavailable (in ::mozilla): Masked by unavailable (In a repository which is unavailable)

You can even search (on description, at least — searching on other metadata keys won’t find anything):

$ sudo inquisitio browser
* net-www/firefox
    unavailable:             (3.0_rc1 (in ::mozilla))X* {:0} 
    Description:             The firefox web browser
    Owning repository:       mozilla
    Repository homepage:     http://git.exherbo.org/?p=mozilla.git
    Masked by unavailable:   In a repository which is unavailable

* net-www/w3m
    unavailable:             (0.5.2 (in ::haskell))X* {:0} 
    Description:             Text based WWW browser, supports tables and frames
    Owning repository:       haskell
    Repository homepage:     http://git.exherbo.org/?p=haskell.git
    Masked by unavailable:   In a repository which is unavailable

So what’s in those magic sync tarballs?

$ ll /var/db/paludis/repositories/unavailable/
total 76K
-rw-r--r-- 1 root root  369 2008-06-12 04:47 alsa.repository
-rw-r--r-- 1 root root 3.2K 2008-06-12 04:47 haskell.repository
-rw-r--r-- 1 root root  195 2008-06-12 04:47 kde.repository
-rw-r--r-- 1 root root  666 2008-06-12 04:47 media.repository
-rw-r--r-- 1 root root  238 2008-06-12 04:47 mozilla.repository
-rw-r--r-- 1 root root  334 2008-06-12 04:47 perl.repository
-rw-r--r-- 1 root root 1.2K 2008-06-12 04:47 rbrown.repository
-rw-r--r-- 1 root root  338 2008-06-12 04:47 scientific.repository
-rw-r--r-- 1 root root  20K 2008-06-12 04:47 x11.repository
-rw-r--r-- 1 root root 1.5K 2008-06-12 04:47 xfce.repository

One file per repository, fairly simple. And the files themselves are nice and clean too:

$ cat /var/db/paludis/repositories/unavailable/mozilla.repository 
format = unavailable-1
repo_name = mozilla
homepage = http://git.exherbo.org/?p=mozilla.git

dev-libs/
    libIDL/
        :0 0.8.10 ; IDL parsing and compilation library
net-www/
    firefox/
        :0 3.0_rc1 ; The firefox web browser

There’s a bit of metadata about the repository in question (not very much — repositories don’t currently have descriptions or anything like that, and even the homepage is a bit of a hack in a lot of cases), and then data about all the versions.

For each package name, we store each version, its slot, and the description of the best version in each slot (all descriptions is probably a waste of space, considering how little descriptions vary between versions). There aren’t any packages with multiple versions or slots in the above example, but when there are they look like:

sys-devel/
    automake/
        :1.4 1.4_p6 ; Tool used to automatically generate Makefile.in files
        :1.5 1.5 ; Tool used to automatically generate Makefile.in files
        :1.6 1.6.3 ; Tool used to automatically generate Makefile.in files
        :1.7 1.7.9 ; Tool used to automatically generate Makefile.in files
        :1.8 1.8.5 ; Tool used to automatically generate Makefile.in files
        :1.9 1.9.6 ; Tool used to automatically generate Makefile.in files
        :1.10 1.10 1.10.1 ; Tool used to automatically generate Makefile.in files

And that’s all there is to it.

(Well, not entirely. There still has to be a tool to generate the repository content files. Fortunately, dleverton wrote a simple Ruby script using the Paludis bindings that generates them all automatically from the layman master file.)