Blag
He's not dead, he's resting
Dealing with Lots of Repositories
June 12, 2008
Posted by on With the explosion of overlays used by Gentoo, finding a package can get quite messy. Most users won’t want to set up lots and lots of repositories, so they won’t necessarily know when a package (or an ebuild for an scm or beta version of a package whose stable versions are in the main tree) is available.
Exherbo has similar issues. It’s likely that not-widely-used applications will remain permanently in individual developers’ personal repositories, with only reasonably important packages making it into Arbor. This has further implications for dependencies — for example, if X11 remains in its own repository, but core Arbor packages have optional dependencies upon X, how will that work?
Enter yesterday’s Paludis project: UnavailableRepository
.
The idea is simple:
- Have a repository that contains all the packages in all the repositories you don’t have.
- Make it know enough about those packages to show you that they’re available, but not enough to let you do the install. In Paludis terms, this means making the packages be masked with a non-overridable mask, but still support
InstallAction
. - Make it less important than any ‘proper’ repository.
- Do something clever to skip doing unavailable IDs for any repository you have configured. (Not strictly speaking necessary, but a lot nicer.)
Configuration is simple. For Exherbo:
format = unavailable location = /var/db/paludis/repositories/unavailable sync = tar+http://git.exherbo.org/exherbo_repositories.tar.bz2 importance = -100
And for Gentoo:
format = unavailable name = layman location = /var/paludis/repositories/layman sync = tar+http://git.exherbo.org/layman_repositories.tar.bz2 importance = -100
(Both sync URLs will probably change soon.)
Then you can do this:
$ paludis -q firefox * net-www/firefox unavailable: (3.0_rc1 (in ::mozilla))X* {:0} Description: The firefox web browser Owning repository: mozilla Repository homepage: http://git.exherbo.org/?p=mozilla.git Masked by unavailable: In a repository which is unavailable
And this:
$ paludis -pi firefox Building target list... Building dependency list... Query error: * In program paludis -pi firefox: * When performing install action from command line: * When executing install task: * When building dependency list: * When adding PackageDepSpec 'net-www/firefox': * All versions of 'net-www/firefox' are masked. Candidates are: * net-www/firefox-3.0_rc1:0::unavailable (in ::mozilla): Masked by unavailable (In a repository which is unavailable)
You can even search (on description, at least — searching on other metadata keys won’t find anything):
$ sudo inquisitio browser * net-www/firefox unavailable: (3.0_rc1 (in ::mozilla))X* {:0} Description: The firefox web browser Owning repository: mozilla Repository homepage: http://git.exherbo.org/?p=mozilla.git Masked by unavailable: In a repository which is unavailable * net-www/w3m unavailable: (0.5.2 (in ::haskell))X* {:0} Description: Text based WWW browser, supports tables and frames Owning repository: haskell Repository homepage: http://git.exherbo.org/?p=haskell.git Masked by unavailable: In a repository which is unavailable
So what’s in those magic sync tarballs?
$ ll /var/db/paludis/repositories/unavailable/ total 76K -rw-r--r-- 1 root root 369 2008-06-12 04:47 alsa.repository -rw-r--r-- 1 root root 3.2K 2008-06-12 04:47 haskell.repository -rw-r--r-- 1 root root 195 2008-06-12 04:47 kde.repository -rw-r--r-- 1 root root 666 2008-06-12 04:47 media.repository -rw-r--r-- 1 root root 238 2008-06-12 04:47 mozilla.repository -rw-r--r-- 1 root root 334 2008-06-12 04:47 perl.repository -rw-r--r-- 1 root root 1.2K 2008-06-12 04:47 rbrown.repository -rw-r--r-- 1 root root 338 2008-06-12 04:47 scientific.repository -rw-r--r-- 1 root root 20K 2008-06-12 04:47 x11.repository -rw-r--r-- 1 root root 1.5K 2008-06-12 04:47 xfce.repository
One file per repository, fairly simple. And the files themselves are nice and clean too:
$ cat /var/db/paludis/repositories/unavailable/mozilla.repository format = unavailable-1 repo_name = mozilla homepage = http://git.exherbo.org/?p=mozilla.git dev-libs/ libIDL/ :0 0.8.10 ; IDL parsing and compilation library net-www/ firefox/ :0 3.0_rc1 ; The firefox web browser
There’s a bit of metadata about the repository in question (not very much — repositories don’t currently have descriptions or anything like that, and even the homepage is a bit of a hack in a lot of cases), and then data about all the versions.
For each package name, we store each version, its slot, and the description of the best version in each slot (all descriptions is probably a waste of space, considering how little descriptions vary between versions). There aren’t any packages with multiple versions or slots in the above example, but when there are they look like:
sys-devel/ automake/ :1.4 1.4_p6 ; Tool used to automatically generate Makefile.in files :1.5 1.5 ; Tool used to automatically generate Makefile.in files :1.6 1.6.3 ; Tool used to automatically generate Makefile.in files :1.7 1.7.9 ; Tool used to automatically generate Makefile.in files :1.8 1.8.5 ; Tool used to automatically generate Makefile.in files :1.9 1.9.6 ; Tool used to automatically generate Makefile.in files :1.10 1.10 1.10.1 ; Tool used to automatically generate Makefile.in files
And that’s all there is to it.
(Well, not entirely. There still has to be a tool to generate the repository content files. Fortunately, dleverton wrote a simple Ruby script using the Paludis bindings that generates them all automatically from the layman master file.)
Pingback: From-Repository Dependencies « Ciaran McCreesh’s Blag
Pingback: Dealing with Unwritten Packages « Ciaran McCreesh’s Blag
Pingback: Managing Accounts with the Package Manager « Ciaran McCreesh’s Blag
Pingback: /dev/otaku » Blog Archive » Exherbo in numbers
Pingback: SUMMER-time in Exherbo « Ingmar’s shameless self-blagvertisement.
Pingback: Configuring Repositories Automatically via RepositoryRepository « Ciaran McCreesh’s Blag