Blag
He's not dead, he's resting
Category Archives: paludis for users
Paludis 0.62.1 Released
June 16, 2011
Posted by on Paludis 0.62.1 has been released:
- ‘cave resolve –hide’ now affects resolvent selection too.
- Various minor bug fixes.
Configuring Repositories Automatically via RepositoryRepository
June 4, 2010
Posted by on Paludis is aware of packages that are in repositories you don’t have configured thanks to the unavailable repository. However, once Paludis has shown you that the package you want is in a repository you don’t have configured, you need to set up a configuration file for that repository (and any repositories it requires) and then sync. This is more work than is really necessary.
Enter RepositoryRepository, also known as r^2. Conceptually, it works as follows:
As well as providing special packages for packages in unavailable repositories, the unavailable repository also now provides packages named ‘repository/blah’ for repositories you don’t have configured. The metadata for these packages includes dependency information etc, along with useful things like the repository’s sync URI.
A new repository, using format = repository
, provides special packages for repositories you do have configured.
Repository packages in unavailable
repositories can be ‘installed’ to repository
repositories. ‘Installing’ a repository creates a configuration file for it, and then syncs the newly created repository.
The configuration file it creates is controlled by a simple template, so it can contain anything you want it to contain.
Exherbo users can follow the setup instructions to start using this. On Gentoo this functionality is not yet available, since we won’t be switching the generated unavailable data to the new format until we’re reasonably sure that everyone is using a Paludis release that supports it.
Paludis is Going Into a Cave
March 31, 2010
Posted by on The following applies to Gentoo, not Exherbo. Exherbo developers (Exherbo has no users) already know what’s going on there. I figure it’s worth having a clear source of information on this for Gentoo users, though, rather than making people rely upon rumours and third hand transcriptions of what’s been said on IRC.
The original Paludis client was more or less designed to be a less perverse version of emerge
, with additional options for things like querying. On the plus side, it made things slightly easier for Gentoo users to pick up. However:
- Users would be confused as to why things like
--show-use-descriptions
, which is an option for the--install
action, would not also do something with--query
. Funnily enough, users don’t seem to wonder whyemerge --sync --changelog --pretend
doesn’t work. - We had to be extremely careful allocating short options. Thus, we only allocated short options when we were sure an option would regularly be used both enabled and disabled, since otherwise it could just go in
PALUDIS_OPTIONS
. This worked well enough for users who were paying attention, but seemed to scare off certain people who jumped in without understanding what they were doing. - It meant we regularly had to decide between breaking familiarity or doing something dumb just to make Portage converts comfortable.
I’ve never especially liked the result. It’s not close enough to emerge
to make learning zero cost, and it’s too close to emerge
to be a pleasant user interface. Fortunately, the UI and the library are mostly nicely separated, so we can fix this. The plan is as follows:
- A new client, named
cave
, will be provided. All pretence of feeling anything likeemerge
has been abandoned forcave
. Instead, the interface is roughly based upongit
and similar tools. - Depending upon whether or not we can be bothered, a second new client named
egress
may or may not be produced. If it is, it will be command-line-compatible withemerge
, and with a reasonably similar output format (within reason). I’m not convinced something like this would be worth the effort to write, although there are some Gentoo developers who insist that something that behaves almost identically toemerge
is a critical requirement for any official Gentoo package manager… - The
paludis
client will be deprecated and phased out.
To make things simpler, we’ve also decided to use the introduction of cave
to switch to the shiny new resolver. The shiny new resolver is a lot more flexible, a lot more powerful and a heck of a lot easier to maintain and modify than the old resolver. However, making the paludis
client use it would be quite a bit of work, so we’re not going to backport it.
There is no specific timeframe for any of this, and no estimates will be provided. There are, however, lists of things that need to be done.
- The Basic Functionality milestone contains all of the things that have to be done before we consider
cave
to be ‘more or less usable’. Until this milestone is complete, we won’t be enablingcave
in any of the ebuilds on Gentoo, and we don’t recommend users sneakily enabling it themselves. Note in particular that some of the things not yet implemented include “displaying a user-readable error message rather than a big fat “UI not implemented!” when attempting an unsafe package uninstall, and “handle virtual blockers”. - The Useful Functionality milestone contains all of the things that have to be done before we remove the
paludis
client. We will be makingcave
easily available before it includes every bit of functionality present inpaludis
, but we won’t be forcing a switch until we’ve either implemented equivalents for every feature or decided we’re not going to support a particular feature at all. - The Long Term Extras milestone contains things we’ll be doing at some point.
These milestones are probably incomplete, and are definitely open to arbitrary additions, removals, changes and being ignored based upon developer whims, available development time, bribery, patches, lack of interest, features looking fun or boring to code and the phase of the moon.
I realise this is probably futile, but I should stress that users should not attempt to enable or use cave
on Gentoo until we pass the Basic Functionality milestone. This isn’t taking away your fun — some of the not yet completed Basic Functionality items are necessary for basic usability and correct operation on Gentoo.
Paludis kdebuild-1 Removal
December 14, 2009
Posted by on kdebuild-1 was an EAPI used by the Gentoo KDE team and the Genkdesvn project for various live ebuilds in overlays. It was created following a request from the Gentoo KDE project, who needed certain features that Portage had not yet implemented. It was a documented, public standard, in the same way that EAPIs 0, 1 and 2 are.
The Gentoo Council has decided that all mention of the kdebuild-1 EAPI is to be removed from the Package Manager Specification immediately. A request that this be done only after a deprecation period to give users more time to catch up was rejected. As such, support for this EAPI, including support for uninstalling packages with that EAPI, will be removed from Paludis in version 0.44.
Users should verify that they have no kdebuild-1 packages installed. If Paludis was built with the ‘inquisitio’ use flag enabled, you can use:
$ inquisitio -K installed -k EAPI kdebuild-1
Otherwise, use:
$ paludis -q '*/*::/[.EAPI=kdebuild-1]'
and, if any such packages are installed, they must be uninstalled before Paludis is upgraded.
Changes to the Paludis Git Syncer
September 26, 2009
Posted by on I’ve just committed two changes to the Paludis Git syncer.
The first allows you to do sync_options = --branch=foo
to specify a particular branch. I don’t expect this to be widely used, since branching for repositories usually means you’re not taking advantage of the better facilities available for that kind of thing. Still, it has its uses.
The second is sync_options = --reset
. Unlike with, say, rsync
, syncing via Git will merge any changes you’ve made to the repository with new upstream changes (it uses git pull
). With --reset
, it will instead discard any local changes and just become equal to whatever you’re syncing against (using instead git fetch
and git reset --hard
).
It’s a matter of considerable debate as to whether the reset behaviour is the right thing to do.
On the one hand, some people like working directly on a checkout to which Paludis is pointed, but still want cave sync
to bring in updates.
On the other hand, doing that is horrible and evil, and a much better workflow is this:
- Have a local checkout for development work. Commit your changes to it.
- Have a separate checkout that is synced against your local checkout for Paludis use. After making changes, commit them to your local checkout and sync.
- When happy with your changes, rebase and squash them in your local checkout, and then push them upstream.
Doing that requires --reset
, since otherwise the checkout Paludis sees will end up as some horrible auto-merged mess that includes changes you thought you’d discarded ages ago.
I’m strongly considering making --reset
by default sometime in the future. However, this will make syncing a destructive operation for anyone who hates puppies enough to be doing work on a Paludis-synced checkout (in the same way that it’s already destructive for rsync).
Managing Accounts with the Package Manager
January 26, 2009
Posted by on Paludis is a multi-format package manager. One beneficial side effect of this is that the core code is sufficiently flexible to make handling things that aren’t really ‘packages’ in the conventional sense very easy; in the past, this has been used to deliver unavailable, unwritten and unpackaged repositories.
One of the things Exherbo inherited from Gentoo without modification was user and group management. In Gentoo, this is done by functions called enewuser
and enewgroup
from eutils.eclass
; a package that needs a user or group ID must call these functions from pkg_setup
. Although usable, this is moderately icky; Exherbo can do better than that.
Really, user and group accounts are just resources. A package that needs a particular user ID can be thought of as depending upon that ID — the only disconnect is that currently dependencies are for packages, not resources. Can we find a way of representing resources as something like packages, in a way that makes sense?
Fortunately, the obvious solution works. Having user/paludisbuild
and group/paludisbuild
as packages makes sense; adding the user or group is equivalent to installing the appropriate package, and if the user or group is present on the system, it shows up as installed. Then, instead of calling functions, the exheres can just do:
DEPENDENCIES=" build+run: user/paludisbuild group/paludisbuild "
What about defaults? Different users need different shells, home directories, groups and so on. We could represent these a bit like options, but there’s a better way.
If two or more ebuilds need the same user, they all have to do the useradd
call. This means duplicating things like home directory information and preferred uid over lots of different ebuilds, which is bad. It would be better to place the users somewhere else. For Exherbo, we’ve gone with metadata/accounts/{users,groups}/*.conf
. A user’s settings look something like this (the username is taken from a filename, so this would be metadata/accounts/users/paludisbuild.conf
):
shell = /bin/bash gecos = Used by Paludis for operations that require an unprivileged user home = /var/tmp/paludis primary_group = paludisbuild extra_groups = preferred_uid =
And a group, metadata/accounts/groups/paludisbuild.conf
:
preferred_gid =
We only specify ’empty’ keys for demonstration purposes; ordinarily they would be omitted.
We automatically make users depend upon the groups they use. The existing dependency abstractions are sufficient for this. There’s a bit of trickery in Paludis to allow supplemental repositories to override user defaults found in their masters; details are in the source for those who care.
One more thing to note: with accounts specified this way, we can be sure that the package manager only manages relevant accounts. There’s no danger of having the package manager accidentally start messing with your user accounts.
So what are the implications?
- We’re no longer tied to a particular method of adding users. If a user doesn’t want to use
useradd
andgroupadd
, they can write their own handler for the package manager to update users via LDAP or whatever. Paludis supports multiple handlers here. - Users who would rather manage a particular account manually can add it themselves, and the package manager will treat it as being already installed and won’t try to mess with it.
- User and group defaults are in one place, not everywhere that uses them.
- It’s much more obvious when an account is going to be added.
- Accounts that are no longer required can be purged using the usual
uninstall-unused
mechanism.
And what does it look like?
$ paludis -pi test-pkg Building target list... Building dependency list... These packages will be installed: * group/alsogroupdemo [N 0] Reasons: *user/accountsdemo-0:0::accounts "alsogroupdemo" * group/groupdemo [N 0] Reasons: *user/accountsdemo-0:0::accounts "groupdemo" * group/thirdgroupdemo [N 0] Reasons: *user/accountsdemo-0:0::accounts "thirdgroupdemo" * user/accountsdemo [N 0] Reasons: *test-cat/test-pkg-2:2::ciaranm_exheres_test "A demo account" * test-cat/test-pkg::ciaranm_exheres_test :2 [N 2] <target> -foo build_options: recommended_tests split strip "Dummy test package"
We can have a look at the accounts before they’re installed:
$ paludis -q accountsdemo groupdemo * user/accountsdemo accounts: 0* {:0} Username: accountsdemo Description: A demo account Default Group: groupdemo Extra Groups: alsogroupdemo thirdgroupdemo Shell: /sbin/nologin Home Directory: /dev/null * group/groupdemo accounts: 0* {:0} Groupname: groupdemo Preferred GID: 123
Note the dependencies:
$ paludis -qDM accountsdemo * user/accountsdemo accounts: 0* {:0} username: accountsdemo gecos: A demo account default_group: groupdemo extra_groups: alsogroupdemo thirdgroupdemo shell: /sbin/nologin home: /dev/null dependencies: group/alsogroupdemo, group/groupdemo, group/thirdgroupdemo location: /var/db/paludis/repositories/ciaranm_exheres_test/metadata/accounts/users/accountsdemo.conf defined_by: ciaranm_exheres_test
The install is fairly boring:
(4 of 5) Installing user/accountsdemo-0:0::accounts * Executing phase 'merge' as instructed >>> Installing user/accountsdemo-0:0::accounts using passwd handler useradd -r accountsdemo -c 'A demo account' -G 'alsogroupdemo,thirdgroupdemo' -s '/sbin/nologin' -d '/dev/null' >>> Finished installing user/accountsdemo-0:0::accounts
And once they’re installed:
$ paludis -q accountsdemo groupdemo * user/accountsdemo installed-accounts: 0* {:0} * group/groupdemo installed-accounts: 0* {:0}
Exherbo will be migrating to this new mechanism shortly — package manager support is already there (it was only a few hours’ work), so it’s just a case of gradually hunting down and killing those enew*
function calls.
UIs for Parallelism
October 27, 2008
Posted by on Getting the UI right for parallel execution is tricky.
Make, for example, doesn’t even try. It just displays output from all the children all mixed together, and relies upon commands prefixing their output with a filename or similar identifier. This generally works for what make needs to do, but it’s a bit too crude for some things.
For the secret “not supposed to use it” parallelism in Paludis, we’re currently using something slightly more sophisticated: each line of output from each child job is prefixed by the job’s name. So you’ll see things like:
sync unavailable> wget -T 30 -t 1 -O /var/tmp/paludis-tarsync-cvh9gX/exherbo_repositories.tar.bz2 http://git.exherbo.org/exherbo_repositories.tar.bz2 sync unavailable-unofficial> wget -T 30 -t 1 -O /var/tmp/paludis-tarsync-kvEzRZ/exherbo_unofficial_repositories.tar.bz2 sync unavailable-unofficial> --14:24:06-- http://git.exherbo.org/exherbo_unofficial_repositories.tar.bz2 sync unavailable-unofficial> => `/var/tmp/paludis-tarsync-kvEzRZ/exherbo_unofficial_repositories.tar.bz2' sync unavailable> --14:24:06-- http://git.exherbo.org/exherbo_repositories.tar.bz2
Something like this is necessary because the commands involved don’t do their own prefixing. But whilst it’s an improvement, the output’s rather cluttered and it’s hard to work out what’s going on. I’ve been experimenting with something more like this (slowdown induced artificially to make the point):
Starting sync Repository Status Pending Active Done -> alsa starting 16 1 0 -> ciaranm starting 15 2 0 -> ferdy starting 14 3 0 -> compnerd starting 13 4 0 -> arbor starting 12 5 0 -> compnerd success 12 4 1 -> gnome starting 11 5 1 -> alsa failed 11 4 2 ... fatal: The remote end hung up unexpectedly -> hardware starting 10 5 2 -> ferdy success 10 4 3 -> media starting 9 5 3 -> gnome success 9 4 4 -> mozilla starting 8 5 4 -> hardware success 8 4 5 -> pioto-exheres starting 7 5 5 -> media success 7 4 6 -> python starting 6 5 6 -> pioto-exheres success 6 4 7 -> rbrown starting 5 5 7 -> mozilla success 5 4 8 -> texlive starting 4 5 8 -> ciaranm active 4 5 8 -> python success 4 4 9 -> unavailable starting 3 5 9 -> texlive success 3 4 10 -> unavailable-unofficial starting 2 5 10 -> rbrown success 2 4 11 -> unwritten starting 1 5 11 -> arbor active 1 5 11 -> unwritten success 1 4 12 -> x11 starting 0 5 12 -> ciaranm active 0 5 12 ... Initialized empty Git repository in /var/db/paludis/repositories/ciaranm/.git/ -> unavailable active 0 5 12 ... Literal data: 0 bytes ... Matched data: 0 bytes ... File list size: 749 ... File list generation time: 0.001 seconds ... File list transfer time: 0.000 seconds ... Total bytes sent: 771 ... Total bytes received: 26 ... ... sent 771 bytes received 26 bytes 1594.00 bytes/sec ... total size is 57557 speedup is 72.22 -> unavailable-unofficial active 0 5 12 ... Literal data: 0 bytes ... Matched data: 0 bytes ... File list size: 110 ... File list generation time: 0.001 seconds ... File list transfer time: 0.000 seconds ... Total bytes sent: 132 ... Total bytes received: 26 ... ... sent 132 bytes received 26 bytes 316.00 bytes/sec ... total size is 4013 speedup is 25.40 -> arbor active 0 5 12 -> unavailable success 0 4 13 -> unavailable-unofficial success 0 3 14 -> x11 active 0 3 14 -> ciaranm active 0 3 14 -> ciaranm success 0 2 15 -> x11 success 0 1 16 -> arbor success 0 0 17 * Cleaning write cache for ebuild format repositories... * Done cleaning write cache for ebuild format repositories * No unread news items found Sync results * alsa: sync of '/var/db/paludis/repositories/alsa' from 'git://git.exherbo.org/demonstrate/failure/alsa.git' failed (paludis::SyncFailedError) Log file: /var/log/sync-alsa.1225118654.log * arbor: success * ciaranm: success * compnerd: success * ferdy: success * gnome: success * hardware: success * media: success * mozilla: success * pioto-exheres: success * python: success * rbrown: success * texlive: success * unavailable: success * unavailable-unofficial: success * unwritten: success * x11: success
In particular:
- Output is automatically logged to a file, rather than dumping everything to the screen. That file can be automatically removed if the job succeeds.
- A summary of what’s going on is displayed every time a job starts or finishes.
- Every ten seconds (for some value of ten), if a job hasn’t finished, we automatically display the last ten (for some other value of ten) lines of its output (but not any output we’ve already displayed) to the screen, along with a note that it’s still active. We also tail the log if a job fails.
This seems to be a comfortable balance between not showing anything except job statuses (which for jobs that can take an hour to run might well leave you wondering whether it’s hung) and just dumping everything to the screen. Now it’s just a case of finding appropriate values of ten…
Paludis is about Choices
October 24, 2008
Posted by on Paludis 0.32 will include a new mechanism called ‘choices’. Examples of choices on Gentoo include use flags, the various USE_EXPAND
settings (linguas
, video_cards
and so on) and a shiny new setting called build_options
. For use flags and the USE_EXPAND
settings, there’s no change from a user perspective, and you can carry on using use.conf
as normal — choices are merely a more general mechanism for handling package settings.
On Exherbo, things do change a bit, and if you’re one of those naughty people who ignored the “Exherbo has no users” warning you’d better have been paying attention.
So what’s this build_options
thing? You’ll see it cropping up for most packages:
* net-print/cups::gentoo [U 1.3.8-r1 -> 1.3.8-r2] <target>
Reasons: app-text/ghostscript-gpl-8.62:0::installed, net-print/foomatic-filters-3.0.20080507:0::installed, 2 more
X acl -avahi -dbus -gnutls -java jpeg -kerberos -ldap pam perl -php png ppds python -samba
-slp ssl -static -tiff -xinetd -zeroconf LINGUAS: -de en -es -et -fr -he -id -it -ja -pl -sv
-zh_TW build_options: -optional_tests split strip
Packages using 0-based EAPIs will usually have three build options:
optional_tests
. This handles what--checks
used to do. If this choice is set,src_test
will be run.split
. This used to be--debug-build split
.strip
. This used to be--debug-build internal
.
Packages using kdebuild-1 or exheres-0 will have recommended_tests
instead of optional_tests
, which defaults to being enabled rather than disabled.
Despite what you might expect from older Paludis versions, these are not use flags. You can set them in use.conf
, though — this means you can select whether to run tests or whether to keep debug information on a per-package basis, which was tricky to do with the old commandline options.
Path and installable-to dependencies
October 16, 2008
Posted by on A while ago, Paludis got from-repository dependencies. 0.32 will extend this further with path and installable-to dependencies.
Path dependencies are fairly simple: cat/pkg::/
will match cat/pkg
installed at root /
. A full path can be used too, so cat/pkg::/var/mychroot
will work (as well as anything involving non-/ roots works, anyway…).
Installable-to dependencies take a bit more explaining. There are two kinds of installable-to dependency: repository and path. An installable-to-repository dependency looks like cat/pkg::repo?
and is matched by any cat/pkg
that could be installed to the repository named repo
. An installable-to-path dependency looks like cat/pkg::/?
and is matched by any cat/pkg
that could be installed to any repository with root /
.
Installable-to dependencies will not match masked packages. To specify “packages that would be installable, ignoring masks”, use a double question mark, like cat/pkg::/??
. At present the double question mark ignores all masks, even those that cannot be overridden (unsupported EAPI, unavailable or unwritten). This may change to only include overridable masks; I’m not sure which is better behaviour yet.
Dealing with Unwritten Packages
October 6, 2008
Posted by on A while ago Exherbo came up with a solution for the “large number of repositories” problem. It turns out this solution works rather well, and I’m now convinced that anyone rushing off and assuming that Git on its own will magically solve anything other than CVS sucking is going in the wrong direction.
Paludis also has ways of tracking packages installed by hand. Whilst useful on Gentoo, this really comes into play on Exherbo, where we don’t want to end up with a massive unmaintainable tree full of packages used by only two people.
The logical next step is tracking packages that don’t exist yet.
An unimaginative person might think that the way to solve this would be to have a bug for each requested package. But that gets messy — it’s not integrated with the package manager, and requires considerable extra effort from the user. A while ago Bernd suggested something more interesting: tracking unwritten packages in a repository in a similar way to how we do unavailable packages.
The implications:
- Most obviously, we can keep track of things we want that haven’t been written yet in a way that doesn’t involve leaving the package manager to look things up.
- We can also use it to track version bumps that we know will take a while. Doing
paludis --query '>=hugescarypackage-4'
, say, would show that it’s being worked upon. - We can also depend upon things that don’t exist, rather than leaving incomplete dependency strings around. This is fine in at least two cases — if a dependency is conditional upon an option that should probably be implemented but isn’t yet, we can add the option and make it unusable. And we can handle obscure suggested dependencies (e.g. git has lots of optional dependencies upon weird perl modules, so we can say “if you want support for git-send-email with SSL, you need to write such and such a package”).
- Bored developers could simply
paludis --query '*/*::unwritten'
and get ideas for what to do.
Adding support to Paludis for this only took a couple of hours. So now Exherbo developers can use this:
format = unwritten location = /var/db/paludis/repositories/unwritten sync = git://git.exherbo.org/unwritten.git importance = -100
And see this:
$ paludis -q genesis * sys-apps/genesis unwritten: (1.0)X* {:0} Description: The daddy of all init systems Homepage: http://www.exherbo.org/ Comment: We need an init system. Masked by unwritten: Package has not been written yet
As with unavailable, installing behaves sensibly:
$ paludis -pi genesis Building target list... Building dependency list... Query error: * In program paludis -pi genesis: * When performing install action from command line: * When executing install task: * When building dependency list: * When adding PackageDepSpec 'sys-apps/genesis': * All versions of 'sys-apps/genesis' are masked. Candidates are: * sys-apps/genesis-1.0:0::unwritten: Masked by unwritten (Package has not been written yet)
Those interested in the repository format can browse the tree. The observant might notice that the file format is quite similar to the one used by unavailable repositories, and wonder whether I was feeling lazy and swiped a load of code rather than implementing a new repository from scratch.
In the spirit of silly buzzwords, one could argue that this increases distributivity and democratisation of the repository, since there’s nothing to stop motivated users from creating their own wishlist repositories. It wouldn’t be hard to make a simple web interface that lets users request and vote on packages and version bumps, automatically generating a repository that anyone can use. But fortunately Exherbo is a dictatorship and has no users, so we don’t have to put up with that kind of nonsense.