mass-deletion of distfiles/ -- what now?

[mounty1]

I wanted to reduce the size of my system as the backup was taking longer each week so:

Code: Select all

# find distfiles/ -mtime +354 | xargs rm

Now in the old days of portage by rsync, that was OK because distfiles/ was just all the packages you had ever downloaded but now the command echoes stuff like

Code: Select all

rm: distfiles/git3-src/proj_portage.git/objects/ae is a directory
rm: distfiles/git3-src/proj_portage.git/objects/63 is a directory

so:

• What have I done? (trashed a git repo. most likely) and
• What should I do about it?

[NeddySeagoon]

mounty1,

The easy one first

What should I do about it?

Do nothing. The upstream git repo will be alive and well.

Some ebuilds fetch from git. Either by release or tag or even by commit ID. The code will be fetched again if required.

You have removed your copy of the objects fetched from git. That's just like removing a tarball.
distfiles/ is still all the packages you had ever downloaded but git is used as well are tarballs now.

[Zucca]

Instead of

Code: Select all

find distfiles/ -mtime +354 | xargs rm

... I'd use eclean-dist. Pass --help first to see the options or read the man page of eclean.
However I'm not certain if it cleans subdirectories like git3-src for example.

EDIT:

Code: Select all

# @ECLASS_VARIABLE: EGIT3_STORE_DIR
# @USER_VARIABLE
# @DEFAULT_UNSET
# @DESCRIPTION:
# Storage directory for git sources.
#
# This is intended to be set by user in make.conf. Ebuilds must not set
# it.
#
# EGIT3_STORE_DIR=${DISTDIR}/git3-src

... so you could place sources fetched by git into an another directory.

[mounty1]

Instead of

Code: Select all

find distfiles/ -mtime +354 | xargs rm

... I'd use eclean-dist. Pass --help first to see the options or read the man page of eclean.
However I'm not certain if it cleans subdirectories like git3-src for example.

I will do in future but tools such as eclean-dist are imperfect in my situation since my local repository is shared (NFS-mounted) amongst several machines, and none is the source-of-truth of what packages are required, obsolete etc.

[wjb]

I wrote a python script that cleans up a shared nfs distfiles directory. Runs on each pc to look at the environment.bz2 files under /var/db/pkg and figure out what's being used by each pc. Then one more run to compare whats in distfiles with whats actually needed; this writes a bash script to delete the unwanted stuff.

It doesn't care about the file dates, only that they're for an installed package.

About 200 lines, needs some edits to make it less specific to my setup - say if interested.

[Zucca]

tools such as eclean-dist are imperfect in my situation since my local repository is shared (NFS-mounted) amongst several machines, and none is the source-of-truth of what packages are required, obsolete etc.

Closest to your find -command equivalent can be achieved by passing -t or --time-limit=<time> for eclean-dist.
Although carefully crafting a find -command can achieve almost same results.
For example:

Code: Select all

find "$(portageq envvar DISTDIR)" -maxdepth 1 -mindepth 1 -type f -atime +180 -delete

... would delete distfiles which haven't been accessed in about half a year.
That would take NFS accessed files into account. Unless your filesystem doesn't support access times or you have mounted the filesystem with noatime. This, however, can't take installed packages into account. So, not perfect either. Also access time can be updated by others than just portage doing an install.

One other way, you could create a list of distfiles each client thinks it doesn't need by running:

Code: Select all

eclean-dist --quiet --pretend | sed 's|^/.*/||'

... but that "will escalate very quickly" into a complex coding project. You'd need to make comparisons and determine that if each client thinks the file is now useless, then remove it.

It's a simple problem for one client, but if multiple are accessing then same $DISTDIR via network it gets complicated really quick.

@wjb: What's your implementation, roughly?

[Hu]

I wanted to reduce the size of my system as the backup was taking longer each week so

If backup times are the only concern, you could just tell the backup tool not to archive this directory.

As regards the problem of deleting distfiles only when they are unnecessary, one approach I once experimented with was:

• Create a shared directory via NFS, which all systems can see.
• Point each system's DISTDIR to a unique subdirectory, say NFSMNT/$hostname.
• Override FETCHCOMMAND so that, before downloading from the Internet, it would try to find the requisite file in a peer's subdirectory. If found, create a hard link to it. Otherwise, download.

This way, any given client can run eclean-dist whenever no one else is fetching. A file that no one needs will drop to a link count of 0, and be deleted. A file that this system does not need, but that others do, will be deleted from this system's $hostname directory, but the link count will not go to 0, so the file is not deleted.

This does not play nicely with git-r3, and further work could be needed there.

[wjb]

@zucca

Code: Select all

# On each client
# Look through all the environment.bz2 files under /var/db/pkg/*/*
#  - Find line: declare -x A="<space delimited list>"
#  - Add all the files in this list to a set of in-use files
# Save the in-use set under distfiles, in a file named for the host
#
# Then, on one PC 
# - Load all the client file lists into a set, REQUIRED
# - Glob all the filenames in distfiles into a set, PRESENT
# - Find filenames in PRESENT but not in REQUIRED, into a list UNWANTED
# - Delete all the UNWANTED files.

Actually generates a bash script to do the deletes, to allow for manual review and tweaking.

Edit: github wwjjbb/wjbtools

[Zucca]

This way, any given client can run eclean-dist whenever no one else is fetching. A file that no one needs will drop to a link count of 0, and be deleted. A file that this system does not need, but that others do, will be deleted from this system's $hostname directory, but the link count will not go to 0, so the file is not deleted.

This is so elegant and simple solution.

This does not play nicely with git-r3, and further work could be needed there.

Maybe easiest is to move git3-src -directory out of $DISTDIR. But there may be other directories too.

[mounty1]

Do nothing. The upstream git repo will be alive and well.

Thanks Neddy.

[mounty1]

Thanks everyone for the ideas. My shared-portage system extends to NFS-mounted /etc/portage and /var/db/pkg on all machines, so I could write my own script, like wjb, to perform the clean-up. I just didn't know, until yesterday, that I ought to do so.

[Zucca]

Code: Select all

       By default, eclean will protect all distfiles or
       binary packages corresponding  to  some  ebuilds
       available  in  the  Portage  tree.   This is the
       safest mode, since it will protect whatever  may
       still  be  useful,  for  instance to downgrade a
       package without downloading its sources for  the
       second  time,  or to reinstall a package you un‐
       merge by mistake without recompiling it.   Sure,
       it’s  also  a  mode  in  which  your DISTDIR and
       PKGDIR will stay rather big (although still  not
       growing  infinitely).  For the ’distfiles’, this
       mode is also quite slow because it requires some
       access to the whole Portage tree.

       If you use the --deep option, eclean  will  only
       protect  files  corresponding  to some currently
       installed package (taking  their  exact  version
       into  account).   It  will save much more space,
       while still preserving sources files around  for
       minor revision bumps, and binaries for reinstal‐
       lation of corrupted packages.  But it won’t keep
       files for less usual operations like downgrading
       or  reinstalling  an  unmerged package.

So maybe running eclean without --deep might also get you close to what you want? In short: without --deep eclean only removes distfiles that aren't used by any ebuild on any active repositories (overlays) you have.
Adding --time-limit would further avoid deleting too new distfiles.
Problem with this approach is, if you have enabled different repos on your machines using the same NFS shared distdir, some distfiles might still get deleted which would still be available from some repo that's active on some other machine than the one running eclean.

For a bullet-proof method, you will need to query all the other machines.

[wjb]

I wanted to reduce the size of my system as the backup was taking longer each week

Oh. So, that can actually be sorted by using a deduplicating backup tool, like app-backup/borgbackup - where a chunk of data is already in the backup repository, it's simply referenced rather than adding a new copy of the data. On average this is way quicker than a trad backup tool, every backup archive is effectively assembled as if incremental but behaves later as a full backup.

There's also app-backup/borgmatic which is a simplified layer on top of borg.

I don't actually bother backing up distfiles as I figure it's easy enough for portage to retrieve them from the mirrors. Current ones anyway. All I'm trying to do with distfiles is purge when it's partition fills up. Anyway, I just put my script here on github.