cleaning up /var/cache/http-replicator [SOLVED]

Tony0945 · Last edited by Tony0945 on Mon Dec 05, 2016 1:07 am; edited 1 time in total

80G of disk space is used on my distfile server. 29G is /var/cache/http-replicator which is four times as big as a typical /usr/portage/distfiles on my boxes.

The tail of "ls -lt /var/cache/http-replicator is:

NeddySeagoon · Posted: Fri Dec 02, 2016 6:32 pm Post subject:

Tony0945,

Wow, a file from before Unix time began :)

Tony0945 · Posted: Fri Dec 02, 2016 10:32 pm Post subject:

Saw that myself.

Something had to be wrong with the unix time (probably zero). But gtk 1.2, it's like a time warp.

This is a new SSD. I copied everything from the old HDD by "cp -a" I was surprised that it seemed to take forever. So everything is still there on the HDD (now on the shelf).

NeddySeagoon · Posted: Fri Dec 02, 2016 11:45 pm Post subject:

Tony0945,

That file from before the beginning of time will less than your timezone offset into very early Unix time.
Times are stored in UTC then timezome offset for display.
If you are in the USA, that UTC-5 to UTC-8, its easy to get such files.

Is it worth having distfiles on SSD? Reading/writing distfiles is only a very small part of the build time.
They may still be in the disk cache at first use too.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

Tony0945 · Posted: Sat Dec 03, 2016 12:32 am Post subject:

UTC-6 here.

I've (obviously) used the http-replicator for many years now and only switched to SSD two weeks ago. This machines reason for existing (it runs 24 7) is to serve portage files, a few local pages, and centralized local ebuilds. I'm inclined to add minidlna server so that a second machine doesn't have to run 24 7. That would entail physically moving a 5TB drive (obviously HDD) over. If you think it advisable, I could reconnect the old HDD and use it for /var/cache or better yet, when I connect the 5TB, point /var/cache/http-replicator to a directory on that drive via symlink. Please give me your thoughts on this.

I sync three other Gentoo machines via this one so that I don't get banned for over using the portage servers.

cboldt · Veteran Joined: 24 Aug 2005 Posts: 1046

Not sure if this is what you are asking ... I pare both the distdir and http-replicator cache using eclean-dist. To get to the http-replicator directory, define DISTDIR before (or as) running eclean (or eclean-dist).

Tony0945 · Posted: Sat Dec 03, 2016 1:34 am Post subject:

cboldt · Veteran Joined: 24 Aug 2005 Posts: 1046

My pleasure. Your local network sounds similar to mine. Just a few Gentoo machines here, and a handful of Win, Android, and whatever. One of the Gentoo machines does "portage serving" for both the portage tree and source files. I doubt I have an optimal setup, as the portage tree is served using rsyncd, and distfiles via http-replicator.

I like use of eclean rather than tmpwatch for distfiles and htto-replicator, because some source files have a darn long life. The only potential "abuse of remote source code server" from the network here is that http-replicator is pared according to source files on that machine, which isn't exactly the same as what is installed on the other machines. This might create a duplicate fetch in a rare instance, but I figure it would be really rare, because each machine's DISTFILE directory should have all of the source code for what is installed on that particular machine. Even after being pared, http-replicator is always totally redundant looking at the machine it sits on. The "gap" is that http-replicator isn't quite comprehensive comparing it to the DISTFILE directory on the other machines.

I clean 'em all out (using eclean) once a month, in a cronjob.