HOWTO:Download Cache for your LAN-Http-Replicator (ver 3.0)

[flybynite]

- What happens if client 1 wants a (huge) file and client 2 as well, requesting it before it has been completely downloaded by http-replicator? client 2 gets a partial (invalid) file?

NEVER!

http-replicator is smart. both clients get exactly the correct file and only 1 copy is ever downloaded from the net. If a long download is in progress and another client requests the same file, the new client will receive the downloaded part quickly from the cache and will "catch up" with the first client download still in progress. Then both clients will then get the file streamed simultaneously as it is downloaded from the net and saved to the cache! This works where 2 clients or 200+ clients make requests.

- I often do an emerge -avuDf (=complete update, but fetch only), just to have the necessary files already in place for when eventually do update. One package (ivtv) keeps downloading two files over ftp, although they are in the cache. What is the reason for this and how do I "solve" this?

this is in the long(ish) HOWTO itself, although probably not obvious

from the howto

Code: Select all

Also, some packages in portage have a RESTRICT="nomirror" option which will prevent portage from checking replicator for those packages. The following will override this behavior. Create the file "/etc/portage/mirrors" containing:
Code:

# Http-Replicator Override for FTP and RESTRICT="nomirror packages
local http://gentoo.osuosl.org

You can replace gento.osuosl.org with your favorite HTTP:// mirror. If you already have a local setting, don't worry, as long as it is an http mirror this will still be effective.

It isn't obvious this is the case with ivtv but this will confirm.

Code: Select all

# grep mirror  /usr/portage/media-tv/ivtv/ivtv-0.7.0.ebuild
RESTRICT="nomirror"

man portage

Code: Select all

/etc/portage/mirrors
                     Whenever  portage  encounters  a mirror:// style URL it will look up the actual
                     hosts here.  If the mirror set is not found here, it will check the global mir-
                     rors  file at /usr/portage/profiles/thirdpartymirrors.  You may also set a spe-
                     cial mirror type called "local".  This list of mirrors will be  checked  before
                     GENTOO_MIRRORS and will be used even if the package has RESTRICT="mirror".

restrict=nomirror is a portage function and I have no control over it or why it exists. This workaround must be done on all clients.

At least there is a workaround for restrict=nomirror, there is also a restrict=nofetch which has no workaround , but there are only a few such ebuilds.

[tek0]

Is is usual that repcacheman.py needs an eternity for extracting the digests? On my box, it takes about 20 minutes at full CPU usage...a niceness param would be helpful

[dalek]

Code: Select all

nice -5 repcacheman

That should work.

[flybynite]

Is is usual that repcacheman.py needs an eternity for extracting the digests?

repcacheman uses portage database functions to read every single ebuild in the whole portage tree. This takes time. This will take more time as the tree grows. Also portage isn't optimized to do the database lookups repcacheman does. Repcacheman could really use its own database to optimize the queries, but I haven't had many other complaints about the speed since 99.9% of the files should be on an http mirror and repcacheman should only rarely have to do the full blown md5sum check.



You may not need to run repcachman as often if you are finding the run time troublesome. Once replicator is installed and running healthy, repcacheman only serves to transfer a couple of files that may have been only available on ftp to replicators cache. If you don't run repcachman, the worst that will happen is that individual boxes may rarely have to ftp one or two files themselves. This might mean that only 99.9% of the files are downloaded once and cached, instead of 100%. If your running hundreds of clients and paying per MB downloaded that might be unacceptable, otherwise running repcacheman less often could be for you.


Also make sure you delete the distfiles after running repcachman. A partially downloaded file left in the distfile dir will cause the full blown md5 check every time repcacheman is run. I don't do this automatically because some people use replicator to cache files that are not in portage.

I do it like this

Code: Select all

repcacheman && rm -r /usr/portage/distfiles

Unless you use replicator to cache other files, anything left in your distfile dir after repcacheman runs is

1. No longer in Portage
2. Incomplete or corrupt
3. Just plain junk
4. Wasting your time and disk space

[tek0]

Is is usual that repcacheman.py needs an eternity for extracting the digests?

repcacheman uses portage database functions to read every single ebuild in the whole portage tree. This takes time. This will take more time as the tree grows. Also portage isn't optimized to do the database lookups repcacheman does. Repcacheman could really use its own database to optimize the queries, but I haven't had many other complaints about the speed since 99.9% of the files should be on an http mirror and repcacheman should only rarely have to do the full blown md5sum check.



You may not need to run repcachman as often if you are finding the run time troublesome. Once replicator is installed and running healthy, repcacheman only serves to transfer a couple of files that may have been only available on ftp to replicators cache. If you don't run repcachman, the worst that will happen is that individual boxes may rarely have to ftp one or two files themselves. This might mean that only 99.9% of the files are downloaded once and cached, instead of 100%. If your running hundreds of clients and paying per MB downloaded that might be unacceptable, otherwise running repcacheman less often could be for you.


Also make sure you delete the distfiles after running repcachman. A partially downloaded file left in the distfile dir will cause the full blown md5 check every time repcacheman is run. I don't do this automatically because some people use replicator to cache files that are not in portage.

I do it like this

Code: Select all

repcacheman && rm -r /usr/portage/distfiles

Unless you use replicator to cache other files, anything left in your distfile dir after repcacheman runs is

1. No longer in Portage
2. Incomplete or corrupt
3. Just plain junk
4. Wasting your time and disk space

Oh, I see. I thought repcacheman was essential for all files, so I ran it daily through cron, and there were about 10 corrupt or incomplete files in $DISTDIR that seem to have caused the trouble. Guess I'll just nice 15 && rm $DISTDIR (I want to work meanwhile, dalek ) and it will be fine.

[ribx]

Bandwidth limiting wasn't one being considered Nobody ever asked for it before... Http-Replicator is all Python and doesn't use any external programs such as wget.

i found this on page 4. i would also like to see this feature would be much easier than doing this throught the kernel or something. its just that my brother cannot play and my mother cannot surf while i'm downloading updates.

but also without that feature, http-replicator is still a realy good thing - and so easy to set up. good job!

-ribx

[RichieB]

Greetings.

After installing http-replicator as per the howto, along with a sync mirror, I get the following error when attempting to emerge -uDag:

======
These are the packages that would be merged, in order:

Fetching binary packages info...
date: Fri, 06 Oct 2006 11:16:14 GMT
connection: close

<html><body><pre><h1>Not found</h1></pre></body></html>

address: /All
Traceback (most recent call last):
File "/usr/bin/emerge", line 4049, in ?
emerge_main()
File "/usr/bin/emerge", line 4044, in emerge_main
myopts, myaction, myfiles, spinner)
File "/usr/bin/emerge", line 3452, in action_build
myopts, myparams, spinner)
File "/usr/bin/emerge", line 675, in __init__
trees["/"]["bintree"].populate(
File "/usr/lib/portage/pym/portage.py", line 5434, in populate
self.remotepkgs = getbinpkg.dir_get_metadata(
File "/usr/lib/portage/pym/getbinpkg.py", line 450, in dir_get_metadata
filelist = dir_get_list(baseurl, conn)
File "/usr/lib/portage/pym/getbinpkg.py", line 295, in dir_get_list
raise Exception, "Unable to get listing: %s %s" % (rc,msg)
Exception: Unable to get listing: 404 Server did not respond successfully (404: Not Found)
======

The server /etc/conf.d/http-replicator:

======
## Config file for http-replicator
## sourced by init scripts automatically
## GENERAL_OPTS used by repcacheman
## DAEMON_OPTS used by http-replicator


## Set the cache dir
GENERAL_OPTS="--dir /var/cache/http-replicator"

## Change UID/GID to user after opening the log and pid file.
## 'user' must have read/write access to cache dir:
GENERAL_OPTS="$GENERAL_OPTS --user portage"

## Don't change or comment this out:
DAEMON_OPTS="$GENERAL_OPTS"

## Do you need a proxy to reach the internet?
## This will forward requests to an external proxy server:
## Use one of the following, not both:
#DAEMON_OPTS="$DAEMON_OPTS --external somehost:1234"
#DAEMON_OPTS="$DAEMON_OPTS --external username:password@host:port"

## Local dir to serve clients. Great for serving binary packages
## See PKDIR and PORTAGE_BINHOST settings in 'man make.conf'
## --alias /path/to/serve:location will make /path/to/serve
## browsable at http://http-replicator.com:port/location
DAEMON_OPTS="$DAEMON_OPTS --alias /usr/portage/packages/All:All"

## Dir to hold the log file:
DAEMON_OPTS="$DAEMON_OPTS --log /var/log/http-replicator.log"

## Make the log messages less and less verbose.
## Up to four times to make it extremely quiet.
#DAEMON_OPTS="$DAEMON_OPTS --quiet"
#DAEMON_OPTS="$DAEMON_OPTS --quiet"

## Make the log messages extra verbose for debugging.
DAEMON_OPTS="$DAEMON_OPTS --debug"

## The ip addresses from which access is allowed. Can be used as many times
## as necessary. Access from localhost is allowed by default.
DAEMON_OPTS="$DAEMON_OPTS --ip 192.168.*.*"
DAEMON_OPTS="$DAEMON_OPTS --ip 10.*.*.*"
DAEMON_OPTS="$DAEMON_OPTS --ip 127.*.*.*"

## The proxy port on which the server listens for http requests:
DAEMON_OPTS="$DAEMON_OPTS --port 8001"
======

The server /etc/rsyncd.conf:

======
# Copyright 1999-2004 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: /var/cvsroot/gentoo-x86/app-admin/gentoo-rsync-mirror/files/rsyncd.conf,v 1.6 2004/07/14 21:12:47 agriffis Exp $

uid = nobody
gid = nobody
use chroot = yes
max connections = 20
pid file = /var/run/rsyncd.pid
motd file = /etc/rsync/rsyncd.motd
transfer logging = no
log format = %t %a %m %f %b
syslog facility = local3
timeout = 300

#[gentoo-x86-portage]
#this entry is for compatibility
#path = /usr/portage
#comment = Gentoo Linux Portage tree

[gentoo]
#modern versions of portage use this entry
path = /usr/portage
comment = Gentoo Linux Portage tree mirror
exclude = distfiles
======

And the server /etc/make.conf:

======
# These settings were set by the catalyst build script that automatically built this stage
# Please consult /etc/make.conf.example for a more detailed example
#CFLAGS="-O2 -march=i686 -pipe"
CFLAGS="-march=athlon-mp -O2 -fomit-frame-pointer -pipe"
CHOST="i686-pc-linux-gnu"
CXXFLAGS="${CFLAGS}"
MAKEOPTS=""
ACCEPT_KEYWORDS=""
USE=" -* -arts -gnome -gtk 3dnow 3dnowext 3ds X aac acpi alsa apm amr audiofile asf bitmap-fonts bzip2 cdparanoia cdr clamav crypt ctype cups dbus dvd dvdread effects encode exif expat fam fastbuild ffmpeg foomaticdb force-cgi-redirect gd gdbm gif glut gmp gpm gtk gtk2 hal idn imlib ipv6 jpeg jpeg2k kde kdgraphics lcms libg++ libvisual libwww lm_sensors mad memlimit mikmod mmx mmxext mng motif mozilla mp3 mpeg mplayer msn ncurses nls nptl nptlonly nsplugin ogg openal opengl oss pam pcre perl png posix qt3 quicktime readline reiserfs scanner sdl session simplexml slang sockets spamassassin spell sse ssl svg svga tcltk tcpd theora threads tiff truetype truetype-fonts type1-fonts udev usb v4l visualization vorbis win32codecs wmf x264 xine xml xml2 xsl xv xvid xvmc yahoo zlib java faad xcomposite dvd cdrom ivman pmount"
FEATURES="-sandbox ccache buildpkg"
LINGUAS="en_GB"
GENTOO_MIRRORS="ftp://ftp.mirrorservice.org/sites/www.i ... rg/gentoo/ http://gentoo.osuosl.org/"
INPUT_DEVICES="keyboard mouse"
HTTP_PROXY="http://localhost:8001"
VIDEO_CARDS="nvidia"
PORTDIR_OVERLAY=/usr/local/portage
======

and the client /etc/make.conf:

======
# These settings were set by the catalyst build script that automatically built this stage
# Please consult /etc/make.conf.example for a more detailed example
#CFLAGS="-O2 -march=i686 -pipe"
CFLAGS="-march=athlon-mp -O2 -fomit-frame-pointer -pipe"
CHOST="i686-pc-linux-gnu"
CXXFLAGS="${CFLAGS}"
MAKEOPTS=""
ACCEPT_KEYWORDS=""
USE=" -* -arts -gnome -gtk 3dnow 3dnowext 3ds X aac acpi alsa apm amr audiofile asf bitmap-fonts bzip2 cdparanoia cdr clamav crypt ctype cups dbus dvd dvdread effects encode exif expat fam fastbuild ffmpeg foomaticdb force-cgi-redirect gd gdbm gif glut gmp gpm gtk gtk2 hal idn imlib ipv6 jpeg jpeg2k kde kdgraphics lcms libg++ libvisual libwww lm_sensors mad memlimit mikmod mmx mmxext mng motif mozilla mp3 mpeg mplayer msn ncurses nls nptl nptlonly nsplugin ogg openal opengl oss pam pcre perl png posix qt3 quicktime readline reiserfs scanner sdl session simplexml slang sockets spamassassin spell sse ssl svg svga tcltk tcpd theora threads tiff truetype truetype-fonts type1-fonts udev usb v4l visualization vorbis win32codecs wmf x264 xine xml xml2 xsl xv xvid xvmc yahoo zlib java faad xcomposite dvd cdrom ivman pmount"
FEATURES="-sandbox ccache"
LINGUAS="en_GB"

SYNC="rsync://localhost/gentoo"
HTTP_PROXY="http://localhost:8001"
PORTAGE_BINHOST="http://localhost:8001/All"

INPUT_DEVICES="keyboard mouse"
VIDEO_CARDS="nvidia"
PORTDIR_OVERLAY=/usr/local/portage
======

Please note that the http-replicator port has been set to 8001, as the client is running an HTTP proxy of its own for internet access.

There are 50 packages in the server's /usr/portage/packagas/All directory.

On oddity you may notice is, in fact, that the server is running within a chroot environment on the client machine. This is simply due to the fact that this machine has a particular configuration that I'm building for and, at the moment, I only have one of them! There are not shared directories betwixt the 'parent' environement (the 'client') and the chroot environment (the 'server').

Also, when running an emerge -av world --deep on the server (running within its chroot), then all is fine - presumably the http-replicator proxy is doing its job?

Many thanks for any and all help!!

RichieB

[golding]

//snip
HTTP_PROXY="http://localhost:8001"
//snip
======

and the client /etc/make.conf:
//snip
HTTP_PROXY="http://localhost:8001"
//snip

When I installed http-replicator I had to put "http_proxy" in lower case, i.e.

Code: Select all

http_proxy="http://server.DNS/Name:8080"

has that changed?

[RichieB]

Thanks Robert, but I don't think that's the issue. http-replicator is successfully negotiating with the client because the error is coming from the proxy server itself.

RB

[RichieB]

*bump*

[think4urs11]

Thanks Robert, but I don't think that's the issue. http-replicator is successfully negotiating with the client because the error is coming from the proxy server itself.

Nevertheless http_proxy is not the same thing as HTTP_PROXY. Have you tried to exchange your uppercase version with the lowercase version (on both client AND server) already?

[RichieB]

Yes, I have - both lower and upper case! Is there any debug I could find which would help?

[RichieB]

As an aside, why would I need http_proxy to be lowercase when the other variables in make.conf are upper case?

Additionally, here is the tail of http-replicator.log on the server:

10 Oct 2006 01:10:31 INFO: HttpReplicator started
10 Oct 2006 00:10:38 STAT: HttpClient 1 bound to 127.0.0.1
10 Oct 2006 00:10:38 INFO: HttpClient 1 direct request for /All
10 Oct 2006 00:10:38 DEBUG: HttpClient 1 cache position: /usr/portage/packages/All
10 Oct 2006 00:10:38 DEBUG: HttpServer 1 received header:

GET /All HTTP/1.1
host: localhost:8001
accept-encoding: identity

10 Oct 2006 00:10:38 DEBUG: HttpClient 1 received header:

HTTP/1.1 404 Not Found
date: Tue, 10 Oct 2006 00:10:38 GMT
connection: close

10 Oct 2006 00:10:38 DEBUG: HttpClient 1 closed
10 Oct 2006 00:10:38 STAT: HttpClient 1 received 56 bytes

There are, however, plenty of packages in /usr/portage/packages/All, so why I should get this message (reproduced below) is not clear to me. Please help!

Fetching binary packages info...
date: Tue, 10 Oct 2006 00:13:56 GMT
connection: close

<html><body><pre><h1>Not found</h1></pre></body></html>

address: /All
Traceback (most recent call last):
File "/usr/bin/emerge", line 4049, in ?
emerge_main()
File "/usr/bin/emerge", line 4044, in emerge_main
myopts, myaction, myfiles, spinner)
File "/usr/bin/emerge", line 3452, in action_build
myopts, myparams, spinner)
File "/usr/bin/emerge", line 675, in __init__
trees["/"]["bintree"].populate(
File "/usr/lib/portage/pym/portage.py", line 5434, in populate
self.remotepkgs = getbinpkg.dir_get_metadata(
File "/usr/lib/portage/pym/getbinpkg.py", line 450, in dir_get_metadata
filelist = dir_get_list(baseurl, conn)
File "/usr/lib/portage/pym/getbinpkg.py", line 295, in dir_get_list
raise Exception, "Unable to get listing: %s %s" % (rc,msg)
Exception: Unable to get listing: 404 Server did not respond successfully (404: Not Found)

[golding]

As an aside, why would I need http_proxy to be lowercase when the other variables in make.conf are upper case?

I have no idea why, I just know I tried my system with the variable in upper case today and it stopped working. I put it back to lower case and it resumed working. I also tried changing the port (to 8001 and relevant client changes to keep everything clean) and this also stopped it working, except on the server, until I put everything back to vanilla.

My attitude tends to be:
If in doubt (I usually am), or if you don't know what you are doing (ditto), follow the manual to the last dotted 'i' and crossed 't'.
Change NOTHING!
Once it is working, then you might be able to tweak things for other variables, but not before.

[flybynite]

Please note that the http-replicator port has been set to 8001, as the client is running an HTTP proxy of its own for internet access.

On oddity you may notice is, in fact, that the server is running within a chroot environment on the client machine.

The problem is most likely specific to a problem in your unusual configuration. Http-replicator just works so your having a CHROOT problem, not a replicator problem, and most likely not worth trying to fix in this temporary chroot situation????

If you want to try and fix it anyway, there are two separate problems.

1. is http-replicator working for normal package downloads?

NO, because of the "HTTP_PROXY" your using. Portage uses wget to download packages unless you've changed it. wget only responds to "http_proxy" in lower case as you've already been told. Check the man page for wget or try

Code: Select all

$ HTTP_PROXY=http://bogus wget http://www.gentoo.org
--04:29:57--  http://www.gentoo.org/
           => `index.html'
Resolving www.gentoo.org... 38.99.64.201, 66.219.59.46, 66.241.137.77
Connecting to www.gentoo.org|38.99.64.201|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 14,358 (14K) [text/html]

100%[======================================================================================================================================>] 14,358        71.76K/s

04:29:57 (71.74 KB/s) - `index.html' saved [14358/14358]




$ http_proxy=http://bogus wget http://www.gentoo.org
--04:30:13--  http://www.gentoo.org/
           => `index.html.1'
Resolving bogus... failed: Name or service not known.

You see this code works with HTTP_PROXY=bogus because wget IGNORES it. The command fails as it should with the lower case http_proxy. This error means that http-replicator isn't used by your client for portage downloads and can be verified because the server /var/cache/http-replicator dir will be empty or not updated with each client emerge.

If by some chance it is being used, it is probably caused by a conflict with the proxy setup your using to access the net or some weird chroot interaction!!

2. Your trying to get BINHOST working which is the error you posted.

replicator should just return a web page file listing. Point your browser to:

Code: Select all

http://localhost:8001/All

It should return a listing of your /usr/portage/packages/All directory. If it doesn't, its because that dir isn't available to replicator probably because of the chroot setup or a permission problem.

Even if the dir were empty replicator would return an empty dir listing, not the "not found" error your getting. Check the dir permissions and also /var/log/http-replicator.log for any possible clues to the error.

Also check the simple stuff.

1. Make sure that since your running the server in the chroot, you didn't start http-replicator before chroot'ing or you started it outside the chroot?
2. How is the internet proxy configured? Any routing or ipchains rules in effect?
3. Are there any other portions of the howto you chose to ignore?

[RichieB]

flybynite et al: thanks [b]so[/b] much for your replies.

Of course, I changed HTTP_PROXY to http_proxy. And, of course, to no avail!

What I found [i]very[/i] interesting, although I cannot explain it, is if I change the client's make.conf from

PORTAGE_BINHOST="http://localhost:8001/All"

to

PORTAGE_BINHOST="http://localhost:8001"

With this alteration, I no longer get the "Not found" (404) message from http-replicator running on the chroot 'server' environment. At that point, emerge -uvaG is met with a "there are no packages available.." error, which I suppose is what you'd expect.

From another machine, I can browse to http://<chroot-http-replicator-ip-address>:8001 and see a list of .bz2's. But if I go to http://<chroot...>:8001/All then http-replicator responds with the "Not found" message above.

So, clearly, http-replicator [b]is[/b] working, but there is something awry with its access to the various directories ... changing DAEMON_OPTS=" ... --alias /usr/portage/packages/All:All" to "/usr/portage/packages:All" in the chroot server's http-replicator.conf seems to make no difference at all.

[flybynite]

Of course, I changed HTTP_PROXY to http_proxy. And, of course, to no avail!

Yes, this won't fix the BINHOST problem but it did fix the problem you aren't aware of yet. It will make sure http-replicator is actually caching packages. Did you check to see if any packages were in /var/cache/http-replicator?


Were there any clues in replicators log?

[RichieB]

Hi, fly, I appreciate your time!

Yes, there are over 600 packages in /var/cache/http-replicator.

I find it confusing that if I set PORTAGE_BINHOST to http://localhost:8001/ then the emerge process kind of works - but rather than emerge binary builds it emerges source and tries to compile it. Resetting it to the HOWTO suggested http://localhost:8001/All/ produces the Not Found error. Interestingly, the http-replicator log shows:

11 Oct 2006 14:04:10 STAT: HttpClient 22 bound to 127.0.0.1
11 Oct 2006 14:04:10 INFO: HttpClient 22 direct request for /All/
11 Oct 2006 14:04:10 DEBUG: HttpClient 22 cache position: /usr/portage/packages/All/index.html
11 Oct 2006 14:04:10 DEBUG: HttpServer 22 received header:

GET /All/ HTTP/1.1
host: localhost:8001
accept-encoding: identity

11 Oct 2006 14:04:10 DEBUG: HttpClient 22 received header:

HTTP/1.1 404 Not Found
date: Wed, 11 Oct 2006 14:04:10 GMT
connection: close

11 Oct 2006 14:04:10 DEBUG: HttpClient 22 closed
11 Oct 2006 14:04:10 STAT: HttpClient 22 received 56 bytes


And, of course, there is no /usr/portage/packages/All/index.html on the http-replicator server. This is the problem - but what is causing it? I believe my configuration files (above) are correct for a binary repository ... I'm not sure, but I think this shows the client is OK but the error *is* with the server ...

[flybynite]

Yes, there are over 600 packages in /var/cache/http-replicator.

Is the chroot a complete filesystem? I mean is there a /var/cache/http-replicator in the host and the chroot or is there only one such dir on the box?

I find it confusing that if I set PORTAGE_BINHOST to http://localhost:8001/ then the emerge process kind of works - but rather than emerge binary builds it emerges source and tries to compile it. Resetting it to the HOWTO suggested http://localhost:8001/All/ produces the Not Found error. Interestingly, the http-replicator log shows:

The reason it works is because without the "ALL" your browsing replicators cache. So we know that dir is available to replicator. With the "ALL" your browsing the .../packages/All dir which replicator can't enter/list for some reason.

So replicator probably can't enter or list the packages dir. Check the permissions on the dir and also enter the chroot with a shell and try and list the dir.

How do you have the outbound net proxy setup. Is it ipchains or a ENV variable?

Show me the output of :

Code: Select all

env

[depontius]

Is there a tool to clean up the http-replicator cache?

I've done a quick search through (not all 20 pages) this thread, Gento Forums, and Google to see if there's a tool to do this, but nothing has jumped out. A while back I found a thing called distclean.py that cleans stale files out of /usr/portage/distfiles. /var/cache/http-replicator tends to have the same problem, of accumulating every version of every package that has ever been installed, anywhere on the cluster.

Obviously distclean.py has an easier time, in that it can check machine inventory and then delete stale files from that same machine's disfiles. An idea tool would accept package lists from each client machine, as well as its own package list, and then clean packages out of the cache that are not associated with that combined list.

Does such a tool exist?

Assuming someone (me?) has to try tweaking distclean.py, can someone point me to a tutorial for special-purpose ssh key management? It seems to me that each client could generate its own package list, and use scp to move it to a dedicated directory/file, and the server would take it from there. To run this under cron, it seems a good idea to use ssh with dedicated-purpose keys.

[flybynite]

Is there a tool to clean up the http-replicator cache?

No dedicated tool for the reasons you thinking about. You would have to know what every client needs. Not only that, but you would have to choose from the many different ways of trimming the cache. Would you expire based on time, version in use, version in portage, newest version, etc,etc.

This is the reason I haven't tried to make my own cleaner. The basic cleaners such as distclean.py will work if you just point them to replicator cache. Others have used simple bash scripts to delete files based on age or keeping the cache to a max size and deleting the oldest files.

[RichieB]

My apologies, flybynite, for not replying sooner. Have been a touch ill. Your continued help is very much appreciated.

Yes, there are over 600 packages in /var/cache/http-replicator.

Is the chroot a complete filesystem? I mean is there a /var/cache/http-replicator in the host and the chroot or is there only one such dir on the box?

Yes, it's a complete filesystem, from the root down. It only shares the 'virtual' filesystems (/proc etc) with the non-chroot environment.

I find it confusing that if I set PORTAGE_BINHOST to http://localhost:8001/ then the emerge process kind of works - but rather than emerge binary builds it emerges source and tries to compile it. Resetting it to the HOWTO suggested http://localhost:8001/All/ produces the Not Found error. Interestingly, the http-replicator log shows:

The reason it works is because without the "ALL" your browsing replicators cache. So we know that dir is available to replicator. With the "ALL" your browsing the .../packages/All dir which replicator can't enter/list for some reason.

So replicator probably can't enter or list the packages dir. Check the permissions on the dir and also enter the chroot with a shell and try and list the dir.

drwxrwxr-x 2 root portage 2344 Oct 11 01:53 All (/usr/portages/packages/All)

And the files within are chown root:portage and chmod 755

How do you have the outbound net proxy setup. Is it ipchains or a ENV variable?

Show me the output of :

Code: Select all

env

server 'chroot' env:

Code: Select all

MANPATH=/usr/local/share/man:/usr/share/man:/usr/share/binutils-data/i686-pc-linux-gnu/2.16.1/man:/usr/share/gcc-data/i686-pc-linux-gnu/3.4.6/man::/opt/blackdown-jdk-1.4.2.03/man:/usr/qt/3/doc/man
SHELL=/bin/bash
TERM=linux
HUSHLOGIN=FALSE
OLDPWD=/
QTDIR=/usr/qt/3
USER=root
GDK_USE_XFT=1
PAGER=/usr/bin/less
CONFIG_PROTECT_MASK=/etc/terminfo /etc/revdep-rebuild
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/i686-pc-linux-gnu/gcc-bin/3.4.6:/opt/blackdown-jdk-1.4.2.03/bin:/opt/blackdown-jdk-1.4.2.03/jre/bin:/usr/kde/3.5/sbin:/usr/kde/3.5/bin:/usr/qt/3/bin
MAIL=/var/mail/root
PWD=/etc
JAVA_HOME=/opt/blackdown-jdk-1.4.2.03
EDITOR=/bin/nano
JAVAC=/opt/blackdown-jdk-1.4.2.03/bin/javac
QMAKESPEC=linux-g++
KDEDIRS=/usr
JDK_HOME=/opt/blackdown-jdk-1.4.2.03
SHLVL=3
HOME=/root
G_FILENAME_ENCODING=UTF-8
LESS=-R -M --shift 5
PYTHONPATH=/usr/lib/portage/pym
LOGNAME=root
CVS_RSH=ssh
GCC_SPECS=
CLASSPATH=.
LESSOPEN=|lesspipe.sh %s
INFOPATH=/usr/share/info:/usr/share/binutils-data/i686-pc-linux-gnu/2.16.1/info:/usr/share/gcc-data/i686-pc-linux-gnu/3.4.6/info
OPENGL_PROFILE=nvidia
G_BROKEN_FILENAMES=1
CONFIG_PROTECT=/usr/share/X11/xkb /usr/kde/3.5/share/config /usr/kde/3.5/env /usr/kde/3.5/shutdown /usr/share/config
_=/usr/bin/env

... and the 'client' non-chroot env:

Code: Select all

MANPATH=/usr/local/share/man:/usr/share/man:/usr/share/binutils-data/i686-pc-linux-gnu/2.16.1/man:/usr/share/gcc-data/i686-pc-linux-gnu/3.4.6/man::/opt/blackdown-jdk-1.4.2.03/man:/usr/qt/3/doc/man
SHELL=/bin/bash
TERM=linux
HUSHLOGIN=FALSE
QTDIR=/usr/qt/3
OLDPWD=/root
USER=root
GDK_USE_XFT=1
PAGER=/usr/bin/less
CONFIG_PROTECT_MASK=/etc/terminfo /etc/revdep-rebuild
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/i686-pc-linux-gnu/gcc-bin/3.4.6:/opt/blackdown-jdk-1.4.2.03/bin:/opt/blackdown-jdk-1.4.2.03/jre/bin:/usr/kde/3.5/sbin:/usr/kde/3.5/bin:/usr/qt/3/bin
MAIL=/var/mail/root
PWD=/etc
JAVA_HOME=/opt/blackdown-jdk-1.4.2.03
EDITOR=/bin/nano
JAVAC=/opt/blackdown-jdk-1.4.2.03/bin/javac
QMAKESPEC=linux-g++
KDEDIRS=/usr
JDK_HOME=/opt/blackdown-jdk-1.4.2.03
SHLVL=1
HOME=/root
G_FILENAME_ENCODING=UTF-8
LESS=-R -M --shift 5
PYTHONPATH=/usr/lib/portage/pym
LOGNAME=root
CVS_RSH=ssh
GCC_SPECS=
CLASSPATH=.
LESSOPEN=|lesspipe.sh %s
INFOPATH=/usr/share/info:/usr/share/binutils-data/i686-pc-linux-gnu/2.16.1/info:/usr/share/gcc-data/i686-pc-linux-gnu/3.4.6/info
OPENGL_PROFILE=nvidia
G_BROKEN_FILENAMES=1
CONFIG_PROTECT=/usr/share/X11/xkb /usr/kde/3.5/share/config /usr/kde/3.5/env /usr/kde/3.5/shutdown /usr/share/config
_=/usr/bin/env

[RichieB]

Hi! So sorry, but ...bump ?

[Underdone]

Under the client /etc/make.conf you have this

SYNC="rsync://localhost/gentoo"
HTTP_PROXY="http://localhost:8001"
PORTAGE_BINHOST="http://localhost:8001/All"

Shouldn't it be the server's ip address and not local host since this is the client?

[bunkacid]

Is there a tool to clean up the http-replicator cache?
...
Obviously distclean.py has an easier time

I've used distclean.py version 0.2 to clean out the downloaded sources before the eclean tool came along.

Code: Select all

# as root.
DISTDIR=/usr/portage/.http-replicator/ ./distclean.py

This works for me, obviously you will have to change the paths to where you keep your files.

You should be able to use the eclean utility which comes with gentoolkit-0.2.2

Unfortunately because eclean will not use the command line ENVIRONMENT VARIABLES as emerge does, you will have to change your make.conf temporarily to point to the location(s) you wish to clean. Or even hardcore the changes into the eclean utility, which I wouldn't recommend.

Perhaps we file a bug to allow for eclean to have the ability to read the ENVIRONMENT VARIABLES added, which would solve a lot.

...
Assuming someone (me?) has to try tweaking distclean.py, can someone point me to a tutorial for special-purpose ssh key management? It seems to me that each client could generate its own package list, and use scp to move it to a dedicated directory/file, and the server would take it from there. To run this under cron, it seems a good idea to use ssh with dedicated-purpose keys.

I'm not sure about the automatically updating a master server of all the packages that are being used throughout a network, but I would suggest you start with trying out the ~/.ssh/authorized_keys for some automation tasks
This is a good walkthrough on howto setup the keys.
http://geekpit.blogspot.com/2006/04/fiv ... e-ssh.html