HOWTO:Download Cache for your LAN-Http-Replicator (ver 3.0)

BlinkEye · Posted: Sat Oct 30, 2004 10:14 am Post subject:

i do have a problem: repcacheman runs every 5 minutes but even though it should have nothing to do (i.e. i didn't download any packages and /usr/portage/distfiles/ is empy) it eats a lot of my precious memory (my server is an oldie):

flybynite · l33t Joined: 06 Dec 2002 Posts: 620

The release of portage 2.0.51 has caused a number of problems.

These problems haven't affected how well http-replicator itself runs, only with the support script repcacheman.

I have updated ebuilds with changes to support portage 2.0.51. I need help testing:

http://www.updatedlinux.com/replicator/http-replicator-flybynite-1.6a.tar.bz2

This release contains 5 versions:
* net-misc/http-replicator :
[ ] 2.0-r2 (0) OVERLAY
[ ] 2.0-r3 (0) OVERLAY
[ ] 2.1_rc3 (0) OVERLAY
[ ] 2.1_rc3-r1 (0) OVERLAY
[ I] 2.1 (0) OVERLAY

2.0-r2 and 2.1_rc3 DEPEND on portage 2.0.50 and won't work with the current stable portage 2.0.51 I left these in only for the 1 or 2 gentooers who don't like to upgrade :-)

2.0-r3 , 2.1_rc3_r1 and 2.1 will work with any current portage version. 2.1 primarily adds external proxy support and improved logging.

2.1 is the latest and is stable, but the changes aren't fully documented in the howto yet.

Follow the howto at the start of this thread for all versions. Then If you upgraded to 2.1 -look at these changes:

**Note The config file location has changed in 2.1. Edit /etc/conf.d/http-replicator to add your external proxy and check other defaults.

**Note repcacheman has changed in 2.1. If you changed the default cache dir or the default user, you must call repcacheman with those options.

flybynite · l33t Joined: 06 Dec 2002 Posts: 620

flybynite · l33t Joined: 06 Dec 2002 Posts: 620

I had a question about repcacheman and what it does and how often it should be run.

Repcacheman only needs to run after emerge is run on the server ( except emerge sync)

Here is some more info on repcacheman:

meowsqueak · Posted: Mon Nov 01, 2004 4:33 am Post subject:

I upgraded to 2.1 but I'm getting this problem on one of my LANs. The following spews out about 20 times:

meowsqueak · Posted: Mon Nov 01, 2004 8:12 pm Post subject:

Ok, I did some reading into the 301 response:

http://www.checkupdown.com/status/E301.html

According to the output, wget first tries to download http://gentoo.oregonstate.edu/distfiles/fontconfig-2.2.3.tar.gz. The proxy request is sent and the response is a 301 error (presumably from the original server?). The Location specifies http://gentoo.osuosl.org/distfiles/fontconfig-2.2.3.tar.gz as the alternative URL, and wget follows this. But when it tries to download this, it gets another 301 with the location set to the exact same URL and the process repeats until the limit (20) is hit. So, is this a problem with this server in particular, or is it a problem with the http-replicator proxy? I suspect the server is fine because a manual wget of the osuosl.org file works fine (no followed location). Does http-replicator cache server responses or something?

flybynite · l33t Joined: 06 Dec 2002 Posts: 620

First, go ahead and update the link to http://gentoo.osuosl.org/ - since http://gentoo.oregonstate.edu/ is outdated anyway. I've updated the howto.

The response isn't cached by http-replicator- might be cached by your ISP or somewhere else.

Then if you still have this problem, upgrade to http-replicator 2.1, set debug in the config and post the logs...

meowsqueak · Posted: Wed Nov 03, 2004 8:26 pm Post subject:

Hmmm, I wonder if my squid proxy is caching something - here's the log:

flybynite · l33t Joined: 06 Dec 2002 Posts: 620

meowsqueak · Posted: Thu Nov 04, 2004 9:30 pm Post subject:

Maxwell · Tux's lil' helper Joined: 10 Dec 2003 Posts: 97

Well, i suppose i must be doing something stupid, but i need some help.

I installed http_replicator as i should, but as i have an http proxy port=8080, i switched replicator's port to 5000. But it looks like http_replicator isn't used by my notebook, in my internal lan. the value "http_proxy" in make.conf is the ip and door of the server, correct?

Both my /usr/portage/distfiles and my /var/log/http-replicator are empty. The log file also doesn't report anything when i emerge a file at the server.

Suggestions?
_________________
Freedom works. Use it!
Linux, by Gentoo

meowsqueak · Posted: Mon Nov 08, 2004 9:50 pm Post subject: Re: Problems using it

Maxwell · Tux's lil' helper Joined: 10 Dec 2003 Posts: 97

yes, i did start it

The output of lsof -i is that it is listening at port 5000.
But it doesn't get connected by any client...
_________________
Freedom works. Use it!
Linux, by Gentoo

flybynite · l33t Joined: 06 Dec 2002 Posts: 620

First, I need to see the client's emerge output. Copy the output showing where the file is downloaded.

Maxwell · Tux's lil' helper Joined: 10 Dec 2003 Posts: 97

Ok
In a client, when i emerge something, the download part output is:

flybynite · l33t Joined: 06 Dec 2002 Posts: 620

Maxwell · Tux's lil' helper Joined: 10 Dec 2003 Posts: 97

Ok, big mistake!
http_proxy now is in lower case. Now i can connect to http_replicator!! (yes!)
But i need a proxy to connet to net. How i set http_replicator to use it? I've already set it in http_replicator.conf and it didn't fetch anything.

Thanks in advance
_________________
Freedom works. Use it!
Linux, by Gentoo

meowsqueak · Posted: Wed Nov 10, 2004 10:37 pm Post subject:

In /etc/conf.d/http-replicator (not /etc/http-replicator.conf) you simply add a line that looks like this:

Maxwell · Tux's lil' helper Joined: 10 Dec 2003 Posts: 97

Now i've done it!

I was using an ultra old version. It didn't had /etc/conf.d/http_replicator.conf file. Now it works as it should. If i find any problems i will tell you guys!

Thank you all!!
_________________
Freedom works. Use it!
Linux, by Gentoo

johntramp · Guru Joined: 03 Feb 2004 Posts: 457 Location: New Zealand

Hi. I am trying to set up Http-Replicator with not much success. At the moment I am just trying with a client and a server. I am sure I have something wrong with the make.conf's as I was a little confused with the wording of the guide.

meowsqueak · Posted: Mon Nov 15, 2004 10:15 pm Post subject:

YourProxyHere.com:Port is the IP address or DNS name of your http-replicator server and the access port. You set this on all client machines that you want to use the replicator. You can set this on the http-replicator server too, so that it uses itself to download source.

Keep the mirrors as is, just make sure they are http mirrors. The client makes a request for a file from one of your mirrors via the http-replicator proxy which then downloads and caches the file (or retrieves it from the cache if it's already downloaded), as well as relaying the incoming file to the client.

I hope this makes more sense?

johntramp · Guru Joined: 03 Feb 2004 Posts: 457 Location: New Zealand

so there is no difference between the server and client's make.conf?

JayBee · Posted: Tue Nov 16, 2004 2:53 am Post subject:

Not sure if this has been covered in the thread - too long to wade through, and if so, could it be put near the front?

When I start an emerge from a client machine, I get a 401 not found error (from memory - away from the box at the moment), and it tries from all the mirrors it has listed. It appears that http-replicator is requesting the file on the server and downloading it, but the client isn't waiting to receive from the server.

Any ideas as to how to solve this?

Cheers
_________________
"There is no patch for stupidity"

meowsqueak · Posted: Tue Nov 16, 2004 3:31 am Post subject:

johntramp · Guru Joined: 03 Feb 2004 Posts: 457 Location: New Zealand

ah

I understand now, thanks meowsqueak.
Q: what will happen if I set this up but the computer with the daemon on is offline? Will my computer just emerge as it would with out the replicator? - also what will happen when it needs to use the ftp?