HOWTO:Download Cache for your LAN-Http-Replicator (ver 3.0)

Message

melinux · Post by **melinux** » Sat Feb 14, 2009 5:28 pm

flybynite wrote:I seem to remember fixing this a while ago, I guess it might be changing again? What version of replicator and portage are you running?

portage version 2.1.6.4

and http-replicator version 3.0-r1 (~x86)

Thanks,
melinux

flybynite · Post by **flybynite** » Fri Feb 20, 2009 1:18 am

melinux wrote: portage version 2.1.6.4

Portage switched from portage_manifest to portage.manifest in version 2.2-r6 or greater and repcacheman is fixed for that version. Now the change seems to have been backported to at least your portage version 2.1.6.4

Could you check to see if the current fix works for you to confirm this?

Just cp the file /usr/portage/net-proxy/http-replicator/files/http-replicator-3.0-repcacheman-0.44-r1 somewhere and run it instead of the system version.

Here is some code if you need it.

Code: Select all

mkdir /root/repcachemantest
cd /root/repcachemantest
cp  /usr/portage/net-proxy/http-replicator/files/http-replicator-3.0-repcacheman-0.44-r1 .
./http-replicator-3.0-repcacheman-0.44-r1

jongeek · Post by **jongeek** » Fri Mar 13, 2009 5:32 pm

flybynite wrote: Could you check to see if the current fix works for you to confirm this?

Just cp the file /usr/portage/net-proxy/http-replicator/files/http-replicator-3.0-repcacheman-0.44-r1 somewhere and run it instead of the system version.

I was having the same warnings with portage 2.1.6.7. I ran the repcacheman script as you described, and then it did not display the warnings.

melinux · Post by **melinux** » Fri Mar 13, 2009 6:34 pm

jongeek wrote:
flybynite wrote: Could you check to see if the current fix works for you to confirm this?

Just cp the file /usr/portage/net-proxy/http-replicator/files/http-replicator-3.0-repcacheman-0.44-r1 somewhere and run it instead of the system version.
I was having the same warnings with portage 2.1.6.7. I ran the repcacheman script as you described, and then it did not display the warnings.

At present I can't test that pc as its not working (hardware - overheating problems) right now...
I installed the current http-replicator and I have no problems as yet, on another machine. Thanks anyway.

Onkl · Post by **Onkl** » Tue Dec 01, 2009 9:24 am

I have set up http-replicator to also serve as PORTAGE_BINHOST. Works as advertised.

However, when I use

Code: Select all

quickpkg --include-config=y <package>

to make a package with my adapted settings to my LAN, the package gets permissions of 600, which http-replicator will not serve. Changing the permissions to 644 solves this, but it is a hassle that I forget to do most of the time.

Is there a way to make http-replicator also serve those files, or make quickpkg to set the correct permissions?

flybynite · Post by **flybynite** » Fri Dec 04, 2009 3:09 am

Onkl wrote:Is there a way to make http-replicator also serve those files, or make quickpkg to set the correct permissions?

Code: Select all

h2 ~ # quickpkg --help

  --umask=UMASK         umask used during package creation (default is 0077)

Quickpkg's default umask of 0077 denies all users except root from even reading the file. I can't explain why it does that. Replicator can't change or work around such drastic file permissions.

There must be many others who feel that default umask is too drastic because Quickpkg does give you the option to change the umask to a more normal 022.

Try

Code: Select all

h2 ~ # quickpkg --umask=022  nano
h2 ~ # ls -l /var/tmp/packages/All/nano*
-rw-r--r-- 1 root root 202145 2009-12-03 20:53 /var/tmp/packages/All/nano-2.1.10.tbz2

Now replicator can read the file and serve it to others.

You can make this easy by putting

Code: Select all

alias quickpkg='quickpkg --umask=022'

in your ~/.bashrc

Onkl · Post by **Onkl** » Fri Dec 04, 2009 8:46 am

Thanks a lot.

And since an "emerge -b" builds the package with 644 it is indeed strange that quickpkg is so restrictive.

sam_i_am · Post by **sam_i_am** » Sun Jun 20, 2010 10:31 pm

Hi,

Thanks for this great program. I'm using at home and at work without any issues except:

On one host (x86), the http-replicator doesn't start. It says "failed to start service". Log says "HttpReplicator started", but it seems to die. I enabled debug option, but no other error messages. However, on a different host (x86_64) on the same network, it works with the exact same config parameters.

Any tips on how to go about finding out why?

Other than the being x86, the failing host also runs apache on port 80 and 443, but port 8080 is clear.

Sam

knight77 · Post by **knight77** » Thu Sep 30, 2010 1:49 pm

Hi there.

The problem is still there (http-replicator-3.0-r2 stable in portage x86).

I filed a bug on it: http://bugs.gentoo.org/show_bug.cgi?id=339079

There is also another thread here about this problem, workaround included: http://forums.gentoo.org/viewtopic-t-787761.html

Havin_it · Post by **Havin_it** » Wed Mar 13, 2013 3:48 pm

Thought this might be worth a mention, even though it's a bit non-standard...

I've used http-replicator for years, and I've just realised I wasn't doing so in the prescribed way. I've never even been aware of repcacheman until today, in fact. I never liked the idea of the distfiles ending up duplicated on the server, so I do things this way:

Set http-replicator's cache dir to the same as my $DISTDIR
Local clients use the proxy, server machine does not

The upshot is that when clients fetch a distfile, it goes in (or is already in, and gets served from) the cache via http-replicator; when the server fetches, the files are saved in the same dir and http-replicator will serve these for local clients as well, even though it didn't fetch them personally.

This seems to have worked fine for me over the years, though I'd welcome any drawbacks to this approach that y'all veterans might be able to identify. Hope it helps someone.

depontius · Post by **depontius** » Wed Mar 13, 2013 4:59 pm

I had http-replicator fail to come up a week or two ago, also. The problem was that the lockfile is at /var/log/http-replicator/http-replicator.pid, and the portage user didn't have write access.

A while back Linux adopted a tmpfs-based /run directory, and /var/run became a symlink to /run. When it was in /var/run it was persistent, and the emerge process could chown /var/run/http-replicator to portage, and it would stick. Now that /run is tmpfs, it's recreated on every boot, and such things don't stick. The fix would be to do a chown against /run/http-replicator in the initscript, prior to changing to the portage userid.

The problem I have is that this doesn't universally fail, so I don't understand what's going on. A few days ago that system lost power, and when I powered it back up, I made a mental note about http-replicator, since I hadn't actually changed the initscript. Then I forgot. Upon seeing this thread, I checked that system and http-replicator is running, even though it doesn't seem that it should be. Moreover, now the lockfile is /run/http-replicator.pid, owned by root, instead of /run/http-replicator/http-replicator.pid, owned by portage.

On another system where I'm running http-replicator it's still on the deeper path. Looking into the initscripts, the locations I see for the lockfile are correct, yet they're all 3.0-r3 - I don't quite get it.

khayyam · Post by **khayyam** » Wed Mar 13, 2013 6:04 pm

depontius wrote:A while back Linux adopted a tmpfs-based /run directory, and /var/run became a symlink to /run. When it was in /var/run it was persistent, and the emerge process could chown /var/run/http-replicator to portage, and it would stick. Now that /run is tmpfs, it's recreated on every boot, and such things don't stick. The fix would be to do a chown against /run/http-replicator in the initscript, prior to changing to the portage userid.

depontius ... when this migration happend openrc intoduced an implimentation of systemd's tmpfiles.d. This implimentation is 100% compatable with the above linked manpage, though openrc doesn't create the configuration dir /etc/tmpfiles.d on install.

Code: Select all

# equery files =sys-apps/openrc-0.11.8 |grep tmpfiles
/etc/conf.d/tmpfiles
/etc/init.d/tmpfiles.setup
/lib/rc/sh/tmpfiles.sh
/usr/share/openrc/runlevels/boot/tmpfiles.setup

So, using the above, and setting tempfiles.setup in the runlevel, you should be able have /run/http-replicator created with the correct ownership, etc.

HTH & best ... khay

depontius · Post by **depontius** » Wed Mar 13, 2013 6:16 pm

Thanks for the info. I'd seen stuff on /etc/tmpfiles.d as a systemd thing, but didn't realize that OpenRC had it, as well.

What bothers me more is that right now is that I have 2 systems running http-replicator-3.0-r3, both x86. One has /run/http-replicator.pid and the other has /run/http-replicator/http-replicator.pid - in the initscript.

Havin_it · Post by **Havin_it** » Wed Mar 13, 2013 10:06 pm

Just wondering, is it possible/supported to run multiple instances of http-replicator? I've got another caching-proxying task it might be useful for, but I want to use a different cache dir for it.

Possible?

mbar · Post by **mbar** » Mon Mar 18, 2013 2:17 pm

Has anybody the same problem as me (~amd64)? http-replicator-4.0_alpha2

Code: Select all

/etc/init.d/http-replicator start
 * Caching service dependencies ...                                                                                                                                                     [ ok ]
 * Starting Http-Replicator ...
Traceback (most recent call last):
  File "/usr/bin/http-replicator", line 4, in <module>
    import Params, Request, Response, fiber, weakref
ImportError: No module named Params
 * start-stop-daemon: failed to start `/usr/bin/http-replicator'
 * Failed to start Http-Replicator                                                                                                                                                      [ !! ]
 * ERROR: http-replicator failed to start

ToeiRei · Post by **ToeiRei** » Tue Mar 19, 2013 7:46 am

same problem here. Trying to track it down.

Update 1:
Looks like a Namespace / Path problem to me as removing 'Params' did make it complain about the next missing module

Solved by downgrading to net-proxy/http-replicator-3.0-r3

TomWij · Post by **TomWij** » Wed Jun 05, 2013 1:33 pm

Please report bugs at Bugzilla, we don't keep track of bugs on the forum. Thanks.

+ 05 Jun 2013; Tom Wijsman <TomWij@gentoo.org>
+ +http-replicator-4.0_alpha2-r1.ebuild:
+ Revision bump, added missing Python modules. Fixes bug #472122 reported by
+ Russell Knighton.

jpc22 · Post by **jpc22** » Mon Dec 08, 2014 10:54 pm

ive read the whole thread and a lot of other documentation for hours and i cant get the damn thing to work

CLIENT make.conf

http_proxy="http://192.168.1.138:8080"
PORTAGE_BINHOST="192.168.1.138:8080/usr/portage/packages" ####(tried all possible variations, ip whitout port , / /usr /all... )

CLIENT ~ # emerge -g jfsutils
--2014-12-08 18:07:57-- http://192.168.1.138:8080/usr/portage/packages/Packages
Connecting to 192.168.1.138:8080... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

--2014-12-08 18:07:58-- (try: 2) http://192.168.1.138:8080/usr/portage/packages/Packages
Connecting to 192.168.1.138:8080... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

--2014-12-08 18:08:00-- (try: 3) http://192.168.1.138:8080/usr/portage/packages/Packages
Connecting to 192.168.1.138:8080... connected.
HTTP request sent, awaiting response... No data received.
Giving up.

Fetcher exited with a failure condition.

!!! Error fetching binhost package info from '192.168.1.138:8080/usr/portage/packages'
!!! FETCHCOMMAND_192.168.1.138 failed

Havin_it · Post by **Havin_it** » Tue Dec 09, 2014 12:39 pm

It appears you're trying to use http-replicator as a regular HTTP server to serve up your binpkgs, which won't work: it is only a proxy, and does nothing unless you have a proper server running to serve the files from. Http-replicator simply caches a copy of the file as you download it from the proper server (or serves its cached copy if it's already present). Either way, you need to be able to fetch the file without http-replicator, or you will not be able to do so with it.

So you need to install apache, lighttpd or another http server and configure it to serve your binpkgs. And if that server is on your LAN, then there's nothing to gain by using http-replicator, and you'll just end up with two copies of every file on your binhost: one wherever your http server is serving it from, and one in the cache.

flybynite · Post by **flybynite** » Tue Apr 26, 2016 10:34 pm

I know this is an old post, but the previous post has been possibly misleading new users for too long already.

It appears you're trying to use http-replicator as a regular HTTP server to serve up your binpkgs, which won't work: it is only a proxy, and does nothing unless you have a proper server

This is incorrect - http-replicator will serve binpkgs by itself and doesn't require installing apache,lighttpd, or any other http server. It is way more than just a simple proxy.

http-replicator continues to work just fine and I'm glad gentoo has kept it updated.

Thistled · Post by **Thistled** » Sun Feb 02, 2020 8:21 pm

This is really beginning to annoy me now.
I'm beginning to think the new layout for portage, i.e /var/db/repos/gentoo and /var/cache/distfiles
is breaking the replicator.

Code: Select all

 000C   Accepted request from [192.168.1.2]:34974
  000C   Waiting at 16: RECV(4,20:03:29)
  000C   Client sends GET /distfiles/layout.conf HTTP/1.1
  000C   Error: invalid url: /distfiles/layout.conf
[ IDLE ] Sun Feb  2 20:03:14 2020
[ BUSY ] Sun Feb  2 20:03:15 2020
[ 000D ] Sun Feb  2 20:03:15 2020
  000D   Accepted request from [192.168.1.2]:34976
  000D   Waiting at 16: RECV(4,20:03:30)
  000D   Client sends GET /distfiles/layout.conf HTTP/1.1
  000D   Error: invalid url: /distfiles/layout.conf
[ IDLE ] Sun Feb  2 20:03:15 2020
[ BUSY ] Sun Feb  2 20:03:17 2020
[ 000E ] Sun Feb  2 20:03:17 2020
  000E   Accepted request from [192.168.1.2]:34978
  000E   Waiting at 16: RECV(4,20:03:32)
  000E   Client sends GET /distfiles/layout.conf HTTP/1.1
  000E   Error: invalid url: /distfiles/layout.conf
[ IDLE ] Sun Feb  2 20:03:17 2020
[ BUSY ] Sun Feb  2 20:03:17 2020
[ 000F ] Sun Feb  2 20:03:17 2020
  000F   Accepted request from [192.168.1.2]:34980
  000F   Waiting at 16: RECV(4,20:03:32)
  000F   Client sends GET /distfiles/libgweather-3.32.2.tar.xz HTTP/1.1
  000F   Error: invalid url: /distfiles/libgweather-3.32.2.tar.xz
[ IDLE ] Sun Feb  2 20:03:17 2020

The above is what I am seeing in my logs.

This is the output of emerge:

Code: Select all

>>> Emerging (1 of 1) dev-libs/libgweather-3.32.2-r1::gentoo
>>> Downloading 'http://pig2:8080/distfiles/layout.conf'
--2020-02-02 20:03:14--  http://pig2:8080/distfiles/layout.conf
Resolving pig2... 192.168.1.4
Connecting to pig2|192.168.1.4|:8080... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

--2020-02-02 20:03:15--  (try: 2)  http://pig2:8080/distfiles/layout.conf
Connecting to pig2|192.168.1.4|:8080... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

--2020-02-02 20:03:17--  (try: 3)  http://pig2:8080/distfiles/layout.conf
Connecting to pig2|192.168.1.4|:8080... connected.
HTTP request sent, awaiting response... No data received.
Giving up.

!!! Couldn't download '.layout.conf.pig2'. Aborting.
>>> Downloading 'http://pig2:8080/distfiles/libgweather-3.32.2.tar.xz'
--2020-02-02 20:03:17--  http://pig2:8080/distfiles/libgweather-3.32.2.tar.xz
Resolving pig2... 192.168.1.4
Connecting to pig2|192.168.1.4|:8080... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

It finally pulls in libgweather from a mirror hosting gnome packages.

The thing is, libgweather definitely resides in /var/cache/distfiles, and it is also in /var/cache/http-replicator.

I think the version 4 alpha of replicator has changed, as some of the properties in the config file (/etc/conf.d/http-replicator) no longer work.

My setup...

/var/db/repos/gentoo is an NFS share on my main server, all gentoo clients use this share.
I'm trying to get each client to request a distfile from http-replicator on the server, but the replicator is
suggesting the requested url is invalid.

Any idea?

Gatak · Post by **Gatak** » Mon Feb 03, 2020 10:51 am

I did an alternative setup. I exported my distfiles over samba with a password so that each client can fetch and update the central repository. This also works with the portage tree itself, if you like. This way only one client needs to run six-sync, and all others just do eix-update.

Thistled · Post by **Thistled** » Mon Feb 03, 2020 5:45 pm

Gatak wrote:I did an alternative setup. I exported my distfiles over samba with a password so that each client can fetch and update the central repository. This also works with the portage tree itself, if you like. This way only one client needs to run six-sync, and all others just do eix-update.

Yes, mines was very similar.

The NFS server hosted/shared /usr/portage and /usr/portage/distfiles. This was mounted as /mnt/nfs_portage on all clients.
The server sync'ed using rsync against mirrors.
Therefore the clients just had to eix-update and their databases were updated.

All clients could install / update packages without issue. When a package is called it would be downloaded into the distfiles folder.
I considered the shared /usr/portage/distfiles folder to be the central repository of source files for all clients on the network.
With this approach, I was saving considerable disk space on each client, as there was no need to store distfiles locally.

The problem I now have is the migration away from /usr/portage and /usr/portage/distfiles, to /var/db/repos/gentoo and /var/cache/distfiles respectively.
This is where I now have an opportunity to move the distfiles away from the server share, and hopefully use http-replicator to dispatch the packages to each client when needed.

The server is still hosting /usr/portage > /var/db/repos/gentoo to all clients who continue to access via /mnt/nfs_portage.
The server does the syncing as usual to the mirrors, and all the clients need to do is eix-update.

The problem lies with the clients requesting source files (distfiles) via http-replicator. The replicator will not dispatch the file, even though it exists.

Here is what I see from the http-replicator log:

Code: Select all

 [ IDLE ] Mon Feb  3 16:58:02 2020
[ BUSY ] Mon Feb  3 16:58:02 2020
[ 0001 ] Mon Feb  3 16:58:02 2020
  0001   Accepted request from [192.168.1.2]:37812
  0001   Waiting at 16: RECV(4,16:58:17)
  0001   Client sends GET http://pig2:8080/distfiles/libgweather-3.32.2.tar.xz HTTP/1.1
  0001   Switching to HttpProtocol
  0001   Cache position: libgweather-3.32.2.tar.xz
  0001   Requesting address info for pig2:8080
  0001   Connecting to [127.0.0.1]:8080
  0001   Waiting at 36: SEND(5,16:58:17)
[ 0002 ] Mon Feb  3 16:58:02 2020
  0001   Waiting at 39: RECV(5,16:58:17)
  0002   Accepted request from [127.0.0.1]:49830
  0002   Waiting at 16: RECV(6,16:58:17)
  0002   Client sends GET /distfiles/libgweather-3.32.2.tar.xz HTTP/1.1
  0002   Error: invalid url: /distfiles/libgweather-3.32.2.tar.xz
  0001   Switching to ExceptionResponse
  0001   Traceback (most recent call last):
  0001     File "/usr/lib/python-exec/python2.7/http-replicator", line 40, in Replicator
  0001       protocol.recv( server )
  0001     File "/usr/lib/python2.7/site-packages/Protocol.py", line 149, in recv
  0001       assert chunk, 'server closed connection before sending a complete message header'
  0001   AssertionError: server closed connection before sending a complete message header
  0001   Waiting at 52: SEND(4,16:58:17)
  0001   Transaction successfully completed
[ IDLE ] Mon Feb  3 16:58:02 2020

I get the impression python is borking out because a request is being made for a distfiles folder at the specified URL. ??
Is this because of the new behaviour portage has adopted. i.e. layout.conf, distfiles, __download__ etc etc?
If so, then http-replicator should be marked as unstable, as it is not working.

Does anyone actually have http-replicator working out of the box today?
I do observe this particular forum post has gone very quiet, is it because the replicator just doesn't work now and is no longer maintained?

unheatedgarage · Post by **unheatedgarage** » Fri Apr 03, 2020 7:01 pm

Thistled wrote:Does anyone actually have http-replicator working out of the box today?
I do observe this particular forum post has gone very quiet, is it because the replicator just doesn't work now and is no longer maintained?

It's a real shame The Replicator doesn't work anymore.

I'm running systemd and it hasn't been able to start for...maybe a year now? Anyway, I removed it a long time ago, and will be looking into using Samba or Squid in the future, unless someone steps up and reignites it.

It's a beautiful program that worked flawlessly for a long, long time, and I miss it.

Let's all keep being good netizens!

HOWTO:Download Cache for your LAN-Http-Replicator (ver 3.0)

same thing here

Re: same thing here

service doesn't start on one host

Re: service doesn't start on one host