HOWTO:Download Cache for your LAN-Http-Replicator (ver 3.0)

Message

hielvc · Post by **hielvc** » Fri Dec 22, 2006 7:49 pm

RichieB and any others who useing a chroot you have to point to your chroots packages directory not the servers package dirtectory unless thats where you are copying your packages to. My chroot is /home/gentoo and my packages dir for it is /home/gentoo/usr/packages ( dont ask me why I put it there ) so in my /etc/conf.d/http-replicator I have DAEMON_OPTS="$DAEMON_OPTS --alias /home/gentoo/usr/packages/All:All" and it works fine. That is when I can get portage to use it

naim · Post by **naim** » Sat Feb 10, 2007 12:33 pm

i keep getting this message, i checked, the partition i assigned to /var/cache/http-replicator is 50% full, with 5 Gb free.
/dev/sda7 11G 5.1G 5.2G 50% /var/cache/http-replicator
i switched to verbose logging and i found the following in the logs::

Traceback (most recent call last):
File "/usr/lib/python2.4/asyncore.py", line 69, in read
obj.handle_read_event()
File "/usr/lib/python2.4/asyncore.py", line 391, in handle_read_event
self.handle_read()
File "/usr/bin/http-replicator", line 160, in handle_read
self.data.seek(0) # move to beginning
IOError: [Errno 28] No space left on device

and on client side ::

Calculating dependencies... done!

>>> Emerging (1 of 4) dev-perl/Archive-Zip-1.16 to /
>>> Downloading 'http://192.168.20.251:8083/distfiles/Ar ... .16.tar.gz'
--14:17:52-- http://192.168.20.251:8083/distfiles/Ar ... .16.tar.gz
=> `/usr/portage/distfiles/Archive-Zip-1.16.tar.gz'
Connecting to 192.168.20.251:8083... connected.
Proxy request sent, awaiting response... No data received.
Retrying.

My /etc/conf.d/httpreplicator.conf

GENERAL_OPTS="--dir /var/cache/http-replicator"
GENERAL_OPTS="$GENERAL_OPTS --user portage"
DAEMON_OPTS="$GENERAL_OPTS"
DAEMON_OPTS="$DAEMON_OPTS --external proxy.network.company:8080"
DAEMON_OPTS="$DAEMON_OPTS --alias /usr/portage/packages/All:All"
DAEMON_OPTS="$DAEMON_OPTS --log /var/log/http-replicator.log"
DAEMON_OPTS="$DAEMON_OPTS --debug"
DAEMON_OPTS="$DAEMON_OPTS --ip 172.16.36.*"
DAEMON_OPTS="$DAEMON_OPTS --ip 192.168.20.*"
DAEMON_OPTS="$DAEMON_OPTS --port 8083"

Any ideas ??

bunkacid · Post by **bunkacid** » Sat Feb 10, 2007 4:49 pm

naim wrote:i keep getting this message, i checked, the partition i assigned to /var/cache/http-replicator is 50% full, with 5 Gb free.
/dev/sda7 11G 5.1G 5.2G 50% /var/cache/http-replicator
i switched to verbose logging and i found the following in the logs::

Any ideas ??

You've run out of space somewhere that http-replicator is outputting data/logs/information/etc to.
What are the results of;

Code: Select all

df -m

naim · Post by **naim** » Mon Feb 12, 2007 2:54 pm

oops, root partition full, i was looking the wrong way, thanks

dalek · Post by **dalek** » Sun Mar 04, 2007 7:15 am

I have been using this for a while but all of a sudden I get this:

Code: Select all

root@smoker / # repcacheman


Replicator's cache directory: /var/cache/http-replicator/
Portage's DISTDIR: /usr/portage/distfiles/

Comparing directories....
Done!

Deleting duplicate file(s) in /usr/portage/distfiles/
http-replicator_3.0.tar.gz
Done!

New files in DISTDIR:
seamonkey-1.1-patches-0.1.tar.bz2
mozilla-launcher-1.56.bz2
seamonkey-1.1.1.source.tar.bz2

Checking authenticity and integrity of new files...
Searching for ebuilds...
Done!

Found 22919 ebuilds.

Extracting the checksums....
Done!

Verifying checksum's....
/usr/portage/distfiles/seamonkey-1.1-patches-0.1.tar.bz2
Traceback (most recent call last):
  File "/usr/bin/repcacheman.py", line 203, in ?
    if t["MD5"]:
KeyError: 'MD5'
root@smoker / #

I tried reemerging it but I still get the same error. What is this? Is this me or something else?

Thanks

flybynite · Post by **flybynite** » Sun Mar 04, 2007 7:56 am

dalek wrote: /usr/portage/distfiles/seamonkey-1.1-patches-0.1.tar.bz2
KeyError: 'MD5'
[/code]

This type of error is probably caused by an error in the portage data. Wait a while, re-sync and see if it goes away.

dalek · Post by **dalek** » Sun Mar 04, 2007 6:30 pm

Hmmm, I should have mentioned this has been going on for a week or better. I have synced three or four times since this started.

I do seem to recall that portage and gentoolkit was upgraded recently. Could that have something to do with this? I may go back a version of gentoolkit and try it. I think that was the last one upgraded and was about the time this started.

Open to any ideas though. I really need to fix this so I can update my other rigs.

dalek · Post by **dalek** » Mon Mar 05, 2007 5:48 am

Well, I went back to a older version of portage and gentoolkit and still no joy. I also synced again.

Any more ideas? I need some.

dalek · Post by **dalek** » Tue Mar 06, 2007 7:57 am

OK. This fixed it:

Code: Select all

mv -v /usr/portage/distfiles/seamonkey-1.1-patches-0.1.tar.bz2 /root/

That will work until I need that patch again. Does nobody have a clue what is up with that? Is the patch corrupt or something?

Thanks

bunkacid · Post by **bunkacid** » Tue Mar 06, 2007 11:58 pm

I believe some of the portage Devs are making changes to the Manifest files which could explain the MD5 error.

dalek · Post by **dalek** » Wed Mar 07, 2007 2:51 am

Thanks for the info. I'm surprised no one else has posted they are getting the same error though. Maybe it will sort itself out pretty soon.

flybynite · Post by **flybynite** » Wed Mar 07, 2007 4:41 am

dalek wrote:I'm surprised no one else has posted they are getting the same error though.

repcacheman uses portage functions, which means this error could be specific to your portage system or setup.

1. Does this file belong to a local portage package? A package that you made yourself or copied from somewhere or maybe even from a portage overlay?

2. Since you upgraded portage lately, according to http://www.gentoo.org/proj/en/portage/d ... oblems.xml , it could be caused by a old version of portage or maybe an incomplete upgrade of portage:

Receiving
portage.db["/"]["porttree"].dbapi.auxdb[porttree_root][cat].clear()
KeyError: 'app-dicts'
while updating portage cache

* Bugreport: 100444
* Caused by: old portage version
* Solution: update portage and run emerge --sync again

I know you don't have this exact error, but I believe the KeyError seems to be the important part and is the error you reported

dalek · Post by **dalek** » Wed Mar 07, 2007 5:37 am

I went back to a older version of portage, just to see if it was a change made in portage that just kicked in. Same thing, even after doing a sync again. I have since went back to the new version of portage. I also did the same thing with gentoolkit just in case.

I read the link you posted, since the fix was to reemerge portage, I would hope that downgrading and upgrading would have taken care of that. The key word is hope though.

The file did come from emergeing Seamonkey. It was a recent upgrade. I don't even know how to do a overlay. After moving the file as mentioned above, I have not had the error since, even after syncing.

Still open to ideas though. I can move the file back if needed.

Thanks

bunkacid · Post by **bunkacid** » Wed Mar 07, 2007 5:54 am

Have you tried removing the offending file?

dalek · Post by **dalek** » Wed Mar 07, 2007 6:13 am

Yup, that was how I got rid of the error. I have ran into corrupt files before though and it would just delete them or not copy them over and go on. On this one, it made the CPU go to about 100%, sit there for a really loooong time and puke out that error. It has never took that long to run repcacheman before though. It just seemed that this file triggered something nasty.

edit: forgot the error part.

flybynite · Post by **flybynite** » Wed Mar 07, 2007 5:27 pm

Hmmm, seems I tried to duplicate the error and I can't even get the same file.

Did the dev's update the ebuild last night or are you using an old version of the patches so you haven't emerge sync'd??

I only have patches 1.1.1 ??

repcacheman

Replicator's cache directory: /var/cache/http-replicator/
Portage's DISTDIR: /var/tmp/distfiles/

Comparing directories....
Done!

Deleting duplicate file(s) in /var/tmp/distfiles/
(snip)
seamonkey-1.1.1-patches-0.1.tar.bz2

Done!

SUMMARY:
Found 113 duplicate file(s).
Deleted 113 dupe(s).

Done!

dalek · Post by **dalek** » Wed Mar 07, 2007 7:18 pm

You may be on to something. I did a emerge -f seamonkey and it redownloaded the patch, it worked fine when I ran repcacheman. I would have to guess that the old one is corrupt but how come portage used it to install Seamonkey and not notice that I wonder?

That is weird. I guess that is why it happened to me.

enkryptz · Post by **enkryptz** » Sun Mar 11, 2007 2:33 pm

Can anyone hook me up with the latest sources for http replicator?
The webpage seems to be inactive since a long time, there is some news about a 3.1 development version,
but I can only find 3.0 which is quite old.
Any public CVS or SVN repository available where I can follow the development?

dalek · Post by **dalek** » Sun Mar 11, 2007 10:51 pm

Try here:

http://cudlug.cudenver.edu/gentoo/distfiles/

and just look for the http-replicator tarball. It's long list but it's in there.

dahoste · Post by **dahoste** » Sat Mar 17, 2007 2:57 pm

Regarding dalek's 'MD5' error while running repcacheman -- you're not the only one! I *have* been having that exact problem, but it isn't with the seamonkey file, it's with all kinds of different files. At first, I just did what I usually do with an offending file in distfiles - delete it, but the problem seems pretty rampant right now (at least, with the set of stuff I've gotten lately in distfiles), so I'm looking at it as a more serious problem.

I haven't done any rebuilds of portage or gentoolkit. That is, as an attempt to fix the problem, I did get the latest portage and gentoolkit updates, but I think the MD5 problem was happening before that anyway. I've also done multiple syncs and 'emerge -uDav world' over the last week.

Here's hoping it goes away in the near future. I'm not sure what to try that might help.

hielvc · Post by **hielvc** » Sun Mar 18, 2007 1:01 am

Same here. I have had 2 offeneding packages. The second one was

Verifying checksum's....
/usr/portage/distfiles/eix-0.9.1.tar.bz2
Traceback (most recent call last):
File "/usr/bin/repcacheman.py", line 203, in ?
if t["MD5"]:
KeyError: 'MD5'

dalek · Post by **dalek** » Sun Mar 18, 2007 2:37 am

This is sort of funny. I keyworded and unmasked that eix package and did a fetch for it. I then ran repcacheman. I got a error but not for the eix package. I got this:

Code: Select all

Deleting duplicate file(s) in /usr/portage/distfiles/
eix-0.9.1.tar.bz2
java-config-wrapper-0.12.tar.bz2
javatoolkit-0.2.0.tar.bz2
java-config-1.3.7.tar.bz2
java-config-2.0.31.tar.bz2
Done!

New files in DISTDIR:
j2sdk-1_4_2-doc.zip
jdk-1_5_0-doc-r1.zip

Checking authenticity and integrity of new files...
Searching for ebuilds...
Done!

Found 22902 ebuilds.

Extracting the checksums....
Done!

Verifying checksum's....
/usr/portage/distfiles/j2sdk-1_4_2-doc.zip
Traceback (most recent call last):
  File "/usr/bin/repcacheman.py", line 203, in ?
    if t["MD5"]:
KeyError: 'MD5'
root@smoker / #

So mine was fine with eix where your wasn't. I synced last night to. Do you get the same error for j2sdk?

I'm hoping this will help them figure out if it is a local thing specific to us or if it is something else.

dahoste · Post by **dahoste** » Sun Mar 18, 2007 3:27 pm

Another bit of info: I just ran repcacheman on one of my other servers (which I hadn't done in awhile - maybe a week or two) and it plowed through everything in distfiles just fine. The interesting piece is that this server is actually the one that feeds packages to all of my LAN machines -- so it had the same set of files in distfiles that caused the 'MD5' error when running repcacheman on the other machine. I wonder if that indicates that perhaps the problem might originate when http-replicator serves package files. The two machines on which I have http-replicator running are as near identical as I can get (that's actually the purpose - one is a backup/test for the other), with the exception that the one (with no 'MD5' errors) sucks packages down from the internet, and the other (with 'MD5' errors) sucks packages down from the first machine.

Too busy right now to run targeted experiments to see if I can narrow down the circumstances surrounding the problem, but if I get a chance, I'll report back here.

flybynite · Post by **flybynite** » Mon Mar 19, 2007 4:27 am

enkryptz wrote:Can anyone hook me up with the latest sources for http replicator?

The current ebuild is the latest sources.

http://gertjan.freezope.org/replicator/ is the last known web page for Gertjan. He authored http-replicator specifically for debian and is not a gentoo user.

I worked with him to improve the code and make it gentoo friendly. I am the author of repcacheman and other gentoo specific code such as the ebuild and config's.

Gertjan, despite my best effort, wanted to completely rewrite the code from scratch rather than refactor and add much needed features. As far as I know, he never completed any updates.

Are you interested in hacking the code? Do you need some specific update?

flybynite · Post by **flybynite** » Mon Mar 19, 2007 6:19 am

dalek,dahoste,hielvc,

Congrats, you've found a BUG!

It seems there is a change in portage working its way into the portage tree somehow related to GLEP 44. It changes the way portage stores the checksums of files.

This wouldn't cause any problems by itself since repcacheman only calls the portage database functions, but it seems MD5's are either no longer going to be used or portage dev's are not bothering to add them to the new manifest format anymore.

I'll see what I can find out about the change in portage.

Thanks again!

HOWTO:Download Cache for your LAN-Http-Replicator (ver 3.0)

[Errno 28] No space left on device

Re: [Errno 28] No space left on device