View previous topic :: View next topic |
Author |
Message |
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Mon Oct 18, 2004 7:08 pm Post subject: |
|
|
cdunham,
Ok, I guess the mirror you sync to must be using some program that leaves the .locks dir. Do you sync to a specific mirror or rotation? I would like to sync there myself to see what else might be in there...
It Seems there is nothing special about your libsoup file error. It's just failing on the first file.....
cdunham wrote: | Just a little more information....
changing t[0] to t['MD5'] did seem to fix things... |
Ah yes, an always welcome API change in unstable portage Shame on me for not asking which portage version your using, you obviously run ~
Relavent part of diff between stable 2.0.50-r11 and unstable portage-2.0.51-rc9
Code: |
< if (md5_list[mybn][0] != md5sums[mybn][0])
---
> if (md5_list[mybn]["MD5"] != md5sums[mybn]["MD5"])
|
So, please edit repcacheman and change t[0] to t["MD5"] to confirm that this works.
If so I'll update http-replicator's ebuild to include an unstable version for unstable portage.
cdunham, thanks for the help fixing this!!
Last edited by flybynite on Fri Oct 29, 2004 7:56 pm; edited 1 time in total |
|
Back to top |
|
|
cdunham Apprentice
Joined: 06 Jun 2003 Posts: 211 Location: Rhode Island
|
Posted: Tue Oct 19, 2004 6:01 am Post subject: |
|
|
Hey, that seems to have done it.
Thanks, flybynite! _________________ This post more meaningful in a scalar context. |
|
Back to top |
|
|
VinnieNZ Tux's lil' helper
Joined: 11 Mar 2004 Posts: 126 Location: New Zealand
|
Posted: Fri Oct 22, 2004 2:40 am Post subject: |
|
|
I've just upgraded to the latest version of portage (2.0.51-r2) and when running repcacheman I get the following error:
Quote: | *SNIP*
Extracting the checksums....
Done!
Verifying checksum's....
/usr/portage/distfiles/linux-2.6.9.tar.bz2
Traceback (most recent call last):
File "/usr/bin/repcacheman", line 198, in ?
if t[0]:
KeyError: 0 |
Are there any fixes for this?
*EDIT Just read the post 2 above this one and realised that this looks like the likely solution. Problem is that I don't have any lines that look like the first set in my /usr/bin/repcacheman file.
Last edited by VinnieNZ on Wed Apr 25, 2007 11:31 pm; edited 1 time in total |
|
Back to top |
|
|
JohnHerdy n00b
Joined: 11 Jul 2002 Posts: 61
|
Posted: Fri Oct 22, 2004 8:43 am Post subject: |
|
|
flybynite wrote: | If so I'll update http-replicator's ebuild to include an unstable version for unstable portage. |
Hi flybynite,
We are using http-replicator on our network and it just works great. I want to thank you for the time and effort you put in this great program. Maybe the readers on this thread need to give more feedback on the bug-report so that this great program is added to the tree.
Portage 2.0.51 is marked stable so the current version doesn't work anymore. Fortunately you already have found a solution to this problem. Would you please let us know when you have a new ebuild ready.
<shameless plug> If it possible I would like to make a feature request; provide an entry in the log to see which IP-adresses have used the http-replicator for what file</shameless plug> |
|
Back to top |
|
|
kevquinn Retired Dev
Joined: 27 Jun 2003 Posts: 52
|
Posted: Sat Oct 23, 2004 1:29 pm Post subject: warning, for portage upgraders |
|
|
Don't know if it's just me, but when I upgraded to portage 2.0.51 (since it's now gone stable), my cache directory was automatically deleted, presumably by portage
I'd placed it at /usr/portage/distcache. My /var partition is 8Gb in size, and the replicator cache got rather large, enough to cause larger ebuilds to bomb due to lack of space in /var/tmp/portage. I've now changed it to /usr/cache/distfiles. Of course, now I've lost all my downloads - I knew I should have backed it up to a DVD-R!
So the warning is, don't put your cache directory in /usr/portage! I guess it was a stupid thing to do anyway, in hindsight... |
|
Back to top |
|
|
Bob Paddock n00b
Joined: 25 Aug 2002 Posts: 49
|
Posted: Sun Oct 24, 2004 1:55 pm Post subject: Not fetching if not in cache. What is the latest version? |
|
|
I'm having trouble getting http-replicator to work consistently. I can fetch from my laptop to get files from the desktop machine if the package is in the cache. However if the package
is not in the cache then the desktop machine does not go out on Internet and get it, and the
laptop just says "waiting..." forever.
I suspect I have a mixture of versions for example, from page four of this thread:
"Edit /etc/conf.d/http-replicator to add your external proxy and check other defaults."
but I find it as /etc/http-replicator.
Are there *CURRENT* install instructions for the *LATESTS* version? I've tried to
piece that together from the long thread here, without success.
Is there an ebuild that works for the *LATESTS* version? |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Sun Oct 24, 2004 3:27 pm Post subject: Re: Not fetching if not in cache. What is the latest version |
|
|
Bob Paddock wrote: |
Are there *CURRENT* install instructions for the *LATESTS* version? |
How are you defining current? I defined STABLE and UNSTABLE. Each version has it own instructions.
The howto at the start of this thread is the "stable" version and it ONLY has a download link for http://www.updatedlinux.com/replicator/http-replicator-flybynite-1.3.tar.bz2
and complete instructions.
The "unstable" version post on page 4 ( https://forums.gentoo.org/viewtopic.php?t=173226&start=76 ) is the only place you can find a download link for http://www.updatedlinux.com/replicator/http-replicator-flybynite-1.5.tar.bz2 and says this which I thought would be clear:
Quote: |
New version ready for testing!! Although the init code,logging, and options are cleaned up, this version primarily adds external proxy support.
Current users not needing external proxy support should wait for the next stable release. Experienced users can help test.
|
If that doesn't make sense to you then
Code: |
etcat -v http-replicator
|
Which will show you your version.
Code: |
* net-misc/http-replicator :
[ ] 2.0-r2 (0) OVERLAY
[ I] 2.1_rc3 (0) OVERLAY
|
If you have 2.0-r2 then read only the HOWTO
If you have 2.1-rc3 then read the HOWTO PLUS the changes in the "unstable version" post at https://forums.gentoo.org/viewtopic.php?t=173226&start=76 |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Sun Oct 24, 2004 3:58 pm Post subject: Re: warning, for portage upgraders |
|
|
kevquinn wrote: | Don't know if it's just me, but when I upgraded to portage 2.0.51 (since it's now gone stable), my cache directory was automatically deleted, presumably by portage
|
I confirmed this. Portage actually deleted /usr/portage/distcache when you ran emerge sync.
It makes sense, really. "emerge sync" makes /usr/portage on your box the same as the /usr/portage on the server. Since /usr/portage/distcache doesn't exist on the server, rsync deleted it the same way it would delete an outdated package.
Just to make this clear to all users, this had nothing to do with http-replicator, just a normal function of an emerge sync. The lesson is portage owns /usr/portage and don't put anything there. |
|
Back to top |
|
|
acaito_con n00b
Joined: 27 Aug 2004 Posts: 4
|
Posted: Tue Oct 26, 2004 4:45 am Post subject: Glad i found the help |
|
|
cdunham wrote: |
However, changing t[0] to t['MD5'] did seem to fix things... |
I did have to change this reference on both line 198 and 201, may help others as well.
Glad i came across this post, i'd already resync'd, updated python. and viewed the source for obvious errors, wasnt famliar w/ the MD5 lib.
glad this works, great tool
btw. you may want to stop by the gentoo-wiki might need some slight update |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Tue Oct 26, 2004 7:38 am Post subject: |
|
|
JohnHerdy wrote: |
<shameless plug> If it possible I would like to make a feature request; provide an entry in the log to see which IP-adresses have used the http-replicator for what file</shameless plug> |
Already included in 2.1_rc3 , Posted: Wed Aug 04 2004! at https://forums.gentoo.org/viewtopic.php?t=173226&start=76
It's part of the stats package that is coming. You may need to parse a couple lines to get the complete info you want. But you probably just need to wait for that stats package in the next version of http-replicator.
Here is just part of what is logged in that version:
Code: |
24 Oct 2004 19:09:23 DEBUG: HttpClient 161 connected to 192.168.2.123:34224
24 Oct 2004 19:09:23 INFO: HttpClient 161 received request for http://gentoo.osuosl.org/distfiles/prelink-20040707.tar.bz2
24 Oct 2004 19:09:23 STAT: HttpServer 161 serving 902324 bytes from cache to 192.168.2.123
|
|
|
Back to top |
|
|
JohnHerdy n00b
Joined: 11 Jul 2002 Posts: 61
|
Posted: Tue Oct 26, 2004 10:03 am Post subject: |
|
|
flybynite wrote: | JohnHerdy wrote: |
<shameless plug> If it possible I would like to make a feature request; provide an entry in the log to see which IP-adresses have used the http-replicator for what file</shameless plug> |
Already included in 2.1_rc3 |
Man I love you!!! (or at least your work). Sorry for asking a stupid question, but I couldn't find it on your website; is the latest version available on your website compliant with 2.0.51-clients? |
|
Back to top |
|
|
kmarasco n00b
Joined: 25 Jul 2003 Posts: 65 Location: Fernandina Beach, FL
|
Posted: Tue Oct 26, 2004 10:21 am Post subject: What is the Purpose of the -O mod to the resume statement? |
|
|
I added
Code: | RESUMECOMMAND="/usr/bin/wget -t 5 --passive-ftp \${URI} -O \${DISTDIR}/\${FILE}" |
to my make,conf, and http replicator worked fine. However, I am serving binary packages from Apache from the same server, and my emerge -gK fails with that setting.
Code: | kristel root # emerge -gK beaver
Fetching binary packages info...
Loaded metadata pickle.
cache miss: 'x' --- cache hit: 'o'
xoo
-- DONE!
Calculating dependencies ...done!
>>> emerge (1 of 1) app-editors/beaver-0.2.6 to /
Fetching 'app-editors/beaver-0.2.6'
--01:07:23-- http://192.168.2.2/portage/packages/All/beaver-0.2.6.tbz2
=> `/usr/portage/packages/All//${FILE}'
Connecting to 192.168.2.2:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 144,983 [text/plain]
100%[=================================================================================>] 144,983 --.--K/s
01:07:23 (1.02 MB/s) - `/usr/portage/packages/All//${FILE}' saved [144983/144983]
!!! CATEGORY info missing from info chunk, aborting...
kristel root # |
Note that it actually uses the variable name as the name of the file, when trying to save the binary to the package directory.
If I use this syntax for the resume command:
Code: | RESUMECOMMAND="/usr/bin/wget -t 5 --passive-ftp \${URI} -P \${DISTDIR}" |
Both http-replicator and my binary pull seem to work fine, but I am concerned that I may be creating a problem for http-replicator that I am unaware, and that may bite me later.
Is the second syntax ok with http-replicator? I'm using portage portage-2.0.51-r2. |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Tue Oct 26, 2004 4:20 pm Post subject: |
|
|
JohnHerdy wrote: |
Sorry for asking a stupid question, but I couldn't find it on your website; is the latest version available on your website compliant with 2.0.51-clients? |
Sorry, the website is out of date, There are two versions out, both are only listed or linked in the howto at the start of this thread.
All versions of http-replicator will work with the latest portage. But, repcacheman will barf with an error on >=portage-2.0.51. repcacheman errors will not affect the operation of http-replicator.
You can download a new repcacheman here:
http://www.updatedlinux.com/replicator/portagefix/repcacheman
You'll need to copy it to /usr/bin/repcacheman and chmod +x
Or, just edit 2 lines on your existing /usr/bin/repcacheman. Change t[0] to t["MD5"]
I'll have the fix in the ebuild soon.....
Last edited by flybynite on Fri Oct 29, 2004 7:53 pm; edited 1 time in total |
|
Back to top |
|
|
JohnHerdy n00b
Joined: 11 Jul 2002 Posts: 61
|
Posted: Tue Oct 26, 2004 5:58 pm Post subject: |
|
|
flybynite wrote: | All versions of http-replicator will work with the latest portage |
Hi flybynite,
Unfortunately this is not the case. 100% of the 2.0.51-clients don't work, 100% of the 2.0.50-clients work. The moment I migrate a client to 2.0.51 http-replicator doesn't work anymore. In our configuration we use a proxy-server (with validation). It seems that something is going wrong with the connection to the proxy. Our proxy resolves the ip-address for the mirrors but when I use http-replicator on a 2.0.51-client the proxy always returns unknown host. When I do a emerge-webrsync on a 2.0.51-client the tarball is fetched from the http-replicator server.
Thanks,
John. |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Tue Oct 26, 2004 8:34 pm Post subject: Re: What is the Purpose of the -O mod to the resume statemen |
|
|
kmarasco wrote: |
Is the second syntax ok with http-replicator? I'm using portage portage-2.0.51-r2. |
It's going to be a long day
This should fix it...
Code: |
rm /usr/portage/packages/app-editors/beaver-0.2.6.tbz2
rm /usr/portage/packages/All/beaver-0.2.6.tbz2
|
Then try the replicator syntax again. This will help me figure out where the problem is.
Http-replicator doesn't support resuming yet. This syntax disables resuming and forces wget to overwrite the partial file. |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Tue Oct 26, 2004 8:54 pm Post subject: |
|
|
JohnHerdy wrote: |
Unfortunately this is not the case. 100% of the 2.0.51-clients don't work |
2.0.51 works for me
I need some output and logs before I can even guess at whats wrong.... |
|
Back to top |
|
|
JohnHerdy n00b
Joined: 11 Jul 2002 Posts: 61
|
Posted: Tue Oct 26, 2004 9:42 pm Post subject: |
|
|
Ok did some more research, it seems that the new version of portage doesn't respect my .wgetrc-settings anymore. After adding the following lines to /etc/make.conf 2.0.51-clients work as well;
http_proxy="http://YourMirrorHere.com:8080"
RESUMECOMMAND=" /usr/bin/wget -t 5 --passive-ftp \${URI} -O \${DISTDIR}/\${FILE}"
With 2.0.50 this wasn't neccesary the proxy-settings where read from .wgetrc. Hmmm, strange... |
|
Back to top |
|
|
kmarasco n00b
Joined: 25 Jul 2003 Posts: 65 Location: Fernandina Beach, FL
|
Posted: Tue Oct 26, 2004 10:10 pm Post subject: Re: What is the Purpose of the -O mod to the resume statemen |
|
|
flybynite wrote: |
This should fix it...
Code: |
rm /usr/portage/packages/app-editors/beaver-0.2.6.tbz2
rm /usr/portage/packages/All/beaver-0.2.6.tbz2
|
Then try the replicator syntax again. This will help me figure out where the problem is. |
Below code after deleting the files from both the client and the server.
Code: | kristel root # emerge -gK beaver
Fetching binary packages info...
Loaded metadata pickle.
cache miss: 'x' --- cache hit: 'o'
ooo
-- DONE!
Calculating dependencies ...done!
>>> emerge (1 of 1) app-editors/beaver-0.2.6 to /
Fetching 'app-editors/beaver-0.2.6'
--12:29:48-- http://192.168.2.2/portage/packages/All/beaver-0.2.6.tbz2
=> `/usr/portage/packages/All//${FILE}'
Connecting to 192.168.2.2:80... connected.
HTTP request sent, awaiting response... 404 Not Found
12:29:48 ERROR 404: Not Found.
Fetcher exited with a failure condition.
!!! CATEGORY info missing from info chunk, aborting...
kristel root #
|
If I recreate the binary package, I get back to the originally posted error.
Note: I had ethereal observe the traffic and the http-replicator proxy does not appear to get involved with the locally served binaries from apache. I'm a rookie at ethereal, so I may have missed something, but it appears that when merging binaries using the PORTAGE_BINHOST variable in make.conf that the http_proxy variable is ignored (only for emerge -gK. The proxy is definitely used when source packages are pulled.). |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Tue Oct 26, 2004 11:13 pm Post subject: Re: What is the Purpose of the -O mod to the resume statemen |
|
|
kmarasco wrote: |
If I recreate the binary package, I get back to the originally posted error.
|
Thanks, that confirms some things for me.
You don't have FETCHCOMMAND set do you?
kmarasco wrote: |
Note: I had ethereal observe the traffic and the http-replicator proxy does not appear to get involved with the locally served binaries from apache.
|
Http-replicator isn't involved at all in this transaction. However RESUMECOMMAND seems to be and that is the problem. I find it "interesting" that portage uses RESUMECOMMAND when there is nothing to resume, I've filed a portage bug to get more info.
I use PORTAGE_BINHOST also, but with an ftp mirror. If you need a workaround, use an FTP server as your BINHOST till I get this worked out. |
|
Back to top |
|
|
kmarasco n00b
Joined: 25 Jul 2003 Posts: 65 Location: Fernandina Beach, FL
|
Posted: Wed Oct 27, 2004 9:21 am Post subject: Re: What is the Purpose of the -O mod to the resume statemen |
|
|
flybynite wrote: |
I use PORTAGE_BINHOST also, but with an ftp mirror. If you need a workaround, use an FTP server as your BINHOST till I get this worked out. |
I set up vsftp and I have the same issue.
Code: |
kristel root # emerge -gK beaver
Fetching binary packages info...
* No password provided for username 'anonymous'
Loaded metadata pickle.
cache miss: 'x' --- cache hit: 'o'
oooo
-- DONE!
Calculating dependencies ...done!
>>> emerge (1 of 1) app-editors/beaver-0.2.6 to /
Fetching 'app-editors/beaver-0.2.6'
--23:29:42-- ftp://192.168.2.2/All/beaver-0.2.6.tbz2
=> `/usr/portage/packages/All//${FILE}'
Connecting to 192.168.2.2:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD /All ... done.
==> PASV ... done. ==> RETR beaver-0.2.6.tbz2 ... done.
Length: 144,983 (unauthoritative)
100%[=================================================================================>] 144,983 107.91K/s
23:29:43 (107.85 KB/s) - `/usr/portage/packages/All//${FILE}' saved [144983]
!!! CATEGORY info missing from info chunk, aborting...
kristel root # |
Hmmm, any more thoughts? Typo/poor attention to detail on my part?
Code: | RESUMECOMMAND="/usr/bin/wget -t 5 --passive-ftp \${URI} -O \${DISTDIR}/\${FILE}" | [/quote] |
|
Back to top |
|
|
kmarasco n00b
Joined: 25 Jul 2003 Posts: 65 Location: Fernandina Beach, FL
|
Posted: Thu Oct 28, 2004 10:24 am Post subject: Alternative Resume Syntax |
|
|
Thanks for your help flybynite. Since I experienced the bug with both ftp and http, I tried to find an alternative syntax for the resume command. I think that the following syntax should have the same effect, but does not require that the file name variable be passed in order to replace a partial file. It is as follows:
Code: | RESUMECOMMAND="/usr/bin/wget -N -t 5 --passive-ftp \${URI} -P \${DISTDIR}" |
With the -N option, Only new files will be downloaded in the place of the old ones. Using -N, a file is considered new if one of these two conditions are met:
1. A file of that name does not already exist locally.
2. A file of that name does exist, but the remote file was modified more recently than the local file.
More importantly, if the local file does not exist, or the sizes of the files do not match, Wget will download the remote file no matter what the time-stamps say. |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
Posted: Fri Oct 29, 2004 12:42 am Post subject: Re: What is the Purpose of the -O mod to the resume statemen |
|
|
kmarasco wrote: | I set up vsftp and I have the same issue.
Typo/poor attention to detail on my part?
|
Nope, my part, sorry. I was using ftp without error on one of my boxes - because I had changed the RESUMECOMMAND so many times in testing that I left it wrong. The portage bug affects both ftp and http.
kmarasco wrote: |
I tried to find an alternative syntax for the resume command. I think that the following syntax should have the same effect, but does not require that the file name variable be passed in order to replace a partial file.
|
I appreciate the help. Wget interacting with portage and RESUMECOMMAND together had so many options finding the right combination is complex. I wish the wget doc's were better....
The good news is that this problem should go away soon!!
Thanks to you, I reported this bug and the portage dev's have deemed it worthy. It has been fixed in CVS. This means the next release of portage should contain the fix and then you can go back to using $FILE!!
Until then, your -N suggestion is probably the best fix. It's not a perfect solution for http-replicator because of a couple side effects that limit http-replicator. One such side effect is that with -N, an internet connection must exist. With the original RESUMECOMMAND, http-replicator doesn't need an internet connection to serve from the cache. |
|
Back to top |
|
|
matbintang n00b
Joined: 12 Aug 2003 Posts: 22 Location: Canada
|
Posted: Fri Oct 29, 2004 3:42 am Post subject: |
|
|
flybynite!
You Da Man!!
_________________ $>cd /pub
$>more beer |
|
Back to top |
|
|
kmarasco n00b
Joined: 25 Jul 2003 Posts: 65 Location: Fernandina Beach, FL
|
Posted: Fri Oct 29, 2004 5:09 am Post subject: |
|
|
Thanks again for all of your help. I commend you for your excellent work and ongoing support effort |
|
Back to top |
|
|
flybynite l33t
Joined: 06 Dec 2002 Posts: 620
|
|
Back to top |
|
|
|