Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
HOWTO:Download Cache for your LAN-Http-Replicator (ver 3.0)
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3 ... 7, 8, 9 ... 22, 23, 24  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Mon Oct 18, 2004 7:08 pm    Post subject: Reply with quote

cdunham,

Ok, I guess the mirror you sync to must be using some program that leaves the .locks dir. Do you sync to a specific mirror or rotation? I would like to sync there myself to see what else might be in there...

It Seems there is nothing special about your libsoup file error. It's just failing on the first file.....

cdunham wrote:
Just a little more information....
changing t[0] to t['MD5'] did seem to fix things...


Ah yes, an always welcome API change in unstable portage :-) Shame on me for not asking which portage version your using, you obviously run ~


Relavent part of diff between stable 2.0.50-r11 and unstable portage-2.0.51-rc9
Code:

< if (md5_list[mybn][0] != md5sums[mybn][0])
---
> if (md5_list[mybn]["MD5"]  != md5sums[mybn]["MD5"])



So, please edit repcacheman and change t[0] to t["MD5"] to confirm that this works.

If so I'll update http-replicator's ebuild to include an unstable version for unstable portage.

cdunham, thanks for the help fixing this!!


Last edited by flybynite on Fri Oct 29, 2004 7:56 pm; edited 1 time in total
Back to top
View user's profile Send private message
cdunham
Apprentice
Apprentice


Joined: 06 Jun 2003
Posts: 211
Location: Rhode Island

PostPosted: Tue Oct 19, 2004 6:01 am    Post subject: Reply with quote

Hey, that seems to have done it.

Thanks, flybynite!
_________________
This post more meaningful in a scalar context.
Back to top
View user's profile Send private message
VinnieNZ
Tux's lil' helper
Tux's lil' helper


Joined: 11 Mar 2004
Posts: 126
Location: New Zealand

PostPosted: Fri Oct 22, 2004 2:40 am    Post subject: Reply with quote

I've just upgraded to the latest version of portage (2.0.51-r2) and when running repcacheman I get the following error:

Quote:
*SNIP*

Extracting the checksums....
Done!

Verifying checksum's....
/usr/portage/distfiles/linux-2.6.9.tar.bz2
Traceback (most recent call last):
File "/usr/bin/repcacheman", line 198, in ?
if t[0]:
KeyError: 0


Are there any fixes for this?


*EDIT Just read the post 2 above this one and realised that this looks like the likely solution. Problem is that I don't have any lines that look like the first set in my /usr/bin/repcacheman file.


Last edited by VinnieNZ on Wed Apr 25, 2007 11:31 pm; edited 1 time in total
Back to top
View user's profile Send private message
JohnHerdy
n00b
n00b


Joined: 11 Jul 2002
Posts: 61

PostPosted: Fri Oct 22, 2004 8:43 am    Post subject: Reply with quote

flybynite wrote:
If so I'll update http-replicator's ebuild to include an unstable version for unstable portage.


Hi flybynite,

We are using http-replicator on our network and it just works great. I want to thank you for the time and effort you put in this great program. Maybe the readers on this thread need to give more feedback on the bug-report so that this great program is added to the tree.

Portage 2.0.51 is marked stable so the current version doesn't work anymore. Fortunately you already have found a solution to this problem. Would you please let us know when you have a new ebuild ready.

<shameless plug> If it possible I would like to make a feature request; provide an entry in the log to see which IP-adresses have used the http-replicator for what file</shameless plug>
Back to top
View user's profile Send private message
kevquinn
Retired Dev
Retired Dev


Joined: 27 Jun 2003
Posts: 52

PostPosted: Sat Oct 23, 2004 1:29 pm    Post subject: warning, for portage upgraders Reply with quote

Don't know if it's just me, but when I upgraded to portage 2.0.51 (since it's now gone stable), my cache directory was automatically deleted, presumably by portage :(

I'd placed it at /usr/portage/distcache. My /var partition is 8Gb in size, and the replicator cache got rather large, enough to cause larger ebuilds to bomb due to lack of space in /var/tmp/portage. I've now changed it to /usr/cache/distfiles. Of course, now I've lost all my downloads - I knew I should have backed it up to a DVD-R!

So the warning is, don't put your cache directory in /usr/portage! I guess it was a stupid thing to do anyway, in hindsight...
Back to top
View user's profile Send private message
Bob Paddock
n00b
n00b


Joined: 25 Aug 2002
Posts: 49

PostPosted: Sun Oct 24, 2004 1:55 pm    Post subject: Not fetching if not in cache. What is the latest version? Reply with quote

I'm having trouble getting http-replicator to work consistently. I can fetch from my laptop to get files from the desktop machine if the package is in the cache. However if the package
is not in the cache then the desktop machine does not go out on Internet and get it, and the
laptop just says "waiting..." forever.

I suspect I have a mixture of versions for example, from page four of this thread:

"Edit /etc/conf.d/http-replicator to add your external proxy and check other defaults."

but I find it as /etc/http-replicator.

Are there *CURRENT* install instructions for the *LATESTS* version? I've tried to
piece that together from the long thread here, without success.

Is there an ebuild that works for the *LATESTS* version?
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Sun Oct 24, 2004 3:27 pm    Post subject: Re: Not fetching if not in cache. What is the latest version Reply with quote

Bob Paddock wrote:

Are there *CURRENT* install instructions for the *LATESTS* version?


How are you defining current? I defined STABLE and UNSTABLE. Each version has it own instructions.

The howto at the start of this thread is the "stable" version and it ONLY has a download link for http://www.updatedlinux.com/replicator/http-replicator-flybynite-1.3.tar.bz2
and complete instructions.

The "unstable" version post on page 4 ( https://forums.gentoo.org/viewtopic.php?t=173226&start=76 ) is the only place you can find a download link for http://www.updatedlinux.com/replicator/http-replicator-flybynite-1.5.tar.bz2 and says this which I thought would be clear:

Quote:

New version ready for testing!! Although the init code,logging, and options are cleaned up, this version primarily adds external proxy support.

Current users not needing external proxy support should wait for the next stable release. Experienced users can help test.


If that doesn't make sense to you then
Code:

etcat -v http-replicator


Which will show you your version.
Code:

*  net-misc/http-replicator :
        [   ] 2.0-r2 (0) OVERLAY
        [  I] 2.1_rc3 (0) OVERLAY


If you have 2.0-r2 then read only the HOWTO
If you have 2.1-rc3 then read the HOWTO PLUS the changes in the "unstable version" post at https://forums.gentoo.org/viewtopic.php?t=173226&start=76
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Sun Oct 24, 2004 3:58 pm    Post subject: Re: warning, for portage upgraders Reply with quote

kevquinn wrote:
Don't know if it's just me, but when I upgraded to portage 2.0.51 (since it's now gone stable), my cache directory was automatically deleted, presumably by portage :(



I confirmed this. Portage actually deleted /usr/portage/distcache when you ran emerge sync.

It makes sense, really. "emerge sync" makes /usr/portage on your box the same as the /usr/portage on the server. Since /usr/portage/distcache doesn't exist on the server, rsync deleted it the same way it would delete an outdated package.

Just to make this clear to all users, this had nothing to do with http-replicator, just a normal function of an emerge sync. The lesson is portage owns /usr/portage and don't put anything there.
Back to top
View user's profile Send private message
acaito_con
n00b
n00b


Joined: 27 Aug 2004
Posts: 4

PostPosted: Tue Oct 26, 2004 4:45 am    Post subject: Glad i found the help Reply with quote

cdunham wrote:

However, changing t[0] to t['MD5'] did seem to fix things...


I did have to change this reference on both line 198 and 201, may help others as well.

Glad i came across this post, i'd already resync'd, updated python. and viewed the source for obvious errors, wasnt famliar w/ the MD5 lib.

glad this works, great tool

btw. you may want to stop by the gentoo-wiki might need some slight update
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Tue Oct 26, 2004 7:38 am    Post subject: Reply with quote

JohnHerdy wrote:

<shameless plug> If it possible I would like to make a feature request; provide an entry in the log to see which IP-adresses have used the http-replicator for what file</shameless plug>


Already included in 2.1_rc3 , Posted: Wed Aug 04 2004! at https://forums.gentoo.org/viewtopic.php?t=173226&start=76

It's part of the stats package that is coming. You may need to parse a couple lines to get the complete info you want. But you probably just need to wait for that stats package in the next version of http-replicator.

Here is just part of what is logged in that version:

Code:

24 Oct 2004 19:09:23 DEBUG: HttpClient 161 connected to 192.168.2.123:34224
24 Oct 2004 19:09:23 INFO: HttpClient 161 received request for http://gentoo.osuosl.org/distfiles/prelink-20040707.tar.bz2
24 Oct 2004 19:09:23 STAT: HttpServer 161 serving 902324 bytes from cache to 192.168.2.123
Back to top
View user's profile Send private message
JohnHerdy
n00b
n00b


Joined: 11 Jul 2002
Posts: 61

PostPosted: Tue Oct 26, 2004 10:03 am    Post subject: Reply with quote

flybynite wrote:
JohnHerdy wrote:

<shameless plug> If it possible I would like to make a feature request; provide an entry in the log to see which IP-adresses have used the http-replicator for what file</shameless plug>


Already included in 2.1_rc3


Man I love you!!! (or at least your work). Sorry for asking a stupid question, but I couldn't find it on your website; is the latest version available on your website compliant with 2.0.51-clients?
Back to top
View user's profile Send private message
kmarasco
n00b
n00b


Joined: 25 Jul 2003
Posts: 65
Location: Fernandina Beach, FL

PostPosted: Tue Oct 26, 2004 10:21 am    Post subject: What is the Purpose of the -O mod to the resume statement? Reply with quote

I added

Code:
RESUMECOMMAND="/usr/bin/wget -t 5 --passive-ftp \${URI} -O \${DISTDIR}/\${FILE}"


to my make,conf, and http replicator worked fine. However, I am serving binary packages from Apache from the same server, and my emerge -gK fails with that setting.

Code:
kristel root # emerge -gK beaver
Fetching binary packages info...
Loaded metadata pickle.
cache miss: 'x' --- cache hit: 'o'
xoo
  -- DONE!

Calculating dependencies ...done!
>>> emerge (1 of 1) app-editors/beaver-0.2.6 to /
Fetching 'app-editors/beaver-0.2.6'
--01:07:23--  http://192.168.2.2/portage/packages/All/beaver-0.2.6.tbz2
           => `/usr/portage/packages/All//${FILE}'
Connecting to 192.168.2.2:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 144,983 [text/plain]

100%[=================================================================================>] 144,983       --.--K/s

01:07:23 (1.02 MB/s) - `/usr/portage/packages/All//${FILE}' saved [144983/144983]

!!! CATEGORY info missing from info chunk, aborting...
kristel root #


Note that it actually uses the variable name as the name of the file, when trying to save the binary to the package directory.

If I use this syntax for the resume command:

Code:
RESUMECOMMAND="/usr/bin/wget -t 5 --passive-ftp \${URI} -P \${DISTDIR}"


Both http-replicator and my binary pull seem to work fine, but I am concerned that I may be creating a problem for http-replicator that I am unaware, and that may bite me later.

Is the second syntax ok with http-replicator? I'm using portage portage-2.0.51-r2.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Tue Oct 26, 2004 4:20 pm    Post subject: Reply with quote

JohnHerdy wrote:

Sorry for asking a stupid question, but I couldn't find it on your website; is the latest version available on your website compliant with 2.0.51-clients?


Sorry, the website is out of date, There are two versions out, both are only listed or linked in the howto at the start of this thread.

All versions of http-replicator will work with the latest portage. But, repcacheman will barf with an error on >=portage-2.0.51. repcacheman errors will not affect the operation of http-replicator.

You can download a new repcacheman here:
http://www.updatedlinux.com/replicator/portagefix/repcacheman

You'll need to copy it to /usr/bin/repcacheman and chmod +x

Or, just edit 2 lines on your existing /usr/bin/repcacheman. Change t[0] to t["MD5"]

I'll have the fix in the ebuild soon.....


Last edited by flybynite on Fri Oct 29, 2004 7:53 pm; edited 1 time in total
Back to top
View user's profile Send private message
JohnHerdy
n00b
n00b


Joined: 11 Jul 2002
Posts: 61

PostPosted: Tue Oct 26, 2004 5:58 pm    Post subject: Reply with quote

flybynite wrote:
All versions of http-replicator will work with the latest portage


Hi flybynite,

Unfortunately this is not the case. 100% of the 2.0.51-clients don't work, 100% of the 2.0.50-clients work. The moment I migrate a client to 2.0.51 http-replicator doesn't work anymore. In our configuration we use a proxy-server (with validation). It seems that something is going wrong with the connection to the proxy. Our proxy resolves the ip-address for the mirrors but when I use http-replicator on a 2.0.51-client the proxy always returns unknown host. When I do a emerge-webrsync on a 2.0.51-client the tarball is fetched from the http-replicator server.

Thanks,
John.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Tue Oct 26, 2004 8:34 pm    Post subject: Re: What is the Purpose of the -O mod to the resume statemen Reply with quote

kmarasco wrote:

Is the second syntax ok with http-replicator? I'm using portage portage-2.0.51-r2.


It's going to be a long day :-(

This should fix it...
Code:

rm /usr/portage/packages/app-editors/beaver-0.2.6.tbz2
rm /usr/portage/packages/All/beaver-0.2.6.tbz2


Then try the replicator syntax again. This will help me figure out where the problem is.

Http-replicator doesn't support resuming yet. This syntax disables resuming and forces wget to overwrite the partial file.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Tue Oct 26, 2004 8:54 pm    Post subject: Reply with quote

JohnHerdy wrote:

Unfortunately this is not the case. 100% of the 2.0.51-clients don't work



2.0.51 works for me :-)

I need some output and logs before I can even guess at whats wrong....
Back to top
View user's profile Send private message
JohnHerdy
n00b
n00b


Joined: 11 Jul 2002
Posts: 61

PostPosted: Tue Oct 26, 2004 9:42 pm    Post subject: Reply with quote

Ok did some more research, it seems that the new version of portage doesn't respect my .wgetrc-settings anymore. After adding the following lines to /etc/make.conf 2.0.51-clients work as well;

http_proxy="http://YourMirrorHere.com:8080"
RESUMECOMMAND=" /usr/bin/wget -t 5 --passive-ftp \${URI} -O \${DISTDIR}/\${FILE}"

With 2.0.50 this wasn't neccesary the proxy-settings where read from .wgetrc. Hmmm, strange...
Back to top
View user's profile Send private message
kmarasco
n00b
n00b


Joined: 25 Jul 2003
Posts: 65
Location: Fernandina Beach, FL

PostPosted: Tue Oct 26, 2004 10:10 pm    Post subject: Re: What is the Purpose of the -O mod to the resume statemen Reply with quote

flybynite wrote:

This should fix it...
Code:

rm /usr/portage/packages/app-editors/beaver-0.2.6.tbz2
rm /usr/portage/packages/All/beaver-0.2.6.tbz2

Then try the replicator syntax again. This will help me figure out where the problem is.


Below code after deleting the files from both the client and the server.
Code:
kristel root # emerge -gK beaver
Fetching binary packages info...
Loaded metadata pickle.
cache miss: 'x' --- cache hit: 'o'
ooo
  -- DONE!

Calculating dependencies ...done!
>>> emerge (1 of 1) app-editors/beaver-0.2.6 to /
Fetching 'app-editors/beaver-0.2.6'
--12:29:48--  http://192.168.2.2/portage/packages/All/beaver-0.2.6.tbz2
           => `/usr/portage/packages/All//${FILE}'
Connecting to 192.168.2.2:80... connected.
HTTP request sent, awaiting response... 404 Not Found
12:29:48 ERROR 404: Not Found.

Fetcher exited with a failure condition.
!!! CATEGORY info missing from info chunk, aborting...
kristel root #

If I recreate the binary package, I get back to the originally posted error.

Note: I had ethereal observe the traffic and the http-replicator proxy does not appear to get involved with the locally served binaries from apache. I'm a rookie at ethereal, so I may have missed something, but it appears that when merging binaries using the PORTAGE_BINHOST variable in make.conf that the http_proxy variable is ignored (only for emerge -gK. The proxy is definitely used when source packages are pulled.).
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Tue Oct 26, 2004 11:13 pm    Post subject: Re: What is the Purpose of the -O mod to the resume statemen Reply with quote

kmarasco wrote:

If I recreate the binary package, I get back to the originally posted error.


Thanks, that confirms some things for me.

You don't have FETCHCOMMAND set do you?

kmarasco wrote:

Note: I had ethereal observe the traffic and the http-replicator proxy does not appear to get involved with the locally served binaries from apache.


Http-replicator isn't involved at all in this transaction. However RESUMECOMMAND seems to be and that is the problem. I find it "interesting" that portage uses RESUMECOMMAND when there is nothing to resume, I've filed a portage bug to get more info.

I use PORTAGE_BINHOST also, but with an ftp mirror. If you need a workaround, use an FTP server as your BINHOST till I get this worked out.
Back to top
View user's profile Send private message
kmarasco
n00b
n00b


Joined: 25 Jul 2003
Posts: 65
Location: Fernandina Beach, FL

PostPosted: Wed Oct 27, 2004 9:21 am    Post subject: Re: What is the Purpose of the -O mod to the resume statemen Reply with quote

flybynite wrote:

I use PORTAGE_BINHOST also, but with an ftp mirror. If you need a workaround, use an FTP server as your BINHOST till I get this worked out.

I set up vsftp and I have the same issue.
Code:

kristel root # emerge -gK beaver
Fetching binary packages info...
 * No password provided for username 'anonymous'

Loaded metadata pickle.
cache miss: 'x' --- cache hit: 'o'
oooo
  -- DONE!

Calculating dependencies ...done!
>>> emerge (1 of 1) app-editors/beaver-0.2.6 to /
Fetching 'app-editors/beaver-0.2.6'
--23:29:42--  ftp://192.168.2.2/All/beaver-0.2.6.tbz2
           => `/usr/portage/packages/All//${FILE}'
Connecting to 192.168.2.2:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD /All ... done.
==> PASV ... done.    ==> RETR beaver-0.2.6.tbz2 ... done.
Length: 144,983 (unauthoritative)

100%[=================================================================================>] 144,983      107.91K/s

23:29:43 (107.85 KB/s) - `/usr/portage/packages/All//${FILE}' saved [144983]

!!! CATEGORY info missing from info chunk, aborting...
kristel root #


Hmmm, any more thoughts? Typo/poor attention to detail on my part?
Code:
RESUMECOMMAND="/usr/bin/wget -t 5 --passive-ftp \${URI} -O \${DISTDIR}/\${FILE}"
[/quote]
Back to top
View user's profile Send private message
kmarasco
n00b
n00b


Joined: 25 Jul 2003
Posts: 65
Location: Fernandina Beach, FL

PostPosted: Thu Oct 28, 2004 10:24 am    Post subject: Alternative Resume Syntax Reply with quote

Thanks for your help flybynite. Since I experienced the bug with both ftp and http, I tried to find an alternative syntax for the resume command. I think that the following syntax should have the same effect, but does not require that the file name variable be passed in order to replace a partial file. It is as follows:
Code:
RESUMECOMMAND="/usr/bin/wget -N -t 5 --passive-ftp \${URI} -P \${DISTDIR}"

With the -N option, Only new files will be downloaded in the place of the old ones. Using -N, a file is considered new if one of these two conditions are met:

1. A file of that name does not already exist locally.
2. A file of that name does exist, but the remote file was modified more recently than the local file.

More importantly, if the local file does not exist, or the sizes of the files do not match, Wget will download the remote file no matter what the time-stamps say.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Fri Oct 29, 2004 12:42 am    Post subject: Re: What is the Purpose of the -O mod to the resume statemen Reply with quote

kmarasco wrote:
I set up vsftp and I have the same issue.

Typo/poor attention to detail on my part?


Nope, my part, sorry. I was using ftp without error on one of my boxes - because I had changed the RESUMECOMMAND so many times in testing that I left it wrong. The portage bug affects both ftp and http.


kmarasco wrote:

I tried to find an alternative syntax for the resume command. I think that the following syntax should have the same effect, but does not require that the file name variable be passed in order to replace a partial file.


I appreciate the help. Wget interacting with portage and RESUMECOMMAND together had so many options finding the right combination is complex. I wish the wget doc's were better....

The good news is that this problem should go away soon!!

Thanks to you, I reported this bug and the portage dev's have deemed it worthy. It has been fixed in CVS. This means the next release of portage should contain the fix and then you can go back to using $FILE!!

Until then, your -N suggestion is probably the best fix. It's not a perfect solution for http-replicator because of a couple side effects that limit http-replicator. One such side effect is that with -N, an internet connection must exist. With the original RESUMECOMMAND, http-replicator doesn't need an internet connection to serve from the cache.
Back to top
View user's profile Send private message
matbintang
n00b
n00b


Joined: 12 Aug 2003
Posts: 22
Location: Canada

PostPosted: Fri Oct 29, 2004 3:42 am    Post subject: Reply with quote

flybynite!

You Da Man!!

:D
_________________
$>cd /pub
$>more beer
Back to top
View user's profile Send private message
kmarasco
n00b
n00b


Joined: 25 Jul 2003
Posts: 65
Location: Fernandina Beach, FL

PostPosted: Fri Oct 29, 2004 5:09 am    Post subject: Reply with quote

Thanks again for all of your help. I commend you for your excellent work and ongoing support effort :!:
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 620

PostPosted: Fri Oct 29, 2004 8:06 pm    Post subject: Reply with quote

Thanks ( smiles proudly)

I just hope you still feel that way when I say you need to change portage_util.writemsg back to portage.writemsg !!

Or download repcacheman ver 3.2 at http://www.updatedlinux.com/replicator/portagefix/repcacheman
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3 ... 7, 8, 9 ... 22, 23, 24  Next
Page 8 of 24

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum