Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
HOWTO:Download Cache for your LAN-Http-Replicator (ver 3.0)
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4 ... 22, 23, 24  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
La`res
Arch/Herd Tester
Arch/Herd Tester


Joined: 11 Aug 2003
Posts: 79
Location: ::0

PostPosted: Sun Jul 11, 2004 9:16 pm    Post subject: Problems.. Reply with quote

I have a Problem.

I emerge from 'jayden' (client) and I get the following from the telnet 'monitor'

Code:
HttpClient 109 Received header from 176.24.10.50:35782

  GET http://joey:8080/distfiles/openmotif-2.1.30-4_MLI.src.tar.gz HTTP/1.0
  User-Agent: Wget/1.9.1
  Host: joey:8080
  Accept: */*

HttpClient 109 Connecting to joey
HttpClient 110 Received header from 176.24.10.2:33215

  GET /distfiles/openmotif-2.1.30-4_MLI.src.tar.gz HTTP/1.0
  host: joey
  connection: close
  accept: */*
  user-agent: Wget/1.9.1

HttpClient 110 Error: 'NoneType' object has no attribute 'groups'

  TRACEBACK
  read (file /usr/lib/python2.3/asyncore.py, line 69)
  handle_read_event (file /usr/lib/python2.3/asyncore.py, line 390)
  handle_read (file /usr/bin/HttpReplicator.py, line 216)
  handle_header (file /usr/bin/HttpReplicator.py, line 278)

HttpServer 110 Closed

THen I need to shutdown the emerge from jayden manually. ( there seems to be no timeout)

If I emerge from 'joey' (server) I don't have this problem.

It seems to me that the server is calling itself and then barfing.

Any Ideas??
_________________
Lares Moreau <lares.moreau@gmail.com>
LRU: 400755 http://counter.li.org
lares/irc.freenode.net
Back to top
View user's profile Send private message
xkb
n00b
n00b


Joined: 13 Jun 2002
Posts: 20
Location: Netherlands

PostPosted: Mon Jul 12, 2004 6:36 am    Post subject: Reply with quote

Gherald wrote:
Hmm, have you tried localhost:8080 instead?


That did the trick! Thanks! :D
Back to top
View user's profile Send private message
jpc82
Guru
Guru


Joined: 09 Mar 2003
Posts: 326

PostPosted: Tue Jul 13, 2004 6:55 pm    Post subject: Reply with quote

Problem fixed itself
Back to top
View user's profile Send private message
dripton
n00b
n00b


Joined: 16 Aug 2002
Posts: 65
Location: Virginia USA

PostPosted: Mon Jul 19, 2004 3:55 pm    Post subject: Reply with quote

Thanks for http-replicator. I like it even better than tsp-cache.

For me, the main missing feature is the ability to fetch all distfiles from the local mirror, even those that aren't hosted on Gentoo http mirrors. Manually stuffing files into one cache directory is a lot nicer than spreading them around to many distfiles directories.

I used the tip in bug 50872 to handle ftp and nomirror ebuilds, but there isn't an equivalent workaround for RESTRICT="fetch". (Which is bug 37455.) I hacked my local portage.py to work around this, but that will quickly get annoying to maintain.
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Mon Jul 19, 2004 9:25 pm    Post subject: Reply with quote

flybynite wrote:
Your right, support isn't there for a proxy.

It has been added to the next version already in testing....


Any chance I could assist with the testing by installing it at my site?
Back to top
View user's profile Send private message
JohnHerdy
n00b
n00b


Joined: 11 Jul 2002
Posts: 61

PostPosted: Thu Jul 22, 2004 4:14 pm    Post subject: Reply with quote

flybynite wrote:
Sorry for the delay, my net access has been a problem for almost a week.

Your right, support isn't there for a proxy.

It has been added to the next version already in testing....


Hi flybynite, do you have something we can test/use? I think this is great software!!! however without proxy support I'm not able to use this. :cry:
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Thu Jul 22, 2004 8:26 pm    Post subject: Reply with quote

JohnHerdy wrote:
flybynite wrote:
Sorry for the delay, my net access has been a problem for almost a week.

Your right, support isn't there for a proxy.

It has been added to the next version already in testing....


Hi flybynite, do you have something we can test/use? I think this is great software!!! however without proxy support I'm not able to use this. :cry:


me too 8)
Back to top
View user's profile Send private message
cdunham
Apprentice
Apprentice


Joined: 06 Jun 2003
Posts: 211
Location: Rhode Island

PostPosted: Sat Jul 24, 2004 6:33 pm    Post subject: Reply with quote

Nice HOWTO!

A couple of issues:

- The ebuild requires 'inherit eutils' for epatch to work
- repcacheman works great, but you may want to note that if you choose not to use it, you need to manually create the cache directory and set its permissions.

Thanks! This is what makes Gentoo such a great distribution!
_________________
This post more meaningful in a scalar context.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Sat Jul 24, 2004 10:05 pm    Post subject: Reply with quote

I guess something changed that 'inherit eutils' is now required. I'll add that to the ebuild, thanks.

I promise to find time to get an ebuild out in the next couple of days for the version with proxy support.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Sun Jul 25, 2004 1:44 pm    Post subject: Reply with quote

cdunham wrote:
Nice HOWTO!
- repcacheman works great, but you may want to note that if you choose not to use it, you need to manually create the cache directory and set its permissions.


The ebuild prints this after merging:
Quote:

*
* If you wish to change defaults
* edit /etc/http-replicator.conf
*
* Execute /usr/bin/repcacheman
* to auto create the cache directory
* and prime the cache.
*




Did you miss this or is this not clear?
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Sun Jul 25, 2004 1:56 pm    Post subject: Reply with quote

dripton wrote:
Thanks for http-replicator. I like it even better than tsp-cache.

For me, the main missing feature is the ability to fetch all distfiles from the local mirror, even those that aren't hosted on Gentoo http mirrors.

I used the tip in bug 50872 to handle ftp and nomirror ebuilds, but there isn't an equivalent workaround for RESTRICT="fetch". (Which is bug 37455.)


The tip in bug 50872 was added to the HOWTO in ver 1.4 so others don't need to worry about it, however, If you installed http-replicator prior to howto ver 1.4 you may want to read the /etc/portage/mirrors section.

I may even add the file to the ebuild. Most users don't have the file, smart users can merge the file?

The restrict=fetch problem is strictly a portage issue. This is not the only portage quirk that doesn't work well with others. http-replicator could share /usr/portage/distfiles if portage chown'd files to the portage user :-)
Back to top
View user's profile Send private message
cdunham
Apprentice
Apprentice


Joined: 06 Jun 2003
Posts: 211
Location: Rhode Island

PostPosted: Sun Jul 25, 2004 2:30 pm    Post subject: Reply with quote

flybynite wrote:

The ebuild prints this after merging:
Quote:

*
* If you wish to change defaults
* edit /etc/http-replicator.conf
*
* Execute /usr/bin/repcacheman
* to auto create the cache directory
* and prime the cache.
*




Did you miss this or is this not clear?


No, it's clear about using it, but in the HOWTO you say:

Quote:
repcacheman is nice to help install and manage http-replicator on gentoo but isn't needed for replicator to work.


which is true, as long as the cache dir is set up properly, that's all.
_________________
This post more meaningful in a scalar context.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Mon Jul 26, 2004 5:45 pm    Post subject: Reply with quote

Ok, I'll check the wording in that line.

I added that because another user had trouble with repcacheman and didn't know replicator was still working for him. It is confusing.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Mon Jul 26, 2004 6:50 pm    Post subject: Reply with quote

Updated HOWTO - version 1.5
Back to top
View user's profile Send private message
ShaunC
n00b
n00b


Joined: 21 May 2003
Posts: 3

PostPosted: Thu Jul 29, 2004 10:59 am    Post subject: Reply with quote

flybynite wrote:

Can anyone confirm any other arch's?


Yep. http-replicator/repcacheman work just fine on alpha.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Thu Jul 29, 2004 10:53 pm    Post subject: Reply with quote

Thanks, I add that to the ebuild.
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Thu Jul 29, 2004 11:57 pm    Post subject: Reply with quote

flybynite wrote:
I promise to find time to get an ebuild out in the next couple of days for the version with proxy support.


How's that promise coming along? :wink:
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Sat Jul 31, 2004 4:53 am    Post subject: Reply with quote

I have been working on it. The new version that includes proxy support is so different that the entire ebuild/package/patches will have to be redone :-(

This is a major amount of work when most users will see no improvements.

I know you need proxy support so I'm considering backporting the proxy support. I'm testing to see which is the better option.
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Sat Jul 31, 2004 10:02 am    Post subject: Reply with quote

Thanks for sticking with it. I think as this becomes a more popular solution, there will be increasing desire for proxy support. However at the moment, I appreciate the time and effort you are putting into this and if there's anything I can do to assist (especially testing) then please let me know. Private message me if necessary.
Back to top
View user's profile Send private message
woZa
Guru
Guru


Joined: 18 Nov 2003
Posts: 340
Location: The Steel City - UK

PostPosted: Sat Jul 31, 2004 7:25 pm    Post subject: Reply with quote

Great work flybynite. I had this up and running in 5 mins. Now reclaimed 3.5gigs from /usr... just need to upgrade my servers ide controller (ata 33 for ata 133 drive!) to get the real speeds! Another thing to get round to sometime!
_________________
A few months struggling with gentoo is better than a lifetime struggling with windoze!
Back to top
View user's profile Send private message
JohnHerdy
n00b
n00b


Joined: 11 Jul 2002
Posts: 61

PostPosted: Mon Aug 02, 2004 9:51 am    Post subject: Reply with quote

flybynite wrote:
I have been working on it. The new version that includes proxy support is so different that the entire ebuild/package/patches will have to be redone :-(

This is a major amount of work when most users will see no improvements.

I know you need proxy support so I'm considering backporting the proxy support. I'm testing to see which is the better option.


hi flibynite,

Thanks a lot for all your effort you put in http-replicator.

I agree with you that most users will see no improvements however the users that work in an enterprise environment have the greatest benefit by using this program. I mean it's nice that John Doe with 2 computer at home is able to use this program but real bandwith savings and load reduction on the mirrors can be achieved in the enterprise. Unfortunatelly in an enterprise environment you always have to go through a proxy.

Why don't you post what you have now and we can help you backport it or create an ebuild for the new version.

I hope I don't sound pushy but I really like your program.
Back to top
View user's profile Send private message
scoy
Tux's lil' helper
Tux's lil' helper


Joined: 21 Jun 2002
Posts: 93
Location: USA, PA, Wilkes-Barre

PostPosted: Mon Aug 02, 2004 2:06 pm    Post subject: NFS Reply with quote

What benefits does this have over NFS exports?

I use one machine to export its /usr/portage/distfiles, then the rest(2) of my machines use that. I export it with no_root_squash, so all machines can store the files if they download them.
Back to top
View user's profile Send private message
JohnHerdy
n00b
n00b


Joined: 11 Jul 2002
Posts: 61

PostPosted: Mon Aug 02, 2004 3:53 pm    Post subject: Re: NFS Reply with quote

scoy wrote:
What benefits does this have over NFS exports?


If a file is on a NFS-volume and two or more hosts need that file at the same time all hosts will try to download that file. This will result in an unsuccesfull download. A powerfull feature of http-replicator that it's able to stream partial downloads.
Back to top
View user's profile Send private message
woZa
Guru
Guru


Joined: 18 Nov 2003
Posts: 340
Location: The Steel City - UK

PostPosted: Wed Aug 04, 2004 3:56 pm    Post subject: Reply with quote

When I emerge an app from a client that isn't on the server the file is downloaded to both the server http-replicator cache dir and the client distfiles dir... Is this normal behaviour?

There is no need for the files to remain in the client distfiles dir is there? Should I just use cron to empty the client distfiles dir daily for example.

Thanks in advance
_________________
A few months struggling with gentoo is better than a lifetime struggling with windoze!
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Wed Aug 04, 2004 5:47 pm    Post subject: Reply with quote

woZa wrote:
When I emerge an app from a client that isn't on the server the file is downloaded to both the server http-replicator cache dir and the client distfiles dir... Is this normal behaviour?


Yes, The clients require the file in the distfile dir before portage can emerge it.
woZa wrote:

There is no need for the files to remain in the client distfiles dir is there? Should I just use cron to empty the client distfiles dir daily for example.

Thanks in advance


Once emerged, you can delete all files in the clients distfile dir.

A cron to do this would be fine as long as you don't delete while emerging. It would be rare and small chance, but imagine an emerge -u world going overnight when the cron script kicks in and deletes a file that portage is still downloading or just about to unpack (any other time would be no problem). While not catastrophic after all since the package is still in http-replicators cache, I think it would make portage error and stop the emerge.

Since I do many upgrades overnight, that small window between the start of the package download and the beginning of the unpack is why I don't have a cron script delete the files on my computer. I just do it manually every once in a while which works fine for me. It's Ok if you wanted to use cron, the most that could happen is that on a rare ocassion, that long list of upgrades won't be finished in the morning...

the repcacheman script automatically manages the files in the distfile dir on the server box. Just run repcacheman either occasionally or at most after every upgrade on the server.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3, 4 ... 22, 23, 24  Next
Page 3 of 24

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum