Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
HOWTO:Download Cache for your LAN-Http-Replicator (ver 3.0)
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, ... 22, 23, 24  Next  
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
dhurt
Apprentice
Apprentice


Joined: 14 May 2003
Posts: 278
Location: Davis, CA

PostPosted: Thu Jun 03, 2004 5:13 am    Post subject: Reply with quote

Use FTP and then have a different directory for each architecture. Then specify the location in /etc/make.conf and the bin host optoin:

Quote:

# Portage uses PORTAGE_BINHOST to specify mirrors for prebuilt-binary packages.
# The list is a single extry specifying the full address of the directory
# serving the tbz2's for your system. Running emerge with either '--getbinpkg'
# or '--getbinpkgonly' will cause portage to retrieve the metadata from all
# packages in the directory specified, and use that data to determine what will
# be downloaded and merged. '-g' or '-gK' are the recommend parameters. Please
# consult the man pages and 'emerge --help' for more information.
#PORTAGE_BINHOST="ftp://login:pass@grp.mirror.site/pub/grp/i686/athlon-xp/"
#PORTAGE_BINHOST="http://grp.mirror.site/gentoo/grp/1.4/i686/athlon-xp/"

_________________
"And isn't sanity really just a one-trick pony, anyway? I mean, all you get is one trick, rational thinking, but when you're good and crazy, ooh ooh ooh, the sky's the limit!" -- The Tick
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Sun Jun 06, 2004 6:22 pm    Post subject: Reply with quote

I've submitted the updated ebuild, now it is up to the gentoo dev's to see if this makes it into the portage tree!!

I guess a couple of good words wouldn't hurt :-)
http://bugs.gentoo.org/show_bug.cgi?id=50872
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Thu Jun 10, 2004 1:32 am    Post subject: Reply with quote

mjr104 wrote:

What's the best/simplest way to also share the binary packages that are in the packages directory of the machine hosting the replicator?


http-replicator will serve any file that is in it's cache directory by http, but I never thought of using it with this portage feature.

I will look into this feature, it looks possible....
Back to top
View user's profile Send private message
Ymerej
n00b
n00b


Joined: 17 Oct 2002
Posts: 10

PostPosted: Mon Jun 14, 2004 4:41 am    Post subject: works for emerging packages -- how about rsync Reply with quote

I installed and set up http-replicator, I did an emerge of a package on a client box, and observed that it went through the proxy -- great.

I got here basically by the same path as some other posters: I didn't want to query the rsync servers too much, with "emerge rsync" on each of the three Gentoo machines on my LAN.

I am going to ask a really naive question here: Does this http-replicator also cache the "emerge rsync" so that I can do it once on the http-replicator server, and then on all my other gentoo boxes that use the http-replicator proxy, and not risk overloading rsync servers and getting banned?

Or do I still need to set up an rsync server on one machine on my LAN?

Thanks.
_________________
-Gentoo newbie
Back to top
View user's profile Send private message
dhurt
Apprentice
Apprentice


Joined: 14 May 2003
Posts: 278
Location: Davis, CA

PostPosted: Mon Jun 14, 2004 6:10 am    Post subject: Reply with quote

You still need to set up the rsync mirror. It is not cached by the http proxy. Here is a link to the updated how-to:

http://forums.gentoo.org/viewtopic.php?t=180336&postdays=0&postorder=asc&start=0
_________________
"And isn't sanity really just a one-trick pony, anyway? I mean, all you get is one trick, rational thinking, but when you're good and crazy, ooh ooh ooh, the sky's the limit!" -- The Tick
Back to top
View user's profile Send private message
soulwarrior
Guru
Guru


Joined: 21 Oct 2002
Posts: 331

PostPosted: Mon Jun 14, 2004 6:14 am    Post subject: Re: works for emerging packages -- how about rsync Reply with quote

Ymerej wrote:

Or do I still need to set up an rsync server on one machine on my LAN?
Thanks.


Yes, you do have do install a rsync server on your network to provide this functionality, but as most things on gentoo, it is quite easy to do :wink:
Code:

emerge app-admin/gentoo-rsync-mirror

Make some changes to the /etc/rsync/rsyncd.conf configuration file:
Code:

uid = nobody
gid = nobody
use chroot = yes
max connections = 20
pid file = /var/run/rsyncd.pid
motd file = /etc/rsync/rsyncd.motd
transfer logging = no
log format = %t %a %m %f %b
syslog facility = local3
timeout = 300

[gentoo-x86-portage]
#this entry is for compatibility
#path = /opt/gentoo-rsync/portage
path=/usr/portage
comment = Gentoo Linux Portage tree

[gentoo-portage]
#modern versions of portage use this entry
#path = /opt/gentoo-rsync/portage
path=/usr/portage
comment = Gentoo Linux Portage tree mirror
exclude = distfiles

I have just changed the two path entries (path = /...) to point to my existing portage tree.

Now on every client, add this line to make.conf to point them to the rsync server on your network:
Code:

SYNC="rsync://192.168.0.1/gentoo-portage"


Just change the ip-address with the one from your server and you are done :wink:

Ah, of course you have to start it:
Code:

rc-update add rsyncd default
/etc/init.d/rsyncd start
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Thu Jun 17, 2004 9:07 am    Post subject: Reply with quote

I recommend all users with a LAN setup http-replicator using this HOWTO and also setup a local rsync server using the HOWTO at http://forums.gentoo.org/viewtopic.php?t=180336

This will best serve your lan and gentoo!!
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Mon Jun 21, 2004 10:04 am    Post subject: Reply with quote

I made a small change to the title of this thread. I hope 'download' will make more people stop and look at what http-replicator can do for them!!

Work has started on the next version of http-replicator!!
Back to top
View user's profile Send private message
dhurt
Apprentice
Apprentice


Joined: 14 May 2003
Posts: 278
Location: Davis, CA

PostPosted: Mon Jun 21, 2004 3:07 pm    Post subject: Reply with quote

Thanks for the hardwork :D
_________________
"And isn't sanity really just a one-trick pony, anyway? I mean, all you get is one trick, rational thinking, but when you're good and crazy, ooh ooh ooh, the sky's the limit!" -- The Tick
Back to top
View user's profile Send private message
Gherald2
Guru
Guru


Joined: 02 Jul 2003
Posts: 326
Location: Madison, WI USA

PostPosted: Wed Jun 30, 2004 7:07 am    Post subject: Reply with quote

Is there a way to configure the clients to fetch stuff themselves when the server is down?

I ask because I'd tell other people on our dorm network about my server, but I can only guarantee 90-95% uptime and I don't want their fetches to fail on my account...
Back to top
View user's profile Send private message
deezee
n00b
n00b


Joined: 19 Jul 2002
Posts: 7

PostPosted: Wed Jun 30, 2004 10:09 pm    Post subject: Reply with quote

This is a really good concept, but it seems to me that it would be a lot easier to mount /usr/portage/distfiles (and maybe even /usr/portage to solve the whole rsync thing as well) through NFS or samba, but of course this carries with it other problems. (Although none of them are unsolvable.)

Anywho, that's how I do it... :)
Back to top
View user's profile Send private message
dhurt
Apprentice
Apprentice


Joined: 14 May 2003
Posts: 278
Location: Davis, CA

PostPosted: Thu Jul 01, 2004 12:22 am    Post subject: Reply with quote

One of the problems of NFS, Samba that this solves is when two clients start downloading the same file at the same time.
_________________
"And isn't sanity really just a one-trick pony, anyway? I mean, all you get is one trick, rational thinking, but when you're good and crazy, ooh ooh ooh, the sky's the limit!" -- The Tick
Back to top
View user's profile Send private message
ejona
n00b
n00b


Joined: 09 Jan 2004
Posts: 11
Location: Texas, USA

PostPosted: Thu Jul 01, 2004 3:33 am    Post subject: Wonderful Job & amd64 works. Reply with quote

First of all, I would like to mention that I started reading on the board from the tsp program. I was happy with what that one offered (and was okay with the bugs), but this has no downfalls what-so-ever that I can see.

I can say that using the how-to worked perfectly. I am on a AMD64 processor and since it workes fine I would say the only thing I see that it needs is to add in amd64 to the KEYWORDS="x86" in the ebuild :D.

Thanks for the work (I am especially thankful - for I am on a modem connection and have 4 gentoo boxes - needless to say, I am also using a local rsync mirror) and I hope the developers include the package in portage soon. I will also keep an eye out on ways to break it.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Thu Jul 01, 2004 7:45 am    Post subject: Reply with quote

Thanks for the report ejona, I've updated the ebuild to include amd64 in the keywords.

Can anyone confirm any other arch's?
Back to top
View user's profile Send private message
larand54
l33t
l33t


Joined: 20 Feb 2004
Posts: 688
Location: Sweden

PostPosted: Thu Jul 01, 2004 11:24 am    Post subject: Reply with quote

I just tried to use this "http-mirror" but failed.
The server looks to be running but the client never tried to connect to it.

Code:
 The only changes I did to Make.conf for the client was:
 http_proxy="http://Jupiter:8080"
 RESUMECOMMAND="/usr/bin/wget -t 5 --passive-ftp \${URI} -O \${DISTDIR}/\${FILE}"

I havn't changed the mirror line:
Code:

GENTOO_MIRRORS="http://ftp.rhnet.is/pub/gentoo/ http://212.219.247.19/sites/www.ibiblio.org/ge$

What more do I have to do with the clients?
I also use my server as a sync-server and it works just fine.
Back to top
View user's profile Send private message
larand54
l33t
l33t


Joined: 20 Feb 2004
Posts: 688
Location: Sweden

PostPosted: Thu Jul 01, 2004 2:19 pm    Post subject: Reply with quote

I finally made it! :D
larand54 wrote:

The server looks to be running but the client never tried to connect to it.

It didn't run - a misstake of me :oops:

But when I got it running no data was transfered.
The problem was that I hadn't set up the IP-list in /etc/http-replicator.conf

When I did that and restarted the replicator worked.
Back to top
View user's profile Send private message
Monkeywrench
Apprentice
Apprentice


Joined: 22 Jun 2004
Posts: 205
Location: Florida

PostPosted: Fri Jul 02, 2004 12:43 am    Post subject: Reply with quote

This is awesome! Both this and your rsyncd howto. Thank you flybynite. You should contact the people who do the gentoo weekly newsletter about both of these howtos. It would save the generous mirror providers some bandwidth =)
_________________
Folding@home -Join the Linux team (163)!
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Sun Jul 04, 2004 7:58 am    Post subject: Reply with quote

Thanks monkeywrench!!

I submitted the howto's for inclusion in GWN!!

The newsletter is probably the best way to get the word out!!
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Sun Jul 04, 2004 8:03 am    Post subject: Reply with quote

larand54 wrote:
The problem was that I hadn't set up the IP-list in /etc/http-replicator.conf


I tried to make the config generic so it would work with ALL of the most common private LAN ip's. Did I miss a range or are you using a public ip range on your lan?
Back to top
View user's profile Send private message
dmitrio
Tux's lil' helper
Tux's lil' helper


Joined: 10 Dec 2002
Posts: 115
Location: Pago Pago

PostPosted: Sun Jul 04, 2004 12:23 pm    Post subject: :. copied to gentoo-wiki.com Reply with quote

I have copied this HOWTO, with suggestion of flybynite, to gentoo-wiki.com
http://gentoo-wiki.com/HOWTO_Download_Cache_for_LAN-Http-Replicator

If you see anything that should be added or changed, feel free to do so.

Thank you for a great HOWTO.
_________________

... Leaving ground, destination is unknown,
into the darkness and far away from home,
Will your dream come true and what will you find,
when fate is your guide ...
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Mon Jul 05, 2004 12:01 am    Post subject: Reply with quote

Ok, so http-replicator is set up as a proxy on my LAN for all my Gentoo boxes. But how do I set up the http-replicator itself to use an external proxy? I can't see anything in /etc/http-replicator.conf that might help.
Back to top
View user's profile Send private message
flybynite
l33t
l33t


Joined: 06 Dec 2002
Posts: 618

PostPosted: Sun Jul 11, 2004 8:00 am    Post subject: Reply with quote

Sorry for the delay, my net access has been a problem for almost a week.

Your right, support isn't there for a proxy.

It has been added to the next version already in testing....
Back to top
View user's profile Send private message
xkb
n00b
n00b


Joined: 13 Jun 2002
Posts: 20
Location: Netherlands

PostPosted: Sun Jul 11, 2004 8:19 am    Post subject: Reply with quote

I get the following error on the machine running the proxy:

Code:

Resolving myhostname.net... someip
Connecting to myhostname.net[someip]:8080... connected.
Proxy request sent, awaiting response...
10:16:22 ERROR -1: No data received.


Any ideas whats causing this? I followed the howto.

The proxy works fine for the client machines.

Code:

>>> Downloading http://distro.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/portage-2.0.50-r9.tar.bz2
--10:20:56--  http://distro.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/portage-2.0.50-r9.tar.bz2
           => `/usr/portage/distfiles/portage-2.0.50-r9.tar.bz2'
Resolving mydomain.net... done.
Connecting to mydomain.net[someip]:8080... connected.
Proxy request sent, awaiting response... 404 Not Found
10:20:57 ERROR 404: Not Found.
 
>>> Downloading http://gentoo.twobit.net/portage/portage-2.0.50-r9.tar.bz2
--10:20:57--  http://gentoo.twobit.net/portage/portage-2.0.50-r9.tar.bz2
           => `/usr/portage/distfiles/portage-2.0.50-r9.tar.bz2'
Resolving mydomain.net... done.
Connecting to mydomain.net[someip]:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 222,044 [application/x-tar]

After this it appears in /var/cache/http-replicator
Back to top
View user's profile Send private message
Gherald2
Guru
Guru


Joined: 02 Jul 2003
Posts: 326
Location: Madison, WI USA

PostPosted: Sun Jul 11, 2004 8:30 am    Post subject: Reply with quote

Hmm, have you tried localhost:8080 instead?
_________________
Unregistered Linux User #17598363
Back to top
View user's profile Send private message
meowsqueak
Veteran
Veteran


Joined: 26 Aug 2003
Posts: 1549
Location: New Zealand

PostPosted: Sun Jul 11, 2004 10:51 am    Post subject: Reply with quote

flybynite wrote:
Your right, support isn't there for a proxy.

It has been added to the next version already in testing....


Oh this is good news! Great - I'll definitely be one of the first to test this new feature :)
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Goto page Previous  1, 2, 3, ... 22, 23, 24  Next
Page 2 of 24

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum