Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Really simple LAN distfiles sharing
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
Ant P.
Advocate
Advocate


Joined: 18 Apr 2009
Posts: 4519

PostPosted: Fri Mar 04, 2016 3:12 pm    Post subject: Really simple LAN distfiles sharing Reply with quote

This isn't quite as effective as a full caching proxy or shared NFS distdir, but it takes seconds to set up and can reduce your external net traffic quite a lot.

On each machine you want to serve distfiles from, make a script in /etc/local.d/ (or your rc system's equivalent) like this, and start it up:
Code:
#!/bin/sh
set -e
wwwroot=/srv/www/distfiles-mirror    # feel free to change this

mkdir -p "$wwwroot"
ln -sT $(portageq envvar DISTDIR) "$wwwroot"/distfiles 2> /dev/null || true
busybox httpd -u nobody -p 8080 -h "$wwwroot"


On each client machine, edit /etc/portage/mirrors to list the hostnames or IPs you set up:
Code:
local http://foo:8080 http://bar.local:8080 http://192.168.1.100:8080

Hint: If some of the machines aren't on all the time, you may want to lower the fetch timeout (use `portageq envvar FETCHCOMMAND` to get the default, edit it and put it in make.conf).
_________________
*.ebuild // /etc/service/*
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 39287
Location: 56N 3W

PostPosted: Fri Mar 04, 2016 7:20 pm    Post subject: Reply with quote

Ant P.,

Code:
emerge http-replicator
or an I missing something?
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Buffoon
Veteran
Veteran


Joined: 17 Jun 2015
Posts: 1074
Location: EU or US

PostPosted: Fri Mar 04, 2016 7:27 pm    Post subject: Reply with quote

I have only one portage and only one distfiles on LAN, it is shared over NFS (every machine has its own packages, though). Why should I keep a copy of distfiles in every box?
Back to top
View user's profile Send private message
szatox
Veteran
Veteran


Joined: 27 Aug 2013
Posts: 1493

PostPosted: Fri Mar 04, 2016 9:15 pm    Post subject: Reply with quote

Quote:
but it takes seconds to set up
I guess that's the reason.
And it's a really good tip. Well, it's a poor solution to this particular problem, but seriously.... Did you know busybox can act as a web server? :lol:
That little thing doesn't stop surprising me. It's awesome!
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 5303
Location: Room 101

PostPosted: Fri Mar 04, 2016 10:05 pm    Post subject: Reply with quote

szatox wrote:
[...] but seriously.... Did you know busybox can act as a web server?

szatox ... I did, but never thought to put it to use in this way (thanks Ant). Also worth noting, busybox has a 'netcat' ("network swiss army knife").

grep _NC_ /etc/portage/savedconfig/sys-apps/busybox-1.24.1:
CONFIG_NC_SERVER=y
CONFIG_NC_EXTRA=y
CONFIG_NC_110_COMPAT=y

best ... khay
Back to top
View user's profile Send private message
Dr.Willy
Guru
Guru


Joined: 15 Jul 2007
Posts: 414
Location: NRW, Germany

PostPosted: Sat Mar 05, 2016 1:54 pm    Post subject: Reply with quote

Why
Code:
#!/bin/sh
set -e

over
Code:
#!/bin/sh -e

?
Back to top
View user's profile Send private message
Ant P.
Advocate
Advocate


Joined: 18 Apr 2009
Posts: 4519

PostPosted: Sat Mar 05, 2016 5:27 pm    Post subject: Reply with quote

@Dr.Willy: it's just a convention I've ended up using over time. Maybe I'll stop doing that.

@everyone else: you all make good points, szatox is right that this is just a "quick and dirty" option. I was changing some things around on my own LAN and needed a usable fallback while I was putting the pieces back together, this seemed like a nice option because it's literally already there.
_________________
*.ebuild // /etc/service/*
Back to top
View user's profile Send private message
Syl20
Guru
Guru


Joined: 04 Aug 2005
Posts: 500
Location: France

PostPosted: Wed Mar 23, 2016 5:57 pm    Post subject: Reply with quote

Dr.Willy wrote:
Why
Code:
#!/bin/sh
set -e

over
Code:
#!/bin/sh -e

?

For security reasons : you can ignore the shebang by running
Code:
$ sh /path/script.sh
instead of
Code:
$ /path/script.sh

The "set ..." lines will always be executed.
Back to top
View user's profile Send private message
chithanh
Developer
Developer


Joined: 05 Aug 2006
Posts: 2141
Location: Berlin, Germany

PostPosted: Wed Mar 23, 2016 9:22 pm    Post subject: Reply with quote

Buffoon wrote:
I have only one portage and only one distfiles on LAN, it is shared over NFS (every machine has its own packages, though). Why should I keep a copy of distfiles in every box?
If only trusted hosts have write access to the distfiles directory that is fine. But the checksumming / unpacking of portage is not atomic, so it may create a TOCTTOU race condition.
Back to top
View user's profile Send private message
miket
Guru
Guru


Joined: 28 Apr 2007
Posts: 376
Location: Gainesville, FL, USA

PostPosted: Mon Apr 04, 2016 2:16 am    Post subject: Reply with quote

chithanh wrote:
Buffoon wrote:
I have only one portage and only one distfiles on LAN, it is shared over NFS (every machine has its own packages, though). Why should I keep a copy of distfiles in every box?
If only trusted hosts have write access to the distfiles directory that is fine. But the checksumming / unpacking of portage is not atomic, so it may create a TOCTTOU race condition.

I always operated with the suspicion that my sharing /usr/portage/ directory could make make me vulnerable to races on /usr/portage/distfiles/. This led me to adopt the discipline of letting only one machine at a time run emerge. When I went to a binhost solution, this problem went away--so long as stayed away from doing full package builds on the other hosts at the same time. (You still have to share /usr/portage/ to install packages built on the binhost.)

Right now I'm playing with using the binhost to build multiple sets of host configurations (e.g. desktop and laptop) with overlayfs overlays so as to maximize package reuse among flavors and avoid a lot of duplicate building. Not only does this require a consistent set of compiler flags across all flavors (easy because all are some variety of Intel Core) but it also means that I do emerges for only one of these layers at a time.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum