| View previous topic :: View next topic |
| Author |
Message |
Ant P.
Watchman
Joined: 18 Apr 2009
Posts: 6920
|
 Posted: Fri Mar 04, 2016 3:12 pm Post subject: Really simple LAN distfiles sharing |
 |
|
This isn't quite as effective as a full caching proxy or shared NFS distdir, but it takes seconds to set up and can reduce your external net traffic quite a lot.
On each machine you want to serve distfiles from, make a script in /etc/local.d/ (or your rc system's equivalent) like this, and start it up:
| Code: | #!/bin/sh
set -e
wwwroot=/srv/www/distfiles-mirror # feel free to change this
mkdir -p "$wwwroot"
ln -sT $(portageq envvar DISTDIR) "$wwwroot"/distfiles 2> /dev/null || true
busybox httpd -u nobody -p 8080 -h "$wwwroot" |
On each client machine, edit /etc/portage/mirrors to list the hostnames or IPs you set up:
| Code: | | local http://foo:8080 http://bar.local:8080 http://192.168.1.100:8080 |
Hint: If some of the machines aren't on all the time, you may want to lower the fetch timeout (use `portageq envvar FETCHCOMMAND` to get the default, edit it and put it in make.conf). |
|
| Back to top |
|
 |
NeddySeagoon
Administrator
Joined: 05 Jul 2003
Posts: 54216
Location: 56N 3W
|
| Posted: Fri Mar 04, 2016 7:20 pm Post subject: |
|
|
Ant P.,
| Code: | | emerge http-replicator |
or an I missing something?
_________________
Regards,
NeddySeagoon
Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail. |
|
| Back to top |
|
|
Buffoon
Veteran
Joined: 17 Jun 2015
Posts: 1369
Location: EU or US
|
| Posted: Fri Mar 04, 2016 7:27 pm Post subject: |
|
|
| I have only one portage and only one distfiles on LAN, it is shared over NFS (every machine has its own packages, though). Why should I keep a copy of distfiles in every box? |
|
| Back to top |
|
|
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3131
|
| Posted: Fri Mar 04, 2016 9:15 pm Post subject: |
|
|
| Quote: | | but it takes seconds to set up |
I guess that's the reason.
And it's a really good tip. Well, it's a poor solution to this particular problem, but seriously.... Did you know busybox can act as a web server? 
That little thing doesn't stop surprising me. It's awesome! |
|
| Back to top |
|
|
khayyam
Watchman
Joined: 07 Jun 2012
Posts: 6227
Location: Room 101
|
| Posted: Fri Mar 04, 2016 10:05 pm Post subject: |
|
|
| szatox wrote: | | [...] but seriously.... Did you know busybox can act as a web server? |
szatox ... I did, but never thought to put it to use in this way (thanks Ant). Also worth noting, busybox has a 'netcat' ("network swiss army knife").
| grep _NC_ /etc/portage/savedconfig/sys-apps/busybox-1.24.1: | CONFIG_NC_SERVER=y
CONFIG_NC_EXTRA=y
CONFIG_NC_110_COMPAT=y |
best ... khay |
|
| Back to top |
|
|
Dr.Willy
Guru
Joined: 15 Jul 2007
Posts: 547
Location: NRW, Germany
|
| Posted: Sat Mar 05, 2016 1:54 pm Post subject: |
|
|
Why
over
? |
|
| Back to top |
|
|
Ant P.
Watchman
Joined: 18 Apr 2009
Posts: 6920
|
| Posted: Sat Mar 05, 2016 5:27 pm Post subject: |
|
|
@Dr.Willy: it's just a convention I've ended up using over time. Maybe I'll stop doing that.
@everyone else: you all make good points, szatox is right that this is just a "quick and dirty" option. I was changing some things around on my own LAN and needed a usable fallback while I was putting the pieces back together, this seemed like a nice option because it's literally already there. |
|
| Back to top |
|
|
Syl20
l33t
Joined: 04 Aug 2005
Posts: 619
Location: France
|
| Posted: Wed Mar 23, 2016 5:57 pm Post subject: |
|
|
| Dr.Willy wrote: | Why
over
? |
For security reasons : you can ignore the shebang by running | Code: | | $ sh /path/script.sh |
instead of
The "set ..." lines will always be executed. |
|
| Back to top |
|
|
chithanh
Developer
Joined: 05 Aug 2006
Posts: 2158
Location: Berlin, Germany
|
| Posted: Wed Mar 23, 2016 9:22 pm Post subject: |
|
|
| Buffoon wrote: | | I have only one portage and only one distfiles on LAN, it is shared over NFS (every machine has its own packages, though). Why should I keep a copy of distfiles in every box? |
If only trusted hosts have write access to the distfiles directory that is fine. But the checksumming / unpacking of portage is not atomic, so it may create a TOCTTOU race condition. |
|
| Back to top |
|
|
miket
Guru
Joined: 28 Apr 2007
Posts: 488
Location: Gainesville, FL, USA
|
| Posted: Mon Apr 04, 2016 2:16 am Post subject: |
|
|
| chithanh wrote: | | Buffoon wrote: | | I have only one portage and only one distfiles on LAN, it is shared over NFS (every machine has its own packages, though). Why should I keep a copy of distfiles in every box? |
If only trusted hosts have write access to the distfiles directory that is fine. But the checksumming / unpacking of portage is not atomic, so it may create a TOCTTOU race condition. |
I always operated with the suspicion that my sharing /usr/portage/ directory could make make me vulnerable to races on /usr/portage/distfiles/. This led me to adopt the discipline of letting only one machine at a time run emerge. When I went to a binhost solution, this problem went away--so long as stayed away from doing full package builds on the other hosts at the same time. (You still have to share /usr/portage/ to install packages built on the binhost.)
Right now I'm playing with using the binhost to build multiple sets of host configurations (e.g. desktop and laptop) with overlayfs overlays so as to maximize package reuse among flavors and avoid a lot of duplicate building. Not only does this require a consistent set of compiler flags across all flavors (easy because all are some variety of Intel Core) but it also means that I do emerges for only one of these layers at a time. |
|
| Back to top |
|
|
|
Documentation, Tips & Tricks
