| View previous topic :: View next topic |
| Author |
Message |
Guenther Brunthaler
Apprentice
Joined: 22 Jul 2006
Posts: 217
Location: Vienna
|
 Posted: Sun Sep 03, 2006 4:16 pm Post subject: HOWTO update your portage tree via SSH from another LAN host |
 |
|
I have written a script that works similar to "emerge --sync", but instead of updating your local portage tree with a remote rsync server, it updates your tree from another host (which will typically be a member of your LAN) via SSH. And this does not require running an rsync server on that host!
In addition to that, it will also copy all files in the /usr/bin/distcache directory on the host to your local /usr/bin/distcache directory (skipping any files which are already there).
Why I wrote it: I have two machines in my LAN, both running Gentoo. Until now, I had to run "emerge --sync" on both of them. Which meant, both machines had to fetch any updated files through my internet connection.
Now I only do an "emerge --sync" on one of my machines, then use my script on the other machine, bypassing the internet connection and only using the LAN to update the files. This is much faster.
But this was only half of the problem: After the portage tree being up-to-date, I ran "emerge --update --deep world" on both machines.
As both machines have a similar package selection installed, this usually meant that about the same packages needed updating as well. And updating packages usually means downloading those packages as well (which go to the /usr/portage/distfiles) directory.
Without my script, both machines downloaded the very same packages through my internet connection, transferring all the files twice (once to each computer).
With my script, any new files in the distfiles directory of the host will by copied to the local machine as well. This means when an "emerge --update world" is run on the local machine, it will already find the required packages and does not need to download them.
Here is an example how to use my script to update 3 different computers on your LAN (the prompt indicates on what machine the respective command is entered):
| Code: |
machine1 ~ # emerge --sync
machine1 ~ # emerge --update --deep --fetchonly world
machine1 ~ # emerge --update --deep world
|
The "--fetchonly"-line downloads all the packages first and the next lines starts the update. As soon as the emerge-command in "--fetchonly"-line has been completed, all the packages have been downloaded and the portage tree as well as the downloaded files can now be replicated to the other machines:
| Code: |
machine2 ~ # update-portage-from machine1
machine2 ~ # emerge --update --deep world
|
and
| Code: |
machine3 ~ # update-portage-from machine1
machine3 ~ # emerge --update --deep world
|
And all three machines are running their updates! But the internet connection has only been used to update one of them.
This will be very helpful if you only have a slow internet connection, or if you have a limited transfer quota.
Some caveats: The examples above assume you have set up your ssh-agent already and loaded the required keys into it (or placed the keys in the $HOME/.ssh/id_rsa files on the respective machine), so that a "ssh machine1" works on machine2 and machine3 without prompting for a password.
It further assumes that machine1 runs its SSH server (or xinetd launching sshd) on the SSH standard port 22. Otherwise you have to modify the script, or, better, predefine the required ports on a per-host basis in the /etc/ssh/ssh_config or $HOME/.ssh_config files.
Furthermore, the script copies the owner and groups settings from the specified host to the local machine. This is OK if all machines use the same group/owner-model for the portage files. That is, if all machines use the same FEATURES from /etc/make.conf: Either all of them should have
userpriv userfetch
as part of their FEATURES= line in /etc/make.conf, or none of the machines should have those features enabled.
File listing "/usr/local/sbin/update-portage-from":
| Code: |
#! /bin/sh
# $HeadURL: /caches/xsvn/trunk/usr/local/sbin/update-portage-from $
# $Author: root $
# $Date: 2006-09-03T15:19:14.166588Z $
# $Revision: 350 $
#
# Use rysnc via SSH to synchronize the local portage tree
# from the portage tree on machine $1, except for the
# distfiles subdirectory.
#
# Then all new files in the distfiles subdirectory on the
# machine $1 are mirrored to the local machine. (Files
# which only exist in the local distfiles directory
# are left alone.)
#
# Written in 2006 by Guenther Brunthaler
NICENESS="10"
die() {
echo "ERROR: $*" >& 2
exit 1
}
do_run() {
"$@" || die "Failed: >>>$*<<<!"
}
do_nice() {
do_run nice "-$NICENESS" "$@"
}
do_rsync() {
do_nice rsync \
--rsh='ssh -o Compression=no' \
--archive --safe-links \
--human-readable --human-readable --stats \
"$@"
}
RHOST="$1"
test "`whoami`" = "root" || die "Must be run be root!"
test -n "$RHOST" || die "Usage: $0 <host>"
ssh "$RHOST" true \
|| die "The SSH connection does not work!" \
"(Forgot to set up ssh-agent?)"
do_rsync --compress --whole-file --delete-after \
--exclude "/distfiles/" --exclude "/packages/" --exclude "/local/" \
"$RHOST:/usr/portage/" "/usr/portage/"
emerge --metadata
do_rsync --update \
"$RHOST:/usr/portage/distfiles/" "/usr/portage/distfiles/"
which eupdatedb > /dev/null 2>& 1 && eupdatedb
|
|
|
| Back to top |
|
 |
I.C.Wiener
Tux's lil' helper
Joined: 25 Jul 2004
Posts: 115
Location: Furtwangen (Germany)
|
| Posted: Sun Sep 03, 2006 6:49 pm Post subject: |
|
|
Sharing /usr/portage on my router via nfs works fine for me, even for different architektures (Sparc/x86).
Advantages:
- you can run 'emerge sync' on every box
- no internal syncing, no scripts to run, nothing, changes become visible for every one immediatly
- saves a lot of diskspace (portage tree and distfiles are saved on 1 box only)
- saves bandwith (see post above)
| Code: |
1. install nfs-utils on the clients and server:
# emerge -av nfs-utils
# rc-update add nfs default
2. share your portage-dir on the server:
# echo "/usr/portage 192.168.0.0/24(rw,async,no_root_squash)" >> /etc/exports
3. mount the nfs share on the clients during boot:
(replace <server ip> with your nfs-servers ip)
# echo "<server ip>:/usr/portage /usr/portage nfs hard,intr 0 0" >> /etc/fstab
|
Don't do this on a public network! People might manipulate your portage tree and distfiles, besides no_root_squash issn't save. However, for your lan at home (where no one else has access to) it's ok. |
|
| Back to top |
|
|
Guenther Brunthaler
Apprentice
Joined: 22 Jul 2006
Posts: 217
Location: Vienna
|
| Posted: Sun Sep 03, 2006 7:53 pm Post subject: |
|
|
| I.C.Wiener wrote: | | Sharing /usr/portage on my router via nfs works fine for me, even for different architektures (Sparc/x86). |
Sounds like a good idea!
Still being a Linux newbie, I have not examined all that NFS/portmapper/automount/kerberos/openssl stuff yet which seems to be necessary to set up NFS "the right way". I currently only use SSH for everything.
However, OpenSSL & OpenVPN is next on my list of things to learn.
And then comes NFS (secured by OpenVPN) - and I'll finally be happy to implement your idea!
| I.C.Wiener wrote: | | Don't do this on a public network! |
Unfortunately, my LAN is nearly a public network: There are Windows clients on it ... which makes the LAN only slightly safer than the public Internet. |
|
| Back to top |
|
|
Guenther Brunthaler
Apprentice
Joined: 22 Jul 2006
Posts: 217
Location: Vienna
|
| Posted: Tue Sep 05, 2006 6:22 pm Post subject: |
|
|
Hi I.C.,
Although I really like your idea very much, I have encountered a problem when thinking about possible configurations for such a solution:
| I.C.Wiener wrote: | | no internal syncing, no scripts to run, nothing, changes become visible for every one immediatly |
What about "emerge --metadata"?
When watching the output of "emerge --sync", it seems that an "--emerge metadata" is done at the end of each "emerge --sync" automatically.
But the metadata cache is somewhere below /var, and not a part of /usr/portage.
So it seems each machine using the updated /usr/portage tree should do an "emerge --metadata" as well, or the metadata cache and the portage tree will get out-of-sync.
How do you handle that? Can the metadata cache also be shared by NFS? Or does it contain machine-specific data (such as USE-flag-dependent results) which cannot be shared?
Unfortunately, my knowledge about the metadata cache is very limited. I don't really know what it actually contains or what it's used for. |
|
| Back to top |
|
|
JeliJami
Veteran
Joined: 17 Jan 2006
Posts: 1086
Location: Belgium
|
| Posted: Thu Sep 07, 2006 1:05 pm Post subject: |
|
|
| Guenther Brunthaler wrote: | Hi I.C.,
Although I really like your idea very much, I have encountered a problem when thinking about possible configurations for such a solution:
| I.C.Wiener wrote: | | no internal syncing, no scripts to run, nothing, changes become visible for every one immediatly |
What about "emerge --metadata"?
When watching the output of "emerge --sync", it seems that an "--emerge metadata" is done at the end of each "emerge --sync" automatically.
But the metadata cache is somewhere below /var, and not a part of /usr/portage.
So it seems each machine using the updated /usr/portage tree should do an "emerge --metadata" as well, or the metadata cache and the portage tree will get out-of-sync. |
| I.C.Wiener wrote: | | you can run 'emerge sync' on every box |
his boxes still connect to gentoo mirrors, but there's nothing to sync, just --metadata is performed
_________________
Unanswered Post Initiative | Search | FAQ
Former username: davjel |
|
| Back to top |
|
|
I.C.Wiener
Tux's lil' helper
Joined: 25 Jul 2004
Posts: 115
Location: Furtwangen (Germany)
|
| Posted: Sun Oct 01, 2006 11:54 pm Post subject: |
|
|
Actually I don't take care of metadata at all. Portage works fine withouth it, just a bit slower perhaps.
For searching the portage tree and other useful stuff app-portage/portage-utils turned out to be much faster anyways. It has it's own cache which I rebuild on each box from time to time. Well, ok that could be optimized as well. But my main goal was to save disk space and internet traffic. |
|
| Back to top |
|
|
neophyte46
Apprentice
Joined: 16 Mar 2004
Posts: 171
Location: Australia
|
| Posted: Fri Nov 03, 2006 12:39 pm Post subject: |
|
|
| I.C.Wiener wrote: | Sharing /usr/portage on my router via nfs works fine for me, even for different architektures (Sparc/x86).
Advantages:
- you can run 'emerge sync' on every box
- no internal syncing, no scripts to run, nothing, changes become visible for every one immediatly
- saves a lot of diskspace (portage tree and distfiles are saved on 1 box only)
- saves bandwith (see post above)
| Code: |
1. install nfs-utils on the clients and server:
# emerge -av nfs-utils
# rc-update add nfs default
2. share your portage-dir on the server:
# echo "/usr/portage 192.168.0.0/24(rw,async,no_root_squash)" >> /etc/exports
3. mount the nfs share on the clients during boot:
(replace <server ip> with your nfs-servers ip)
# echo "<server ip>:/usr/portage /usr/portage nfs hard,intr 0 0" >> /etc/fstab
|
Don't do this on a public network! People might manipulate your portage tree and distfiles, besides no_root_squash issn't save. However, for your lan at home (where no one else has access to) it's ok. |
Wow thanks. That was exactly what I was looking for, for my home network (server, a desktop and laptop).
Works perfectly. Initially I tried doing it via samba nad mounting it that way, but it was more a a pita.
Cheers,
_________________
Acer Aspire 1690; Pentium M 1.73Ghz, 1gig DDRII 533mhz dual channel, ATI 128mb x700 PCIE,
Athlon 64 X2 3800+ @ 2.5Ghz, 2gig DDR 400, 4x120gig sata raid, A8N-SLI, ASUS 256mb 7800GTX Extreme
http://codelines.net.au - Software design and development |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Documentation, Tips & Tricks
