JSharku
Apprentice
Joined: 09 Feb 2003
Posts: 189
Location: Belgium
|
 Posted: Wed Feb 14, 2007 9:08 pm Post subject: [TIP] Making distfiles over NFS behave with eclean-dist. |
 |
|
I have a couple of Gentoo machines; now, as much for my own benefit as for Gentoo's, I have these syncing with a local rsync mirror and getting their distfiles over an NFS share. There's one major problem with this setup IMO: if you have a couple of machines with different configurations, how do you figure out who needs what? If I'd run eclean-dist on my server it would e.g. want to delete the xorg tarballs, whereas eclean-dist on a client PC might want to delete the tarballs for e.g. iptables or postfix.
One possible solution to this could be to have all the client PCs generate a file with all the distfiles it needs, upload this to the server and use this file as exclude-file for eclean-dist. This would probably involve lots of scripting and error checking to generate the list of needed files, combine all the different exclusion files, filter out duplicates etc. etc.
Another solution could be to have separate distdirs for each client and have them download to that. This completely destroys the benefit of having a NFS shared distdir, it's wasteful both in bandwidth and diskspace.
My solution is a smarter version (if I do say so myself  ) of that last method: each host gets its own directory to store distfiles in, but the files themselves get hardlinked across all the directories. Hardlinking is particularly useful in this scenario: if a file is deleted by one host, it will still be available in the other hosts' directories in case they should need it; only when the last host deletes the file is it removed from disk. Hardlinks are also cheap in terms of disk space: the files don't get duplicated, only their name does.
Ok, enough talk, let's set this thing up. We'll start with the server side of things.
SERVER.
First, you'll need a fairly recent kernel (>=2.6.13) and you'll need to enable inotify, which can be found here: | Code: | CONFIG_INOTIFY
Location:
-> File systems |
Either use make menuconfig:
| Code: | # mount /boot
# cd /usr/src/linux
# make menuconfig
# make && make install && make modules_install |
or genkernel:
| Code: | # mount /boot
# genkernel --menuconfig <whatever other options you want for genkernel> all |
While our kernel is recompiling, we'll set up the rest. First we'll create directories for each host, including the server:
| Code: | # cd /usr/portage/distfiles
# mkdir server host1 host2 ... hostn |
Next, create this script in /etc/init.d/:
| Code: | #!/sbin/runscript
start() {
ebegin "Starting distfiles duplication"
if [[ ! -e ${PID_DIR} ]]
then
mkdir ${PID_DIR} || eerror "Cannot create ${PID_DIR}"
return 1
fi
for i in ${DIRS}
do
if [[ ! -e ${i} ]]
then
eerror "${i} doesn't exist, make sure to create it first."
return 1
fi
if [[ ! -d ${i} ]]
then
eerror "${i} is not a directory."
return 1
fi
for j in ${DIRS/${i}/}
do
#start-stop-daemon doesn't play nice with piped commands, as it apparently redirects stdout and stderr itself
#start-stop-daemon --start --background --pidfile ${PID_DIR}/`basename ${i}`.`basename ${j}`.pid \
# --make-pidfile --nicelevel 19 --quiet --exec /usr/bin/inotifywait -- -q -e create -m --format '%f' ${i} \
# | xargs -I '{}' ln ${i}/'{}' ${j}
#hence this kludge
[[ ! -e ${PID_DIR}/`basename ${i}`.`basename ${j}`.fifo ]] && mkfifo ${PID_DIR}/`basename ${i}`.`basename ${j}`.fifo
nice -n 19 inotifywait -q -e create,moved_to -m --format '%f' ${i} > ${PID_DIR}/`basename ${i}`.`basename ${j}`.fifo &
echo $! > ${PID_DIR}/`basename ${i}`.`basename ${j}`.pid
nice -n 19 xargs -I '{}' ln ${i}/'{}' ${j} < ${PID_DIR}/`basename ${i}`.`basename ${j}`.fifo &> /dev/null &
done
done
eend $?
}
stop() {
ebegin "Stopping distfiles duplication"
for i in ${PID_DIR}/*.pid
do
start-stop-daemon --stop --exec /usr/sbin/inotifywait --pidfile ${PID_DIR}/${i}
rm ${i}
done
for i in ${PID_DIR}/*.fifo
do
rm ${i}
done
eend $?
} |
I called it combine, but you can name it whatever you want. You can also download it:
| Code: | # cd /etc/init.d/
# wget http://webs.hogent.be/~b29309jl/combine |
Don't forget to make it executable:
| Code: | | # chmod 755 combine |
Next, create a file in /etc/conf.d/ with the following content:
| Code: | PID_DIR="/var/run/combine"
DIRS="/usr/portage/distfiles/server /usr/portage/distfiles/host1 ... /usr/portage/distfiles/hostn" |
It MUST be the same name as the file in /etc/init.d/ or it won't work. Add this script to the default runlevel:
| Code: | | # rc-update add combine default |
We'll also need inotify-tools, which is currently masked, so we'll unmask it and emerge it. Unfortunately, inotify-tools are only available for x86 and amd64, so if you're on a different platform, this won't work for you.
| Code: | # echo "sys-fs/inotify-tools ~x86" >> /etc/portage/package.keywords
# emerge inotify-tools |
Finally, you'll need to change /etc/make.conf to make portage aware of the new location of the distfiles:
| Code: | PORTDIR=/usr/portage
DISTDIR=${PORTDIR}/distfiles/server |
That's it! You can now reboot into your new kernel, and the combine script will automagically watch the directories defined in $DIRS and if a file gets created or moved into one of these, the script will create hardlinks to that file in all the other directories.
Once rebooted you might want to move all your current distfiles into the new directory. It's also a good test to see if everything is working:
| Code: | # cd /usr/portage/distfiles
# for i in *; do if [[ -f ${i} ]]; then mv ${i} server/; fi; done |
The contents of each of the directories should now be identical.
CLIENTS.
On the clients, all we need is to alter /etc/fstab:
| Code: | #change a line in /etc/fstab similar to this
server:/usr/portage/distfiles /usr/portage/distfiles nfs rsize=8192,wsize=8192 0 0
#into something similar to this
server:/usr/portage/distfiles/host1 /usr/portage/distfiles nfs rsize=8192,wsize=8192 0 0 |
Now you can safely run eclean-dist on any of the clients or the server, and a particular tarball will only get truly deleted if none of them need the file anymore.
CAVEATS:
I am well aware that this script isn't very scalable: wether it's a bug or a feature is debatable but in it's current form, this script WILL spawn 2*n!/(n - 2)! processes watching n directories/hosts. While these processes run at nice level 19, and I haven't noticed any significant delays or lag with a couple of directories being watched, it's still something you should carefully consider before using this system on a server with 10's or 100's of clients.
Another possible consideration to take into account is that this system only watches the directories defined in $DIRS, cvs-src won't get duplicated. This isn't too big a deal for me, since I never use CVS sources anyway, but some people might want that functionality. If anyone feels inclined to extend my script, they're more than welcome to do so. 
STANDARD DISCLAIMER: I have tested this configuration and it Works For Me, however I take no responsibility for any damage this script or this tip could cause. In other words: if it breaks your system, you get to keep the pieces.
_________________
If only life were portage-driven: | Code: | | USE="-bitch -in-laws nice gorgeous smart" emerge girlfriend |
*sigh*
--
Open Source for Windows! |
|
Documentation, Tips & Tricks
