View previous topic :: View next topic |
Author |
Message |
magowiz Veteran
Joined: 17 Feb 2005 Posts: 1029 Location: Italy/Milan/Bresso
|
Posted: Sat Sep 01, 2007 12:30 pm Post subject: [TIP]share distfiles dir to optimize getdelta |
|
|
Hi all,
I was thinking about using an external hd shared on a network using nfs to store distfiles to reduce the number of full downloads for getdelta, I was thinking to write a script that does this :
- check if the external hd is available for mounting
- mount it in a temporary mountpoint
- copy all distfiles in local distfiles dir to shared distfiles only when the nfs distfiles doesn't have that file ( i.e. with a cp --reply="no")
- remove all distfiles in local distfiles dir
- unmount the nfs shared distfiles and remount it in /usr/portage/distfiles
Is it possible to do so?
Any suggestion or critics will be accepted.
(sorry if my English is not too good)
Last edited by magowiz on Sat Sep 01, 2007 4:11 pm; edited 2 times in total |
|
Back to top |
|
|
magowiz Veteran
Joined: 17 Feb 2005 Posts: 1029 Location: Italy/Milan/Bresso
|
Posted: Sat Sep 01, 2007 4:10 pm Post subject: |
|
|
I made a first version of the script, It's licensed under GPL.
I assumed that the nfs shared dir is /opt/distfiles/ and you have a temporary mount point under /mnt/other/ , you can launch it passing as first argument the hostname or IP of the pc that shares distfiles.
Here it is :
Code: |
#!/bin/sh
HOST=$1
if [ $HOST = "" ]; then echo "usage sharedistfiles.sh hostname"
else
sudo mount $HOST:/opt/distfiles/ /mnt/other/ -o rw >/dev/null 2>/dev/null
if [ $? -gt 0 ]; then echo "there was an error mounting shared distfiles, check if the "$HOST" host is online and nfsd is started on it I will use the local distfiles dir"
else
yes n | cp -i /usr/portage/distfiles/* /mnt/other/ >/dev/null 2>/dev/null
sudo umount /mnt/other/
sudo mount $HOST:/opt/distfiles/ /usr/portage/distfiles/ -o rw
fi
fi
|
|
|
Back to top |
|
|
magowiz Veteran
Joined: 17 Feb 2005 Posts: 1029 Location: Italy/Milan/Bresso
|
Posted: Sat Sep 01, 2007 4:33 pm Post subject: |
|
|
I changed cp command to cp -r to copy also the *-src/ dirs for live ebuilds and added some output.
Code: |
#!/bin/sh
HOST=$1
if [ "$HOST" = "" ]; then echo "usage sharedistfiles.sh hostname"
else
echo "Try to mounting shared distfiles in a temporary mountpoint"
sudo mount $HOST:/opt/distfiles/ /mnt/other/ -o rw >/dev/null 2>/dev/null
if [ $? -gt 0 ]; then echo "there was an error mounting shared distfiles, check if the "$HOST" host is online and nfsd is started on it I will use the local distfiles dir"
else
echo "Merging local distfiles with the shared one, this could take a while"
yes n | cp -r -i /usr/portage/distfiles/* /mnt/other/ >/dev/null 2>/dev/null
echo "Remounting shared distfiles in /usr/portage/distfiles/"
sudo umount /mnt/other/
sudo mount $HOST:/opt/distfiles/ /usr/portage/distfiles/ -o rw
fi
fi
|
|
|
Back to top |
|
|
Emission n00b
Joined: 01 Sep 2007 Posts: 6 Location: Over the hills and far away...
|
Posted: Sat Sep 01, 2007 5:18 pm Post subject: |
|
|
Interesting concept, your English is fine by the way . |
|
Back to top |
|
|
magowiz Veteran
Joined: 17 Feb 2005 Posts: 1029 Location: Italy/Milan/Bresso
|
Posted: Thu Sep 13, 2007 9:06 am Post subject: |
|
|
I wote also a script that synchronize local *-src dirs with the remote shared one, this to avoid that at a successive mount, local host will overwrite *-src files with old ones.
Code: |
#!/bin/sh
HOST=$1
sudo umount /usr/portage/distfiles/
sudo mount $HOST:/mnt/data/distfiles/ /mnt/other/ -o rw
echo "Updating local *-src dirs"
cp -u -r /mnt/other/cvs-src/* /usr/portage/distfiles/cvs-src/ >/dev/null 2>/dev/null
cp -u -r /mnt/other/git-src/* /usr/portage/distfiles/git-src/ >/dev/null 2>/dev/null
cp -u -r /mnt/other/svn-src/* /usr/portage/distfiles/svn-src/ >/dev/null 2>/dev/null
sudo umount /mnt/other/
|
since the two scripts need to be run after nfs-client (also server for the host that hosts shared distfiles) starts and, respectively before it stops I suggest to invoke them using /etc/conf.d/local.start and /etc/conf.d/local.stop as I did on mine PCs.
I hope somebody will find useful theese scripts. |
|
Back to top |
|
|
magowiz Veteran
Joined: 17 Feb 2005 Posts: 1029 Location: Italy/Milan/Bresso
|
Posted: Tue Apr 01, 2008 7:04 am Post subject: |
|
|
I modified a bit the two scripts, I also made a script that checks if the distfile_server is available:
- if it's available and the shared distfile dir isn't mounted it mounts it
- if it isn't available and the shared distfile dir is still mounted it forces the umount
it's useful to put this last script on a crontab (on my box I execute it every 5 minutes)
let's start with the new script I described above :
Code: | $cat /etc/init.d/sharedistfiletocheck.sh
#!/bin/sh
HOST=$1
if [ ! -f /shared/nfslock ]
then
source /etc/make.globals
source /etc/make.conf
touch /shared/nfslock
n=`mount | grep $DISTDIR | wc -l`
nc -w 1 -z $HOST 2049 &> /dev/null
RESULT=$?
if [[ $n -ne 0 ]]
then
if [[ $RESULT -ne 0 ]]
then
echo "${HOST} unreachable">/shared/nfslog
umount -f ${DISTDIR} &>/dev/null
fi
else
if [[ $RESULT -eq 0 ]]
then
echo "${HOST} is now reachable">/shared/nfslog
/etc/init.d/sharedistfiles.sh $HOST &>/dev/null
fi
fi
rm /shared/nfslock
fi;
|
and now the others two script : Code: |
$cat /etc/init.d/sharedistfiles.sh
#!/bin/sh
source /etc/make.globals
source /etc/make.conf
HOST=$1
if [ "$HOST" = "" ]; then echo "usage sharedistfiles.sh hostname"
else
echo "Try to mounting shared distfiles in a temporary mountpoint"
sudo mount $HOST:/mnt/data/distfiles/ /mnt/other/ -o rw >/dev/null 2>/dev/null
if [ $? -gt 0 ]; then echo "there was an error mounting shared distfiles, check if the "$HOST" host is online and nfsd is started on it I will use the local distfiles dir"
else
echo "Merging local distfiles with the shared one, this could take a while"
yes n | cp -r -i ${DISTDIR}/* /mnt/other/ >/dev/null 2>/dev/null
echo "Remounting shared distfiles in "${DISTDIR}
sudo umount /mnt/other/
sudo mount $HOST:/mnt/data/distfiles/ ${DISTDIR} -o rw
fi
fi
|
Code: | $ cat /etc/init.d/unsharedistfiles.sh
#!/bin/sh
source /etc/make.globals
source /etc/make.conf
HOST=$1
if [ `mount | grep ${DISTDIR} | wc -l ` -ne 0 ]
then
sudo umount ${DISTDIR}
sudo mount $HOST:/mnt/data/distfiles/ /mnt/other/ -o rw
echo "Updating local"
cp -u -r /mnt/other/cvs-src/* ${DISTDIR}/cvs-src/ >/dev/null 2>/dev/null
cp -u -r /mnt/other/git-src/* ${DISTDIR}/git-src/ >/dev/null 2>/dev/null
cp -u -r /mnt/other/svn-src/* ${DISTDIR}/svn-src/ >/dev/null 2>/dev/null
sudo umount /mnt/other/
fi
|
The usage of each script is
Code: | <script-name> <distfile_server> |
I call the sharedistfiles.sh and the unsharedistfiles.sh respectively from local.start and local.stop
If you have any suggestion or doubt please post. |
|
Back to top |
|
|
|