Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[TIP]share distfiles dir to optimize getdelta
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
magowiz
Veteran
Veteran


Joined: 17 Feb 2005
Posts: 1029
Location: Italy/Milan/Bresso

PostPosted: Sat Sep 01, 2007 12:30 pm    Post subject: [TIP]share distfiles dir to optimize getdelta Reply with quote

Hi all,
I was thinking about using an external hd shared on a network using nfs to store distfiles to reduce the number of full downloads for getdelta, I was thinking to write a script that does this :

  • check if the external hd is available for mounting
  • mount it in a temporary mountpoint
  • copy all distfiles in local distfiles dir to shared distfiles only when the nfs distfiles doesn't have that file ( i.e. with a cp --reply="no")
  • remove all distfiles in local distfiles dir
  • unmount the nfs shared distfiles and remount it in /usr/portage/distfiles


Is it possible to do so?
Any suggestion or critics will be accepted.
(sorry if my English is not too good)


Last edited by magowiz on Sat Sep 01, 2007 4:11 pm; edited 2 times in total
Back to top
View user's profile Send private message
magowiz
Veteran
Veteran


Joined: 17 Feb 2005
Posts: 1029
Location: Italy/Milan/Bresso

PostPosted: Sat Sep 01, 2007 4:10 pm    Post subject: Reply with quote

I made a first version of the script, It's licensed under GPL.
I assumed that the nfs shared dir is /opt/distfiles/ and you have a temporary mount point under /mnt/other/ , you can launch it passing as first argument the hostname or IP of the pc that shares distfiles.
Here it is :
Code:

#!/bin/sh
HOST=$1
if [ $HOST = "" ]; then echo "usage sharedistfiles.sh hostname"
else
        sudo mount $HOST:/opt/distfiles/ /mnt/other/ -o rw >/dev/null 2>/dev/null
        if [ $? -gt 0 ]; then echo "there was an error mounting shared distfiles, check if the "$HOST" host is online and nfsd is started on it I will use the local distfiles dir"
        else
                yes n | cp -i /usr/portage/distfiles/* /mnt/other/ >/dev/null 2>/dev/null
                sudo umount /mnt/other/
                sudo mount $HOST:/opt/distfiles/ /usr/portage/distfiles/ -o rw
        fi
fi
Back to top
View user's profile Send private message
magowiz
Veteran
Veteran


Joined: 17 Feb 2005
Posts: 1029
Location: Italy/Milan/Bresso

PostPosted: Sat Sep 01, 2007 4:33 pm    Post subject: Reply with quote

I changed cp command to cp -r to copy also the *-src/ dirs for live ebuilds and added some output.
Code:

#!/bin/sh
HOST=$1
if [ "$HOST" = "" ]; then echo "usage sharedistfiles.sh hostname"
else
        echo "Try to mounting shared distfiles in a temporary mountpoint"
        sudo mount $HOST:/opt/distfiles/ /mnt/other/ -o rw >/dev/null 2>/dev/null
        if [ $? -gt 0 ]; then echo "there was an error mounting shared distfiles, check if the "$HOST" host is online and nfsd is started on it I will use the local distfiles dir"
        else   
                echo "Merging local distfiles with the shared one, this could take a while"
                yes n | cp -r -i /usr/portage/distfiles/* /mnt/other/ >/dev/null 2>/dev/null
                echo "Remounting shared distfiles in /usr/portage/distfiles/"
                sudo umount /mnt/other/
                sudo mount $HOST:/opt/distfiles/ /usr/portage/distfiles/ -o rw
        fi
fi
Back to top
View user's profile Send private message
Emission
n00b
n00b


Joined: 01 Sep 2007
Posts: 6
Location: Over the hills and far away...

PostPosted: Sat Sep 01, 2007 5:18 pm    Post subject: Reply with quote

Interesting concept, your English is fine by the way :) .
Back to top
View user's profile Send private message
magowiz
Veteran
Veteran


Joined: 17 Feb 2005
Posts: 1029
Location: Italy/Milan/Bresso

PostPosted: Thu Sep 13, 2007 9:06 am    Post subject: Reply with quote

I wote also a script that synchronize local *-src dirs with the remote shared one, this to avoid that at a successive mount, local host will overwrite *-src files with old ones.
Code:

#!/bin/sh
HOST=$1
sudo umount /usr/portage/distfiles/
sudo mount $HOST:/mnt/data/distfiles/ /mnt/other/ -o rw
echo "Updating local *-src dirs"
cp -u -r  /mnt/other/cvs-src/* /usr/portage/distfiles/cvs-src/ >/dev/null 2>/dev/null
cp -u -r  /mnt/other/git-src/* /usr/portage/distfiles/git-src/ >/dev/null 2>/dev/null
cp -u -r  /mnt/other/svn-src/* /usr/portage/distfiles/svn-src/  >/dev/null 2>/dev/null
sudo umount /mnt/other/


since the two scripts need to be run after nfs-client (also server for the host that hosts shared distfiles) starts and, respectively before it stops I suggest to invoke them using /etc/conf.d/local.start and /etc/conf.d/local.stop as I did on mine PCs.
I hope somebody will find useful theese scripts.
Back to top
View user's profile Send private message
magowiz
Veteran
Veteran


Joined: 17 Feb 2005
Posts: 1029
Location: Italy/Milan/Bresso

PostPosted: Tue Apr 01, 2008 7:04 am    Post subject: Reply with quote

I modified a bit the two scripts, I also made a script that checks if the distfile_server is available:
- if it's available and the shared distfile dir isn't mounted it mounts it
- if it isn't available and the shared distfile dir is still mounted it forces the umount

it's useful to put this last script on a crontab (on my box I execute it every 5 minutes)
let's start with the new script I described above :
Code:
 $cat /etc/init.d/sharedistfiletocheck.sh
#!/bin/sh
HOST=$1
if [ ! -f /shared/nfslock ]
then
   source /etc/make.globals
   source /etc/make.conf
   touch /shared/nfslock
   n=`mount | grep  $DISTDIR | wc -l`
   nc -w 1 -z $HOST 2049 &> /dev/null
   RESULT=$?
    if [[ $n -ne 0 ]]
   then
       if [[ $RESULT -ne 0 ]]
           then
                echo "${HOST} unreachable">/shared/nfslog
                umount -f ${DISTDIR} &>/dev/null
        fi
   else
       if [[ $RESULT -eq 0 ]]
           then
                echo "${HOST} is now reachable">/shared/nfslog
              /etc/init.d/sharedistfiles.sh $HOST &>/dev/null
        fi
   fi
rm /shared/nfslock
fi;



and now the others two script :
Code:

$cat /etc/init.d/sharedistfiles.sh
#!/bin/sh
source /etc/make.globals
source /etc/make.conf
HOST=$1
if [ "$HOST" = "" ]; then echo "usage sharedistfiles.sh hostname"
else
   echo "Try to mounting shared distfiles in a temporary mountpoint"
   sudo mount $HOST:/mnt/data/distfiles/  /mnt/other/ -o rw >/dev/null 2>/dev/null
   if [ $? -gt 0 ]; then echo "there was an error mounting shared distfiles, check if the "$HOST" host is online and nfsd is started on it I will use the local distfiles dir"
   else   
      echo "Merging local distfiles with the shared one, this could take a while"
      yes n | cp -r -i ${DISTDIR}/* /mnt/other/ >/dev/null 2>/dev/null
      echo "Remounting shared distfiles in "${DISTDIR}
      sudo umount /mnt/other/
      sudo mount $HOST:/mnt/data/distfiles/ ${DISTDIR} -o rw
   fi
fi

Code:
 $ cat /etc/init.d/unsharedistfiles.sh
#!/bin/sh
source /etc/make.globals
source /etc/make.conf
HOST=$1
if [ `mount | grep ${DISTDIR} | wc -l ` -ne 0  ]
then
   sudo umount ${DISTDIR}
   sudo mount $HOST:/mnt/data/distfiles/ /mnt/other/ -o rw
   echo "Updating local"
   cp -u -r  /mnt/other/cvs-src/* ${DISTDIR}/cvs-src/ >/dev/null 2>/dev/null
   cp -u -r  /mnt/other/git-src/* ${DISTDIR}/git-src/ >/dev/null 2>/dev/null
   cp -u -r  /mnt/other/svn-src/* ${DISTDIR}/svn-src/   >/dev/null 2>/dev/null
   sudo umount /mnt/other/
fi

The usage of each script is
Code:
 <script-name> <distfile_server>

I call the sharedistfiles.sh and the unsharedistfiles.sh respectively from local.start and local.stop



If you have any suggestion or doubt please post.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum