View previous topic :: View next topic |
Author |
Message |
miscdebris n00b
Joined: 30 Mar 2005 Posts: 70
|
Posted: Wed Aug 17, 2005 10:00 pm Post subject: Dynamic DISTCC_HOSTS & MAKEOPTS Generation for Portage |
|
|
I have quite a few computers at home, and most of the time, not all of them are on. When I use distcc, I had to reconfigure it every time I turned a computer on or off. Especially MAKEOPTS. So I set out to write a script to take care of that.
This script will reside in /etc/portage/bashrc and will try to detect which computers are up (and with nmap installed, which are running distccd)
This script does the following:
Checks to see if distcc (and not -distcc) is in your features
Reads in your /etc/distcc/hosts file
Auto validates your localhost
Pings any non-localhost in your /etc/distcc/hosts file
If it gets a response and you have nmap installed, it checks to see if distccd is running on that computer (port 3632 default)
Any validated host has it's /limit added to the MAKEOPTS=-j (it understands the defaults)
Writes the valid hosts to your DISTCC_HOSTS variable
Displays the variables DISTCC_HOSTS and MAKEOPTS
Will run once each for each package right before it compiles.
Automaticly finding of active DISTCC hosts (NEW)
Future Features:
Thresholds as to how to use localhost
I appreciate any comments, suggestions, bug fixes, etc.
And without further waiting, here it is.
Code: | #!/bin/sh
# Options most likely to be changed
# Control the amount of information displayed.
# 0 displays nothing
# 1 displays the DISTCC_HOSTS and MAKEOPTS that will be used
# 2 displays the above plus the detection process
VERBOSE_DISPLAY=1
RANDOM_HOSTS=1
DISTCC_NETWORKS=172.20.1.0/24
IGNORE_HOSTS=172.20.1.2
DEFAULT_PROCESSES=2
# Options unlikely to need changed
CAT=/bin/cat
GREP=/bin/grep
AWK=/bin/awk
PING=/bin/ping
NMAP=/usr/bin/nmap
DATE=/usr/bin/date
PORT=3632
if [ `echo ${FEATURES} | $GREP -q -e distcc ; echo $?` == 0 ] && [ `echo ${FEATURES} | $GREP -q -e -distcc ; echo $?` != 0 ] && [ -n "$EBUILD_PHASE" ] && [ $EBUILD_PHASE == "compile" ]; then
if [ $RANDOM_HOSTS -ge 1 ] ; then
DISTCC_HOSTS=""
BUILD_PROCESSES=""
for X in `$NMAP --randomize-hosts -n -sT -p $PORT $DISTCC_NETWORKS --open --exclude $IGNORE_HOSTS | $GREP Interesting| $AWK '{ print $4 }'| $AWK -F: '{ print $1 }'` ; do
DISTCC_HOSTS="$DISTCC_HOSTS ${X}/${DEFAULT_PROCESSES}"
BUILD_PROCESSES=$[$BUILD_PROCESSES+$DEFAULT_PROCESSES]
done
DISTCC_HOSTS="localhost/1 $DISTCC_HOSTS localhost/1"
BUILD_PROCESSES=$[$BUILD_PROCESSES+2]
COMMIT=1
else
if [ -f "/etc/distcc/hosts" ] ; then
DISTCC_CLIENTS=`echo | $CAT /etc/distcc/hosts`
fi
DISTCC_HOSTS=""
for CLIENT in $DISTCC_CLIENTS ; do
HOST=`echo $CLIENT | $AWK -F/ '{ print $1 }'`
if [ $VERBOSE_DISPLAY -ge 2 ] ; then
echo -n "Checking ${CLIENT}..."
fi
VALID=0
PROCESSES=0
if [ $HOST = "localhost" ] ; then
VALID=1
elif [ `$PING -c 1 -n -q $HOST > /dev/null ; echo $?` = 0 ] ; then
if [ -f $NMAP ] ; then
if [ `$NMAP -n -sT -p $PORT $HOST | $GREP $PORT | $AWK '{ print $2 }'` = open ] ; then
VALID=1
else
VALID=0
fi
else
VALID=1
fi
fi
if [ $VALID = 1 ] ; then
if [ $VERBOSE_DISPLAY -ge 2 ] ; then
echo -n " Verified. Checking Processes..."
fi
COMMIT=1
if [ ! -z "$DISTCC_HOSTS" ] ; then
DISTCC_HOSTS="${DISTCC_HOSTS} ${CLIENT}"
else
DISTCC_HOSTS="${CLIENT}"
fi
PROCESSES=`echo $CLIENT | $AWK -F/ '{ print $2 }'`
if [ -z "$PROCESSES" ] ; then
if [ $HOST = "localhost" ] ; then
PROCESSES=2
else
PROCESSES=4
fi
fi
if [ $VERBOSE_DISPLAY -ge 2 ] ; then
echo " $PROCESSES Processes."
fi
BUILD_PROCESSES=$[$BUILD_PROCESSES+$PROCESSES]
else
if [ $VERBOSE_DISPLAY -ge 2 ] ; then
echo " Could not verify host, skipping..."
fi
fi
done
fi
if [ ! -z "$COMMIT" ] ; then
export MAKEOPTS+=" -j${BUILD_PROCESSES}"
export DISTCC_HOSTS
fi
if [ $VERBOSE_DISPLAY -ge 1 ] ; then
echo HOSTS: $DISTCC_HOSTS, MAKEOPTS: $MAKEOPTS
fi
fi
|
MiscDebris _________________ Dynamic DISTCC_HOSTS & MAKEOPTS Generation for Portage
Last edited by miscdebris on Thu Feb 21, 2013 1:51 am; edited 10 times in total |
|
Back to top |
|
|
Renton007 n00b
Joined: 14 Aug 2005 Posts: 27
|
Posted: Thu Aug 18, 2005 8:01 pm Post subject: |
|
|
What a great Idea, I'll have to give it a try once I've got distcc setup on my network.
Thanks for the Script. |
|
Back to top |
|
|
nielchiano Veteran
Joined: 11 Nov 2003 Posts: 1287 Location: 50N 3E
|
Posted: Tue Sep 06, 2005 9:20 pm Post subject: |
|
|
looks nice, but when I do emerge -pDu world it runs a LOT of times and waits every time for a few ping-timeouts.....
it should cache it's response for a while (1 or 2 minutes or so) |
|
Back to top |
|
|
ExZombie Apprentice
Joined: 29 May 2004 Posts: 170
|
Posted: Wed Sep 07, 2005 7:53 am Post subject: |
|
|
nielchiano wrote: | looks nice, but when I do emerge -pDu world it runs a LOT of times and waits every time for a few ping-timeouts.....
it should cache it's response for a while (1 or 2 minutes or so) |
IMHO it would be better if it checked for the -p flag and simply not run in such a case. Adding caching would unnecessarily complicate it. |
|
Back to top |
|
|
nielchiano Veteran
Joined: 11 Nov 2003 Posts: 1287 Location: 50N 3E
|
Posted: Wed Sep 07, 2005 4:45 pm Post subject: |
|
|
ExZombie wrote: | IMHO it would be better if it checked for the -p flag and simply not run in such a case. Adding caching would unnecessarily complicate it. |
I'm not sure about this, but I had the impression that during the "finding dependencies"-phase, this script gets run dozens of times. I don't think (but DIDN'T chech) that it matters if it's pretending or not |
|
Back to top |
|
|
miscdebris n00b
Joined: 30 Mar 2005 Posts: 70
|
Posted: Thu Sep 08, 2005 1:38 am Post subject: |
|
|
Yeah, I've known about that pausing issue. The script gets run each time that it gets sourced (several times per emerge). I generally don't mind it, myself. However, I'll see what I can come up with to get around that. Perhaps I can have it avoid running the script more than once every X minutes or so. Since it saves the results in a variable, it should 'cache' easily enough.
I actually think that a timer and cache would be easier than trying to catch the -p flag. The cache should auto-reset between emerge sessions, and might reset between packages.
Also, whenever I run emerge -pDuv world, the script doesn't run once for me. It seems to only run (I only see the echo line, and nmap/ping in top) when it's actually merging something. Has anyone else seen this?
I won't have time to work on this til Friday, unless work gets slow, but I'll see what I can do then.
MiscDebris |
|
Back to top |
|
|
Dlareh Advocate
Joined: 06 Aug 2005 Posts: 2102
|
Posted: Fri Sep 16, 2005 2:31 pm Post subject: |
|
|
I would suggest adding -W1 or -w1 to ping flags. _________________ "Mr Thomas Edison has been up on the two previous nights discovering 'a bug' in his phonograph." --Pall Mall Gazette (1889)
Are we THERE yet? |
|
Back to top |
|
|
miscdebris n00b
Joined: 30 Mar 2005 Posts: 70
|
Posted: Sat Sep 17, 2005 3:12 am Post subject: |
|
|
Good call. Done and done.
Also, I've not figured out a way to get that script to "cache" or detect -p. Ideas?
MiscDebris |
|
Back to top |
|
|
nielchiano Veteran
Joined: 11 Nov 2003 Posts: 1287 Location: 50N 3E
|
Posted: Sat Sep 17, 2005 9:04 am Post subject: |
|
|
you could save the "export $BLABLA" lines into a file (somewhere in the /var/cache dirs looks like it would be a good place)
also export the current time/date.
when the script starts, read out that file and check ik the saved date/time is older than xxx seconds. if it is not, return the saved values; if it is, run the script.
here is a very simple demo-script that does this: Code: |
#!/bin/bash
if [ -r cache ]; then
source cache
if [[ "$DATE" > "$( date '+%Y-%m-%d %H:%M:%S' --date="-60 seconds" )" ]]; then
# we are up to date
echo "old SAVED: $SAVED"
exit 0
fi
fi
echo "export SAVED=\"$(date)\"" > cache
echo "export DATE=\"$( date '+%Y-%m-%d %H:%M:%S' )\"" >> cache
source cache
echo "new SAVED: $SAVED"
|
|
|
Back to top |
|
|
miscdebris n00b
Joined: 30 Mar 2005 Posts: 70
|
Posted: Mon Feb 20, 2006 12:22 am Post subject: |
|
|
Ok, it won't recheck as often, though it may not automatically recheck between emerge executions. I'm working on that now. |
|
Back to top |
|
|
miscdebris n00b
Joined: 30 Mar 2005 Posts: 70
|
Posted: Mon Feb 20, 2006 12:51 am Post subject: |
|
|
Portage seems to dump the environmental variables when it reloads bashrc, so I can't use a variable. Thus, if you do several emerges rapidly, the ones after the first will use the defaults, as though you weren't using this at all. Do any of you know a way to get around this limitation? Something that resets the wait everytime you run emerge from within bashrc or without an alias/altering emerge? |
|
Back to top |
|
|
miscdebris n00b
Joined: 30 Mar 2005 Posts: 70
|
Posted: Wed Feb 22, 2006 5:25 am Post subject: |
|
|
Ok. I got it working. I found a handy dandy little variable that tells me what stage the ebuild is in. $EBUILD_PHASE. When it equals "compile" it runs my script, finding the distcc clients that you have listed, and alters MAKEOPTS accordingly.
Let me know if something doesn't work or if you have any ideas. |
|
Back to top |
|
|
brot Guru
Joined: 06 Apr 2004 Posts: 322
|
Posted: Wed Feb 22, 2006 11:10 am Post subject: ... |
|
|
i dont know if you will eat me now, but there is also another solution for the same problem.
http://wiki.kde.org/icecream
It is basically the same as distcc with the following differences:
When a pc gets turned on, it regulates everything itself
When it gets turned off, the same
even other arches can be used
imho it is really a great piece of software. worth giving a try |
|
Back to top |
|
|
miscdebris n00b
Joined: 30 Mar 2005 Posts: 70
|
Posted: Wed Feb 22, 2006 8:00 pm Post subject: |
|
|
heh Where was that when I started?
I'll probably just stick with mine. I have spent all this time working on it, after all. heh |
|
Back to top |
|
|
Anon5710 Apprentice
Joined: 23 Jul 2004 Posts: 232 Location: Belgium
|
Posted: Wed Feb 22, 2006 9:30 pm Post subject: |
|
|
Hi,
i got a simelar thing going on here.
Ive got 2 pc's both with gentoo linux (these will be the only 2 for now) but both got a dynamic ip (one i can't control).
Both do have something like this : pc1.no-ip.info or pc2.no-ip.info
If you could rewrite your script so that i works for me ? (since i really can't do such a thing) i would really appraciate it
Or even better, show me a website where this script lanua's is explained so i can build one myself |
|
Back to top |
|
|
miscdebris n00b
Joined: 30 Mar 2005 Posts: 70
|
Posted: Wed Feb 22, 2006 11:05 pm Post subject: |
|
|
Anon5710 wrote: |
Ive got 2 pc's both with gentoo linux (these will be the only 2 for now) but both got a dynamic ip (one i can't control).
Both do have something like this : pc1.no-ip.info or pc2.no-ip.info
|
If you have hostnames that follow your up (as you mentioned above), just put those hostnames in your /etc/distcc/hosts file. Using your example, your /etc/distcc/hosts file will look like this
Code: | pc1.no-ip.info pc2.no-ip.info |
or
Code: | pc1.no-ip.info/4 pc2.no-ip.info/4 |
It's not the most secure, but it should work. |
|
Back to top |
|
|
miscdebris n00b
Joined: 30 Mar 2005 Posts: 70
|
Posted: Fri Jun 09, 2006 4:32 am Post subject: |
|
|
Well, it's been a while since I've looked in on this, and I'm wondering if anyone besides me is using this?
MiscDebris |
|
Back to top |
|
|
nielchiano Veteran
Joined: 11 Nov 2003 Posts: 1287 Location: 50N 3E
|
Posted: Fri Jun 09, 2006 8:22 am Post subject: |
|
|
I'd like to use it. But I removed the script again because of this:
nielchiano wrote: | looks nice, but when I do emerge -pDu world it runs a LOT of times and waits every time for a few ping-timeouts.....
it should cache it's response for a while (1 or 2 minutes or so) |
Is this solved already? (sorry, too lazy to read the whole thread) |
|
Back to top |
|
|
miscdebris n00b
Joined: 30 Mar 2005 Posts: 70
|
Posted: Fri Jun 09, 2006 1:28 pm Post subject: |
|
|
Yep. All solved. It only actually checks when you're compiling. Also, the variable $DISPLAY controls how much you see. I'd recommend setting it to 1. If you want to see it working, set $DISPLAY to 2, but be prepared for alot of noise. Set it to 0 to display nothing. |
|
Back to top |
|
|
nielchiano Veteran
Joined: 11 Nov 2003 Posts: 1287 Location: 50N 3E
|
Posted: Fri Jun 09, 2006 1:32 pm Post subject: |
|
|
miscdebris wrote: | Yep. All solved. It only actually checks when you're compiling. Also, the variable $DISPLAY controls how much you see. I'd recommend setting it to 1. If you want to see it working, set $DISPLAY to 2, but be prepared for alot of noise. Set it to 0 to display nothing. |
I'll check it out, after my exams...
Oh, don't use $DISPLAY, since that is used by all X-programs to find the running X-server. use $VERBOSE or something like that. It works here, since you overwrite it locally, but it's cleaner to use another name. |
|
Back to top |
|
|
miscdebris n00b
Joined: 30 Mar 2005 Posts: 70
|
Posted: Fri Jun 09, 2006 2:00 pm Post subject: |
|
|
Ahh, good point. I'll change it up top.
Thanx. |
|
Back to top |
|
|
miscdebris n00b
Joined: 30 Mar 2005 Posts: 70
|
|
Back to top |
|
|
xaviermiller Bodhisattva
Joined: 23 Jul 2004 Posts: 8711 Location: ~Brussels - Belgique
|
Posted: Thu Mar 15, 2012 8:47 pm Post subject: |
|
|
Old post necromancy...
But I vote to add that feature in Portage ! _________________ Kind regards,
Xavier Miller |
|
Back to top |
|
|
Ant P. Watchman
Joined: 18 Apr 2009 Posts: 6920
|
Posted: Fri Mar 16, 2012 9:31 pm Post subject: |
|
|
I wrote this for myself. Requires Avahi-enabled distcc, but it should work in any package manager. |
|
Back to top |
|
|
miscdebris n00b
Joined: 30 Mar 2005 Posts: 70
|
|
Back to top |
|
|
|