Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Dynamic DISTCC_HOSTS & MAKEOPTS Generation for Portage
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
miscdebris
n00b
n00b


Joined: 30 Mar 2005
Posts: 70

PostPosted: Wed Aug 17, 2005 10:00 pm    Post subject: Dynamic DISTCC_HOSTS & MAKEOPTS Generation for Portage Reply with quote

I have quite a few computers at home, and most of the time, not all of them are on. When I use distcc, I had to reconfigure it every time I turned a computer on or off. Especially MAKEOPTS. So I set out to write a script to take care of that.

This script will reside in /etc/portage/bashrc and will try to detect which computers are up (and with nmap installed, which are running distccd)

This script does the following:

    Checks to see if distcc (and not -distcc) is in your features
    Reads in your /etc/distcc/hosts file
    Auto validates your localhost
    Pings any non-localhost in your /etc/distcc/hosts file
    If it gets a response and you have nmap installed, it checks to see if distccd is running on that computer (port 3632 default)
    Any validated host has it's /limit added to the MAKEOPTS=-j (it understands the defaults)
    Writes the valid hosts to your DISTCC_HOSTS variable
    Displays the variables DISTCC_HOSTS and MAKEOPTS
    Will run once each for each package right before it compiles.
    Automaticly finding of active DISTCC hosts (NEW)



Future Features:

    Thresholds as to how to use localhost


I appreciate any comments, suggestions, bug fixes, etc.

And without further waiting, here it is.

Code:
#!/bin/sh

# Options most likely to be changed
# Control the amount of information displayed.
# 0 displays nothing
# 1 displays the DISTCC_HOSTS and MAKEOPTS that will be used
# 2 displays the above plus the detection process
VERBOSE_DISPLAY=1
RANDOM_HOSTS=1
DISTCC_NETWORKS=172.20.1.0/24
IGNORE_HOSTS=172.20.1.2
DEFAULT_PROCESSES=2


# Options unlikely to need changed

CAT=/bin/cat
GREP=/bin/grep
AWK=/bin/awk
PING=/bin/ping
NMAP=/usr/bin/nmap
DATE=/usr/bin/date
PORT=3632

if [ `echo ${FEATURES} | $GREP -q -e distcc ; echo $?` == 0 ] && [ `echo ${FEATURES} | $GREP -q -e -distcc ; echo $?` != 0 ] && [ -n "$EBUILD_PHASE" ] && [ $EBUILD_PHASE == "compile" ]; then

   if [ $RANDOM_HOSTS -ge 1 ] ; then
      DISTCC_HOSTS=""
      BUILD_PROCESSES=""
      for X in `$NMAP --randomize-hosts -n -sT -p $PORT $DISTCC_NETWORKS --open --exclude $IGNORE_HOSTS | $GREP Interesting| $AWK '{ print $4 }'| $AWK -F: '{ print $1 }'` ; do
         DISTCC_HOSTS="$DISTCC_HOSTS ${X}/${DEFAULT_PROCESSES}"
         BUILD_PROCESSES=$[$BUILD_PROCESSES+$DEFAULT_PROCESSES]
      done
      DISTCC_HOSTS="localhost/1 $DISTCC_HOSTS localhost/1"
      BUILD_PROCESSES=$[$BUILD_PROCESSES+2]
      COMMIT=1
   else

      if [ -f "/etc/distcc/hosts" ] ; then
         DISTCC_CLIENTS=`echo | $CAT /etc/distcc/hosts`
      fi

      DISTCC_HOSTS=""

      for CLIENT in $DISTCC_CLIENTS ; do
         HOST=`echo $CLIENT | $AWK -F/ '{ print $1 }'`

         if [ $VERBOSE_DISPLAY -ge 2 ] ; then
            echo -n "Checking ${CLIENT}..."
         fi
   
         VALID=0
         PROCESSES=0
         if [ $HOST = "localhost" ] ; then
            VALID=1
         elif [ `$PING -c 1 -n -q  $HOST > /dev/null ; echo $?` = 0 ] ; then
            if [ -f $NMAP ] ; then
               if [ `$NMAP -n -sT -p $PORT $HOST | $GREP $PORT | $AWK '{ print $2 }'` = open ] ; then
                  VALID=1
               else
                  VALID=0
               fi
            else
               VALID=1
            fi
         fi

         if [ $VALID = 1 ] ; then

            if [ $VERBOSE_DISPLAY -ge 2 ] ; then
               echo -n " Verified.  Checking Processes..."
            fi

            COMMIT=1

            if [ ! -z "$DISTCC_HOSTS" ] ; then
               DISTCC_HOSTS="${DISTCC_HOSTS} ${CLIENT}"
            else
               DISTCC_HOSTS="${CLIENT}"
            fi

            PROCESSES=`echo $CLIENT | $AWK -F/ '{ print $2 }'`

            if [ -z "$PROCESSES" ] ; then
               if [ $HOST = "localhost" ] ; then
                  PROCESSES=2
               else
                  PROCESSES=4
               fi
            fi

            if [ $VERBOSE_DISPLAY -ge 2 ] ; then
               echo " $PROCESSES Processes."
            fi

            BUILD_PROCESSES=$[$BUILD_PROCESSES+$PROCESSES]
         else
            if [ $VERBOSE_DISPLAY -ge 2 ] ; then
               echo " Could not verify host, skipping..."
            fi
         fi
      done
   fi


   if [ ! -z "$COMMIT" ] ; then
      export MAKEOPTS+=" -j${BUILD_PROCESSES}"
      export DISTCC_HOSTS
   fi

   if [ $VERBOSE_DISPLAY -ge 1 ] ; then
      echo HOSTS: $DISTCC_HOSTS, MAKEOPTS: $MAKEOPTS
   fi
fi


MiscDebris
_________________
Dynamic DISTCC_HOSTS & MAKEOPTS Generation for Portage


Last edited by miscdebris on Thu Feb 21, 2013 1:51 am; edited 10 times in total
Back to top
View user's profile Send private message
Renton007
n00b
n00b


Joined: 14 Aug 2005
Posts: 27

PostPosted: Thu Aug 18, 2005 8:01 pm    Post subject: Reply with quote

What a great Idea, I'll have to give it a try once I've got distcc setup on my network.

Thanks for the Script.
Back to top
View user's profile Send private message
nielchiano
Veteran
Veteran


Joined: 11 Nov 2003
Posts: 1287
Location: 50N 3E

PostPosted: Tue Sep 06, 2005 9:20 pm    Post subject: Reply with quote

looks nice, but when I do emerge -pDu world it runs a LOT of times and waits every time for a few ping-timeouts.....
it should cache it's response for a while (1 or 2 minutes or so)
Back to top
View user's profile Send private message
ExZombie
Apprentice
Apprentice


Joined: 29 May 2004
Posts: 170

PostPosted: Wed Sep 07, 2005 7:53 am    Post subject: Reply with quote

nielchiano wrote:
looks nice, but when I do emerge -pDu world it runs a LOT of times and waits every time for a few ping-timeouts.....
it should cache it's response for a while (1 or 2 minutes or so)


IMHO it would be better if it checked for the -p flag and simply not run in such a case. Adding caching would unnecessarily complicate it.
Back to top
View user's profile Send private message
nielchiano
Veteran
Veteran


Joined: 11 Nov 2003
Posts: 1287
Location: 50N 3E

PostPosted: Wed Sep 07, 2005 4:45 pm    Post subject: Reply with quote

ExZombie wrote:
IMHO it would be better if it checked for the -p flag and simply not run in such a case. Adding caching would unnecessarily complicate it.

I'm not sure about this, but I had the impression that during the "finding dependencies"-phase, this script gets run dozens of times. I don't think (but DIDN'T chech) that it matters if it's pretending or not
Back to top
View user's profile Send private message
miscdebris
n00b
n00b


Joined: 30 Mar 2005
Posts: 70

PostPosted: Thu Sep 08, 2005 1:38 am    Post subject: Reply with quote

Yeah, I've known about that pausing issue. The script gets run each time that it gets sourced (several times per emerge). I generally don't mind it, myself. However, I'll see what I can come up with to get around that. Perhaps I can have it avoid running the script more than once every X minutes or so. Since it saves the results in a variable, it should 'cache' easily enough.

I actually think that a timer and cache would be easier than trying to catch the -p flag. The cache should auto-reset between emerge sessions, and might reset between packages.

Also, whenever I run emerge -pDuv world, the script doesn't run once for me. It seems to only run (I only see the echo line, and nmap/ping in top) when it's actually merging something. Has anyone else seen this?

I won't have time to work on this til Friday, unless work gets slow, but I'll see what I can do then.

MiscDebris
Back to top
View user's profile Send private message
Dlareh
Advocate
Advocate


Joined: 06 Aug 2005
Posts: 2102

PostPosted: Fri Sep 16, 2005 2:31 pm    Post subject: Reply with quote

I would suggest adding -W1 or -w1 to ping flags.
_________________
"Mr Thomas Edison has been up on the two previous nights discovering 'a bug' in his phonograph." --Pall Mall Gazette (1889)
Are we THERE yet?
Back to top
View user's profile Send private message
miscdebris
n00b
n00b


Joined: 30 Mar 2005
Posts: 70

PostPosted: Sat Sep 17, 2005 3:12 am    Post subject: Reply with quote

Good call. Done and done.

Also, I've not figured out a way to get that script to "cache" or detect -p. Ideas?

MiscDebris
Back to top
View user's profile Send private message
nielchiano
Veteran
Veteran


Joined: 11 Nov 2003
Posts: 1287
Location: 50N 3E

PostPosted: Sat Sep 17, 2005 9:04 am    Post subject: Reply with quote

you could save the "export $BLABLA" lines into a file (somewhere in the /var/cache dirs looks like it would be a good place)
also export the current time/date.

when the script starts, read out that file and check ik the saved date/time is older than xxx seconds. if it is not, return the saved values; if it is, run the script.

here is a very simple demo-script that does this:
Code:

#!/bin/bash

if [ -r cache ]; then
        source cache
        if [[ "$DATE" > "$( date '+%Y-%m-%d %H:%M:%S' --date="-60 seconds" )" ]]; then
                # we are up to date
                echo "old SAVED: $SAVED"
                exit 0
        fi
fi

echo "export SAVED=\"$(date)\"" > cache
echo "export DATE=\"$( date '+%Y-%m-%d %H:%M:%S' )\"" >> cache

source cache
echo "new SAVED: $SAVED"
Back to top
View user's profile Send private message
miscdebris
n00b
n00b


Joined: 30 Mar 2005
Posts: 70

PostPosted: Mon Feb 20, 2006 12:22 am    Post subject: Reply with quote

Ok, it won't recheck as often, though it may not automatically recheck between emerge executions. I'm working on that now.
Back to top
View user's profile Send private message
miscdebris
n00b
n00b


Joined: 30 Mar 2005
Posts: 70

PostPosted: Mon Feb 20, 2006 12:51 am    Post subject: Reply with quote

Portage seems to dump the environmental variables when it reloads bashrc, so I can't use a variable. Thus, if you do several emerges rapidly, the ones after the first will use the defaults, as though you weren't using this at all. Do any of you know a way to get around this limitation? Something that resets the wait everytime you run emerge from within bashrc or without an alias/altering emerge?
Back to top
View user's profile Send private message
miscdebris
n00b
n00b


Joined: 30 Mar 2005
Posts: 70

PostPosted: Wed Feb 22, 2006 5:25 am    Post subject: Reply with quote

Ok. I got it working. I found a handy dandy little variable that tells me what stage the ebuild is in. $EBUILD_PHASE. When it equals "compile" it runs my script, finding the distcc clients that you have listed, and alters MAKEOPTS accordingly.

Let me know if something doesn't work or if you have any ideas.
Back to top
View user's profile Send private message
brot
Guru
Guru


Joined: 06 Apr 2004
Posts: 322

PostPosted: Wed Feb 22, 2006 11:10 am    Post subject: ... Reply with quote

i dont know if you will eat me now, but there is also another solution for the same problem.

http://wiki.kde.org/icecream

It is basically the same as distcc with the following differences:
    When a pc gets turned on, it regulates everything itself
    When it gets turned off, the same
    even other arches can be used


imho it is really a great piece of software. worth giving a try ;)
Back to top
View user's profile Send private message
miscdebris
n00b
n00b


Joined: 30 Mar 2005
Posts: 70

PostPosted: Wed Feb 22, 2006 8:00 pm    Post subject: Reply with quote

heh Where was that when I started?

I'll probably just stick with mine. I have spent all this time working on it, after all. heh
Back to top
View user's profile Send private message
Anon5710
Apprentice
Apprentice


Joined: 23 Jul 2004
Posts: 232
Location: Belgium

PostPosted: Wed Feb 22, 2006 9:30 pm    Post subject: Reply with quote

Hi,

i got a simelar thing going on here.

Ive got 2 pc's both with gentoo linux (these will be the only 2 for now) but both got a dynamic ip (one i can't control).
Both do have something like this : pc1.no-ip.info or pc2.no-ip.info


If you could rewrite your script so that i works for me ? (since i really can't do such a thing) i would really appraciate it :)
Or even better, show me a website where this script lanua's is explained so i can build one myself :)
Back to top
View user's profile Send private message
miscdebris
n00b
n00b


Joined: 30 Mar 2005
Posts: 70

PostPosted: Wed Feb 22, 2006 11:05 pm    Post subject: Reply with quote

Anon5710 wrote:

Ive got 2 pc's both with gentoo linux (these will be the only 2 for now) but both got a dynamic ip (one i can't control).
Both do have something like this : pc1.no-ip.info or pc2.no-ip.info


If you have hostnames that follow your up (as you mentioned above), just put those hostnames in your /etc/distcc/hosts file. Using your example, your /etc/distcc/hosts file will look like this
Code:
pc1.no-ip.info pc2.no-ip.info

or
Code:
pc1.no-ip.info/4 pc2.no-ip.info/4

It's not the most secure, but it should work.
Back to top
View user's profile Send private message
miscdebris
n00b
n00b


Joined: 30 Mar 2005
Posts: 70

PostPosted: Fri Jun 09, 2006 4:32 am    Post subject: Reply with quote

Well, it's been a while since I've looked in on this, and I'm wondering if anyone besides me is using this?

MiscDebris
Back to top
View user's profile Send private message
nielchiano
Veteran
Veteran


Joined: 11 Nov 2003
Posts: 1287
Location: 50N 3E

PostPosted: Fri Jun 09, 2006 8:22 am    Post subject: Reply with quote

I'd like to use it. But I removed the script again because of this:
nielchiano wrote:
looks nice, but when I do emerge -pDu world it runs a LOT of times and waits every time for a few ping-timeouts.....
it should cache it's response for a while (1 or 2 minutes or so)

Is this solved already? (sorry, too lazy to read the whole thread)
Back to top
View user's profile Send private message
miscdebris
n00b
n00b


Joined: 30 Mar 2005
Posts: 70

PostPosted: Fri Jun 09, 2006 1:28 pm    Post subject: Reply with quote

Yep. All solved. It only actually checks when you're compiling. Also, the variable $DISPLAY controls how much you see. I'd recommend setting it to 1. If you want to see it working, set $DISPLAY to 2, but be prepared for alot of noise. Set it to 0 to display nothing.
Back to top
View user's profile Send private message
nielchiano
Veteran
Veteran


Joined: 11 Nov 2003
Posts: 1287
Location: 50N 3E

PostPosted: Fri Jun 09, 2006 1:32 pm    Post subject: Reply with quote

miscdebris wrote:
Yep. All solved. It only actually checks when you're compiling. Also, the variable $DISPLAY controls how much you see. I'd recommend setting it to 1. If you want to see it working, set $DISPLAY to 2, but be prepared for alot of noise. Set it to 0 to display nothing.

I'll check it out, after my exams... :-(
Oh, don't use $DISPLAY, since that is used by all X-programs to find the running X-server. use $VERBOSE or something like that. It works here, since you overwrite it locally, but it's cleaner to use another name.
Back to top
View user's profile Send private message
miscdebris
n00b
n00b


Joined: 30 Mar 2005
Posts: 70

PostPosted: Fri Jun 09, 2006 2:00 pm    Post subject: Reply with quote

Ahh, good point. I'll change it up top.

Thanx.
Back to top
View user's profile Send private message
miscdebris
n00b
n00b


Joined: 30 Mar 2005
Posts: 70

PostPosted: Thu Sep 06, 2007 5:31 am    Post subject: Reply with quote

Updated the script to find hosts in a random order. It can still use /etc/distcc/hosts, just set RANDOM_HOSTS=0.
_________________
Dynamic DISTCC_HOSTS & MAKEOPTS Generation for Portage
Back to top
View user's profile Send private message
xaviermiller
Bodhisattva
Bodhisattva


Joined: 23 Jul 2004
Posts: 8706
Location: ~Brussels - Belgique

PostPosted: Thu Mar 15, 2012 8:47 pm    Post subject: Reply with quote

Old post necromancy...

But I vote to add that feature in Portage ! 8)
_________________
Kind regards,
Xavier Miller
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 6920

PostPosted: Fri Mar 16, 2012 9:31 pm    Post subject: Reply with quote

I wrote this for myself. Requires Avahi-enabled distcc, but it should work in any package manager.
Back to top
View user's profile Send private message
miscdebris
n00b
n00b


Joined: 30 Mar 2005
Posts: 70

PostPosted: Wed Mar 21, 2012 6:46 pm    Post subject: Reply with quote

It's been a while since I've used Gentoo. Does my script still work? I can do a gentoo install to get it working again if it doesn't.
_________________
Dynamic DISTCC_HOSTS & MAKEOPTS Generation for Portage
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum