| View previous topic :: View next topic |
| Author |
Message |
Luc484
Veteran
Joined: 26 Mar 2005
Posts: 1035
Location: Italy
|
 Posted: Sat Sep 23, 2006 12:50 am Post subject: distcc and local compilation |
 |
|
Hi guys. I have a question. I'm in this situation: I have a desktop and a laptop, both running Gentoo. The laptop is old, and is absolutely slow (much slower than the desktop). I thought it would have been a good idea to use distcc to distribute the work over the network to compile the software I emerge on my laptop. After some months, I noticed that it seems not a good idea. I tried many solutions, but it seems there is not a improvement, and I would like to understand why. The network is at 100Mbit. I noticed using splat that using distcc doesn't help. I tried using "desktop laptop" and "laptop desktop" is more or less the same, and in some cases this is slower than compiling without distcc. I thought it was due to the use of the laptop, which is far more slow than the desktop, so I thought... well then use the desktop only! But this is slower as well. I mean that any way I put it, using only localhost with this configuration is not that different. Is this normal? Maybe distcc doesn't give improvements with two systems only?
Another question: using distcc to compile only with the desktop, is not the same as compiling a software on the desktop, right? I mean, using as hosts only the desktop, is not a good idea, even if the desktop is much faster than localhost. Would it be possible to export the entire compilation from the laptop to the desktop, so that the desktop compiles the software and gives back the result to the laptop only at the end?
Last edited by Luc484 on Fri Sep 29, 2006 12:06 am; edited 2 times in total |
|
| Back to top |
|
 |
all-inc.
Tux's lil' helper
Joined: 03 Jul 2004
Posts: 138
Location: Darmstadt.Germany.EU
|
| Posted: Sat Sep 23, 2006 1:44 am Post subject: |
|
|
hi,
about your laptop...can you become a little more concrete  ? perhaps ram, cpu type/clock of both of your system.
the problem is that distcc produces some overhead... a lot for baby computers  (i assume they are both of the same arch(x86), right?)
i would suggest localhost/0 dekstop/3, but i need to know more specific about the systems. but you said this doesn't work...well the preprocessor always runs on the target machine(thus your notebook) and if your laptop is REALLY slow distcc won't help.
to the second question, as already said, only compiling and assembling is run on the host machines, linking, preprocessing etc. is run on the target.
but you can let your desktop completely compile for your laptop. there are some howtos out there.
_________________
# make config; not war
registered linux user # 265707
Last edited by all-inc. on Sat Sep 23, 2006 1:45 am; edited 1 time in total |
|
| Back to top |
|
|
psycepa
Veteran
Joined: 27 Dec 2004
Posts: 1637
Location: Poland
|
| Posted: Sat Sep 23, 2006 1:45 am Post subject: |
|
|
Hi,
as far as i remember, the speed of distcc comiplation is limited to the speed of compilation on a slowest machine in the set (although i can be wrong )
and that was what i observed while using distcc
as a proof you could give some
timings for foo package (any, but the best would be some big app, like gcc or ooffice) from the normal compilation (without distcc), from the compilation on one host with distcc, and on the compilation on two hosts with distcc,
then you would have a clear look at what is really going on and not just a 'feeling'
in my case turning off distcc on a laptop (which in contrary to your case is faster than desktop) improved compilation to some extend (there is only 300 MHz difference between CPUs of my machines)
_________________
RLUNo. 376073
bo czytac, to trzeba ze zrozumieniem (c) by psycepa
"nigdy nie kloc sie z glupcem, najpierw sprowadzi cie do swojego poziomu a potem pokona doswiadczeniem"
"Life is complex. It has its real and imaginary part." |
|
| Back to top |
|
|
Luc484
Veteran
Joined: 26 Mar 2005
Posts: 1035
Location: Italy
|
| Posted: Sat Sep 23, 2006 2:05 am Post subject: |
|
|
| all-inc. wrote: | hi,
about your laptop...can you become a little more concrete ? perhaps ram, cpu type/clock of both of your system. |
cpu: Intel 2GHz
ram: 128MB
But I think there is something strange with the hd. I think the access time is very long, and that poor RAM with long access time to hd make performance sink.
| all-inc. wrote: | | the problem is that distcc produces some overhead... a lot for baby computers (i assume they are both of the same arch(x86), right?) |
Yes, they are.
| all-inc. wrote: | | i would suggest localhost/0 dekstop/3, but i need to know more specific about the systems. but you said this doesn't work...well the preprocessor always runs on the target machine(thus your notebook) and if your laptop is REALLY slow distcc won't help. |
Using both from what I saw doesn't help. But what are those "/0" and "/3"? Are those contemporary compilations?
| all-inc. wrote: | to the second question, as already said, only compiling and assembling is run on the host machines, linking, preprocessing etc. is run on the target.
but you can let your desktop completely compile for your laptop. there are some howtos out there. |
So, using distcc using the only desktop as host is not like compiling the package on my desktop completely. There is some more work performed by the laptop. I've been looking around for a guide to do that, but I can't find it. What should I look for?
Thank you for your answer. |
|
| Back to top |
|
|
Luc484
Veteran
Joined: 26 Mar 2005
Posts: 1035
Location: Italy
|
| Posted: Sat Sep 23, 2006 2:16 am Post subject: |
|
|
| psycepa wrote: | Hi,
as far as i remember, the speed of distcc comiplation is limited to the speed of compilation on a slowest machine in the set (although i can be wrong ) |
Wait... you mean that if a, b, and c are three different machines and I use distcc to use them to compile I get that the speed of {a, b, c} is:
| Code: | | speed({a, b, c}) = min{speed(a), speed(b), speed(c)} |
?? Seems useless to me. Maybe I misunderstood.
| all-inc. wrote: | as a proof you could give some
timings for foo package (any, but the best would be some big app, like gcc or ooffice) from the normal compilation (without distcc), from the compilation on one host with distcc, and on the compilation on two hosts with distcc,
then you would have a clear look at what is really going on and not just a 'feeling' |
That is what I did. I used calc to draw diagrams as a function of n where n is the number used as the flag -jn. But I saw there is not a great improvement. In some cases it is slower. I thought that using the only desktop could have helped, but maybe I did it the wrong way. I did it thruogh:
| Code: | | /usr/bin/distcc-config --set-hosts "desktop" |
but the compilation time is far longer than the compilation time of the desktop. Maybe this is not the right way to do this. |
|
| Back to top |
|
|
psycepa
Veteran
Joined: 27 Dec 2004
Posts: 1637
Location: Poland
|
| Posted: Sat Sep 23, 2006 2:28 am Post subject: |
|
|
from the gentoo distcc documentation:
| Quote: |
(There are also several other methods of setting up hosts. See the
distcc manpage for more details.)
If you wish to compile on the local machine you should put 'localhost'
in the hosts list. Conversely if you do not wish to use the local machine to
compile (which is often the case) omit it from the hosts list. On a slow
machine using localhost may actually slow things down. Make sure to test your
settings for performance.
|
what means that if a slow machine is in the set, things will be (can be?) slowed down
that is why one should adjust settings, couse if for example every machine gets same amouint of work, fast machines will do it faster than a slow machine (woo, i have discoverd an America )
but maby if you set fast machines to get twice or three times more work than slow one, things will go to equilibrium state (every machine will end work in more or less the same time)
I don't know that exactly since I haven't done such adjustment.
As far as I can recall discussions on distcc there was a lot of such statements, like my from my first post in this thread.
and you have to remember that what all-inc already wrote, distcc introduces some overhead, division of tasks takes some time, even if there is only division by 1...
greetingz
_________________
RLUNo. 376073
bo czytac, to trzeba ze zrozumieniem (c) by psycepa
"nigdy nie kloc sie z glupcem, najpierw sprowadzi cie do swojego poziomu a potem pokona doswiadczeniem"
"Life is complex. It has its real and imaginary part." |
|
| Back to top |
|
|
all-inc.
Tux's lil' helper
Joined: 03 Jul 2004
Posts: 138
Location: Darmstadt.Germany.EU
|
| Posted: Sat Sep 23, 2006 2:37 am Post subject: |
|
|
the "/n" after the hostnames tell distcc how much work the host should do simultaneously(man-pages... ). like the -j option. it is very important that you set this to minimum 3 for your dekstop. further, set this number also in /etc/conf.d/distccd on your dektop. or your desktop only accepts one job...
i still don't got it -> 128mb ram on the laptop, right? and cpu? and ram on dekstop? cpu is 2GHz, right?
so try to leave localhost out and only use desktop/3. you could also run top while compiling on your notebook and look at the values for wait in the 3. row...this is the percentage of time the cpu didn't do anything in the last second becaue it waits for disk I/O. could be useful if you think your disk is broken.
edit: and don't forget to set MAKEOPTS="-j3" in make.conf on the laptop
_________________
# make config; not war
registered linux user # 265707 |
|
| Back to top |
|
|
Luc484
Veteran
Joined: 26 Mar 2005
Posts: 1035
Location: Italy
|
| Posted: Sat Sep 23, 2006 3:33 am Post subject: |
|
|
| all-inc. wrote: | the "/n" after the hostnames tell distcc how much work the host should do simultaneously(man-pages... ). like the -j option. it is very important that you set this to minimum 3 for your dekstop. further, set this number also in /etc/conf.d/distccd on your dektop. or your desktop only accepts one job...
i still don't got it -> 128mb ram on the laptop, right? and cpu? and ram on dekstop? cpu is 2GHz, right? |
laptop
cpu: intel 2GHz
ram: 128MB
desktop
cpu: amd 2800+
ram 728MB
| all-inc. wrote: | so try to leave localhost out and only use desktop/3. you could also run top while compiling on your notebook and look at the values for wait in the 3. row...this is the percentage of time the cpu didn't do anything in the last second becaue it waits for disk I/O. could be useful if you think your disk is broken.
edit: and don't forget to set MAKEOPTS="-j3" in make.conf on the laptop |
Ok, seems to work. I looked at distccmon-gui. With this configuration it seems to be slower than before. With distccmon-gui I see total compile time is a low percentage of the total time. There is much time in which no compilation is running. This means the problem stays on the laptop during the work left.
Maybe the last chance is to delegate everything to the desktop.
EDIT: Just to be more precise, I made a test with konqueror and it took me 22 minutes and 17 seconds to compile with desktop/3. On my desktop I get a averange time of 6 minutes, and 26 seconds. Compilation without distccd running is more or less 21 minutes. |
|
| Back to top |
|
|
runningwithscissors
Guru
Joined: 21 Apr 2006
Posts: 454
Location: the third world
|
| Posted: Sat Sep 23, 2006 8:32 am Post subject: |
|
|
| 128 MB of memory isn't great for compiles. Compile everything on your desktop. |
|
| Back to top |
|
|
Luc484
Veteran
Joined: 26 Mar 2005
Posts: 1035
Location: Italy
|
| Posted: Sat Sep 23, 2006 9:52 am Post subject: |
|
|
But, is it possible to compile everuthing on the desktop and then bring the result back to the laptop? What should I look for?
Many thanks to everyone. |
|
| Back to top |
|
|
all-inc.
Tux's lil' helper
Joined: 03 Jul 2004
Posts: 138
Location: Darmstadt.Germany.EU
|
| Posted: Tue Sep 26, 2006 1:12 pm Post subject: |
|
|
Have a look at this. It could fit your needs!
_________________
# make config; not war
registered linux user # 265707 |
|
| Back to top |
|
|
Luc484
Veteran
Joined: 26 Mar 2005
Posts: 1035
Location: Italy
|
| Posted: Tue Sep 26, 2006 1:41 pm Post subject: |
|
|
| all-inc. wrote: | | Have a look at this. It could fit your needs! |
I think it is similar to this one I found in the italian forums. Unfortunately I'm having a problem after the chroot executing env-update. I get:
| Code: | cluca ~ # chroot /mnt/nfs/ /bin/bash -c 'env-update && source /etc/profile && exec /bin/bash'
/bin/bash: line 1: 24447 Illegal instruction env-update |
I think I made something wrong, but I can't understand what.
Thanks for your advice. |
|
| Back to top |
|
|
all-inc.
Tux's lil' helper
Joined: 03 Jul 2004
Posts: 138
Location: Darmstadt.Germany.EU
|
| Posted: Tue Sep 26, 2006 2:11 pm Post subject: |
|
|
I think you get this error because the $PATH variable isn't set at this point or similar.
Try to run them in order:
| Code: | # chroot /mnt/nfs/ /bin/bash
# env-update && source /etc/profile |
I can't speak italian . but because of /mnt/nfs it seems that in this method you mount the / of the small machine on your big machine via nfs, right? and then chroot to this? Well, because of space problems once again while emerging openoffice-2.0.3(it's crazy...needs more time than emerge world -e ), I mounted /var/tmp/portage via nfs on a machine with enough free space...it was the hell slow when it came to the install part of emerging... Please report me your experiences when chrooting into a nfs mounted environment. Is it really fast enough? Thanks.
_________________
# make config; not war
registered linux user # 265707 |
|
| Back to top |
|
|
davidgurvich
Veteran
Joined: 23 Apr 2004
Posts: 1063
|
| Posted: Tue Sep 26, 2006 2:18 pm Post subject: |
|
|
| If both systems are compatible (not requiring a cross-compile toolchain), just build packages for everything on the faster machine, then export the packages directory to the slower and use those to install everything. |
|
| Back to top |
|
|
davidgurvich
Veteran
Joined: 23 Apr 2004
Posts: 1063
|
| Posted: Tue Sep 26, 2006 2:19 pm Post subject: |
|
|
| If both systems are compatible (not requiring a cross-compile toolchain), just build packages for everything on the faster machine, then export the packages directory to the slower and use those to install everything. |
|
| Back to top |
|
|
Luc484
Veteran
Joined: 26 Mar 2005
Posts: 1035
Location: Italy
|
| Posted: Tue Sep 26, 2006 2:35 pm Post subject: |
|
|
| all-inc. wrote: | I think you get this error because the $PATH variable isn't set at this point or similar.
Try to run them in order:
| Code: | # chroot /mnt/nfs/ /bin/bash
# env-update && source /etc/profile |
I can't speak italian . but because of /mnt/nfs it seems that in this method you mount the / of the small machine on your big machine via nfs, right? and then chroot to this? Well, because of space problems once again while emerging openoffice-2.0.3(it's crazy...needs more time than emerge world -e ), I mounted /var/tmp/portage via nfs on a machine with enough free space...it was the hell slow when it came to the install part of emerging... Please report me your experiences when chrooting into a nfs mounted environment. Is it really fast enough? Thanks. |
 Yes, that's why I didn't post the link before. It's in Italian language. Anyway I already tried to do that, but I can chroot, but then any command I give I get the same error I reported before, something like this:
| Code: | cluca packages # chroot /mnt/nfs/ /bin/bash
cluca / # env-update
Illegal instruction
cluca / # emerge -a test
Illegal instruction
cluca / # ls
bin boot dev etc home lib lost+found mnt opt proc root sbin sys tmp usr var
cluca reiser # dispatch-conf
Illegal instruction |
|
|
| Back to top |
|
|
Luc484
Veteran
Joined: 26 Mar 2005
Posts: 1035
Location: Italy
|
| Posted: Tue Sep 26, 2006 2:37 pm Post subject: |
|
|
| davidgurvich wrote: | | If both systems are compatible (not requiring a cross-compile toolchain), just build packages for everything on the faster machine, then export the packages directory to the slower and use those to install everything. |
The problem is that I cannot do this. I cannot compile packages and then install them since one is a athlon xp, the other is a pentium4. Would be useful to do this for big packages like openoffice or firefox. |
|
| Back to top |
|
|
all-inc.
Tux's lil' helper
Joined: 03 Jul 2004
Posts: 138
Location: Darmstadt.Germany.EU
|
| Posted: Tue Sep 26, 2006 2:59 pm Post subject: |
|
|
hmm, davidgurvich, this is exactly the method used in the link I posted above!
| Quote: | | I cannot compile packages and then install them since one is a athlon xp, the other is a pentium4. |
Not really. If you read the link I posted, you see that you chroot in a different environment for that , eg., ahton-xp box. and there you have your own make.conf where you can set CFLAGS="-march=athlon-xp -O2 -fomit-frame-pointers -pipe ...". then, you can compile on your p4 box packages for your athlon-xp. gcc doesn't care if it is run on a pentium4, you can still do -march athlon-xp. I don't know if -march=athlon-xp compiled binaries run on a p4(hmm could you try it? would be interesting ). But that's not a problem: You can do this | Code: | mkdir -p /home/gentoo/p4bin/{,usr/}{,s}bin
for p in {/bin,/sbin,/usr/bin,/usr/sbin,whatyouneedalso}; do mount --bind $p /home/gentoo/p4bin$p; done |
if /home/gentoo is the path to your chroot environment.
then chroot and set PATH="/p4bin/bin:/p4bin/sbin:..." and finally you can run all tools that are needed from p4 binaries(like python, sed, install, perl, ...) in your athlon-xp environment.
hmm, some work
_________________
# make config; not war
registered linux user # 265707 |
|
| Back to top |
|
|
all-inc.
Tux's lil' helper
Joined: 03 Jul 2004
Posts: 138
Location: Darmstadt.Germany.EU
|
| Posted: Tue Sep 26, 2006 3:09 pm Post subject: |
|
|
Ah, and to "Illegal instruction"
This error occurs if you try to run, eg. athlon-xp bins on a p4. Just make all the 'mount --bind' i've mentioned at /mnt/nfs instead of /home/gentoo and try | Code: | | chroot /mnt/nfs /p4bin/bin/bash |
_________________
# make config; not war
registered linux user # 265707 |
|
| Back to top |
|
|
Luc484
Veteran
Joined: 26 Mar 2005
Posts: 1035
Location: Italy
|
| Posted: Tue Sep 26, 2006 4:11 pm Post subject: |
|
|
| all-inc. wrote: | Ah, and to "Illegal instruction"
This error occurs if you try to run, eg. athlon-xp bins on a p4. Just make all the 'mount --bind' i've mentioned at /mnt/nfs instead of /home/gentoo and try | Code: | | chroot /mnt/nfs /p4bin/bin/bash |
|
arg! I did something wrong again:
| Code: | cluca packages # mkdir -p /mnt/nfs/p4bin/{,usr/}{,s}bin
cluca packages # for p in {/bin,/sbin,/usr/bin,/usr/sbin}; do mount --bind $p /mnt/nfs/p4bin$p; done
cluca packages # chroot /mnt/nfs/ /p4bin/bin/bash
cluca / # PATH="/p4bin/bin:/p4bin/sbin:/usr/bin:/usr/sbin"
cluca / # env-update
Illegal instruction
cluca / # |
Do you see something wrong here?
Many thanks. Seems great this thing. |
|
| Back to top |
|
|
all-inc.
Tux's lil' helper
Joined: 03 Jul 2004
Posts: 138
Location: Darmstadt.Germany.EU
|
| Posted: Tue Sep 26, 2006 4:44 pm Post subject: |
|
|
| Code: | # which env-update
/usr/sbin/env-update |
look what you've done
| Quote: | | cluca / # PATH="/p4bin/bin:/p4bin/sbin:/usr/bin:/usr/sbin" |
you forgot the things in /usr/. change them also to /p4bin/usr/{,s}bin
A simpler thing came into my mind: instead of making the mount --bind for each bin dir, you can also just | Code: | | mount --bind / /mnt/nfs/p4bin |
then you just have to edit the PATH var.
_________________
# make config; not war
registered linux user # 265707 |
|
| Back to top |
|
|
Luc484
Veteran
Joined: 26 Mar 2005
Posts: 1035
Location: Italy
|
| Posted: Tue Sep 26, 2006 4:54 pm Post subject: |
|
|
| all-inc. wrote: | | Code: | # which env-update
/usr/sbin/env-update |
look what you've done
| Quote: | | cluca / # PATH="/p4bin/bin:/p4bin/sbin:/usr/bin:/usr/sbin" |
you forgot the things in /usr/. change them also to /p4bin/usr/{,s}bin
A simpler thing came into my mind: instead of making the mount --bind for each bin dir, you can also just | Code: | | mount --bind / /mnt/nfs/p4bin |
then you just have to edit the PATH var. |
d'oh! I'm stupid. Ok, maybe I'm understanding what we are doing, now I have this:
| Code: | cluca / # which env-update
/p4bin/usr/sbin/env-update |
in chroot environment. This seems to be ok, right?
But I still have:
| Code: | cluca / # env-update
Illegal instruction |
|
|
| Back to top |
|
|
all-inc.
Tux's lil' helper
Joined: 03 Jul 2004
Posts: 138
Location: Darmstadt.Germany.EU
|
| Posted: Tue Sep 26, 2006 4:55 pm Post subject: |
|
|
damn! we have a new problem! the file /usr/lib/portage/bin/env-update is a python exe script...you knwo what's in its first line?
| Code: | | #!/usr/bin/python -O |
so it doesn't care about PATH, the path is hardcoded. but we can handle this...just give some minutes ^^
another question, does a simple work(of course in the chrooted environment)? then, we know that my method works in general. and what shows ? It should show /p4bin/usr/bin/python ^^
_________________
# make config; not war
registered linux user # 265707 |
|
| Back to top |
|
|
Luc484
Veteran
Joined: 26 Mar 2005
Posts: 1035
Location: Italy
|
| Posted: Tue Sep 26, 2006 5:01 pm Post subject: |
|
|
| all-inc. wrote: | damn! we have a new problem! the file /usr/lib/portage/bin/env-update is a python exe script...you knwo what's in its first line?
| Code: | | #!/usr/bin/python -O |
so it doesn't care about PATH, the path is hardcoded. but we can handle this...just give some minutes ^^ |
ah ah, great! You're a genius! Now I understand.
| all-inc. wrote: | | another question, does a simple work(of course in the chrooted environment)? then, we know that my method works in general. and what shows ? It should show /p4bin/usr/bin/python ^^ |
| Code: | cluca / # python -V
Python 2.4.3
cluca / # which python
/p4bin/usr/bin/python |
|
|
| Back to top |
|
|
all-inc.
Tux's lil' helper
Joined: 03 Jul 2004
Posts: 138
Location: Darmstadt.Germany.EU
|
| Posted: Tue Sep 26, 2006 6:58 pm Post subject: |
|
|
ok, an idea: if you plan to only run emerges in your chroot, no other commands, we can handle this with a wrapper script. puh.
on the p4 NOT chrooted
| Code: | mkdir -p /usr/lib/portage/bin/pseudo-bin/real-bin
cd /usr/lib/portage/bin/pseudo-bin |
then create a file named pseudo.sh in this dir and put this in it:
| Code: | #!/p4bin/bin/sh
P_LIB="/usr/lib/portage/bin/pseudo-bin/"
PFX="/p4bin/usr/bin/"
SCRIPT=$(echo $0 | ${PFX}sed -e "s:$(${PFX}dirname $0)::" )
SCR_F="${P_LIB}real-bin${SCRIPT}"
INT=$(${PFX}head -n1 ${SCR_F} | ${PFX}awk '{ print substr($0, 3) }')
${PFX}sed -e '2,$s:/usr/bin/:/p4bin/usr/bin/:g' ${SCR_F} | /p4bin${INT} - |
now
| Code: | chmod a+x pseudo.sh
cd ..
for f in $(egrep -l -e '^#!.*/bin' *); do (cd pseudo-bin; ln -s pseudo.sh $f); done |
then | Code: | mount --bind /usr/lib/portage/bin/pseudo-bin /mnt/nfs/usr/lib/portage/bin
mount --bind /usr/lib/portage/bin /mnt/nfs/usr/lib/portage/bin/pseudo-bin/real-bin |
I hope you understand the ideas behind all this dirty hacks. I don't give any warranty! First, have a look at everything before applying it.
If you run now a command like env-update, it invokes pseudo.sh, which invokes python (or bash if it is a bash script) with the script put into stdin and all occurences of /usr/bin replaces by /p4bin/usr/bin.
If you have any questions on any step, ask it! i'm quiet sure, I made anywhere a typo ^^ This comes from my bizarre mind, so I really don't give any warranty that it will function. Because if anywhere in the emerge process a scipt is invoked which also has /usr/bin hardcoded, it will break again...then you would end up in many wrapper scripts ^^ another option is to replace the /ustr/bin/python and /bin/sh files with wrapper scripts that detect if they get involved from within your p4 chroot or if they are really called from your athlon-xp machine.
_________________
# make config; not war
registered linux user # 265707 |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Other Things Gentoo
