Distcc Pump: Can't get pump mode working, distcc works great

[seansmr]

My first question!
I have been using Gentoo as a LAMP server for years and recently as a desktop. It is a great system, I love it. Now I am trying to get to know it better.

I have read a lot and tried a lot, but I just can't get distcc pump mode working. Some solutions reported in the forums do not produce the same results for me. There are 3 main ways that I have tried:
1) Set ",cpp" in the conf.d/hosts file or set the variable DISTCC_HOSTS in make.conf. When I try to use pump --startup or pump emerge, either one gets me the error:

Code: Select all

/usr/bin/pump: error: pump mode requested, but distcc hosts list does not contain any hosts with ',cpp' option

2) Tried at the command line:

Code: Select all

DISTCC_POTENTIAL_HOSTS="192.168.2.4,cpp 192.168.2.7,cpp" pump --startup

This seems to produce ok output, below, but does not do anything different than normal distcc.

Code: Select all

export INCLUDE_SERVER_PID='2042'
export INCLUDE_SERVER_DIR='/tmp/distcc-pump.flghAS'
export INCLUDE_SERVER_PORT='/tmp/distcc-pump.flghAS/socket'
export DISTCC_HOSTS='--randomize '
export PATH='/usr/lib/distcc/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/i686-pc-linux-gnu/gcc-bin/4.5.3'

3) Also, tried at the command line:

Code: Select all

DISTCC_POTENTIAL_HOSTS="192.168.2.4,cpp 192.168.2.7,cpp" pump emerge screen -pv

This give the result below:

Code: Select all

__________Using distcc-pump from /usr/bin
__________Found 0 available distcc servers
These are the packages that would be merged, in order:
Calculating dependencies... done!
[ebuild   R    ] app-misc/screen-4.0.3-r4  USE="pam -debug -multiuser -nethack (-selinux)" 0 kB
Total: 1 package (1 reinstall), Size of downloads: 0 kB
__________Shutting down distcc-pump include server

Normal distcc works great for me. Also, I haven't seen anything in the logs when I try to use pump mode.

Using DISTCC_HOSTS does not work:

Code: Select all

DISTCC_HOSTS="192.168.2.4,cpp 192.168.2.7,cpp" pump emerge screen -pv

Produces this error:

Code: Select all

distcc[3916] (dcc_get_protover_from_features) ERROR: pump mode (',cpp') requires compression (',lzo')
distcc[3916] (dcc_parse_options) ERROR: invalid host options: ,cpp 192.168.2.7,cpp
distcc[3916] (dcc_show_hosts) CRITICAL! Failed to get host list
/usr/bin/pump: error: pump mode requested, but distcc hosts list does not contain any hosts with ',cpp' option

OK, so then I tried this:

Code: Select all

DISTCC_HOSTS="192.168.2.4,cpp,lzo 192.168.2.7,cpp,lzo" pump emerge screen -pv

And it seems to work:

Code: Select all

__________Using distcc-pump from /usr/bin
__________Using 2 distcc servers in pump mode
These are the packages that would be merged, in order:
Calculating dependencies... done!
[ebuild   R    ] app-misc/screen-4.0.3-r4  USE="pam -debug -multiuser -nethack (-selinux)" 0 kB
Total: 1 package (1 reinstall), Size of downloads: 0 kB
__________Shutting down distcc-pump include server

However, adding ",lzo" to the hosts file or make.conf does not work. I must be missing something obvious, or I completely misunderstand the whole pump configuration. How can I get pump to take the ",cpp" option from the hosts file or the variable in make.conf?
Also, I do not want compression, is it possible to get pump working without ,lzo?

And, I haven't seen anything in the documentation, is there any setup for pump required on the remote host?

I hope that this is understandable, let me know if you need more information.

[krinn]

Well, it's just adding lzo,cpp to your hosts list.
and if you don't use DISTCC_POTENTIAL_HOSTS, it will default to find them in /etc/distcc/hosts & DISTCC_HOSTSnano
And don't ask me why cpp doesn't simply imply lzo, but you cannot run cpp without lzo, so both are needed for pump.

And you can ask distcc to show you if your hosts are configure fine using distcc --show-hosts

Code: Select all

distcc --show-hosts
@beleg/8,lzo,cpp
@hurin/4,lzo,cpp
localhost/2,lzo,cpp

on running you might also check the hosts are handle, but except informative, don't take it as a proof the host is valid, just one the host will be tried.

Code: Select all

__________Using distcc-pump from /usr/bin
__________Using 3 distcc servers in pump mode

and on the running host, ls /tmp/distcc-pump*

[seansmr]

Thanks for your help. I have added ,lzo to the hosts file and it seems to start ok now.
I can't really tell if it is using pump mode or not. I have checked the logs, but not the monitor.

One more problem, I can't get a separate log file, it still logs to syslog.
I have tried this:

Code: Select all

DISTCCD_OPTS="${DISTCCD_OPTS} --log-file /var/log/distcc.log"

[krinn]

Try gave it a level too.

Code: Select all

DISTCCD_OPTS="${DISTCCD_OPTS} --log-level info --log-file /var/log/distcc.log"

And i don't remember if it need a valid file or not, anyway it won't kill you to touch it in case it's a need.

[seansmr]

I tried log level info, with and without an existing file. Neither worked.

I now have some messages about pump mode. I tried it with today's upgrade, scite-2.28

Code: Select all

pump emerge --oneshot scite

This gives me this result:

Code: Select all

 * GNU info directory index is up-to-date.
__________Warning: 4 pump-mode compilation(s) failed on server, but succeeded locally.
__________Distcc-pump was demoted to plain mode.  See the Distcc Discrepancy Symptoms section in the include_server(1) man page.
__________Shutting down distcc-pump include server

I did read the man page, not something that I know about.
Distccmon shows this:

Code: Select all

 16030  Compile     ContractionState.cxx                        192.168.2.7[0]
 16046  Preprocess                                                localhost[0]
 16078  Compile     RunStyles.cxx                               192.168.2.7[0]
 16129  Compile     LexPS.cxx                                   192.168.2.7[0]
 16142  Compile     LexHTML.cxx                                 192.168.2.7[0]
 16162  Compile     LexBash.cxx                                 192.168.2.7[0]
 16190  Compile     LexTxt2tags.cxx                             192.168.2.7[0]
 16214  Compile     LexD.cxx                                    192.168.2.7[0]
 16238  Compile     LexModula.cxx                               192.168.2.7[0]
 16258  Compile     LexCLW.cxx                                  192.168.2.7[0]
 16283  Compile     LexAU3.cxx                                  192.168.2.7[0]
 16311  Compile     LexMMIXAL.cxx                               192.168.2.7[0]
 16348  Compile     LexErlang.cxx                               192.168.2.7[0]
 16372  Compile     LexScriptol.cxx                             192.168.2.7[0]
 16396  Compile     LexPerl.cxx                                 192.168.2.7[0]
 16416  Compile     LexOthers.cxx                               192.168.2.7[0]
 16436  Compile     LexPascal.cxx                               192.168.2.7[0]

 16480  Blocked                                                            [0]
 16478  Blocked                                                            [0]
 16455  Compile     Widget.cxx                                  192.168.2.7[0]
 16454  Compile     GUIGTK.cxx                                  192.168.2.7[2]
 16453  Compile     SciTEGTK.cxx                                  localhost[1]
 16457  Compile     SciTEBase.cxx                                 localhost[2]

 16480  Blocked                                                            [0]
 16481  Blocked                                                            [0]
 16482  Blocked                                                            [0]
 16478  Compile     SciTEIO.cxx                                 192.168.2.7[0]
 16498  Compile     SciTEProps.cxx                              192.168.2.7[1]

However, each time I run it, distccmon has a different output.
One time I go this:

Code: Select all

... similar lines left out...
 11403  Compile     LexLout.cxx                                 192.168.2.7[0]

 11427  Compile     LexFortran.cxx                              192.168.2.7[0]

 11479  Blocked                                                            [0]
 11471  Preprocess  GUIGTK.cxx                                  192.168.2.4[1]
 11473  Preprocess  FilePath.cxx                                192.168.2.4[3]
 11476  Preprocess  SciTEBase.cxx                               192.168.2.4[4]
 11470  Preprocess  SciTEGTK.cxx                                192.168.2.7[0]
 11472  Preprocess  Widget.cxx                                  192.168.2.7[2]

 11479  Blocked                                                            [0]
 11525  Preprocess  DirectorExtension.cxx                       192.168.2.4[0]
 11528  Preprocess  SciTEProps.cxx                              192.168.2.4[1]
 11476  Compile     SciTEBase.cxx                               192.168.2.4[4]
 11516  Compile     PropSetFile.cxx                             192.168.2.7[0]
 11470  Compile     SciTEGTK.cxx                                192.168.2.7[0]

It appears that it worked correctly, at least sometimes. Is this the expected output for a properly working pump mode?

Thanks for all of your help.

[seansmr]

I have tried distcc pump again, this time with firefox-8.0. Here is what happened
My hosts:

Code: Select all

192.168.2.7/3,cpp,lzo
192.168.2.4/4,cpp,lzo

The distcc monitor outputs in batches like code below. The first 10 or so batched were like this:

Code: Select all

 23872  Blocked                                                            [0]
 23870  Blocked                                                            [0]
 23861  Blocked                                                            [0]
 23868  Blocked                                                            [0]
 23902  Compile     jsfriendapi.cpp                             192.168.2.7[0]
 23888  Compile     jsemit.cpp                                  192.168.2.7[0]
 23833  Compile     jsapi.cpp                                   192.168.2.7[2]

At this point nothing was done on remote the server.
After this, I got activity on the server and the distcc monitor showed like this:

Code: Select all

 24422  Preprocess  ExecutableAllocatorWin.cpp                  192.168.2.4[0]
 24341  Compile     FastOps.cpp                                 192.168.2.4[1]
 24417  Compile     ExecutableAllocatorPosix.cpp                192.168.2.4[1]
 24349  Compile     MonoIC.cpp                                  192.168.2.4[3]
 24338  Compile     FastArithmetic.cpp                          192.168.2.7[0]
 24379  Compile     TrampolineCompiler.cpp                      192.168.2.7[0]
 24352  Compile     PolyIC.cpp                                  192.168.2.7[4]

Then for a few minutes there was no output from distccmon. Emerge continued working, but nothing happened on the server. After this blank period, it continued as in the above code black for the rest of the emerge. The local machine was usually at 100% cpu use and the remote was 60% - 90% About 10% of the status lines were Preprocess, the rest were Compile.

At the end of the, successful, emerge I go this message:

Code: Select all

__________Warning: 4 pump-mode compilation(s) failed on server, but succeeded locally.
__________Distcc-pump was demoted to plain mode.  See the Distcc Discrepancy Symptoms section in the include_server(1) man page.
__________Shutting down distcc-pump include server

So, what does this output mean?
"Distcc-pump was demoted to plain mode." Was distcc running in plain mode for the rest of the emerge? What about the status lines that show Preprocess.
Does the fallback only apply to one job or to the rest of the emerge?

The question is: Is pump mode working correctly or does it need to be fixed?

I also tried it with DISTCC_FALLBACK="1"
It didn't stop and there was no output from the server.

[Goverp]

This is probably either incorrect or irrelevant, but I suspect distcc/pump reverts to local mode if there's a transient network error. I've noticed that I get long compilations (the ones that I want distcc to handle!) reverting to local if I use my netbook too far from my desktop distcc server. My hypothesis is that transient WiFi problems cause distcc to mark the connection as broken.

I see KDE 4.7.3 is about. When I summon up the courage to try it, I'll use a wire connection to my router rather than WiFi and see if that works better.

[krinn]

Pump or not, distcc always default to localhost on a compile failure.
And because pump need to upload to the server the headers and everything to let the linking goes, the preprocess is of course slower than a non-pump running distcc.

It's again, upto you to finetune your farm and decide if it's good or not to let another computer handle the linking for another.
It depend on network/cpu/hdd speed of the slave and its master, and of course the ebuild in use.

[seansmr]

Thanks for your responses to my questions.

Earlier I had seen promises of 3X or more increase in speed when used with pump mode. However, now I have just found a readme, in with the distcc source code. It says that the greatly increased speed only applies to networks with 100 CPUs. It says that one computer can keep up to 10 pcs busy without pump mode.
I am just going to add in one more fast computer and not worry about perfecting pump mode.

Since a few compilations must be done on localhost, it does not appear suitable for a very slow PC. I have Gentoo installed on one of these: http://www.norhtec.com/products/mcjrdx