Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Distcc Performance Optimization
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
Spargeltarzan
Apprentice
Apprentice


Joined: 23 Jul 2017
Posts: 251

PostPosted: Wed Jun 06, 2018 3:41 pm    Post subject: Distcc Performance Optimization Reply with quote

Dear Community,

I am trying to optimize my distcc setup (notebook is client, desktop is server).

I do not want to build something (only what is necessary) on my notebook, but use my server.

My compilation performance is very bad (glibc merged formerly in 5 min on my server, 9 min on my notebook and with distcc it is 25 min without building something parallel), and I hessle how to find the right MAKEOPTS and if needed load averaging.

Originally I used MAKEOPTS=j8 on my server and MAKEOPTS=j4 on my notebook. Emerge Jobs 9 on my server and 5 on my notebook. No load-averaging.

With all recommendations I have found to apply MAKEOPTS="-j13 -l8" in my notebook's make.conf. In /etc/distcc/hosts is the server's ip/16, cpp, lzo, and I apply "-j=8" to emerge default opts on the notebook,

1) Shouldn't I apply only MAKEOPTS="-j8" to make.conf if I do not build on localhost too? (like formerly in my local setup)
2) If the server is unavailable, the MAKEOPTS="-j8" would be used to build locally, what will be too much. How to avoid this issue?
3) Why is my compilation performance so bad when I use this high MAKEOPTS? Also with -l8 I have currently averaging applied, so I guess I also do not over-hessle.
4) I have never used load averaging before, is it recommended for my use case?
5) With all the sending overhead, can I expect to have shorter compilation with my server but no localhost?
6) No matter what I change, I cannot emerge packages parallel with distcc currently.
7) Do I need to change something in my server's make.conf? I have only applied the CFLAGS to my notebook.

Distcc-mon shows always 13 tasks (send/compile, etc)

Please advice! :) Thank you!

ADD: I am playing around and was removing load-average and put MAKEOPTS to -j16, emerge jobs j=16 too, and ip/24 in hosts. Now parallel compilation works, but my server still is unbusy with CPU < 10% with 38 °C while my notebook is hessled on 88 ° C compiling mostly packages like command.c, conftest.cpp and make_hash.c. (excatly opposit effect I wanted to achieve...)
_________________
___________________
Regards

Spargeltarzan

Notebook: Lenovo YOGA 900-13ISK: Gentoo stable amd64, GNOME systemd, KVM/QEMU
Desktop-PC: Intel Core i7-4770K, 8GB Ram, AMD Radeon R9 280X, ZFS Storage, GNOME openrc, Dantrell, Xen
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 41127
Location: 56N 3W

PostPosted: Wed Jun 06, 2018 5:57 pm    Post subject: Reply with quote

Spargeltarzan,

distcc cannot distribute everything. With distcc, its only C and C++ compile phases.
With pump mode, preprocessing can be distributed too.

The client still does the configure and link steps, so helpeps may not be as busy as you might expect.

In theory,
Code:
MAKEOPTS="-j <total_cores>
is recommended. Keep in mind configure and link constraints.
If your MAKEOPTS="-j <total_cores> pushes the client into swapping, things will slow to a crawl.

Parallel make jobs in a single package are limited too. Even with 128G RAM, 96 real cores, and MAKEOPS="-j100" there are very few packages that will push the system load average over 40. To make the load average go over 100, you need to run emerge with at least --jobs=4
This is all one system, no network or distcc involved.

How fast is your network?

On the client, what do you have in
Code:
/etc/distcc/hosts
?
localhost, if its there at all, should be at the end as jobs are sent to the helpers in the order that they appear in that list.
If localhost is first, it will get compile jobs first, which is not what you want .

What does
Code:
 gcc-config -l
and
Code:
binutils-config -l
show on the client.
The helpers need to be *identical*.

glibc is a bad test case. It does support lots of parallelism in make but it has mostly ended up broken for me.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Spargeltarzan
Apprentice
Apprentice


Joined: 23 Jul 2017
Posts: 251

PostPosted: Thu Jun 07, 2018 8:01 am    Post subject: Reply with quote

NeddySeagoon wrote:

With pump mode, preprocessing can be distributed too.

I have set
Code:
 /etc/portage/make.conf
FEATURES="distcc distcc-pump"

according to distcc-wiki. Why do we need to set both, if I want pump mode? Is this now configured correctly for distcc-pump?

NeddySeagoon wrote:

On the client, what do you have in
Code:
/etc/distcc/hosts
?


Code:
 cat /etc/distcc/hosts
192.168.8.150/24,cpp,lzo


NeddySeagoon wrote:

In theory,
Code:
MAKEOPTS="-j <total_cores>
is recommended. Keep in mind configure and link constraints.
If your MAKEOPTS="-j <total_cores> pushes the client into swapping, things will slow to a crawl.

What does it mean to keep in mind configure and link constraints? (for MAKEOPTS?)

Code:
 helper lscpu
Architektur:                   x86_64
CPU Operationsmodus:           32-bit, 64-bit
Byte-Reihenfolge:              Little Endian
CPU(s):                        8
Liste der Online-CPU(s):       0-7
Thread(s) pro Kern:            2
Kern(e) pro Socket:            4
Sockel:                        1
NUMA-Knoten:                   1
Anbieterkennung:               GenuineIntel
Prozessorfamilie:              6
Modell:                        60
Modellname:                    Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz


Code:
 client lscpu
Architektur:                   x86_64
CPU Operationsmodus:           32-bit, 64-bit
Byte-Reihenfolge:              Little Endian
CPU(s):                        4
Liste der Online-CPU(s):       0-3
Thread(s) pro Kern:            2
Kern(e) pro Socket:            2
Sockel:                        1
NUMA-Knoten:                   1
Anbieterkennung:               GenuineIntel
Prozessorfamilie:              6
Modell:                        78
Modellname:                    Intel(R) Core(TM) i7-6500U CPU @ 2.50GHz


Code:
MAKEOPTS="-j <total_cores>

Cores for helper only (because localhost not in /etc/distcc/hosts) or still helper+client? (configure + link constraints?) Cores*threads or only cores?

I would calculate cores*threads = 8 for helper and cores*threads = 4 for client --> 8+4=12 in total for MAKEOPTS in client's make.conf with distcc. With MAKEOPTS=8 is extremely slow, but I do not understand why I should set it to 12 since my client (localhost) is not in /etc/distcc/hosts.

NeddySeagoon wrote:

Parallel make jobs in a single package are limited too. Even with 128G RAM, 96 real cores, and MAKEOPS="-j100" there are very few packages that will push the system load average over 40. To make the load average go over 100, you need to run emerge with at least --jobs=4
This is all one system, no network or distcc involved.

is
Code:
 --load-average
senseful for my setup with distcc?

Currently I have
Code:
 /etc/portage/make.conf
MAKEOPTS="-j12 -l4"
FEATURES="distcc distcc-pump"
EMERGE_DEFAULT_OPTS="-j8 -b"

on my client. I have good experience for --jobs=8 on my helper locally, that's why I have decided for it again in the distcc setup. Is OK?

NeddySeagoon wrote:

How fast is your network?

I have measured ~ 47MB/s with iperf3 in my 5Ghz wifi network. Helper is on Gigabit LAN.

NeddySeagoon wrote:

What does
Code:
 gcc-config -l
and
Code:
binutils-config -l
show on the client.
The helpers need to be *identical*.


Code:
 gcc-config -l
x86_64-pc-linux-gnu-8.1.0 *


Code:
 binutils-config -l
x86_64-pc-linux-gnu-2.30 *

on all machines.
_________________
___________________
Regards

Spargeltarzan

Notebook: Lenovo YOGA 900-13ISK: Gentoo stable amd64, GNOME systemd, KVM/QEMU
Desktop-PC: Intel Core i7-4770K, 8GB Ram, AMD Radeon R9 280X, ZFS Storage, GNOME openrc, Dantrell, Xen
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 41127
Location: 56N 3W

PostPosted: Thu Jun 07, 2018 8:04 pm    Post subject: Reply with quote

Spargeltarzan,

Code:
192.168.8.150/24,cpp,lzo
allows the client to send 24 make jobs to the host at 192.168.8.150 using pump mode with lzo compression.
That helper needs to be set up to accept 24 jobs, or at least, as many as the client will send.
Excess jobs will be rejected and compiled on the client. They are not queued.

Linking, which is performed on the client, can require large amounts of RAM. You must take care not to run out of real RAM on the client when its linking all these modules that it didn't build for itself.
Similarly, configure is run on the client but its not nearly as memory intensive.

I've not used the --load-average option.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
mgorny
Developer
Developer


Joined: 27 Apr 2007
Posts: 80

PostPosted: Mon Jul 16, 2018 8:43 pm    Post subject: Reply with quote

Please don't ever enable distcc-pump. It breaks stuff. Some packages fail to build. Some become corrupted in hardly predictable ways, and cause other packages to fail.
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 6804

PostPosted: Mon Jul 16, 2018 8:51 pm    Post subject: Reply with quote

distcc-pump is known to break specially with that stupid cmake, hopefully, not all packages use that crap.
Back to top
View user's profile Send private message
Spargeltarzan
Apprentice
Apprentice


Joined: 23 Jul 2017
Posts: 251

PostPosted: Sat Jul 21, 2018 9:05 pm    Post subject: Reply with quote

Thanks for the hints!

I am also considering to clone my notebook to a chroot on my helper (Desktop) and use my Desktop PC as a binhost with a common set of CFLAGS, USE flags and CPU Flags. The chroot "notebook setup" might also use binaries from my Desktop itself. Since the desktop and my notebook differ in some packages and I believe --buildpkgonly on the Desktop itself will lead me to issues, so I would guess the chroot is the best solution; what do you think?

What I have no idea about is how much the "perfect set of CFLAGS" values for all my computers, is a common set of CFLAGS something what is somehow realized badly?

My Desktop is a i7-4770 and my notebook is a i7-6500U CPU.

Would you go for the binhost or the dist-cc?
When I would have a binhost I could compile whenever it doesn't disturb me and simply emerge quickly from my notebook, what sounds quite nicely to me... But only if I does not give up much with the common set??

ADD: Another option would be to use the Desktop as a binhost without a chroot and use dist-cc for the remaining missing packages?
_________________
___________________
Regards

Spargeltarzan

Notebook: Lenovo YOGA 900-13ISK: Gentoo stable amd64, GNOME systemd, KVM/QEMU
Desktop-PC: Intel Core i7-4770K, 8GB Ram, AMD Radeon R9 280X, ZFS Storage, GNOME openrc, Dantrell, Xen
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 41127
Location: 56N 3W

PostPosted: Sat Jul 21, 2018 9:19 pm    Post subject: Reply with quote

Spargeltarzan,

Lets see what the damage is.

Read -march=native. When you understand it, run
Code:
gcc -### -E - -march=native 2>&1 | sed -r '/cc1/!d;s/(")|(^.* - )|( -mno-[^\ ]+)//g'
on each system that will share the binhost.

You replace CFLAGS= -march=native with all the output in common. Sometimes it matters more that others.
Do the same comparison for CPU_FLAGS_X86= too.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Spargeltarzan
Apprentice
Apprentice


Joined: 23 Jul 2017
Posts: 251

PostPosted: Sat Jul 21, 2018 9:42 pm    Post subject: Reply with quote

My notebook i7-6500U has got

CFLAGS:
Code:

gcc -march=native -v -E - < /dev/null 2>&1 | grep cc1 | perl -pe 's/ -mno-\S+//g; s/^.* - //g;'

-march=skylake -mmmx -msse -msse2 -msse3 -mssse3 -mcx16 -msahf -mmovbe -maes -mpclmul -mpopcnt -mabm -mfma -mbmi -msgx -mbmi2 -mavx -mavx2 -msse4.2 -msse4.1 -mlzcnt -mrdrnd -mf16c -mfsgsbase -mrdseed -mprfchw -madx -mfxsr -mxsave -mxsaveopt -mclflushopt -mxsavec -mxsaves --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=4096 -mtune=skylake


CPU-Flags:
Code:

cpuid2cpuflags

CPU_FLAGS_X86: aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3


I will need a day or two to check my Desktop i7-4770. :-)
_________________
___________________
Regards

Spargeltarzan

Notebook: Lenovo YOGA 900-13ISK: Gentoo stable amd64, GNOME systemd, KVM/QEMU
Desktop-PC: Intel Core i7-4770K, 8GB Ram, AMD Radeon R9 280X, ZFS Storage, GNOME openrc, Dantrell, Xen
Back to top
View user's profile Send private message
axl
Guru
Guru


Joined: 11 Oct 2002
Posts: 409
Location: Romania

PostPosted: Sat Jul 21, 2018 9:47 pm    Post subject: Reply with quote

it is less cpu intensive to mount rootfs of the laptop on the dsktop as nfs fs, and chroot in it and upgrade it from there.

what u need for this scheme to work is put nfs server on the laptop, share rootfs for desktop with rw rights. then on desktop mount the rootfs of the laptop, dev proc sys run usr/portage binded on the desktop, a tmpfs for var/tmp/portage on the pseudolaptop root of the laptop on the desktop and then chroot in it like in the install guide.

with my method, the laptop only has to provide executables, libs and includes(through nfs). desktop has to do compiling, memory, fs where things are compiled and ofc portage tree.

food for thought.


edit: like neddy said (i think it was neddy, on tablet, cant scroll) but these kind of schemes require a good fast network connection. both distcc and my nfs solution require high bandwidth to work in a satisfactory way.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 41127
Location: 56N 3W

PostPosted: Sat Jul 21, 2018 10:08 pm    Post subject: Reply with quote

axl,

That also requires that the build host can execute the code compiled for the laptop.
It might not, then you get Illegal Instruction exceptions and the kernel kills whatever process caused the exception.
If its bash, that's a very bad thing.

Its safer to do the homework and come up with a lowest common denominator setup that works everywhere.
Then for a few packages that are peformance critical, build them locally or with distcc, using custom settings.
Portage can be configured to do that if its required,
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
axl
Guru
Guru


Joined: 11 Oct 2002
Posts: 409
Location: Romania

PostPosted: Sat Jul 21, 2018 10:13 pm    Post subject: Reply with quote

also, taken the liberty to look at your cpus. a big NONO with distcc is not use mtune native. because it will be native to your host that is compiling your data.

in your case is ok. turns out mtune=skylake that u should be using on your laptop, has more cpu extentions than -mtune=haswell that u should be using on your desktop. use those. trust me.
Back to top
View user's profile Send private message
axl
Guru
Guru


Joined: 11 Oct 2002
Posts: 409
Location: Romania

PostPosted: Sat Jul 21, 2018 10:17 pm    Post subject: Reply with quote

NeddySeagoon wrote:
axl,

That also requires that the build host can execute the code compiled for the laptop.
It might not, then you get Illegal Instruction exceptions and the kernel kills whatever process caused the exception.
If its bash, that's a very bad thing.

Its safer to do the homework and come up with a lowest common denominator setup that works everywhere.
Then for a few packages that are peformance critical, build them locally or with distcc, using custom settings.
Portage can be configured to do that if its required,


true. but i don’t think in this case haswell is very affected gcc wise to compile for skylake. i doubt skylake/haswell generate such massively different code. i could be wrong. foot in mouth.

or he could just use mtune haswell on both machines

edit: RDSEED, ADCX, PREFETCHW, CLFLUSHOPT, XSAVEC and XSAVES these are the extensions skylake has and haswell doesnt. or in other words, the laptop has but the desktop doesn’t. or the client has but the server doesn’t.

any one of these could be in gcc already. or not. but i doubt it will lead to an illegal instruction which is what might happen.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 41127
Location: 56N 3W

PostPosted: Sat Jul 21, 2018 10:33 pm    Post subject: Reply with quote

axl,

distcc has got cleverer, it will no longer distribute if you set -march=native.
-mtune only affects instruction ordering, not the generated code.
I don't know if distcc screens for -mtune.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
axl
Guru
Guru


Joined: 11 Oct 2002
Posts: 409
Location: Romania

PostPosted: Sat Jul 21, 2018 10:47 pm    Post subject: Reply with quote

NeddySeagoon wrote:
axl,

distcc has got cleverer, it will no longer distribute if you set -march=native.
-mtune only affects instruction ordering, not the generated code.
I don't know if distcc screens for -mtune.


thank u.

i never considered mtune march. I just kept both the same and in compliance with each machine.

should be said, gcc/distcc can compile code for a higher cpu. it just cant execute it. not gcc. a lower class cpu can compile code for a higher class cpu. just can't execute that code. (and again the laptop is more advanced in term of cpu extensions).

distcc method doesn't require any execution of code. my nfs method does. so you could run into illegal instruction.


EDIT: i remember openmosix. ohh the nightmares.
Back to top
View user's profile Send private message
Spargeltarzan
Apprentice
Apprentice


Joined: 23 Jul 2017
Posts: 251

PostPosted: Mon Jul 23, 2018 10:27 am    Post subject: Reply with quote

My notebook i7-6500U has got

CFLAGS:
Code:

gcc -march=native -v -E - < /dev/null 2>&1 | grep cc1 | perl -pe 's/ -mno-\S+//g; s/^.* - //g;'

-march=skylake -mmmx -msse -msse2 -msse3 -mssse3 -mcx16 -msahf -mmovbe -maes -mpclmul -mpopcnt -mabm -mfma -mbmi -msgx -mbmi2 -mavx -mavx2 -msse4.2 -msse4.1 -mlzcnt -mrdrnd -mf16c -mfsgsbase -mrdseed -mprfchw -madx -mfxsr -mxsave -mxsaveopt -mclflushopt -mxsavec -mxsaves --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=4096 -mtune=skylake


CPU-Flags:
Code:

cpuid2cpuflags

CPU_FLAGS_X86: aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3


I will need a day or two to check my Desktop i7-4770. :-)
--> Desktop i7-4770:

Code:

gcc -### -E - -march=native 2>&1 | sed -r '/cc1/!d;s/(")|(^.* - )|( -mno-[^\ ]+)//g'
-march=haswell -mmmx -msse -msse2 -msse3 -mssse3 -mcx16 -msahf -mmovbe -maes -mpclmul -mpopcnt -mabm -mfma -mbmi -mbmi2 -mavx -mavx2 -msse4.2 -msse4.1 -mlzcnt -mrdrnd -mf16c -mfsgsbase -mfxsr -mxsave -mxsaveopt --param l1-cache-size=32 --param l1-cache-line-size=64 --param l2-cache-size=8192 -mtune=haswell


Code:

CPU_FLAGS_X86: aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3


It seems I will miss following flags on the notebook, beside the bigger l2-cache-size on the Desktop:
Code:

-msgx --mrdseed -mprfchw -madx -mclflushopt -mxsavec -mxsaves l2-cache-size=8192 (for Desktop) -mtune


-) What do you think is the penalty in this? How to deal with cache size?

The CPU Flags are in common.

Understand with distcc I will not give up Cflags, since the code on the helper will not be executed.
Quote:

in your case is ok. turns out mtune=skylake that u should be using on your laptop, has more cpu extentions than -mtune=haswell that u should be using on your desktop. use those. trust me.

Believe I could try:
-) no mtune
-) mtune=haswell for Desktop, mtune=skylake for Notebook --> Will Portage accept the package with different flag?
-) mtune=haswell for both, which is supposed to work
-) mtune=skylake for both with the risk of Illegal Instruction exceptions on the Desktop

Am I right?
_________________
___________________
Regards

Spargeltarzan

Notebook: Lenovo YOGA 900-13ISK: Gentoo stable amd64, GNOME systemd, KVM/QEMU
Desktop-PC: Intel Core i7-4770K, 8GB Ram, AMD Radeon R9 280X, ZFS Storage, GNOME openrc, Dantrell, Xen
Back to top
View user's profile Send private message
axl
Guru
Guru


Joined: 11 Oct 2002
Posts: 409
Location: Romania

PostPosted: Mon Jul 23, 2018 11:31 am    Post subject: Reply with quote

Spargeltarzan wrote:
-) no mtune


As far as I know, this is the equivalent of what you are doing now. and that it native. and again, native to the computer compiling it, not the computer running it.

Spargeltarzan wrote:
-) mtune=haswell for Desktop, mtune=skylake for Notebook --> Will Portage accept the package with different flag?


I think this is proper way. Each computer compiled to his own specifications. I don't know what you mean by the question at the end.

Spargeltarzan wrote:
-) mtune=haswell for both, which is supposed to work


this is not only supposed to work, it will definitely work. although maybe I shouldn't be so sure of myself. I don't exactly know if you try to compile haswell code on skylake if you will not take a speed hit because of the different cache sizes. haswell has double cache over skylake in l1 line and l2.

Spargeltarzan wrote:
-) mtune=skylake for both with the risk of Illegal Instruction exceptions on the Desktop


I never suggested that. I only suggested you can remote mount on haswell a nfs partition that has been compiled for skylake and update it remotely. In that case you will only run a few console application and i HOPE none of them use -msgx -mrdseed -mprfchw -madx. But to me honest I can't be sure of that.

Ohhh. Now I understand the question. No portage will not accept a package compiled for another architecture.

So wait, if I read correctly:

- your server can compile glibc in 5
- laptop in 9
- laptop with distcc 25?!

Again I ask, what kind of network is between those 2 computers? distcc sux a lot of bandwidth.

Also like others pointed out, on the laptop:
- in /etc/distcc/hosts remove the ",cpp" (but leave lzo - line should be like "myhost/16,lzo"), also make sure the laptop's ip or 127.0.0.1 is not in there. just the haswell ip
- remove distcc-pump from make.conf (leave only distcc), and finally set MAKEOPTS to -j8.

And give it another go. Watch bandwidth between computers, and desktop's cpu should be almost maxed out, and the laptop's computer semi-busy from compressing stuff with lzo and sending it to desktop.

If this doesn't satisfy your needs, perhaps you can try the other method I suggested. Way to much to write about that now, but if you need i can detail a few things.

As far as binhost and that other stuff, honestly I have no experience with that and therefor won't say anything about it. I always avoided binaries of any kind. Even those made by myself. :)
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 6804

PostPosted: Mon Jul 23, 2018 2:09 pm    Post subject: Reply with quote

Spargeltarzan wrote:

-) no mtune
-) mtune=haswell for Desktop, mtune=skylake for Notebook --> Will Portage accept the package with different flag?
-) mtune=haswell for both, which is supposed to work
-) mtune=skylake for both with the risk of Illegal Instruction exceptions on the Desktop

The answer is proper mtune for each host, so best should be option 2.

distcc run on the other host with the cflags from the calling host, that's why you cannot use any "native".
hostA: native -> distcc -> hostB compile for hostA using native (so its own settings and not the ones from hostA), in real, distcc will not distribute to hostB anything with native to prevent this mistake.

when you build
locally desktop: will use haswell as it should
locally laptop: will use skylake as it should
laptop calling desktop: desktop will build with laptop cflags (so desktop will build the job using skylake) ; which is fine because the final result will be run on the laptop which use skylake code.
desktop calling laptop: laptop will build with desktop cflags (so laptop will build the job using haswell)
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 41127
Location: 56N 3W

PostPosted: Mon Jul 23, 2018 7:11 pm    Post subject: Reply with quote

To add to what krinn has contributed, setting -march implies -mtune too, so there is no need to set it separately.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Spargeltarzan
Apprentice
Apprentice


Joined: 23 Jul 2017
Posts: 251

PostPosted: Mon Jul 23, 2018 10:00 pm    Post subject: Reply with quote

axl wrote:

I think this is proper way. Each computer compiled to his own specifications.


Understand :-) I have set it like this.

axl wrote:

Again I ask, what kind of network is between those 2 computers? distcc sux a lot of bandwidth.


I have measured ~ 47MB/s with iperf3 in my 5Ghz wifi network. The helper is on Gigabit LAN, but my notebook bottlenecks with wifi.

For the future I would like to connect to my helper also sometimes with VPN - is this possible? With a bandwith-testing site I measured 10.3 Mb/s upload bandwith of the internet where my helper is.

axl wrote:

Also like others pointed out, on the laptop:
- in /etc/distcc/hosts remove the ",cpp" (but leave lzo - line should be like "myhost/16,lzo"), also make sure the laptop's ip or 127.0.0.1 is not in there. just the haswell ip
- remove distcc-pump from make.conf (leave only distcc), and finally set MAKEOPTS to -j8.

Done. Why do you set myhost up to 16 jobs? What is the reason why you remove cpp?

NeddySeagoon wrote:

That helper needs to be set up to accept 24 jobs, or at least, as many as the client will send.
Excess jobs will be rejected and compiled on the client. They are not queued.

Isn't 16 too much for my helper? According to Neddy packages will be rejected and compiled on the client?

axl wrote:

And give it another go. Watch bandwidth between computers, and desktop's cpu should be almost maxed out, and the laptop's computer semi-busy from compressing stuff with lzo and sending it to desktop.

I tested wine-vanilla and qtgui and I observe network traffic and I can monitor distcc taks. Some *.c files are listed for localhost, they looked special, maybe only possible to compile on localhost.

wine-vanilla-3.0.1 compile time is reduced from 30 min to 20 min.
qtgui compile time is reduced from 6min 35s to 3min 54s

I am happy with the reduced compile times :-).

I hoped to be able to reduce the stress of my notebook a bit more, still I mostly observed 77-100% cpu load and temperature climbing up to 80 °C. My Yoga is quite portable design. With all the compilation, my fan now sometimes starts to make deep low frequency noise - I will try to fix this anyway but it was one reason beside reducing compilation times to reduce the stress in future.

Thank you Neddy, Axl and Krinn for your detailed answers and patience! :-)
_________________
___________________
Regards

Spargeltarzan

Notebook: Lenovo YOGA 900-13ISK: Gentoo stable amd64, GNOME systemd, KVM/QEMU
Desktop-PC: Intel Core i7-4770K, 8GB Ram, AMD Radeon R9 280X, ZFS Storage, GNOME openrc, Dantrell, Xen
Back to top
View user's profile Send private message
axl
Guru
Guru


Joined: 11 Oct 2002
Posts: 409
Location: Romania

PostPosted: Tue Jul 24, 2018 11:35 am    Post subject: Reply with quote

Spargeltarzan wrote:
I have measured ~ 47MB/s with iperf3 in my 5Ghz wifi network. The helper is on Gigabit LAN, but my notebook bottlenecks with wifi.

For the future I would like to connect to my helper also sometimes with VPN - is this possible? With a bandwith-testing site I measured 10.3 Mb/s upload bandwith of the internet where my helper is.

Done. Why do you set myhost up to 16 jobs? What is the reason why you remove cpp?


if those numbers are correct, then that is just as much as your setup can handle.

"cpp" enables distcc pump mode. As stated before, some applications break when you compile them in pump mode. Also some don't compile at all. So to avoid this, I thought better to disable it completely in both make.conf and in /etc/distcc/hosts. Btw, if you want pump mode, and want to test it further, then remember that you have to both set make.conf AND /etc/distcc/hosts, and that also pump-mode is coming together with lzo. so $ip/$threads,lzo,cpp for pump mode and for non pump mode you could write both $ip/$threads or $ip/$threads,lzo.

now concerning threads. I asked you to set 16 in hosts to make sure portage maxes out with it's 8. also because the server has 8.

The test such as it was, proves your laptop cannot compress and send 8 compilation threads reliably. Both because of network bandwidth, and because of cpu overheating. You could set it in make.conf to 6, see if that lowers your temperature issue. But the point is, you cannot max out the server, because of this.

And to put VPN on top of that... would FURTHER increase the cpu usage on client, because it would have to encrypt the data on top of compressing it. Besides it's not super secret data. It's sources. They are readily available on the internet, as well as the resulted binaries. But what do I know.

SO the way u ran the test yesterday is the maximum you could achieve, until either the laptop becomes either more capable to compress data, or the bandwidth increases enough so that you don't have to compress as much data. What you have to figure out now, is how to dial it down to a satisfactory level. It's up to you now to balance temperature, threads and bandwidth.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 41127
Location: 56N 3W

PostPosted: Tue Jul 24, 2018 5:54 pm    Post subject: Reply with quote

Spargeltarzan,

Wifi is a killer for distcc. The bandwidth is good but that's only a part of the story.
As its only ever half duplex, (worse with more than two devices) the latency is poor.

Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway illustrates the difference very well, even if it is told around 1970's technology.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
axl
Guru
Guru


Joined: 11 Oct 2002
Posts: 409
Location: Romania

PostPosted: Tue Jul 24, 2018 9:54 pm    Post subject: Reply with quote

NeddySeagoon wrote:
Spargeltarzan,

Wifi is a killer for distcc. The bandwidth is good but that's only a part of the story.
As its only ever half duplex, (worse with more than two devices) the latency is poor.

Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway illustrates the difference very well, even if it is told around 1970's technology.


Spargeltarzan wrote:
With a bandwith-testing site I measured 10.3 Mb/s upload bandwith of the internet where my helper is.


did you notice this part?
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 41127
Location: 56N 3W

PostPosted: Wed Jul 25, 2018 5:11 pm    Post subject: Reply with quote

axl,

I missed the "of the internet". It says nothing of the latency.

Spargeltarzan, distcc is totally insecure. Its a really bad idea to run it over an untrusted network without adding your own security via a tunnel.
That adds overhead to both ends and makes the latency worse too.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
msst
Apprentice
Apprentice


Joined: 07 Jun 2011
Posts: 189

PostPosted: Wed Jul 25, 2018 9:10 pm    Post subject: Reply with quote

Quote:
distcc sux a lot of bandwidth.


I have once tried distcc on a local network. It not only sux with anything, but a high speed cable bandwidth situation, it also suxx when the local machine is much slower/older than the compile host. Simply because the configure and linking stages take almost more time as the compile then and the extra overhead kills the small advantage in the compile time.

Add to this the configuration complications, then I would only ever recommend distcc when one deals with a cluster of identic machines linked by at minimum GBit Ethernet.

So in the given scenario here I am not overly surprised that it made the whole thing actually slower.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum