Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
no-distcc-env: disable distcc on a per-package basis
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
mattst88
Developer
Developer


Joined: 28 Oct 2004
Posts: 387

PostPosted: Wed Jul 24, 2019 4:18 am    Post subject: no-distcc-env: disable distcc on a per-package basis Reply with quote

I have some slow systems that benefit a lot from FEATURES="distcc" (and perhaps FEATURES="distcc-pump" but not sure), but setting this often leads to problems.

Packages fail to compile under distcc sometimes. Worse, sometimes a package doesn't benefit from distcc and your high MAKEOPTS value now overloads the poor system that thought it was going to get some help.

I've created app-portage/no-distcc-env: a collection of package.env files to disable distcc on a per-package basis. With this installed, the hope is to make it safe to use FEATURES="distcc distcc-pump" globally. Set LOCAL_MAKEOPTS=... to a reasonable value for your system alone, and MAKEOPTS=... for your distcc network.

For example, if my slow system has a single-core CPU but I have 20 CPUs among all the machines in my distcc network, in make.conf I would set

Code:
FEATURES="... distcc ..."
MAKEOPTS="-j20"
LOCAL_MAKEOPTS="-j1"


By default packages will build with distcc, but if one doesn't (e.g., dev-libs/libunistring, bug 620738) it'll automatically have FEATURES="-distcc" and will replace MAKEOPTS with LOCAL_MAKEOPTS.

The package.env files installed by the package are named in the typical cate-gory/package style. E.g., /etc/portage/package.env/no-distcc/dev-libs/libunistring, which contains

Code:
dev-libs/libunistring no-distcc.conf # Bug 620738


This way, using INSTALL_MASK (see "man 5 make.conf") you can selectively disable the installation of these files for specific packages or categories if you're feeling adventurous.

My hope is that people will contribute to the project as they find packages that are incompatible with distcc (but the best method of contributing remains to be seen).
Back to top
View user's profile Send private message
eccerr0r
Watchman
Watchman


Joined: 01 Jul 2004
Posts: 7274
Location: almost Mile High in the USA

PostPosted: Wed Jul 24, 2019 2:05 pm    Post subject: Reply with quote

I know it's probably bad style to reply to tips and tricks, but what about the packages that are simply temporarily do not build properly with distcc?

Some packages continually fail, some packages fail for some people and not others (like www-client/firefox). Ideally it's dynamic, perhaps portage should simply retry packages that fail - if it notices distcc is enabled, disable it and retry? Maybe only in conjunction with --keep-going or perhaps another option to allow alerting of the user that distcc is failing and should be investigated?

As for specific packages, I still am not too sure of the status of waf (sys-libs/tdb sys-libs/ldb sys-libs/talloc sys-libs/tevent and specifically net-fs/samba) because theoretically it should work -- but I have not had much success for many months building through portage with distcc enabled.
_________________
Intel Core i7 2700K@ 4.1GHz/HD3000 graphics/8GB DDR3/180GB SSD
What am I supposed watching?
Back to top
View user's profile Send private message
bunder
Bodhisattva
Bodhisattva


Joined: 10 Apr 2004
Posts: 5882

PostPosted: Wed Jul 24, 2019 3:16 pm    Post subject: Reply with quote

i used to use distcc quite a bit, it's nice to see people still using it, this sounds like it would work, but i wonder if this would be easier to set up in an eclass similar to flag-o-matic's strip-flags...
_________________
Neddyseagoon wrote:
The problem with leaving is that you can only do it once and it reduces your influence.

banned from #gentoo since sept 2017
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum