| View previous topic :: View next topic |
| Author |
Message |
Bircoph
Developer
Joined: 27 Jun 2008
Posts: 261
Location: Moscow
|
 Posted: Wed Jul 15, 2009 11:54 am Post subject: distcc talks |
 |
|
Hello,
it would be nice if we'll share experience on distcc usage here.
I personally encountered some features not mentioned in any gentoo-related distcc howto's I found, so this may be useful to another people. Also I would gladly hear suggestions about my current setup, I'm still relatively new to distcc. There are some distcc-related posts on this forums, but they are discussions of specific problems, not some general/cumulative discussion.
The main reason for me to try it out was gentoo installation on EeePC, but I was amazed by power and speedup of this utility, so I currently use it for my desktop too.
0) Setup:
I have the following network setup:
EeePC <--> desktop <--> server <=> outside world
All connections are wired through 100 MBit ethernet without any third-party hardware, host A acts as a bridge.
EeePC is Atom N270,
desktop is Athlon-XP,
server is Celeron D in 64-bit mode.
All hosts are checked via ip/mac in the iptables rules, thus there is no need in encrypted (e.g. tunneled via ssh) distcc; this gives some speedup (according to well known howtos it is ~25%).
1) Distcc setup:
Current distcc logical setup:
eeepc -- client only;
desktop -- server for eeepc, client ;
server -- server for eeepc, desktop.
In the iptables you should open tcp port for your distcc (3632 by default) and, as usual, you should accept RELATED, ESTABLISHED connections.
Network is fast in my setup (direct 100 MBit/s FD line), max bursts by distcc are ~ 3 MB/s, so there is no slowdown due to network. However, I use the following in /etc/distcc/hosts (on my eeepc):
| Code: |
desktop/3 server/3 localhost/2
|
So, why /3, not /2? As it occurs, preprocessing eats some measurable time too, so /2 doesn't provide full CPU load on helper servers. I tried (helpers/2) localhost/1 (to free one thread on eeepc for preprocessing only), but this provides sligtly worse results.
Of course, -j8 is present in /etc/make.conf.
And you should definitely use ccache together with distcc, you want to gain maximum compilation performance, aren't you? Usually people use ccache widely, but distcc only as a last resort.
2) Standardize your compilers.
distcc doesn't care about gcc versions itself, but you'll surely run into the problems if your compilers will produce incompatible code or will fail to deal with some "extension", like openmp instructions. Some people say you may combine all compilers with the same major number (e.g. 4.x) in a single distcc network, but I don't want to risk. It is unlikely you'll encounter fails after upgrading from 4.3.2 to 4.3.3, but this is for your own sake to keep the difference as little as possible.
Also all options related to cc (C Compiler) and c++ should be the same too: be careful with ntpl,openmp, etc... I had to rebuild compilers on desktop and server to support this features for eeepc compilation, despite I don't need them on these hosts natively.
Also if you want to dist-compile on different and generally binary incompatible OSes (like FreeBSD and Linux), you should setup cross compilers for every those host for every possible case. Even if your gcc's are "slightly" different, like hardened and normal flavour, you should cross compile.
In other words in general you should have sereval gcc installed and multiple distccd running for all possible incompatible combinations of architecture, operating system and compiler options. It's up to you to minimize this set of combinations )).
3) Cross compilation.
a) x86 on amd64 with multilib environment (USE="multilib")
Despite someone may argue it is possible to easy compile x86 on amd64 (with multilib) natively: just make sure -m32 is added to CFLAGS. However, it is not enough to add -m32 to CFLAGS/CXXFLAGS on your client: some ebuilds strips out all or some, but 'sane' CFLAGS, so -m32 goes away and you'll eventually get amd64 object file instead of x86. Here is mine solution:
i) On amd64 host create 3 executable (a+x) files:
i686-pc-linux-gnu-c++:
| Code: |
#!/bin/bash
exec /usr/bin/c++ -m32 "$@"
|
The same for i686-pc-linux-gnu-g++, i686-pc-linux-gnu-gcc (replace c++ as appropriate).
ii) Symlink scripts above to /usr/lib/distcc/bin.
iii) This doesn't work for me with distcc enabled by $PATH entry after ccache's one: /usr/lib/ccache/bin:/usr/lib/distcc/bin/:${PATH}. I haven't investigated in a great detail why this approach fails, and do not want to investigate this, because if you have read manuals for distcc and ccache, you already noticed both manuals recommends to setup ccache with distcc in the following way:
| Code: |
CCACHE_PREFIX="distcc"
|
That's all! ccache will do all other job and it works perfectly. FEATURES="distcc" is uneeded in this approach. You may just add the string above to your make.conf. You do *not* need to tweak something else (like CC, CXX,...).
Really, I can't understand why gentoo devs use non-recommended approach of setting distcc together with ccache via $PATH and not via CCACHE_PREFIX.
Why I do not use crossdev for i686-pc-linux-gnu creation on amd64? Because this is a pain, cross compilation is always a pain it fails for me on both amd64<-x86 and x86<-amd64 (see below). Besides with native compiler you have to support only single instance of compiler, you have to run only single instance of distcc (in case you want to dist-compile for both amd64 and x86), thus you save your time and prevent massive grey hair generation.
b) amd64 on x86
But there is no such nice trick for compilation amd64 on x86. You should perform full cross compilation setup, e.g. via crossdev -t x86_64-pc-linux-gnu. Actually it is better to specify the same versions of all required things as installed in your system (I assume x86 and amd64 are in sync in terms of versions on your net):
| Code: |
crossdev --b 2.19.1-r1 --g 4.3.3-r2 --k 2.6.28-r1 --l 2.10.1 -t x86_64-pc-linux-gnu
|
Unfortunately, this fails for me somewhere inside nptl part of glibc compilation... Well, this is another story, maybe I'll find a way someday to fix this.
Afterwards you need to run multiple distcc instances (for x86 and amd64) or you may run single instance of amd64 compiler in the way described above. To run multiple instances of distcc use this guide: http://www.gentoo-wiki.info/TIP_AMD64-x86-distcc#Compiling_x86_64_packages_on_32bit_x86
But I personally prefer single instace of amd64, because: it allows you to fully control all distcc resources, you need to run one less daemon. Anyway this part of discussion is somewhat theoretical until I'll be able to successfully crossdev amd64 gcc on x86.
4) Dynamic distcc configuration?
It would be nice to use all available hosts, even those not available 24/7, but only for some time, like mine eeepc (indeed, it is not as weak as it seems to be).
In theory this is possible using zeroconf. Currently I do not use this:
a) it is too unsecure, there is no way to cut off "bad" servers on the same lan;
b) you still need to modify MAKEOPTS="-jN" in your /etc/make.conf manually.
Another way is to specify all hosts at /etc/distcc/host and rely on distcc blacklist and timeouts, but the last ones may be too hight and impact performance.
I should investigate this further one day...
5) Why I do not use icecream?
http://en.opensuse.org/Icecream#Icecream_on_gentoo
No comments. Removing CPU-dependand optimizations niveleur half of Gentoo's benefits: you'll be able only to customize packages, not to optimize them.
_________________
Per aspera ad astra! |
|
| Back to top |
|
 |
Yak
Tux's lil' helper
Joined: 01 Sep 2002
Posts: 107
|
| Posted: Wed Feb 10, 2010 1:04 am Post subject: |
|
|
Got to experimenting with distcc for a moment. Must be some kind of (semi?)annual occurrence for me, never can tell when, must depend on the moon phase, the snow depth, and the varying hassle of keeping multiple (cross)compilers in sync. But anyway, you have some very nice ideas here about using multilib, Bircoph! After much searching I was about to try and do something similar when I dug up your post 
Here is what I have so far, with a couple amd64 boxes and an x86 one, on getting the amd64's to help out the x86 box.
I did as you (long ago!) suggested on the amd64 ones to add -m32. Looks like this on each amd64:
| Code: |
# for file in /usr/bin/i686-pc-linux-gnu-*; do echo; echo $file; cat $file;done
/usr/bin/i686-pc-linux-gnu-c++
#!/bin/bash
exec /usr/bin/c++ -m32 "$@"
/usr/bin/i686-pc-linux-gnu-g++
#!/bin/bash
exec /usr/bin/g++ -m32 "$@"
/usr/bin/i686-pc-linux-gnu-gcc
#!/bin/bash
exec /usr/bin/gcc -m32 "$@"
|
Didn't really understand the part ii) Symlink scripts above to /usr/lib/distcc/bin. Here is what mine ended up looking like: (this is on the x86)
| Code: |
ls /usr/lib/distcc/bin/ -l
total 0
lrwxrwxrwx 1 root root 34 Feb 9 14:19 c++ -> /usr/bin/i686-pc-linux-gnu-wrapper
lrwxrwxrwx 1 root root 34 Feb 9 14:19 cc -> /usr/bin/i686-pc-linux-gnu-wrapper
lrwxrwxrwx 1 root root 34 Feb 9 14:19 g++ -> /usr/bin/i686-pc-linux-gnu-wrapper
lrwxrwxrwx 1 root root 34 Feb 9 14:19 gcc -> /usr/bin/i686-pc-linux-gnu-wrapper
lrwxrwxrwx 1 root root 15 Feb 9 14:19 i686-pc-linux-gnu-c++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Feb 9 14:19 i686-pc-linux-gnu-g++ -> /usr/bin/distcc
lrwxrwxrwx 1 root root 15 Feb 9 14:19 i686-pc-linux-gnu-gcc -> /usr/bin/distcc
|
The wrapper is from the distcc cross compiling guide. This works for me with ccache so I haven't tried any other custom configuration of distcc or ccache. |
|
| Back to top |
|
|
Yak
Tux's lil' helper
Joined: 01 Sep 2002
Posts: 107
|
| Posted: Wed Feb 10, 2010 2:15 am Post subject: |
|
|
In case anyone else is snowed in, or feeling adventurous, here is a modified distcc ebuild to put in your local overlay. It's pretty much the same except for the addition of a use flag I called i686wrapper, that, if enabled, installs the wrapper in /usr/bin and symlinks gcc, g++, cc, c++ to the wrapper.
Easiest would probably be to cp the whole /usr/portage/sys-devel/distcc/ to local overlay, overwrite or merge this one, rebuild manifest.
| Code: |
# Copyright 1999-2009 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: /var/cvsroot/gentoo-x86/sys-devel/distcc/distcc-3.1-r4.ebuild,v 1.8 2009/10/19 21:09:13 aballier Exp $
EAPI="2"
inherit eutils fdo-mime flag-o-matic multilib toolchain-funcs
DESCRIPTION="a program to distribute compilation of C code across several machines on a network"
HOMEPAGE="http://distcc.org/"
SRC_URI="http://distcc.googlecode.com/files/${P}.tar.bz2"
LICENSE="GPL-2"
SLOT="0"
KEYWORDS="alpha amd64 arm hppa ia64 ~mips ppc ppc64 s390 sh sparc x86 ~sparc-fbsd ~x86-fbsd"
IUSE="avahi gnome gtk hardened ipv6 selinux xinetd i686wrapper"
RESTRICT="test"
RDEPEND=">=dev-lang/python-2.4
dev-libs/popt
avahi? ( >=net-dns/avahi-0.6[dbus] )
gnome? (
>=gnome-base/libgnome-2
>=gnome-base/libgnomeui-2
>=x11-libs/gtk+-2
x11-libs/pango
)
gtk? (
>=x11-libs/gtk+-2
)"
DEPEND="${RDEPEND}
dev-util/pkgconfig"
RDEPEND="${RDEPEND}
!net-misc/pump
>=sys-devel/gcc-config-1.4.1
selinux? ( sec-policy/selinux-distcc )
xinetd? ( sys-apps/xinetd )"
DISTCC_LOG=""
DCCC_PATH="/usr/$(get_libdir)/distcc/bin"
DISTCC_VERBOSE="0"
pkg_setup() {
enewuser distcc 240 -1 -1 daemon
}
src_prepare() {
epatch "${FILESDIR}/${PN}-3.0-xinetd.patch"
# bug #253786
epatch "${FILESDIR}/${PN}-3.0-fix-fortify.patch"
# bug #255188
epatch "${FILESDIR}/${P}-freedesktop.patch"
# bug #258364
epatch "${FILESDIR}/${P}-python.patch"
sed -i -e "/PATH/s:\$distcc_location:${DCCC_PATH}:" pump.in || die
# Bugs #120001, #167844 and probably more. See patch for description.
use hardened && epatch "${FILESDIR}/distcc-hardened.patch"
}
src_configure() {
local myconf="--disable-Werror --with-docdir=/usr/share/doc/${PF}"
# More legacy stuff?
[ "$(gcc-major-version)" = "2" ] && filter-lfs-flags
# --disable-rfc2553 b0rked, bug #254176
use ipv6 && myconf="${myconf} --enable-rfc2553"
econf \
$(use_with avahi) \
$(use_with gtk) \
$(use_with gnome) \
${myconf} || die "econf failed"
}
src_install() {
# In rare cases, parallel make install failed
emake -j1 DESTDIR="${D}" install || die
dobin "${FILESDIR}/3.0/distcc-config"
newinitd "${FILESDIR}/${PV}/init" distccd
cp "${FILESDIR}/3.0/conf" "${T}/distccd"
if use avahi; then
cat >> "${T}/distccd" <<-EOF
# Enable zeroconf support in distccd
DISTCCD_OPTS="\${DISTCCD_OPTS} --zeroconf"
EOF
fi
doconfd "${T}/distccd"
cat > "${T}/02distcc" <<-EOF
# This file is managed by distcc-config; use it to change these settings.
DISTCC_LOG="${DISTCC_LOG}"
DCCC_PATH="${DCCC_PATH}"
DISTCC_VERBOSE="${DISTCC_VERBOSE}"
EOF
doenvd "${T}/02distcc"
# create the masquerade directory
dodir "${DCCC_PATH}"
# wrapper for i686
if use i686wrapper; then
dobin "${FILESDIR}/i686-pc-linux-gnu-wrapper"
fi
for f in cc c++ gcc g++; do
if use i686wrapper; then
dosym /usr/bin/i686-pc-linux-gnu-wrapper "${DCCC_PATH}/${f}"
else
dosym /usr/bin/distcc "${DCCC_PATH}/${f}"
fi
if [ "${f}" != "cc" ]; then
dosym /usr/bin/distcc "${DCCC_PATH}/${CTARGET:-${CHOST}}-${f}"
fi
done
# create the distccd pid directory
keepdir /var/run/distccd
fowners distcc:daemon /var/run/distccd
if use gnome || use gtk; then
einfo "Renaming /usr/bin/distccmon-gnome to /usr/bin/distccmon-gui"
einfo "This is to have a little sensability in naming schemes between distccmon programs"
mv "${D}/usr/bin/distccmon-gnome" "${D}/usr/bin/distccmon-gui" || die
dosym distccmon-gui /usr/bin/distccmon-gnome
fi
if use xinetd; then
insinto /etc/xinetd.d
newins "doc/example/xinetd" distcc
fi
rm -rf "${D}/etc/default"
rm -f "${D}/etc/distcc/clients.allow"
rm -f "${D}/etc/distcc/commands.allow.sh"
prepalldocs
}
pkg_postinst() {
use gnome && fdo-mime_desktop_database_update
if use ipv6; then
elog
elog "IPv6 has not supported yet by ${P}."
fi
elog
elog "Tips on using distcc with Gentoo can be found at"
elog "http://www.gentoo.org/doc/en/distcc.xml"
elog
elog "How to use pump mode with Gentoo:"
elog "# distcc-config --set-hosts \"foo,cpp,lzo bar,cpp,lzo baz,cpp,lzo\""
elog "# pump emerge -u world"
elog
elog "To use the distccmon programs with Gentoo you should use this command:"
elog "# DISTCC_DIR=\"${DISTCC_DIR}\" distccmon-text 5"
if use gnome || use gtk; then
elog "Or:"
elog "# DISTCC_DIR=\"${DISTCC_DIR}\" distccmon-gnome"
fi
elog
elog "***SECURITY NOTICE***"
elog "If you are upgrading distcc please make sure to run etc-update to"
elog "update your /etc/conf.d/distccd and /etc/init.d/distccd files with"
elog "added security precautions (the --listen and --allow directives)"
elog
}
pkg_postrm() {
use gnome && fdo-mime_desktop_database_update
}
|
The wrapper script, put this in overlay with the rest of the distcc patch files.
| Code: |
cat /usr/local/portage/sys-devel/distcc/files/i686-pc-linux-gnu-wrapper
#!/bin/bash
exec /usr/lib/distcc/bin/i686-pc-linux-gnu-g${0:$[-2]} "$@"
|
So with this modified ebuild, just turn on i686wrapper in package.use when installing distcc on x86 and the wrapper will be installed in /usr/bin and the symlinks in /usr/lib/distcc/bin will use the wrapper. This solves the problem of things breaking when distcc overwrites the symlinks when re-merging distcc.
Also here is a pretty basic ebuild for the amd64 hosts. I put it in /usr/local/portage/sys-devel/distcc-i686-on-amd64/distcc-i686-on-amd64-0.1.ebuild
| Code: |
# Copyright 1999-2010 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $
EAPI="2"
DESCRIPTION="wrappers for amd64 hosts so distcc can compile i686 code by calling gcc with -m32"
HOMEPAGE=""
SRC_URI=""
LICENSE=""
SLOT="0"
KEYWORDS="amd64"
IUSE=""
DEPEND=""
RDEPEND="sys-devel/gcc[multilib]"
src_install() {
dobin "${FILESDIR}/i686-pc-linux-gnu-c++"
dobin "${FILESDIR}/i686-pc-linux-gnu-g++"
dobin "${FILESDIR}/i686-pc-linux-gnu-gcc"
}
|
If using the ebuild, of course instead of putting the three scripts straight into /usr/bin/ on the amd64s, put them into /usr/local/portage/sys-devel/distcc-i686-on-amd64/files and let emerge put them into /usr/bin. Here's the three scripts for amd64, thanks again Bircoph
| Code: |
../files/i686-pc-linux-gnu-c++
#!/bin/bash
exec /usr/bin/c++ -m32 "$@"
../files/i686-pc-linux-gnu-g++
#!/bin/bash
exec /usr/bin/g++ -m32 "$@"
../files/i686-pc-linux-gnu-gcc
#!/bin/bash
exec /usr/bin/gcc -m32 "$@"
|
That's all for now. If it continues to work for me this may be all I do with it. Just putting it all up here, maybe someone else will find it useful. Make scripts executable, build digests, etc, as appropriate. Use at your own risk, and happy compiling. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Documentation, Tips & Tricks
