Get a higher performance for a particular program

Message

linux_dream · Post by **linux_dream** » Thu Jun 18, 2015 8:52 pm

Hello people,
I am very new to gentoo. I have 2 linux distributions installed on my laptop (gentoo and Manjaro which is based on Arch linux), in both of them I installed "stockfish", a chess engine. In both cases the compilation was done on my laptop. However the Manjaro version is about 2% faster than my gentoo's one and I'm not sure why it is so.
I've checked in Manjaro the compilation was done with -O3 level of optimization while gentoo was at first done with -O2 but I later changed this for -O3, to no avail: there's still a 2 or 3% slowdown on gentoo.
I compiled the gentoo kernel myself and I've set the processors to be "on demand" instead of performance. From what I understand this should reduce the processors speed when I'm not running any high cpu usage programs and should not hurt the performance when I run a heavily resources hungry program. I think I have the same parameter in Manjaro.
So I really have no clue on why the chess program runs noticeably slower in gentoo than in Manjaro. I'd like to make a better Stockfish compilation (if that's possible), any help is appreciated.

eccerr0r · Post by **eccerr0r** » Thu Jun 18, 2015 9:02 pm

What version of GCC?
Are you glibc heavy, are they using the same versions of glibc?
What other compiler options are you using?
What background tasks/window manager/desktop environments are being run?
Kernel compiled the same way for both setups? Same kernel version?

linux_dream · Post by **linux_dream** » Thu Jun 18, 2015 9:13 pm

eccerr0r wrote:What version of GCC?
Are you glibc heavy, are they using the same versions of glibc?
What other compiler options are you using?
What background tasks/window manager/desktop environments are being run?
Kernel compiled the same way for both setups? Same kernel version?

Everything is different, Manjaro is more bleeding edge.
I am not sure of the versions exactly, out of memory:
Manjaro: gcc 5.0 or 5.1. xfce running with no particular program in the background. The kernel is a generic one already compiled, I don't know the details of it.
Gentoo: gcc 4.8 I believe, lxde running with no particular program in the background. The kernel was built by myself (with make menuconfig).
Ok I see that these may be the reasons why Stockfish performs better on Manjaro.

Post by **John R. Graham** » Thu Jun 18, 2015 9:58 pm

Gentoo can be more bleeding edge if you want it to be. If you keyword gcc, you'll get up to version 4.9.2 (as of this writing). Gentoo Unstable is pretty stable in my experience. The gcc 5.x series is hard masked in Gentoo, though, which usually means something is critically broken. (In this case, it's a feature: an ABI change.) Also, you may not have taken advantage of all of the available optimizations. What does

Code: Select all

emerge -1vp stockfish

show you?

- John

linux_dream · Post by **linux_dream** » Fri Jun 19, 2015 12:37 am

John R. Graham wrote:Gentoo can be more bleeding edge if you want it to be. If you keyword gcc, you'll get up to version 4.9.2 (as of this writing). Gentoo Unstable is pretty stable in my experience. The gcc 5.x series is hard masked in Gentoo, though, which usually means something is critically broken. (In this case, it's a feature: an ABI change.) Also, you may not have taken advantage of all of the available optimizations. What does
Code: Select all
emerge -1vp stockfish
show you?

- John

I see. I'd rather stick with the default stable gentoo though, even though some programs have a slightly lower performance.
The command gives me:

Code: Select all

[ebuild   R   ~] games-board/stockfish-6-r1::gentoo  USE="optimize (-armv7) -debug -general-32 -general-64" CPU_FLAGS_X86="popcnt sse -avx2" 0 KiB

Total: 1 package (1 reinstall), Size of downloads: 0 KiB

What I've just done is to download the executable (of a development version of Stockfish) instead of compiling it, and I ran it on both Manjaro and Gentoo. Manjaro again came ahead by about 2% speed up : the nodes per second were 1225271, 12234486 and 1226166 for Manjaro and 1189259, 1198383 and 1197955 for Gentoo. (I ran the benchmark 3 times in a row only because the variance was low enough).
Manjaro uses the kernel 4.0.5 and gentoo 3.18.12. I'll reboot and try the gentoo kernel 4.0 even though in that kernel I didn't enable huge page.

Edit: Apparently I had huge page enabled also on gentoo kernel 4.0.5. Unfortunately it is signficantly slower than both manjaro and gentoo with kernel 3.18.12. The benchmarks were 1175523, 1174495 and 1172240. I am somehow puzzled. I feel like I messed up somewhere in the kernel configuration or something like that.

ct85711 · Post by **ct85711** » Fri Jun 19, 2015 1:52 am

If you want, you can see what kernel configurations Manjaro used, by making a copy of the /proc/config.gz. That has the running kernel's configuration gzipped.

Note: I just checked on my copy of Manjaro to make sure that file was available.

linux_dream · Post by **linux_dream** » Fri Jun 19, 2015 2:17 am

ct85711 wrote:If you want, you can see what kernel configurations Manjaro used, by making a copy of the /proc/config.gz. That has the running kernel's configuration gzipped.

Note: I just checked on my copy of Manjaro to make sure that file was available.

Yeah I had done so but honestly I didn't bother to check out all the differences. In fact I did it to see if I could got rid of a spam of USB related messages in dmesg in gentoo while I didn't get those spams in Manjaro. (I fixed that problem).

steveL · Post by **steveL** » Fri Jun 19, 2015 2:58 pm

Show us the emerge --info please (in a code tag or a pastebin.)

spectromas · Post by **spectromas** » Fri Jun 19, 2015 6:14 pm

As far as I remember manjaro uses BFS and BFQ in its default kernel as well.

CaptainBlood · Post by **CaptainBlood** » Fri Jun 19, 2015 7:38 pm

They are a lot of gcc compilation options not activated by default, possibly even at -03.
Some may speed up a specific program and slow down another.
They are also gcc compilation options that have an effect on how sub programs and the program as a whole are linked together : -flto is the major key here. it reduces size when applicable. Using /etc/portage/packages.env directory eases its activation at package level.
Not sure if -lfto makes a program always faster, but I won't be suprised if it does from a general perspective.

Thks 4 ur attention, interest & support.

CaptainBlood · Post by **CaptainBlood** » Fri Jun 19, 2015 7:45 pm

As for stable against unstable, especially regarding gcc, same cpu as different options as in:

Code: Select all

#gcc-4.9
#CFLAGS_GCC_BASE="-march=ivybridge -mmmx -mno-3dnow -msse -msse2 -msse3 -mssse3 -mno-sse4a -mcx16 -msahf -mno-movbe -mno-aes -mno-sha -mpclmul -mpopcnt -mno-abm -mno-lwp -mno-fma -mno-fma4 -mno-xop -mno-bmi -mno-bmi2 -mno-tbm -mavx -mno-avx2 -msse4.2 -msse4.1 -mno-lzcnt -mno-rtm -mno-hle -mno-rdrnd -mf16c -mfsgsbase -mno-rdseed -mno-prfchw -mno-adx -mfxsr -mxsave -mxsaveopt -mno-avx512f -mno-avx512er -mno-avx512cd -mno-avx512pf -mno-prefetchwt1 --param l1-cache-size=32 --param l1-cache-line-size=64 
--param l2-cache-size=3072 -mtune=ivybridge"

#gcc-4.8
#CFLAGS_GCC_BASE="-march=core-avx-i -mcx16 -msahf -mno-movbe -mno-aes -mpclmul -mpopcnt -mno-abm -mno-lwp -mno-fma -mno-fma4 -mno-xop -mno-bmi -mno-bmi2 -mno-tbm -mavx -mno-avx2 -msse4.2 -msse4.1 -mno-lzcnt -mno-rtm -mno-hle -mno-rdrnd -mf16c -mfsgsbase -mno-rdseed -mno-prfchw -mno-adx -mfxsr -mxsave -mxsaveopt --param l1-cache-size=3 
2 --param l1-cache-line-size=64 --param l2-cache-size=3072 -mtune=core-avx-i"

No idea what impact it has on generated code, still it may have in e insecure code sitting unused in the kernel waiting to be exploited down the road because SYSTEMD decided for you.
That is funny. How again has systemd decided that kdbus is suddenly in the kernel? Isn't that up to Linus Torvalds to decide that? Didn't see him accepting it, so far.

And how again is it unused if systemd uses it and delivers shims so that any DBUS software can use it? some cases.

N.B. above values have been extracted via

Code: Select all

gcc -march=native -E -v - </dev/null 2>&1 | grep cc1

linux_dream · Post by **linux_dream** » Fri Jun 19, 2015 10:49 pm

steveL wrote:Show us the emerge --info please (in a code tag or a pastebin.)

Here it is:

Code: Select all

~ $ sudo emerge --info
Password: 
Portage 2.2.18 (python 2.7.9-final-0, default/linux/amd64/13.0/systemd, gcc-4.8.4, glibc-2.20-r2, 3.18.12-gentoo x86_64)
=================================================================
System uname: Linux-3.18.12-gentoo-x86_64-Intel-R-_Core-TM-_i3-3217U_CPU_@_1.80GHz-with-gentoo-2.2
KiB Mem:     3935096 total,   3198616 free
KiB Swap:    4097020 total,   4097020 free
Timestamp of repository gentoo: Mon, 15 Jun 2015 12:30:01 +0000
sh bash 4.3_p33-r2
ld GNU ld (Gentoo 2.24 p1.4) 2.24
app-shells/bash:          4.3_p33-r2::gentoo
dev-java/java-config:     2.2.0::gentoo
dev-lang/perl:            5.20.2::gentoo
dev-lang/python:          2.7.9-r1::gentoo, 3.3.5-r1::gentoo
dev-util/cmake:           3.2.2::gentoo
dev-util/pkgconfig:       0.28-r2::gentoo
sys-apps/baselayout:      2.2::gentoo
sys-apps/openrc:          0.16.4::gentoo
sys-apps/sandbox:         2.6-r1::gentoo
sys-devel/autoconf:       2.13::gentoo, 2.69::gentoo
sys-devel/automake:       1.12.6::gentoo, 1.13.4::gentoo, 1.14.1::gentoo
sys-devel/binutils:       2.24-r3::gentoo
sys-devel/gcc:            4.8.4::gentoo
sys-devel/gcc-config:     1.7.3::gentoo
sys-devel/libtool:        2.4.6::gentoo
sys-devel/make:           4.1-r1::gentoo
sys-kernel/linux-headers: 3.18::gentoo (virtual/os-headers)
sys-libs/glibc:           2.20-r2::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=native -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LANG="en_US.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j2"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
USE="X acl alsa amd64 berkdb bzip2 cli cracklib crypt cxx dri fortran gdbm gudev iconv ipv6 mmx mmxext modules multilib ncurses nls nptl openmp pam pcre qt4 qt5 readline session sse sse2 ssl systemd tcpd udev unicode zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="avx mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="synaptics evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_3" RUBY_TARGETS="ruby19 ruby20" USERLAND="GNU" VIDEO_CARDS="fbdev glint intel mach64 mga nouveau nv r128 radeon savage sis tdfx trident vesa via vmware dummy v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON

As far as I remember manjaro uses BFS and BFQ in its default kernel as well.

I've just enabled BFS and rebuilt the kernel. It gives almost exactly the same performance, I can't notice any difference even after benchmarking. So I still haven't found out why Manjaro runs Stockfish 2 to 4% faster than my gentoo...
Also in gentoo I've selected the kernel for core2 and newer processors, something that's not done in Manjaro I believe.

CaptainBlood · Post by **CaptainBlood** » Sat Jun 20, 2015 12:40 am

Seems like you're focusing on kernel issue.
I didn't read all your env, but isn't possible to copy Manjaro kernel and dependancies to gentoo for a cross-distro boot?
I guess not ... Maybe someone else knows.

Isn't .config somehow available @ Manjaro for an import to gentoo?

Thks 4 ur attention, interest & support.

linux_dream · Post by **linux_dream** » Sat Jun 20, 2015 12:44 am

CaptainBlood wrote:Seems like you're focusing on kernel issue.
I didn't read all your env, but isn't possible to copy Manjaro kernel and dependancies to gentoo for a cross-distro boot?
I guess not ...
Maybe someone else knows.

Isnt .config somehow available @ Manjaro for an import to gentoo?

Thks 4 ur attention, interest & support.

Yes it's possible but I've just learned that it uses "BFQ" which is available in the AUR but apparently not enabled in the default gentoo kernel configuration. I guess it would be possible to add it although I wouldn't know how, and I don't know whether this is responsible for the speed difference.
Meanwhile I've tweaked some parameters in the kernel, retried benchmarks but I get no difference whatsoever.

steveL · Post by **steveL** » Sat Jun 20, 2015 2:21 am

One thing that stands out is you don't have:

Code: Select all

FFLAGS="$CFLAGS"
FCFLAGS="$FFLAGS"

in make.conf, and I believe fortran is used in some mathematical stuff, including in the backend for either gmp or mpfr, though I might be wrong on that.

I doubt it's going to make much difference, but if you're doing much scientific work it could; either way you want them set the same as CFLAGS in the general case (ie: unless you have a reason to do otherwise.) "-march=native" is important here.

Another one, that won't help benchmarks, but may improve responsiveness:

Code: Select all

# default: -Wl,-O1 -Wl,--as-needed
LDFLAGS="$LDFLAGS -Wl,--sort-common -Wl,--hash-style=gnu -Wl,-z,relro -Wl,-z,now -Wl,-z,combreloc"

The only controversial one is "-Wl,-z,now" which means binding is done at startup, not lazily which can be slower if many things need to be loaded, and may break some C++ (I hear) though it's never affected KDE ime; pretty sure there's support to remove things from LDFLAGS in ebuilds, but just a heads-up in case you want to remove it.

That and "-z relro" (as ld sees it) are better for security, the others help loader performance, as does relro overall (since the more that's mapped read-only, the better.)

I've been using those flags for years without hassle, fwtw.

spectromas · Post by **spectromas** » Sat Jun 20, 2015 3:26 am

linux_dream wrote:
CaptainBlood wrote:Seems like you're focusing on kernel issue.
I didn't read all your env, but isn't possible to copy Manjaro kernel and dependancies to gentoo for a cross-distro boot?
I guess not ...
Maybe someone else knows.

Isnt .config somehow available @ Manjaro for an import to gentoo?

Thks 4 ur attention, interest & support.
Yes it's possible but I've just learned that it uses "BFQ" which is available in the AUR but apparently not enabled in the default gentoo kernel configuration. I guess it would be possible to add it although I wouldn't know how, and I don't know whether this is responsible for the speed difference.
Meanwhile I've tweaked some parameters in the kernel, retried benchmarks but I get no difference whatsoever.

I doubt it would make much difference but BFQ comes with their kernel built in, there's nothing that involves AUR. Gentoo-sources comes with BFQ if you use the 'experimental' flag, then you just need to enable it in the kernel config.
BFS on the other hand, which manjaro also uses is a patch not available in gentoo sources but is available in ck-sources. Ck-sources also includes BFQ so you could just build your kernel with ck-sources, make sure BFQ and BFS are enabled and you should have something similar to the manjaro kernel, not sure what other patches they may use though.

CaptainBlood · Post by **CaptainBlood** » Sat Jun 20, 2015 8:43 am

Sorry to ask why stockfish compiling env seems discarded, as it almost seems only kernel is focused on?**
My very little investigation on the sheduler option would indicate it mostly has to do witch UI responsivness.
EDIT slimmed down
Thks 4 ur attention, interest & support

Post by **John R. Graham** » Sat Jun 20, 2015 11:16 am

CaptainBlood wrote:Sorry to ask why stockfish compiling env seems discarded, as it almost seems only kernel is focused on?

Because he imported the same binary he was running under Manjaro and saw approximately the same 2% difference, which implicates the kernel.

- John

Post by **NeddySeagoon** » Sat Jun 20, 2015 1:44 pm

John R. Graham,

... or one or more libraries used by stockfish.
For a CPU bound application, the kernel should mostly stay out of the way, so a 2% performace reduction, due to the kernel is unlikely.

If Manjaro is using gcc-5 it might also be using LTO, which will make a differece to lots of things except the kernel as the kernel is a stand alone app.
There are also some non standard CPU targets for the kernel which might have some performace effects.

Post by **John R. Graham** » Sat Jun 20, 2015 2:50 pm

Good point, but applications like stockfish are mostly computational and shouldn't, I think, be dependent for a lot of their computation on external libraries. Chess analysis is simple mathematics yet deeply iterated and/or recursed. Admittedly 2% seems like a lot of overhead for a kernel configuration difference to make up. especially on a lightly-loaded system.

- John

Post by **NeddySeagoon** » Sat Jun 20, 2015 3:02 pm

John R. Graham,

Code: Select all

g++ -march=native -O2 -pipe -Wall -Wcast-qual -fno-exceptions -fno-rtti  -ansi -pedantic -Wno-long-long -Wextra -Wshadow -DNDEBUG -O3 -DIS_64BIT -msse -DUSE_BSFQ -msse3 -mpopcnt -DUSE_POPCNT -flto -Wall -Wcast-qual -fno-exceptions -fno-rtti  -ansi -pedantic -Wno-long-long -Wextra -Wshadow -DNDEBUG -O3 -DIS_64BIT -msse -DUSE_BSFQ -msse3 -mpopcnt -DUSE_POPCNT -flto   -c -o benchmark.o benchmark.cpp

That fairly agressive ... -O3 -flto

It does use a few libraries

Code: Select all

 ldd /usr/bin/stockfish
	linux-vdso.so.1 (0x00007ffcaede7000)
	libc.so.6 => /lib64/libc.so.6 (0x00007ff4e2893000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff4e2678000)
	libstdc++.so.6 => /usr/lib/gcc/x86_64-pc-linux-gnu/4.9.2/libstdc++.so.6 (0x00007ff4e2364000)
	libm.so.6 => /lib64/libm.so.6 (0x00007ff4e2061000)
	/lib64/ld-linux-x86-64.so.2 (0x00007ff4e2c30000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007ff4e1e4a000)

and I don't have system wide -flto.
I don'f feel like experimenting much further as -flto that doesn't work on those libraries will need my static busybox to fix.

CaptainBlood · Post by **CaptainBlood** » Tue Jun 23, 2015 9:14 am

NeddySeagoon wrote: ....I don'f feel like experimenting much further as -flto that doesn't work on those libraries will need my static busybox to fix.

Despite out of topics, could you plz explain 'static busybox part' from a general perspective, or a crystal clear link maybe ...

Thks 4 ur attention, interest & support;

Post by **NeddySeagoon** » Tue Jun 23, 2015 7:39 pm

CaptainBlood,

When you make a mess of glibc, almost nothing works because almost everything is linnked against it. That includes bash, so even if your box comes up with a broken glibc, you won't have a shell.

busybox providle lots of tools in a single binary. Further, it can be statically linked, so it has no runtime dependancies.
This is a good thing for initrds, where it normally provides several functions an a fall back shell.

In a oops, I broke my box by building key libraries in a way that doesn't work, a statically linked busybox saves you finding a liveCD to fix it.You boot to

Code: Select all

busybox sh

knowing that the broken libries are not required.

Code: Select all

$ lddtree /bin/busybox
/bin/busybox (interpreter => None)

Code: Select all

$ eix busybox
[I] sys-apps/busybox
     Available versions:  1.20.2^t 1.21.0^t{tbz2} (~)1.21.1^t{tbz2} (~)1.22.0^t (~)1.22.1^t{tbz2} (~)1.22.1-r1^t{tbz2} 1.23.0-r1^t{tbz2} 1.23.1^t 1.23.1-r1^t{tbz2} (~)1.23.2^t{tbz2} **9999^t {debug ipv6 livecd make-symlinks math mdev -pam savedconfig selinux sep-usr +static syslog systemd}
     Installed versions:  1.23.2^t{tbz2}(21:05:08 06/03/15)(ipv6 pam static -debug -livecd -make-symlinks -math -mdev -savedconfig -selinux -sep-usr -syslog -systemd)
     Homepage:            http://www.busybox.net/
     Description:         Utilities for rescue and embedded system

Thats what the static USE flag does.

The Doctor · Post by **The Doctor** » Wed Jun 24, 2015 4:54 pm

NeddySeagoon wrote:
Code: Select all
...-O2 ... -O3 

Isn't that going to cause problems?

Post by **NeddySeagoon** » Wed Jun 24, 2015 8:16 pm

The Doctor,

-O3 will be used. One would hope that as its the build system setting -O3 for itself, the package authors know it works with -O3.