Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
LTO: We are almost there
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4, 5, 6, 7  Next  
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
saboya
Guru
Guru


Joined: 28 Nov 2006
Posts: 552
Location: Brazil

PostPosted: Fri Dec 30, 2016 8:38 pm    Post subject: Reply with quote

Just did emerge -e @system.

Packages used:
Code:
=sys-devel/gcc-6.3.0 **
=dev-libs/isl-0.17.1 ~amd64
=sys-libs/glibc-2.24* **
=sys-devel/binutils-2.27* ~amd64
=sys-libs/binutils-libs-2.27* ~amd64
=dev-libs/mpfr-3.1.5* ~amd64


make.conf
Code:
AR="/usr/bin/gcc-ar"
NM="/usr/bin/gcc-nm"
RANLIB="/usr/bin/gcc-ranlib"

FLTO="-flto=4 -fuse-linker-plugin -fno-fat-lto-objects"
FGRAPHITE="-fgraphite-identity -floop-interchange -floop-strip-mine -floop-block"

CFLAGS="-march=ivybridge -O2 -pipe ${FLTO} ${FGRAPHITE} -ftree-vectorize"
CXXFLAGS="${CFLAGS} -fno-delete-null-pointer-checks -flifetime-dse=1"
LDFLAGS="${LDFLAGS} ${CXXFLAGS}"


The following pacakges need to be bumped in order to compile:
Code:
=dev-lang/spidermonkey-1.8.5-r6 ~amd64
=sys-fs/eudev-3.2.1* ~amd64
=sys-apps/util-linux-2.27.1* ~amd64


And the following packages needed to be built with bfd, no LTO or no Graphite:
Code:
media-video/avidemux linker-bfd
sys-devel/gettext linker-bfd
sys-fs/fuse linker-bfd

app-arch/cpio no-graphite
app-arch/tar no-graphite
sys-apps/gawk no-graphite
dev-libs/libgcrypt no-lto
dev-lang/spidermonkey:0/mozjs185 no-lto
sys-apps/groff no-graphite
sys-apps/man-db no-lto no-graphite
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Sat Dec 31, 2016 4:47 am    Post subject: Reply with quote

saboya wrote:
And the following packages needed to be built with bfd, no LTO or no Graphite:

I wonder how you come to such small lists. I am using the same toolchain with gold, and due to the postings here, I recently retried whether I could use flto. My list of not-compiling packages still has 450 packages listed (from about 1500 installed), and many things like e.g. qpdfview would have runtime issues with it or cause other packages to fail (e.g. python with -flto will break lilypond, since lilypond inherits some CFLAGS from python). Actually, the number of my entries is larger than 450, since I didn't count the following lines
/etc/portage/package.cflags wrote:
*-libs/* +fwhole-program
dev-perl/* +fwhole-program
dev-python/* +fwhole-program
dev-qt/* +flto*
kde-frameworks/* +fwhole-program
media-plugins/* +fwhole-program
perl-core/* +fwhole-program

(Yes, I am also using -fwhole-program by default, but this alone doesn't make the huge number: Note that e.g. practically none of dev-qt/* does not contain only +fwhole-program, not even combined with -ffat-lto-objects: Neither is enough for the vast majority of qt packages, neither in :4 nor in :5. Note that +flto* implies +fwhole-program in portage-bashrc-mv...)

Just an example: In the app-arch category, I still need
Quote:
app-arch/bzip2 +flto*
app-arch/cabextract +fwhole-program -ffat-lto-objects
app-arch/cpio +fwhole-program -ffat-lto-objects
app-arch/gzip +fwhole-program -ffat-lto-objects
app-arch/lha !+flto* # ICE on x86 without it
app-arch/libarchive +fwhole-program -ffat-lto-objects
app-arch/lrzip +fwhole-program -ffat-lto-objects
app-arch/lz4 +fwhole-program -ffat-lto-objects
app-arch/sharutils +fwhole-program ~-ffat-lto-objects !+flto* # ICE on x86, link breakage on amd64
app-arch/star +fwhole-program -ffat-lto-objects
app-arch/tar +fwhole-program -ffat-lto-objects
app-arch/zpaq +fwhole-program

(and no: +fwhole-program alone is not enough for any of these packages).
The list indeed became slightly smaller: Some years ago, I still had in this list
Quote:
#app-arch/p7zip +flto*
#app-arch/par2cmdline +flto*
#app-arch/unrar +flto*
#app-arch/unzip +fwhole-program -ffat-lto-objects

and, moreover, many of the packages which have now +fwhole-program -flat-lto-objects required the usage of +flto*

For comparison: eix -IC# arch shows 32 packages:
Code:
app-arch/afio
app-arch/archwrap
app-arch/arj
app-arch/bzip2
app-arch/bzwc
app-arch/cabextract
app-arch/cpio
app-arch/gzip
app-arch/lha
app-arch/libarchive
app-arch/lrzip
app-arch/lz4
app-arch/lzip
app-arch/ncompress
app-arch/p7zip
app-arch/paq8l
app-arch/par2cmdline
app-arch/rar
app-arch/rpm2targz
app-arch/sharutils
app-arch/snappy
app-arch/star
app-arch/tar
app-arch/tarsync
app-arch/unace
app-arch/unarj
app-arch/unrar
app-arch/unzip
app-arch/xz-utils
app-arch/zip
app-arch/zoo
app-arch/zpaq

So this category seems to be rather representative: About half of the packages did not work previously. With recent toolchain (or recent package upgrades - I don't know the reason) the number of non-working packages has reduced by only 25% (i.e. by 12.5% from the total number of packages): 32->16->12. I wouldn't call this a dramatic improvement or even as the thread title suggests "we are almost there"...
Back to top
View user's profile Send private message
saboya
Guru
Guru


Joined: 28 Nov 2006
Posts: 552
Location: Brazil

PostPosted: Sat Dec 31, 2016 6:34 am    Post subject: Reply with quote

mv wrote:
saboya wrote:
And the following packages needed to be built with bfd, no LTO or no Graphite:

I wonder how you come to such small lists. I am using the same toolchain with gold, and due to the postings here, I recently retried whether I could use flto. My list of not-compiling packages still has 450 packages listed (from about 1500 installed), and many things like e.g. qpdfview would have runtime issues with it or cause other packages to fail (e.g. python with -flto will break lilypond, since lilypond inherits some CFLAGS from python). Actually, the number of my entries is larger than 450, since I didn't count the following lines

My list is only about @system packages, I did not compile @world yet. But I don't think is gonna get much bigger than that. But my world has ~900 packages, far from your 1500. I'm also on Gnome 3.22, and I'm very picky about my USE.
Back to top
View user's profile Send private message
NTU
Apprentice
Apprentice


Joined: 17 Jul 2015
Posts: 187

PostPosted: Sat Dec 31, 2016 8:05 am    Post subject: Reply with quote

costel78 wrote:
I have free time until 3rd of January.
One small sugestion regarding build script: why not make profiledbootstrap -j${CORES} or even make autoprofiledback, perf and autofdo are needed in the last case, see: https://gcc.gnu.org/install/build.html - Building with profile feedback
It is very unlikely that someone using your script are still using x86, or if architecture is x86 go with the simple make to satisfy both cases.
I always build gcc with GCC_MAKE_TARGET="profiledbootstrap", via package.env and autofdo would be really something :D .


The info at the GCC build page is written so backwards I'm having a hard time understanding it. Replace make -j${CORES} with make -j{CORES} profiledbootstrap and make -j{CORES} autoprofiledback, or does autoprofiledback include a regular make build and profiledbootstrap? Or is autoprofiledback a replacement for profiledbootstrap? I'm surprised that you're still learning English and can understand the broken English and incredibly unclear writing by the GCC devs. If you can be more specific, sure no problem! I just can't follow and don't want to spend the time deciphering such things. Thanks for the tip! Including a testcase for perf and autofdo is easy. Also, I should re-work the Gentoo GCC detecting thing a bit, because with a non-hardened profile, there is no *-vanilla GCC, so the script will fail for non-hardened builds, as it will assume it's hardened, with a non-vanilla GCC. I recently discovered this :P

If anyone else is using that script for testing out the incremental LTO linking with -r, could you star the project page on github? Thanks!
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Sat Dec 31, 2016 9:57 am    Post subject: Reply with quote

The full /etc/portage/package.cflags is now public.

In particular, it contains the LTO related list.

All data with the exception of "graphite" will presumably be updated quite regularly.
Back to top
View user's profile Send private message
costel78
Guru
Guru


Joined: 20 Apr 2007
Posts: 402

PostPosted: Sat Dec 31, 2016 1:43 pm    Post subject: Reply with quote

@NTU
make profiledbootstrap and make autoprofiledbootstrap are different "branches". They collect profile info differently and exclude each other. Both of them use the fact that gcc itself is built in 3 stages, and both collect data on stage 2, as far I understand from Makefile
Maybe if you read https://gcc.gnu.org/wiki/AutoFDO/Tutorial you will understand better than I can explain.

Auto-Feedback-Directed-Optimization it is new and I haven't had much succes with it, yet. First documentation is misleading. There is no autoprofiledback, but autoprofiledbootstrap.
I am interested about it because I wanted to mix Howto compile packages with PGO with AutoFDO and optimize, at least important packages, in Intel Clear Linux way.

profiledbootstrap is the old good way to make gcc compile faster. You can export GCC_MAKE_TARGET="profiledbootstrap" emerge gcc, or use package.env for that. If you manually compile gcc, make -jN profiledbootstrap is sufficient. At least for me, it is rock stable and I used it for years.

@mv The list of "usual" packages which failed with lto decreased over years. sci-*/, video/audio editors packages still tend to fail.
Also, -ffat-lto-objects help a lot.
Is liblto_plugin.so.0.0.0 present in /usr/x86_64-pc-linux-gnu/binutils-bin/lib/bfd-plugins/, assuming you are on amd64 ?

For me, saboya list seems to small, and yours to big. 450 of 1500 is 30%
Here is my list:
Code:
app-arch/cpio lto-no-graphite
app-arch/tar lto-no-graphite
app-crypt/pinentry no-lto-graphite
dev-lang/orc lto-no-graphite
dev-lang/php php
dev-lang/spidermonkey:0/mozjs185 no-lto-graphite
dev-libs/elfutils no-lto-graphite
dev-libs/libgcrypt no-lto-graphite
dev-python/notify-python no-lto-graphite
games-strategy/dunelegacy std98
gnome-base/gnome-shell no-lto-graphite
mail-mta/postfix fat-lto-graphite
media-libs/alsa-lib no-lto-graphite
media-libs/flac lto-no-graphite
media-libs/x264 fat-lto-graphite
media-sound/pulseaudio no-lto-graphite
media-video/ffmpeg lto-no-graphite
net-mail/mailutils lto-no-graphite
net-misc/dhcp no-lto-graphite
sys-apps/gawk lto-no-graphite
sys-apps/groff lto-no-graphite
sys-apps/man-db no-lto-no-graphite
sys-apps/pciutils no-lto-graphite
sys-boot/gnu-efi no-lto-no-graphite
sys-boot/refind std98
sys-process/numactl no-lto-graphite
www-client/w3m no-plugins


and my emerge --info
Code:
Portage 2.3.3 (python 3.5.2-final-0, default/linux/amd64/13.0/no-multilib, gcc-6.3.0, glibc-2.24, 4.9.0-gentoo-costel x86_64)
=================================================================
System uname: Linux-4.9.0-gentoo-costel-x86_64-Intel-R-_Core-TM-_i7-6700K_CPU_@_4.00GHz-with-gentoo-2.3
KiB Mem:    31848368 total,  22508784 free
KiB Swap:          0 total,         0 free
Timestamp of repository gentoo: Sat, 31 Dec 2016 11:00:01 +0000
sh bash 4.4_p5-r1
ld GNU ld (Gentoo 2.27 p1.0) 2.27
app-shells/bash:          4.4_p5-r1::gentoo
dev-lang/perl:            5.24.1_rc4::gentoo
dev-lang/python:          2.7.12::gentoo, 3.5.2::gentoo
dev-util/cmake:           3.7.1::gentoo
dev-util/pkgconfig:       0.29.1::gentoo
sys-apps/baselayout:      2.3::gentoo
sys-apps/sandbox:         2.10-r2::gentoo
sys-devel/autoconf:       2.13::gentoo, 2.69-r2::gentoo
sys-devel/automake:       1.13.4-r1::gentoo, 1.14.1-r1::gentoo, 1.15-r2::gentoo
sys-devel/binutils:       2.27::gentoo
sys-devel/gcc:            6.3.0::gentoo
sys-devel/gcc-config:     1.8-r1::gentoo
sys-devel/libtool:        2.4.6-r2::gentoo
sys-devel/make:           4.2.1::gentoo
sys-kernel/linux-headers: 4.9::gentoo (virtual/os-headers)
sys-libs/glibc:           2.24::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://192.168.78.252/gentoo-portage/
    priority: -1000
    sync-rsync-extra-opts: --human-readable --delete-before --progress
    sync-rsync-vcs-ignore: true

added
    location: /usr/portage/local/added
    masters: gentoo
    priority: 30

newer
    location: /usr/portage/local/newer
    masters: gentoo
    priority: 30

Installed sets: @minimal, @minimal-desktop, @squashmount, @sunflower
ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe -march=native -mtune=skylake -fomit-frame-pointer -w -fgraphite-identity -floop-nest-optimize -flto=8 -fuse-linker-plugin -fno-fat-lto-objects -fpredictive-commoning -fgcse-after-reload -fvect-cost-model -ftree-partial-pre -ftree-vectorize -fgcse-las"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -pipe -march=native -mtune=skylake -fomit-frame-pointer -w -fgraphite-identity -floop-nest-optimize -flto=8 -fuse-linker-plugin -fno-fat-lto-objects -fpredictive-commoning -fgcse-after-reload -fvect-cost-model -ftree-partial-pre -ftree-vectorize -fgcse-las -fno-delete-null-pointer-checks -flifetime-dse=1"
DISTDIR="/mnt/linux/distfiles"
EMERGE_DEFAULT_OPTS="--jobs=4 --load-average=12 --keep-going --with-bdeps=y --complete-graph --quiet-build=n --autounmask-write"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs candy cgroup collision-protect config-protect-if-modified distlocks fail-clean fixlafiles ipc-sandbox merge-sync news nodoc noinfo parallel-fetch parallel-install preserve-libs protect-owned sandbox sfperms strict unknown-features-filter unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://localhost/ http://mirrors.evowise.com/gentoo/ http://mirrors.xservers.ro/gentoo/ http://ftp.romnet.org/gentoo/ ftp://ftp.romnet.org/gentoo/ http://distfiles.gentoo.org/"
LANG="ro_RO.UTF-8"
LC_ALL="ro_RO.UTF-8"
LDFLAGS="-Wl,-O1,--sort-common,--hash-style=gnu,--as-needed,-z,relro,-z,now -O2 -pipe -march=native -mtune=skylake -fomit-frame-pointer -w -fgraphite-identity -floop-nest-optimize -flto=8 -fuse-linker-plugin -fno-fat-lto-objects -fpredictive-commoning -fgcse-after-reload -fvect-cost-model -ftree-partial-pre -ftree-vectorize -fgcse-las -fno-delete-null-pointer-checks -flifetime-dse=1"
MAKEOPTS="-j8 --load-average=16"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_EXTRA_OPTS="--human-readable --delete-before --progress"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="X a52 aac acl aes alsa amd64 avx avx2 berkdb branding bzip2 cairo cdda cli colord cracklib crypt cups custom-cflags custom-optimization cxx dbus dri dvd egl encode exif fam flac fma3 gdbm gif glamor gles gnome-keyring gpm gstreamer gtk gtk3 iconv icu ipv6 jemalloc jpeg logrotate lz4 mmx mmxext modules mp3 mp4 mpeg ncurses nls nptl ogg opengl openmp pam pango pcre pdf png policykit popcnt pulseaudio python readline sdl seccomp session spell sse sse2 sse3 sse4_1 sse4_2 ssl ssse3 startup-notification svg systemd tcpd threads tiff truetype udev udisks unicode usb vaapi vala vdpau vhosts vorbis wayland x264 xattr xcb xml xv xvid zlib" ABI_X86="64" ALSA_CARDS="hda-intel usb-audio" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 fma3 mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="evdev keyboard mouse" KERNEL="linux" L10N="ro en" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="ro en" NETBEANS_MODULES="apisupport extide php webcommon" NGINX_MODULES_HTTP="access addition auth_basic autoindex browser cache_purge charset dav dav_ext echo empty_gif fancyindex fastcgi flv geo geoip gunzip gzip headers_more image_filter limit_conn limit_req map memc memcached metrics mp4 naxsi perl random_index realip referer rewrite scgi secure_link split_clients ssi sticky stub_status sub upload_progress upstream_ip_hash userid uwsgi xslt" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-0" PYTHON_SINGLE_TARGET="python3_5" PYTHON_TARGETS="python2_7 python3_5" RUBY_TARGETS="ruby23 ruby24" USERLAND="GNU" VIDEO_CARDS="intel i965" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
USE_PYTHON="2.7 3.5"
Unset:  CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS


Once again, thanks for your work at squashmount.

Happy New Year!

Later: From toolchain.eclass:
Code:
toolchain_src_compile() {
        touch "${S}"/gcc/c-gperf.h

        # Do not make manpages if we do not have perl ...
        [[ ! -x /usr/bin/perl ]] \
                && find "${WORKDIR}"/build -name '*.[17]' -exec touch {} +

        gcc_do_make ${GCC_MAKE_TARGET}
}


and in gcc_do_make there is
Code:
emake \
                LDFLAGS="${LDFLAGS}" \
                STAGE1_CFLAGS="${STAGE1_CFLAGS}" \
                LIBPATH="${LIBPATH}" \
                BOOT_CFLAGS="${BOOT_CFLAGS}" \
                ${GCC_MAKE_TARGET} \
                || die "emake failed with ${GCC_MAKE_TARGET}"


So yes, there is either make or make profiledboostrap or make autoprofiledbootstrap

I'll give another try to autoprofiledbootstrap, let see if I can get it build. It wold be nice to work, not because of speed - it should be equal to profiledboostrap, but for chance for "(auto)PGO for all packages".
_________________
Sorry for my English. I'm still learning this language.


Last edited by costel78 on Sat Dec 31, 2016 1:57 pm; edited 1 time in total
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Sat Dec 31, 2016 1:54 pm    Post subject: Reply with quote

costel78 wrote:
Is liblto_plugin.so.0.0.0 present in /usr/x86_64-pc-linux-gnu/binutils-bin/lib/bfd-plugins/

No, there is no file "liblto*" on my system, though I am using sys-devel/binutils-2.27[cxx -multitarget -nls -static-libs -test -vanilla].
The only file in /usr/x86_64-pc-linux-gnu/binutils-bin/lib/bfd-plugins is a symlink to LLVMgold.so
However, I am not sure whether this is important at all, because I switched to gold linker with eselect.
Back to top
View user's profile Send private message
costel78
Guru
Guru


Joined: 20 Apr 2007
Posts: 402

PostPosted: Sat Dec 31, 2016 2:12 pm    Post subject: Reply with quote

Yes, in this case is not important.
liblto_plugin.so.0.0.0 belong to gcc and is recomand to make a symlink to /usr/x86_64-pc-linux-gnu/binutils-bin/lib/bfd-plugins/ because that was the location where old gcc-ar|nm|randlib wrappers was looking for plugin. And some of them still search there.
Code:
gentoo ~ # locate liblto* -b
/home/toolchain_builder/toolchain-src/gcc/build/prev-gcc/liblto_plugin.so.0.0.0
/home/toolchain_builder/toolchain-src/gcc/build/prev-lto-plugin/.libs/liblto_plugin.so.0.0.0
/mnt/usb/usr/libexec/gcc/x86_64-pc-linux-gnu/6.3.0/liblto_plugin.so.0.0.0
/usr/libexec/gcc/x86_64-pc-linux-gnu/6.3.0/liblto_plugin.so.0.0.0
/usr/x86_64-pc-linux-gnu/binutils-bin/lib/bfd-plugins/liblto_plugin.so.0.0.0

_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
costel78
Guru
Guru


Joined: 20 Apr 2007
Posts: 402

PostPosted: Sat Dec 31, 2016 3:06 pm    Post subject: Reply with quote

No succes with autoprofiledbootstrap.

Permission error mapping pages.
Consider increasing /proc/sys/kernel/perf_event_mlock_kb,
or try again with a smaller value of -m/--mmap_pages.
(current value: 4294967295,0)
make[2]: *** [Makefile:6651: all-stageautoprofile-intl] Error 255
make[2]: *** Waiting for unfinished jobs....

cat /proc/sys/kernel/perf_event_mlock_kb
516000
Default is 516

[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 2.365 MB perf.data (2985 samples) ]

cat /proc/sys/kernel/perf_event_max_sample_rate
22000
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Sun Jan 01, 2017 8:10 am    Post subject: Reply with quote

@costel: Thanks for the information.
costel78 wrote:
liblto_plugin.so.0.0.0 belong to gcc

I found it; my locate database was not recent enough. Unfortunately, this means that the plugin is available only in a gcc-version specific directory. Hence, the following is very inconvenient:
Quote:
and is recomand to make a symlink to /usr/x86_64-pc-linux-gnu/binutils-bin/lib/bfd-plugins/

That's awful: one has to remember to update the link whenever one switches/updates gcc. If this is recommended, then gcc-config should do it. Did anybody write a wrapper/patch for gcc-config?
Back to top
View user's profile Send private message
trippels
Tux's lil' helper
Tux's lil' helper


Joined: 24 Nov 2010
Posts: 137
Location: Berlin

PostPosted: Sun Jan 01, 2017 8:35 am    Post subject: Reply with quote

mv wrote:
@costel: Thanks for the information.
costel78 wrote:
liblto_plugin.so.0.0.0 belong to gcc

I found it; my locate database was not recent enough. Unfortunately, this means that the plugin is available only in a gcc-version specific directory. Hence, the following is very inconvenient:
Quote:
and is recomand to make a symlink to /usr/x86_64-pc-linux-gnu/binutils-bin/lib/bfd-plugins/

That's awful: one has to remember to update the link whenever one switches/updates gcc. If this is recommended, then gcc-config should do it. Did anybody write a wrapper/patch for gcc-config?


Currently there is no reason, because liblto_plugin.so from any gcc version can be used interchangeably.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Sun Jan 01, 2017 9:21 am    Post subject: Reply with quote

trippels wrote:
Currently there is no reason, because liblto_plugin.so from any gcc version can be used interchangeably.

Typical gcc updgrade scenario: emerge -1 gcc ; gcc-config ; emerge --depclean -> symlink is broken.
Back to top
View user's profile Send private message
trippels
Tux's lil' helper
Tux's lil' helper


Joined: 24 Nov 2010
Posts: 137
Location: Berlin

PostPosted: Sun Jan 01, 2017 9:34 am    Post subject: Reply with quote

mv wrote:
trippels wrote:
Currently there is no reason, because liblto_plugin.so from any gcc version can be used interchangeably.

Typical gcc updgrade scenario: emerge -1 gcc ; gcc-config ; emerge --depclean -> symlink is broken.


You're right. So the best solution would be if a new gcc ebuild would set up the symlink automatically, overwriting any existing ones.
Back to top
View user's profile Send private message
costel78
Guru
Guru


Joined: 20 Apr 2007
Posts: 402

PostPosted: Sun Jan 01, 2017 10:33 am    Post subject: Reply with quote

Well, the situation is very unclear. Since gcc-6 neighter gcc-plugins (AR="/usr/bin/gcc-ar" NM="/usr/bin/gcc-nm" RANLIB="/usr/bin/gcc-ranlib" ) or the infamous symlink of liblto_plugin.so.0.0.0 are not required according to documentation. And it is true as long as you are using -ffat-lto-objects. The compile failure rate is very low, but a quite high number of packages are bigger than without lto, probably glimpse parts are preserved.
When you are using -fno-fat-lto-objects, which is the default, without gcc-plugins and the symlink, the compile failure rate is very high. I do not know who is to blame, gcc or package build system.
The interested part is that even with symllink and gcc-plugins wrappers sometimes gcc-strip still having troubles (plugin needed to handle lto object) - python is an example that cross to my mind right now.

On every major gcc version since gcc5 I keep a small stage4 compiled without lto, do a equery s for every package and compare the size with the same system compiled with lto, that's how I drew these conclusions.

I was wrong (remembered wrong): only wrappers are not required, the symlink is. Sources: https://gcc.gnu.org/ml/gcc-help/2016-02/msg00003.html and https://forums.gentoo.org/viewtopic-t-1035544-start-0.html
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
trippels
Tux's lil' helper
Tux's lil' helper


Joined: 24 Nov 2010
Posts: 137
Location: Berlin

PostPosted: Sun Jan 01, 2017 11:13 am    Post subject: Reply with quote

costel78 wrote:
Well, the situation is very unclear. Since gcc-6 neighter gcc-plugins (AR="/usr/bin/gcc-ar" NM="/usr/bin/gcc-nm" RANLIB="/usr/bin/gcc-ranlib" ) or the infamous symlink of liblto_plugin.so.0.0.0 are not required according to documentation. And it is true as long as you are using -ffat-lto-objects. The compile failure rate is very low, but a quite high number of packages are bigger than without lto, probably glimpse parts are preserved.
When you are using -fno-fat-lto-objects, which is the default, without gcc-plugins and the symlink, the compile failure rate is very high. I do not know who is to blame, gcc or package build system.
The interested part is that even with symllink and gcc-plugins wrappers sometimes gcc-strip still having troubles (plugin needed to handle lto object) - python is an example that cross to my mind right now.

On every major gcc version since gcc5 I keep a small stage4 compiled without lto, do a equery s for every package and compare the size with the same system compiled with lto, that's how I drew these conclusions.

I was wrong (remembered wrong): only wrappers are not required, the symlink is. Sources: https://gcc.gnu.org/ml/gcc-help/2016-02/msg00003.html and https://forums.gentoo.org/viewtopic-t-1035544-start-0.html


Just to make thinks more clear:

If you use the symlink, you don't need to use the gcc wrappers anymore. That is the whole point of the symlink.
Just use the binutils ar, nm, ranlib directly.

For -ffat-lto-objects it works without the symlink. But in this case you're not using any LTO optimizations,
because only the non-lto part of the "fat" object is used and the IR (GIMPLE) part is simply discarded.
Back to top
View user's profile Send private message
costel78
Guru
Guru


Joined: 20 Apr 2007
Posts: 402

PostPosted: Sun Jan 01, 2017 11:57 am    Post subject: Reply with quote

Quote:
For -ffat-lto-objects it works without the symlink. But in this case you're not using any LTO optimizations,
because only the non-lto part of the "fat" object is used and the IR (GIMPLE) part is simply discarded.


Then why some packages compiled with ffat-lto-objects are larger that the same package compiled without lto at all ?
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
trippels
Tux's lil' helper
Tux's lil' helper


Joined: 24 Nov 2010
Posts: 137
Location: Berlin

PostPosted: Sun Jan 01, 2017 12:11 pm    Post subject: Reply with quote

costel78 wrote:
Quote:
For -ffat-lto-objects it works without the symlink. But in this case you're not using any LTO optimizations,
because only the non-lto part of the "fat" object is used and the IR (GIMPLE) part is simply discarded.


Then why some packages compiled with ffat-lto-objects are larger that the same package compiled without lto at all ?


You mean the object files and archives? Of course they are larger, because they contain normal object code and compressed GIMPLE code.
Back to top
View user's profile Send private message
costel78
Guru
Guru


Joined: 20 Apr 2007
Posts: 402

PostPosted: Sun Jan 01, 2017 12:51 pm    Post subject: Reply with quote

Oh, I see. Sorry, my bad English, I taken discarded to literary, as drop/eliminated, instead as ignored.
Anybody have a clue how to fix perf error I encountered ? It still bugging me.
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
saboya
Guru
Guru


Joined: 28 Nov 2006
Posts: 552
Location: Brazil

PostPosted: Mon Jan 02, 2017 3:47 am    Post subject: Reply with quote

This is the full list of packages that fail for me:

Code:
app-arch/cpio no-graphite
app-arch/tar no-graphite
sys-apps/gawk no-graphite
dev-libs/libgcrypt no-lto
dev-lang/spidermonkey:0/mozjs185 no-lto
sys-apps/groff no-graphite
sys-apps/man-db no-lto no-graphite

media-libs/alsa-lib no-lto
dev-lang/orc no-graphite
dev-qt/qtscript no-lto
media-libs/flac no-graphite
x11-base/xorg-server no-lto
x11-drivers/xf86-video-intel no-lto
net-misc/dhcp no-lto
dev-libs/elfutils no-lto std98
media-sound/pulseaudio no-lto
media-sound/audacity no-lto
media-libs/x264 no-lto
media-libs/x265 std98
media-libs/audiofile std98
mail-mta/nullmailer no-lto
dev-lang/php no-lto
dev-java/icedtea no-lto
media-sound/musepack-tools lm
media-libs/avidemux-core no-lto no-graphite

#dev-db/mysql-workbench no-lto
#net-firewall/ipsec-tools no-graphite no-lto

gnome-base/gnome-shell no-lto


dev-util/ragel - Needs patches to compile with GCC6. - https://bugs.gentoo.org/show_bug.cgi?id=582606
deb-db/mysqlworkbench - Couldn't get it to work with GCC6. Got a patch from Debian that fixed a couple things, but still fails. https://bugs.gentoo.org/show_bug.cgi?id=601938
net-firewall/ipsec-tools - Some failure with flex. I have no idea how to fix this. Will open a bug report.
app-admin/sysklogd - Needs a patch for Glibc 2.24 - https://bugs.gentoo.org/show_bug.cgi?id=604232

avidemux / avidemux-core / avidemix-plugins need version bumps that are outside the tree atm.
Back to top
View user's profile Send private message
costel78
Guru
Guru


Joined: 20 Apr 2007
Posts: 402

PostPosted: Mon Jan 02, 2017 1:47 pm    Post subject: Reply with quote

xorg-server it is a surprise, at least for me, it compiles with lto without a glitch for a while now and php it is compiling with lto with -flto-partition=max and disabled imap useflag.
media-libs/x264 x265 requires -ffat-lto-objects.
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
saboya
Guru
Guru


Joined: 28 Nov 2006
Posts: 552
Location: Brazil

PostPosted: Mon Jan 02, 2017 2:26 pm    Post subject: Reply with quote

Maybe it's xorg version? I'm on 1.19. Admittedly, I'm probably over my head fiddling with these CFLAGS since my techincal knowledge about this is limited, but we learn as we go :)

*edit* - Compiled it again just to be sure, still failing! Here's the build.log: http://filebin.ca/37Vu64d5ewnG
Back to top
View user's profile Send private message
costel78
Guru
Guru


Joined: 20 Apr 2007
Posts: 402

PostPosted: Mon Jan 02, 2017 5:08 pm    Post subject: Reply with quote

It does not look like "usuals" lto errors (different symbols missing/unknow), but error: array subscript is below array bounds [-Werror=array-bounds]; lto1: some warnings being treated as errors.
Are you on hardened profile ? If so, try with vanilla gcc.
Difference between our useflags:
Code:
USE: abi_x86_64 amd64 dmx elibc_glibc glamor ipv6 kernel_linux      systemd udev userland_GNU wayland xorg
USE: abi_x86_64 amd64     elibc_glibc glamor ipv6 kernel_linux suid         udev userland_GNU         xorg

but it does not make any difference. I tried with USE="-dmx suid -wayland" and it compiled.

Code:
gentoo costel # emerge -pvO xorg-server

These are the packages that would be merged, in order:

[ebuild   R   #] x11-base/xorg-server-1.19.0:0/1.19.0::gentoo  USE="dmx glamor ipv6 systemd udev wayland xorg -doc -kdrive -libressl -minimal (-selinux) -static-libs -suid -tslib -unwind -xephyr -xnest -xvfb" 0 KiB

Total: 1 package (1 reinstall), Size of downloads: 0 KiB


We are on same version, except I am using kernel module and not xf86-video-intel.
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
saboya
Guru
Guru


Joined: 28 Nov 2006
Posts: 552
Location: Brazil

PostPosted: Mon Jan 02, 2017 5:42 pm    Post subject: Reply with quote

Seems like a known issue: https://bugs.gentoo.org/show_bug.cgi?id=557486

I have no idea why it fails for me but not for you. I'm not on hardened btw.
Back to top
View user's profile Send private message
costel78
Guru
Guru


Joined: 20 Apr 2007
Posts: 402

PostPosted: Mon Jan 02, 2017 9:09 pm    Post subject: Reply with quote

A old, nasty bug. I played with lto since gcc-4.6 and never hit it.
Maybe you should fix it via package.env or an ebuild in your local overlay, as it not seems to be fixed soon. :)
Once again, Link Time Optimization show an error which otherwise remained undiscovered.

What about system stability ? Impressions, other than long time involved to fix/circumvent all lto errors ?
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6747

PostPosted: Mon Jan 02, 2017 9:43 pm    Post subject: Reply with quote

trippels wrote:
the best solution would be if a new gcc ebuild would set up the symlink automatically, overwriting any existing ones.

This is not good, either, because if you reemerge some older gcc and then later it gets cleaned, you have a broken symlink.
The only clean solution is to include setting the symlink in gcc-config. I have currently written a gcc-config wrapper which calls the original gcc-config and then sets the symlink (depending on the arguments), but this is of course just a temporary hack; the "real" solution must be a patch/extension of gcc-config...
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Goto page Previous  1, 2, 3, 4, 5, 6, 7  Next
Page 3 of 7

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum