Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
LTO: We are almost there
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3, 4, 5, 6  Next  
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
costel78
Guru
Guru


Joined: 20 Apr 2007
Posts: 308

PostPosted: Sun Oct 23, 2016 7:47 am    Post subject: Reply with quote

Today, with gentoo-sources-4.8.4 and lto patch kernel completely freeze, no crash dump, no error message, it not react on SysReq key, nothing.
Code:
gentoo costel # lsmod
Module                  Size  Used by
vboxpci                13934  0
vboxnetadp             18310  0
vboxnetflt             16324  0
vboxdrv               356141  3 vboxnetadp,vboxnetflt,vboxpci


And I enabled zram as built-in, was compiled as module before lto-kernel.
Playing with it was fun, but LTO kernel it is not safe enough yet.
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
NTU
Tux's lil' helper
Tux's lil' helper


Joined: 17 Jul 2015
Posts: 126

PostPosted: Sun Oct 23, 2016 8:20 pm    Post subject: Reply with quote

Sounds like a fun problem for me to fix! I'll do some more testing, don't worry. I've got a serial port and everything I need to get to the bottom of it, kernel development is so much fun, definitely my favorite area. I'll be helping out the KSPP project as well, security and performance is what I'm shooting for. If people are interested in LTO support for the Linux kernel, then why not go for it. I had no idea that this thread would be as popular as it is, let alone people being crazy enough to try it.

https://gcc.gnu.org/gcc-6/changes.html ->
Quote:
GCC 7 will support incremental link-time optimization with gcc -r.


Sounds like I'll be testing some builds with GCC trunk branch! costel78, I hear you, everything we've discussed in this page should not be deployed on a production system, expect breakage and instability, development and testing purposes only. You're a really awesome guy, it's a pleasure working with you, and I hope you're having as much fun with this as I am! Definitely one of my most fun side projects, lol! I think this is the one of the very few places where you and I can do this kind of stuff and not have everyone call us maniacs or insult us for going off the deep end.

Big shout out to the Gentoo developers for putting in 9999 ebuilds for binutils btw! I'll iron out bugs along the way but please remember that this is a volunteer project and may take some time depending on my schedule. If you'd like to help out, feel free, but please note that none of this LTO work may ever see a day of light in mainline, at least not for awhile.

All the best!

Your crazy kernel developer,

NTU
Back to top
View user's profile Send private message
costel78
Guru
Guru


Joined: 20 Apr 2007
Posts: 308

PostPosted: Mon Oct 24, 2016 2:46 pm    Post subject: Reply with quote

It would be a fun problem to me to fix, too, only if I have had your experience :lol: . Count on me to do the testing, if necessary.
What can I say more ? Likewise, a real pleasure to talk and learn from you. Right now I am not sure what I love more at Gentoo: opportunity to learn, speed or the community.

Thank you for your work! It is really appreciate.

All the best!
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
NTU
Tux's lil' helper
Tux's lil' helper


Joined: 17 Jul 2015
Posts: 126

PostPosted: Tue Oct 25, 2016 2:52 am    Post subject: Reply with quote

Well this isn't good, I never ran into problems with the kernel starting as early as jump_label.c, I can't even get it to boot on my system.

Code:
Unexpected op at jump_label_test+0x53/0x187 [ffffffff81cb896d] (0f 1f 44 00 00)
PANIC: early exception 0x06 IP 10:ffffffff8101f2d2 error 0 cr2 0xffff88043e1ff00


I just compile tested it when I posted that patch.. Heh, much more difficult than I thought. It was able to initialize ACPI and APIC, PM timer and SMP at least, wow!

addr2line says it's coming from line 584:

Code:
WARN_ON(static_key_enabled(&sk_false.key) != false);


which is only executed if CONFIG_STATIC_KEYS_SELFTEST is enabled (which it is.) LTO is definitely incredibly dangerous. Disabled that, this is the result:

https://www.youtube.com/watch?v=9_Qj-KY62yU

Fatal enough to make me want to stop messing with this rofl.
Back to top
View user's profile Send private message
costel78
Guru
Guru


Joined: 20 Apr 2007
Posts: 308

PostPosted: Tue Oct 25, 2016 1:41 pm    Post subject: Reply with quote

Plenty of errors, indeed. At least you manage to see them.
I can think only at two future hopes:
1. Last time when saw so many pointer null references was when migrated to gcc-6.2.0 and -fno-delete-null-pointer-checks became mandatory. But as I recall you are on gcc-5.4.0 and the kernel code can not be so buggy to require -fno-delete-null-pointer-checks;
2. Maybe, link posted by charles17 https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/?id=84d69848c97faab0c25aa2667b273404d2e2a64a will bring some improvements with kernel-4.9.
Is there any info why original author, Andi Kleen, stopped maintain the patch ? Was he discouraged by Linus Torvalds patch rejection or there was something else ?

Anyway, bright side, LTO and graphite are mature enough to be deployed system-wide. I haven't any glitch or error with them (except annoyance with those in package.env). I believe that, alone, is a step forward.
Kernel, I believe, we have to wait and hope. If "malamutes" projects as libreoffice, firefox, thunderbird, mesa, even no so big, but older code as x11 xorg etc managed to be lto ready, is just a matter of time for kernel.

As always, congratulations for you efforts! Cheer up! :)
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
altblitz
n00b
n00b


Joined: 25 Oct 2016
Posts: 4

PostPosted: Tue Oct 25, 2016 6:15 pm    Post subject: Re: LTO: We are almost there Reply with quote

costel78 wrote:
Using gcc-6.2.0, glibc-2.24 and binutils-2.27 is quite easy to compile firefox, thunderbird, webkit-gtk and libreoffice with Link Time Optimization.

Firefox and thunderbird require a small patch from firefox bugzilla.

The same firefox codebase allows to produce mobile fennec LTOptimized build too.
There is room for the further improvement in LTO build, but for now Mozilla chooses to go with restrained configuration, perphaps due to the only bug in the last step - libxul.so linkage:
Code:
/usr/include/bits/fcntl2.h: In function 'open.constprop':
/usr/include/bits/fcntl2.h:43:0: error: invalid use of '__builtin_va_arg_pack_len ()'
   if (__va_arg_pack_len () > 1)

/usr/include/bits/fcntl2.h:44:0: error: call to '__open_too_many_args' declared with attribute error: open can be called either with 2 or 3 argume
nts, not more
     __open_too_many_args ();
lto-wrapper: fatal error: /usr/bin/g++ returned 1 exit status
compilation terminated.
../../build/unix/gold/ld: fatal error: lto-wrapper failed
collect2: error: ld returned 1 exit status

This bug is present since long ago but never shows unless some optimization flags were given in mozconfig and triggers only in LTO build.

Regarding firefox LTO build, would you mind to post about:buildconfig content from the built browser?
Wonder if your chosen flags keep your build safe from this bug.
Back to top
View user's profile Send private message
costel78
Guru
Guru


Joined: 20 Apr 2007
Posts: 308

PostPosted: Tue Oct 25, 2016 6:20 pm    Post subject: Reply with quote

about:buildconfig
Build platform
target
x86_64-pc-linux-gnu
Build tools
Compiler Version Compiler flags
/usr/bin/x86_64-pc-linux-gnu-gcc -std=gnu99 6.2.0 -Wall -Wempty-body -Wignored-qualifiers -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wno-error=maybe-uninitialized -Wno-error=deprecated-declarations -Wno-error=array-bounds -pipe -march=native -mtune=btver2 -w -flto=4 -fuse-linker-plugin -fno-fat-lto-objects -fgraphite-identity -floop-interchange -floop-strip-mine -floop-block -fno-strict-aliasing -fno-math-errno -pthread -pipe
/usr/bin/x86_64-pc-linux-gnu-g++ -std=gnu++11 6.2.0 -Wall -Wc++11-compat -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wwrite-strings -Wc++14-compat -Wno-invalid-offsetof -Wno-error=maybe-uninitialized -Wno-error=deprecated-declarations -Wno-error=array-bounds -pipe -march=native -mtune=btver2 -w -flto=4 -fuse-linker-plugin -fno-fat-lto-objects -fgraphite-identity -floop-interchange -floop-strip-mine -floop-block -fno-delete-null-pointer-checks -flifetime-dse=1 -fno-exceptions -fno-strict-aliasing -fno-rtti -fno-exceptions -fno-math-errno -pthread -pipe -O2 -fomit-frame-pointer
Configure options

--host=x86_64-pc-linux-gnu --target=x86_64-pc-linux-gnu --enable-application=browser PKG_CONFIG=x86_64-pc-linux-gnu-pkg-config --enable-default-toolkit=cairo-gtk3 --disable-tests CC=x86_64-pc-linux-gnu-gcc CXX=x86_64-pc-linux-gnu-g++ HOST_CC=x86_64-pc-linux-gnu-gcc HOST_CXX=x86_64-pc-linux-gnu-g++ MOZ_JEMALLOC4=1 --enable-replace-malloc XARGS=/usr/bin/xargs --enable-system-hunspell --disable-crashreporter --enable-dbus --disable-debug --disable-debug-symbols --enable-extensions=default --disable-gconf --enable-gio --disable-gnomeui --disable-gold --disable-install-strip --enable-ion --disable-necko-wifi --enable-official-branding --enable-optimize=-O2 --enable-pulseaudio --enable-skia --enable-startup-notification --disable-strip --disable-system-cairo --enable-system-ffi --enable-system-sqlite --disable-updater --libdir=/usr/lib64 --prefix=/usr --with-default-mozilla-five-home=/usr/lib64/firefox --with-google-api-keyfile=/var/tmp/portage/www-client/firefox-49.0/work/firefox-49.0/google-api-key --with-intl-api --with-nspr-prefix=/usr --with-nss-prefix=/usr --with-system-bz2 --with-system-graphite2 --with-system-harfbuzz --with-system-icu --with-system-jpeg --with-system-libvpx --with-system-nspr --with-system-nss --with-system-png --with-system-zlib --x-includes=/usr/include --x-libraries=/usr/lib64

Code:
mini costel # emerge -pvO firefox

These are the packages that would be merged, in order:

[ebuild   R    ] www-client/firefox-49.0::gentoo  USE="custom-cflags custom-optimization dbus gmp-autoupdate hwaccel jemalloc jit pulseaudio skia startup-notification system-harfbuzz system-icu system-jpeg system-libvpx system-sqlite -bindist -debug -gtk2 -hardened (-neon) -pgo (-selinux) (-system-cairo) -system-libevent {-test} -wifi" L10N="ro -ach -af -an -ar -as -ast -az -be -bg -bn-BD -bn-IN -br -bs -ca -cak -cs -cy -da -de -dsb -el -en-GB -en-ZA -eo -es-AR -es-CL -es-ES -es-MX -et -eu -fa -ff -fi -fr -fy -ga -gd -gl -gn -gu -he -hi -hr -hsb -hu -hy -id -is -it -ja -kk -km -kn -ko -lij -lt -lv -mai -mk -ml -mr -ms -nb -nl -nn -or -pa -pl -pt-BR -pt-PT -rm -ru -si -sk -sl -son -sq -sr -sv -ta -te -th -tr -uk -uz -vi -xh -zh-CN -zh-TW" 0 KiB

Total: 1 package (1 reinstall), Size of downloads: 0 KiB


This is from my NAS/multimedia PC

and

about:buildconfig
Build platform
target
x86_64-pc-linux-gnu
Build tools
Compiler Version Compiler flags
/usr/bin/x86_64-pc-linux-gnu-gcc -std=gnu99 6.2.0 -Wall -Wempty-body -Wignored-qualifiers -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wno-error=maybe-uninitialized -Wno-error=deprecated-declarations -Wno-error=array-bounds -pipe -march=native -mtune=skylake -fomit-frame-pointer -w -flto=8 -fuse-linker-plugin -fno-fat-lto-objects -fgraphite-identity -floop-nest-optimize -ftracer -fpredictive-commoning -fgcse-after-reload -fvect-cost-model -ftree-partial-pre -ftree-vectorize -fgcse-las -fno-strict-aliasing -fno-math-errno -pthread -pipe
/usr/bin/x86_64-pc-linux-gnu-g++ -std=gnu++11 6.2.0 -Wall -Wc++11-compat -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wwrite-strings -Wc++14-compat -Wno-invalid-offsetof -Wno-error=maybe-uninitialized -Wno-error=deprecated-declarations -Wno-error=array-bounds -pipe -march=native -mtune=skylake -fomit-frame-pointer -w -flto=8 -fuse-linker-plugin -fno-fat-lto-objects -fgraphite-identity -floop-nest-optimize -ftracer -fpredictive-commoning -fgcse-after-reload -fvect-cost-model -ftree-partial-pre -ftree-vectorize -fgcse-las -fno-delete-null-pointer-checks -flifetime-dse=1 -fno-exceptions -fno-strict-aliasing -fno-rtti -fno-exceptions -fno-math-errno -pthread -pipe -O2 -fomit-frame-pointer
Configure options

--host=x86_64-pc-linux-gnu --target=x86_64-pc-linux-gnu --enable-application=browser PKG_CONFIG=x86_64-pc-linux-gnu-pkg-config --enable-default-toolkit=cairo-gtk3 --disable-tests CC=x86_64-pc-linux-gnu-gcc CXX=x86_64-pc-linux-gnu-g++ HOST_CC=x86_64-pc-linux-gnu-gcc HOST_CXX=x86_64-pc-linux-gnu-g++ MOZ_JEMALLOC4=1 --enable-replace-malloc XARGS=/usr/bin/xargs --enable-system-hunspell --disable-crashreporter --enable-dbus --disable-debug --disable-debug-symbols --enable-extensions=default --disable-gconf --enable-gio --disable-gnomeui --disable-gold --disable-install-strip --enable-ion --disable-necko-wifi --enable-official-branding --enable-optimize=-O2 --enable-pulseaudio --enable-skia --enable-startup-notification --disable-strip --disable-system-cairo --enable-system-ffi --enable-system-sqlite --disable-updater --libdir=/usr/lib64 --prefix=/usr --with-default-mozilla-five-home=/usr/lib64/firefox --with-google-api-keyfile=/var/tmp/portage/www-client/firefox-49.0/work/firefox-49.0/google-api-key --with-intl-api --with-nspr-prefix=/usr --with-nss-prefix=/usr --with-system-bz2 --with-system-graphite2 --with-system-harfbuzz --with-system-icu --with-system-jpeg --with-system-libvpx --with-system-nspr --with-system-nss --with-system-png --with-system-zlib --x-includes=/usr/include --x-libraries=/usr/lib64
Code:
gentoo costel # emerge -pvO firefox

These are the packages that would be merged, in order:

[ebuild   R    ] www-client/firefox-49.0::gentoo  USE="custom-cflags custom-optimization dbus gmp-autoupdate hwaccel jemalloc jit pulseaudio skia startup-notification system-harfbuzz system-icu system-jpeg system-libvpx system-sqlite -bindist -debug -gtk2 -hardened (-neon) -pgo (-selinux) (-system-cairo) -system-libevent {-test} -wifi" L10N="ro -ach -af -an -ar -as -ast -az -be -bg -bn-BD -bn-IN -br -bs -ca -cak -cs -cy -da -de -dsb -el -en-GB -en-ZA -eo -es-AR -es-CL -es-ES -es-MX -et -eu -fa -ff -fi -fr -fy -ga -gd -gl -gn -gu -he -hi -hr -hsb -hu -hy -id -is -it -ja -kk -km -kn -ko -lij -lt -lv -mai -mk -ml -mr -ms -nb -nl -nn -or -pa -pl -pt-BR -pt-PT -rm -ru -si -sk -sl -son -sq -sr -sv -ta -te -th -tr -uk -uz -vi -xh -zh-CN -zh-TW" 0 KiB

Total: 1 package (1 reinstall), Size of downloads: 0 KiB


More aggresive flags from main desktop.
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
altblitz
n00b
n00b


Joined: 25 Oct 2016
Posts: 4

PostPosted: Tue Oct 25, 2016 9:39 pm    Post subject: Reply with quote

costel78 wrote:
about:buildconfig
Build platform
target
x86_64-pc-linux-gnu
Build tools
Compiler Version Compiler flags
/usr/bin/x86_64-pc-linux-gnu-gcc -std=gnu99 6.2.0 -march=native -mtune=btver2 -w -flto=4 -fuse-linker-plugin -fno-fat-lto-objects -fgraphite-identity -floop-interchange -floop-strip-mine -floop-block -O2
Configure options
--disable-gold --enable-optimize=-O2

This is from my NAS/multimedia PC

about:buildconfig
Build platform
target
x86_64-pc-linux-gnu
Build tools
Compiler Version Compiler flags
/usr/bin/x86_64-pc-linux-gnu-g++ -std=gnu++11 6.2.0 -march=native -mtune=skylake -flto=8 -fuse-linker-plugin -fno-fat-lto-objects -fgraphite-identity -floop-nest-optimize -ftracer -fpredictive-commoning -fgcse-after-reload -fvect-cost-model -ftree-partial-pre -ftree-vectorize -fgcse-las -O2
Configure options
--disable-gold
More aggresive flags from main desktop.


Now, it's clear why linkage does not happened.
I put -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector-strong and -march=native -O3 together for LTO build and that leads to aforementioned ld error when linking largest library with gcc 6.2.0.
There are reports that hardening flags restricts firefox build somehow coupling with -mnative...

Sigh, firefox links just well with all these flags - but with clang.

Quote:
about:buildconfig
Build platform
target
x86_64-pc-linux-gnu
Build tools
Compiler Version Compiler flags
/usr/bin/ccache /usr/bin/clang -std=gnu99 4.0.0 -Qunused-arguments -Wall -Wempty-body -Wignored-qualifiers -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wclass-varargs -Wloop-analysis -Wthread-safety -Wno-error=deprecated-declarations -Wno-error=array-bounds -march=native -O3 -pipe -fstack-protector-strong -fdiagnostics-color -Wp,-D_FORTIFY_SOURCE=2 -flto -Wl,-plugin-opt=O3 -fno-strict-aliasing -ffunction-sections -fdata-sections -fno-math-errno -pthread -pipe
/usr/bin/ccache /usr/bin/clang++ -std=gnu++11 4.0.0 -Qunused-arguments -Qunused-arguments -Wall -Wc++11-compat -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wtype-limits -Wunreachable-code -Wwrite-strings -Wc++11-compat-pedantic -Wc++14-compat -Wc++14-compat-pedantic -Wc++1z-compat -Wclass-varargs -Wimplicit-fallthrough -Wloop-analysis -Wthread-safety -Wno-invalid-offsetof -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-unknown-warning-option -Wno-return-type-c-linkage -march=native -O3 -pipe -fstack-protector-strong -fdiagnostics-color -Wp,-D_FORTIFY_SOURCE=2 -flto -Wl,-plugin-opt=O3 -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -O3 -fomit-frame-pointer

Tried LTO flag -fwhole-program/-fwhole-program-vtables with no avail.
That shall improve (on paper at least) efficiency of the LTO build. Chromium uses this flag for LTO build.
Back to top
View user's profile Send private message
NTU
Tux's lil' helper
Tux's lil' helper


Joined: 17 Jul 2015
Posts: 126

PostPosted: Wed Oct 26, 2016 12:10 am    Post subject: Reply with quote

costel78 thanks for the commit! Wouldn't hurt to give it another pass, will try with GCC trunk as well if it's still buggy.

As far as seeing the output, earlyprintk is your friend. ;)
Back to top
View user's profile Send private message
costel78
Guru
Guru


Joined: 20 Apr 2007
Posts: 308

PostPosted: Wed Oct 26, 2016 6:49 am    Post subject: Reply with quote

@altblitz In my experience, lto and hardened did not play well together on gcc-4.8.0 days. I do not know nothing regarding status this days.
Also, globally enabled O3 break things, for example, right now it breaks openssh (you can not disconnect cleanly), some parts of libreoffice, at least on my setup.
Will try with -D_FORTIFY_SOURCE=2 -fstack-protector-strong (but I am not on hardened profile) later today and report back.

NTU. It is set:
Code:
grep EARLY /boot/.config
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_GENERIC_EARLY_IOREMAP=y
CONFIG_EARLY_PRINTK=y
# CONFIG_EARLY_PRINTK_DBGP is not set
CONFIG_EARLY_PRINTK_EFI=y

I presume it is not enough, different config ?

Built kernel on 3 machines, but, on reboot, none of them booted. Only on 4.9-rc1 I was able to see errors. Celebrated to early, rebooted only a day after...
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
costel78
Guru
Guru


Joined: 20 Apr 2007
Posts: 308

PostPosted: Wed Oct 26, 2016 3:31 pm    Post subject: Reply with quote

@altblitz
If firefox fail to build on hardened, -D_FORTIFY_SOURCE=2 -fstack-protector-strong are not the root cause.
Code:
emerge --info firefox
Portage 2.3.2 (python 3.5.2-final-0, default/linux/amd64/13.0/no-multilib, gcc-6.2.0, glibc-2.24, 4.8.4-gentoo-costel x86_64)
=================================================================
                         System Settings
=================================================================
System uname: Linux-4.8.4-gentoo-costel-x86_64-Intel-R-_Core-TM-_i7-6700K_CPU_@_4.00GHz-with-gentoo-2.3
KiB Mem:    31848492 total,  18120688 free
KiB Swap:          0 total,         0 free
Timestamp of repository gentoo: Wed, 26 Oct 2016 12:45:01 +0000
sh bash 4.4-r1
ld GNU ld (Gentoo 2.27 p1.0) 2.27
app-shells/bash:          4.4-r1::gentoo
dev-java/java-config:     2.2.0-r3::gentoo
dev-lang/perl:            5.24.0-r2::gentoo
dev-lang/python:          2.7.12::gentoo, 3.5.2::gentoo
dev-util/cmake:           3.6.2::gentoo
dev-util/pkgconfig:       0.29.1::gentoo
sys-apps/baselayout:      2.3::gentoo
sys-apps/sandbox:         2.10-r2::gentoo
sys-devel/autoconf:       2.13::gentoo, 2.69-r2::gentoo
sys-devel/automake:       1.13.4-r1::gentoo, 1.15-r2::gentoo
sys-devel/binutils:       2.27::newer
sys-devel/gcc:            6.2.0-r1::gentoo
sys-devel/gcc-config:     1.8-r1::gentoo
sys-devel/libtool:        2.4.6-r2::gentoo
sys-devel/make:           4.2.1::gentoo
sys-kernel/linux-headers: 4.7::gentoo (virtual/os-headers)
sys-libs/glibc:           2.24::newer
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.ro.gentoo.org/gentoo-portage/
    priority: -1000
    sync-rsync-vcs-ignore: true
    sync-rsync-extra-opts: --human-readable --delete-before --progress

added
    location: /usr/portage/local/added
    masters: gentoo
    priority: 30

newer
    location: /usr/portage/local/newer
    masters: gentoo
    priority: 30

python
    location: /usr/portage/local/python
    masters: gentoo
    priority: 30

mv
    location: /usr/portage/local/mv
    sync-type: laymansync
    sync-uri: git://anongit.gentoo.org/user/mv.git
    masters: gentoo
    priority: 50

Installed sets: @dev, @gnome-features, @media, @minimal, @office, @parallelcompress, @squashmount, @sunflower, @utilities, @virtualbox
ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe -march=native -mtune=skylake -w -D_FORTIFY_SOURCE=2 -fstack-protector-strong -flto=8 -fuse-linker-plugin -fno-fat-lto-objects -fgraphite-identity -floop-nest-optimize -ftracer -fpredictive-commoning -fgcse-after-reload -fvect-cost-model -ftree-partial-pre -ftree-vectorize -fgcse-las"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/gnupg/qualified.txt /var/lib/hsqldb"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php7.0/ext-active/ /etc/php/cgi-php7.0/ext-active/ /etc/php/cli-php7.0/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -pipe -march=native -mtune=skylake -w -D_FORTIFY_SOURCE=2 -fstack-protector-strong -flto=8 -fuse-linker-plugin -fno-fat-lto-objects -fgraphite-identity -floop-nest-optimize -ftracer -fpredictive-commoning -fgcse-after-reload -fvect-cost-model -ftree-partial-pre -ftree-vectorize -fgcse-las -fno-delete-null-pointer-checks -flifetime-dse=1"
DISTDIR="/mnt/linux/distfiles"
EMERGE_DEFAULT_OPTS="--jobs=3 --load-average=12 --keep-going --with-bdeps=y --complete-graph --quiet-build=n --autounmask-write"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs candy cgroup collision-protect config-protect-if-modified distlocks fail-clean fixlafiles ipc-sandbox merge-sync news nodoc noinfo parallel-fetch parallel-install preserve-libs protect-owned sandbox sfperms strict unknown-features-filter unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://mirrors.evowise.com/gentoo/ http://mirrors.xservers.ro/gentoo/ http://ftp.romnet.org/gentoo/ ftp://ftp.romnet.org/gentoo/ http://distfiles.gentoo.org/"
LANG="ro_RO.UTF-8"
LC_ALL="ro_RO.UTF-8"
LDFLAGS="-Wl,-O1,--sort-common,--hash-style=gnu,--as-needed,-z,now,-z,relro -O2 -pipe -march=native -mtune=skylake -w -D_FORTIFY_SOURCE=2 -fstack-protector-strong -flto=8 -fuse-linker-plugin -fno-fat-lto-objects -fgraphite-identity -floop-nest-optimize -ftracer -fpredictive-commoning -fgcse-after-reload -fvect-cost-model -ftree-partial-pre -ftree-vectorize -fgcse-las -fno-delete-null-pointer-checks -flifetime-dse=1"
MAKEOPTS="-j8 --load-average=16"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_EXTRA_OPTS="--human-readable --delete-before --progress"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="X a52 aac acl acpi aes alsa amd64 avx avx2 berkdb branding bzip2 cairo caps cdda cdr cli colord cracklib crypt cups custom-cflags custom-optimization cxx dbus dri dts dvd dvdr eds egl emboss encode evo exif fam ffmpeg firefox flac fma3 gdbm gif glamor gles gmp gnome gnome-keyring gnome-online-accounts gpm gstreamer gtk gtk3 iconv icu introspection ipv6 jemalloc jpeg lcms libnotify libsecret logrotate lz4 mad mmx mmxext mng modules mp3 mp4 mpeg mysql nautilus ncurses networkmanager nls nptl ogg opengl openmp pam pch pcre pdf png policykit popcnt ppds pulseaudio python qt5 readline sdl seccomp server session spell sse sse2 sse3 sse4_1 sse4_2 ssl ssse3 startup-notification svg systemd tcpd theora threads tiff truetype udev udisks unicode upower usb vaapi vdpau vhosts vorbis wayland wxwidgets x264 xattr xcb xml xv xvid zlib" ABI_X86="64" ALSA_CARDS="hda-intel usb-audio" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 fma3 mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="evdev keyboard mouse" KERNEL="linux" L10N="ro en" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="ro en" NETBEANS_MODULES="apisupport extide php webcommon" NGINX_MODULES_HTTP="access addition auth_basic autoindex browser cache_purge charset dav dav_ext echo empty_gif fancyindex fastcgi flv geo geoip gunzip gzip headers_more image_filter limit_conn limit_req map memc memcached metrics mp4 naxsi perl random_index realip referer rewrite scgi secure_link split_clients ssi sticky stub_status sub upload_progress upstream_ip_hash userid uwsgi xslt" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-0" PYTHON_SINGLE_TARGET="python3_5" PYTHON_TARGETS="python2_7 python3_5" RUBY_TARGETS="ruby23" USERLAND="GNU" VIDEO_CARDS="intel i965" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
USE_PYTHON="2.7 3.5"
Unset:  CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS

=================================================================
                        Package Settings
=================================================================

www-client/firefox-49.0::gentoo was built with the following:
USE="custom-cflags custom-optimization dbus gmp-autoupdate hwaccel jemalloc jit pulseaudio skia startup-notification system-harfbuzz system-icu system-jpeg system-libvpx system-sqlite -bindist -debug -gtk2 -hardened (-neon) -pgo (-selinux) (-system-cairo) -system-libevent -test -wifi" L10N="ro -ach -af -an -ar -as -ast -az -be -bg -bn-BD -bn-IN -br -bs -ca -cak -cs -cy -da -de -dsb -el -en-GB -en-ZA -eo -es-AR -es-CL -es-ES -es-MX -et -eu -fa -ff -fi -fr -fy -ga -gd -gl -gn -gu -he -hi -hr -hsb -hu -hy -id -is -it -ja -kk -km -kn -ko -lij -lt -lv -mai -mk -ml -mr -ms -nb -nl -nn -or -pa -pl -pt-BR -pt-PT -rm -ru -si -sk -sl -son -sq -sr -sv -ta -te -th -tr -uk -uz -vi -xh -zh-CN -zh-TW"
CFLAGS="-pipe -march=native -mtune=skylake -w -D_FORTIFY_SOURCE=2 -fstack-protector-strong -flto=8 -fuse-linker-plugin -fno-fat-lto-objects -fgraphite-identity -floop-nest-optimize -ftracer -fpredictive-commoning -fgcse-after-reload -fvect-cost-model -ftree-partial-pre -ftree-vectorize -fgcse-las"
CXXFLAGS="-pipe -march=native -mtune=skylake -w -D_FORTIFY_SOURCE=2 -fstack-protector-strong -flto=8 -fuse-linker-plugin -fno-fat-lto-objects -fgraphite-identity -floop-nest-optimize -ftracer -fpredictive-commoning -fgcse-after-reload -fvect-cost-model -ftree-partial-pre -ftree-vectorize -fgcse-las -fno-delete-null-pointer-checks -flifetime-dse=1 -fno-delete-null-pointer-checks -fno-lifetime-dse -fno-schedule-insns2"
LDFLAGS="-Wl,-O1,--sort-common,--hash-style=gnu,--as-needed,-z,now,-z,relro -pipe -march=native -mtune=skylake -w -D_FORTIFY_SOURCE=2 -fstack-protector-strong -flto=8 -fuse-linker-plugin -fno-fat-lto-objects -fgraphite-identity -floop-nest-optimize -ftracer -fpredictive-commoning -fgcse-after-reload -fvect-cost-model -ftree-partial-pre -ftree-vectorize -fgcse-las -fno-delete-null-pointer-checks -flifetime-dse=1 -Wl,-rpath=/usr/lib64/firefox,--enable-new-dtags"


I briefly test youtube and I am posting from firefox right now. Maybe the culprit are default "hidden" (sorry, I don't know how to call them) flags of hardened ?
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
altblitz
n00b
n00b


Joined: 25 Oct 2016
Posts: 4

PostPosted: Thu Oct 27, 2016 8:21 pm    Post subject: Reply with quote

costel78 wrote:
@altblitz
If firefox fail to build on hardened, -D_FORTIFY_SOURCE=2 -fstack-protector-strong are not the root cause.
Code:

CXXFLAGS="-O2 -pipe -march=native -mtune=skylake -w -D_FORTIFY_SOURCE=2 -fstack-protector-strong -flto=8 -fuse-linker-plugin -fno-fat-lto-objects -fgraphite-identity -floop-nest-optimize -ftracer -fpredictive-commoning -fgcse-after-reload -fvect-cost-model -ftree-partial-pre -ftree-vectorize -fgcse-las -fno-delete-null-pointer-checks -flifetime-dse=1"


I briefly test youtube and I am posting from firefox right now. Maybe the culprit are default "hidden" (sorry, I don't know how to call them) flags of hardened ?

Firefox calls function open() from different cpp files as function with four arguments,
Code:
static bool
open(JSContext* cx, JS::Handle<JSObject*> obj, mozilla::dom::cache::CacheStorage* self, const JSJitMethodCallArgs& args)

while function open() in /usr/include/bits/fcntl2.h declared with attribute __fortify_function and with three arguments.
Code:
open (const char *__path, int __oflag, ...)
{
  if (__va_arg_pack_len () > 1)
    __open_too_many_args ();

Enhance application security with FORTIFY_SOURCE
Quote:
With _FORTIFY_SOURCE set to 2 some more checking is added, but some conforming programs might fail. Some of the checks can be performed at compile time, and result in compiler warnings; other checks take place at run time, and result in a run-time error if the check fails.

That is the case of error.
Optimization flag -O3 and -flto do trigger FORTIFY_SOURCE check error at compile time, while -O2 does not.
Clang however, pass the check with -O3 flag.

If the flag FORTIFY_SOURCE not used, then firefox with -O3 builds and works as normally.

costel78 wrote:

Also, globally enabled O3 break things, for example, right now it breaks openssh (you can not disconnect cleanly), some parts of libreoffice, at least on my setup.

Mozilla builds firefox-nightly with O3 flag.

Several builds were build with -Ox/-D_FORTIFY-SOURCE variations and tested.
One of the flags or certain combination of them
Code:
-finline-functions, -funswitch-loops, -fpredictive-commoning, -fgcse-after-reload, -ftree-loop-vectorize, -ftree-loop-distribute-patterns, -fsplit-paths -ftree-slp-vectorize, -fvect-cost-model, -ftree-partial-pre and -fipa-cp-clone

specific only to -O3 trigger FORTIFY_SOURCE error. Should investigate, which one.
Back to top
View user's profile Send private message
roarinelk
Guru
Guru


Joined: 04 Mar 2004
Posts: 497

PostPosted: Mon Nov 07, 2016 2:52 pm    Post subject: Re: LTO: We are almost there Reply with quote

costel78 wrote:


I have only nine packages with failed with lto in my package.env:
Code:
dev-lang/spidermonkey:0/mozjs185 no-lto-graphite
dev-libs/libgcrypt no-lto-graphite
dev-python/notify-python no-lto-graphite
gnome-base/gnome-shell no-lto-graphite
media-libs/alsa-lib no-lto-graphite
net-misc/dhcp no-lto-graphite
sys-apps/man-db no-lto-no-graphite
sys-apps/pciutils no-lto-graphite
x11-drivers/xf86-video-intel no-lto-graphite



I have a few more to add. Some don't build, some don't work right or cause build failures in other things:
Code:

app-crypt/efitools
app-text/texlive-core
dev-libs/glib  (if using static-libs)
dev-qt/qtgui
dev-qt/qtwayland
net-fs/nfs-utils
net-libs/libasyncns
net-libs/libnfsidmap
net-libs/libtirpc
net-nds/rpcbind
sys-boot/gnu-efi
sys-libs/zlib  (if using static-libs)
sys-power/upower
Back to top
View user's profile Send private message
costel78
Guru
Guru


Joined: 20 Apr 2007
Posts: 308

PostPosted: Wed Nov 09, 2016 11:31 am    Post subject: Reply with quote

Let's hope things will be better with gcc-7.
Anyway, at this stage, it is more likely to be package fault, rather than gcc.
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
CaptainBlood
l33t
l33t


Joined: 24 Jan 2010
Posts: 878

PostPosted: Wed Nov 09, 2016 2:01 pm    Post subject: Reply with quote

I'm sure I have more to add that doesn't support lto but since I stuck to 4.9.4, I'm not sure someone is interested.

Just ask if you please for current list of 74 packages.

Thks 4 ur attention.
Back to top
View user's profile Send private message
NTU
Tux's lil' helper
Tux's lil' helper


Joined: 17 Jul 2015
Posts: 126

PostPosted: Sun Dec 04, 2016 5:08 pm    Post subject: Reply with quote

Well to rule out GCC being too old I've spent the past short while working on a bleeding edge native toolchain builder:

https://github.com/NTULINUX/toolchain_builder

I'll try out an LTO kernel build using that when I get around to it, feel free to muck around with it in the mean time!
Back to top
View user's profile Send private message
costel78
Guru
Guru


Joined: 20 Apr 2007
Posts: 308

PostPosted: Wed Dec 07, 2016 10:19 pm    Post subject: Reply with quote

Wow, you did a lot of work!
But I have to admit a failed to get the big picture. :? You want to try to compile the kernel (with lto) using git-versions of toolchains. How is this different than a -9999 ebuild ? Are you expecting some performance gain doing a stage two ? Also, are you planing to introduce something like gcc-config to be able to use that separate toolchain like the gentoo one, or switching them ?
Yes, I know, a lot of questions. :) Can you please give more details ?
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
NTU
Tux's lil' helper
Tux's lil' helper


Joined: 17 Jul 2015
Posts: 126

PostPosted: Thu Dec 08, 2016 8:47 am    Post subject: Reply with quote

The idea is to create a development toolchain that doesn't touch the host system, and crosstool-ng has yet to support a native build, so I wrote my own builder in bash. Installing a -9999 ebuild requires root permissions and changes to root fs, mine doesn't :P As for gcc-config, probably not as specifying the PATH variable works well (tested a kernel build with it, all was fine.)

I'll write a chroot builder as well that uses that toolchain to build the necessities for a bare minimum chroot, then have it build a complete system from there, inside a home directory for example. It's really just a playground is all, plus you can easily change branches of which one you want to be on and some extra fine tweaking. The toolchain eclasses do this for you, my builder leaves it up to the user. I already have a few hundred lines of the chroot builder written, that way people can test out developer features of compilers or whatever it may be without messing with their main system, it's also all Linux distribution independent :wink:
Back to top
View user's profile Send private message
altblitz
n00b
n00b


Joined: 25 Oct 2016
Posts: 4

PostPosted: Sun Dec 25, 2016 10:20 pm    Post subject: Re: LTO: We are almost there Reply with quote

Blitz did it. Yet again.
Firefox 50.1 with -O3 LTO PGO march=native
and Chromium LTO PGO.

From the benchmarks, Firefox is only ~20% lower than Chromium.

All in all - Blitz have the most speedy browser on the Earth now.
Privacy and security patches applied.
Back to top
View user's profile Send private message
NTU
Tux's lil' helper
Tux's lil' helper


Joined: 17 Jul 2015
Posts: 126

PostPosted: Wed Dec 28, 2016 9:17 am    Post subject: Reply with quote

Might be even quicker with a toolchain using all the latest git sources :P
Back to top
View user's profile Send private message
costel78
Guru
Guru


Joined: 20 Apr 2007
Posts: 308

PostPosted: Thu Dec 29, 2016 7:17 pm    Post subject: Reply with quote

@altblitz I was unable to compile chromium with lto. Which version ? Can you share the procedure, the patches ?
@NTU Found, on the hard way, that newest isn't always the best. Tried to recompile everything with git sources and I got enough segmentation faults to give up.
Well, of course, I should pick a "lucky day" to grab all those git sources, which it seems I failed miserably. :D
Any progress on lto kernel ?

Later: At a closer look chromium error is similar with firefox one a while back: lto partitioning on asm code, both encountered on sandbox code. But I am still interested on those security and privacy patches. Thank you!
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
NTU
Tux's lil' helper
Tux's lil' helper


Joined: 17 Jul 2015
Posts: 126

PostPosted: Fri Dec 30, 2016 4:38 am    Post subject: Reply with quote

You had problems with the toolchain script I posted? Also Gentoo doesn't have a 9999 ebuild for GCC, another reason why I wrote that :P
Back to top
View user's profile Send private message
costel78
Guru
Guru


Joined: 20 Apr 2007
Posts: 308

PostPosted: Fri Dec 30, 2016 8:59 am    Post subject: Reply with quote

No, no with the script itself, but with stability of the system, of the generated binaries.
I'll give another try in the next days.
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
NTU
Tux's lil' helper
Tux's lil' helper


Joined: 17 Jul 2015
Posts: 126

PostPosted: Fri Dec 30, 2016 9:31 am    Post subject: Reply with quote

Run git pull on the builder before giving it another go, just updated a few things.
Back to top
View user's profile Send private message
costel78
Guru
Guru


Joined: 20 Apr 2007
Posts: 308

PostPosted: Fri Dec 30, 2016 1:43 pm    Post subject: Reply with quote

I have free time until 3rd of January.
One small sugestion regarding build script: why not make profiledbootstrap -j${CORES} or even make autoprofiledback, perf and autofdo are needed in the last case, see: https://gcc.gnu.org/install/build.html - Building with profile feedback
It is very unlikely that someone using your script are still using x86, or if architecture is x86 go with the simple make to satisfy both cases.
I always build gcc with GCC_MAKE_TARGET="profiledbootstrap", via package.env and autofdo would be really something :D .

Later:
nice -n 19 when call make it would be... well, nice.
I can confirm that make profiledbootstrap it's working. AutoFDO is next.
_________________
Sorry for my English. I'm still learning this language.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Goto page Previous  1, 2, 3, 4, 5, 6  Next
Page 2 of 6

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum