LTO: We are almost there

costel78 · Guru Joined: 20 Apr 2007 Posts: 402

@mv Until gcc-config will behave proper I put in my /etc/portage/bashrc a BIG warning everytime sys-devel/gcc is emergeged, uninstalled or so. Also

NTU · Apprentice Joined: 17 Jul 2015 Posts: 187

System wide instability with Gold? I've been using it for a year or so now with no issues except compiling problems. You're getting runtime problems with Gold? What kinds???

As for LTO kernel development, I'm in the middle of moving right now, there's a lot of kernel development I've been meaning to catch up on, backporting all the latest and greatest security features of 4.8 and 4.9 - vmapped stacks, hardened usercopy, etc etc. also splitting up the Grsecurity patch into separate files for easier maintainability and dissecting the enormous patch blob, might even try and join the KSPP team, kernel self protection project (if they aren't too crazy)

People have been asking about a secure kernel that you don't need to constantly update or switch releases, so with a barrage of backport patches, people can have all the same security features of 4.9 to older kernels, they'll be hosted here (empty tree right now, haven't worked on it yet)

https://github.com/NTULINUX/kernel-patches/tree/master/backports

There's a lot to do, I've just been busy so I haven't had the time. LTO (fun project) and security patches (real project) are both a priority for me right now, might be a few more weeks until I get around to it. Don't worry!

Cheers guys, and thanks for testing out that script. Please star the project if you can, it will feature a chroot/linux filesystem builder as well, also written entirely in bash, practically LFS in a script, and btw, all the LFS docs in regards to building a toolchain couldn't be further from the truth.

1.) GCC does not support newlib, I was wondering why it wasn't building, GCC bug report I filed confirmed this.

2.) GMP, MPC, and MPFR are not extracted in the GCC builds in the LFS docs, so you get decimal float errors (IIRC)

3.) LFS does not install GLIBC headers before GCC, so GCC does not build

4.) Configure lines are all goofy

5.) Unncessesary modifications to source trees

Tons and tons of issues I can't even remember them all with LFS, but it's a complete mess so I'm basically re-doing LFS entirely and converting it all in a script because those guys can't build a Linux system to save their life, the order is all goofed up on the build process and countless steps are missing. The crosstool-ng tree is in much better shape.

Cheers mates!

Jaglover · Posted: Wed Jan 04, 2017 5:11 pm Post subject:

FWIW I just noticed gcc-6.3.0 with gold linker produces an error in kernel compiling. Just happened to look at the screen when I saw it, compiling didn't stop. Not good.
_________________
My Gentoo installation notes.
Please learn how to denote units correctly!

saboya · Guru Joined: 28 Nov 2006 Posts: 552 Location: Brazil

NTU · Apprentice Joined: 17 Jul 2015 Posts: 187

Jaglover, if you compile with multiple jobs, the compilation won't stop as soon as there's an error until the other jobs are complete. OT: XJ220 FTW!

Child_of_Sun_24 · Guru Joined: 28 Jul 2004 Posts: 578

[quote="saboya"]

saboya · Guru Joined: 28 Nov 2006 Posts: 552 Location: Brazil

mv · Watchman Joined: 20 Apr 2005 Posts: 6747

After the hint of costel78, I made the symlink and re-emerged world on amd64 and x86

There is a corresponding new version of the flto failure list

Result:

Several packages succeeded which failed previously, most remarkably most qt packages (though not all).

On the other hand, the link caused additional failures: Packages failing with

Child_of_Sun_24 · Guru Joined: 28 Jul 2004 Posts: 578

costel78 · Guru Joined: 20 Apr 2007 Posts: 402

@NTU I will give another chance to Google Linker. I enable it on chromium compilation (via -fuse-ld=gold) but I haven't seen any improvements. Of course it could be due to lto-partition=1. When I'll have time will try with libreoffice. If it link/compile faster I'll try a world recompile with gold.
Of course, any work on kernel security or lto if always appreciated.
One more question: what exactly happened between gcc 5 and 6 regarding Gentoo old hardened patches ? Were they accepted upstream ? Where did they disappear ?

@mv First, great statistic (and effort).
Are you using a hardened profile ? That's the only profile where I got numerous runtime errors with lto. I should specify earlier that I am using default/linux/amd64/13.0/no-multilib profile. gnome-shell is the only package that exhibit runtime errors.
I was looking at yours flto failure list. As far as I understand both -flto and -fwhole-program don't work very well together and that combination is discouraged.

The reason I asked then and now about impressions and stability is no matter what benefits any compilation compilation flag may bring, stability is sacred, so please notify us there of such cases.
_________________
Sorry for my English. I'm still learning this language.

mv · Watchman Joined: 20 Apr 2005 Posts: 6747

NTU · Apprentice Joined: 17 Jul 2015 Posts: 187

Yeah I'd definitely give it another go, Gold is stable as all hell for me, my only issues with compiler flags has been forcing fPIE in CFLAGS and CXXFLAGS and -pie in LDFLAGS, I went through all the packages though and only about 20 or 30 of them didn't build with fPIE+pie out of about 1100 installed. Setting fstack-protector-all had no issues what so ever. I have noticed though that the certain packages that didnt build with fPIE and -pie explicility specified in make.conf actually were position independent executables in Debian, I have to figure out how they did that and were able to build with pie when I couldn't.

Like double cherry -fPIE yeah there she was. (lul)

Sent from my Droid 4 XT894. iPhones suck.

costel78 · Guru Joined: 20 Apr 2007 Posts: 402

I did a full recompile of my system with gold. No stability problems, so far.
Aside fpc and lazarus which does not accept gold, only gettext and kernel failed with gold.
There were a few others problems related to the new gperf-3.1, well systemd error scare me a bit until the real cause was elucidated.

What I observed doing comparatives genlop -t, or better said, what I did not observed, is the speed gain of ld.gold. Did bfd catch up in the mean time ?
_________________
Sorry for my English. I'm still learning this language.

saboya · Guru Joined: 28 Nov 2006 Posts: 552 Location: Brazil

costel78 · Guru Joined: 20 Apr 2007 Posts: 402

If that's the case it mean there are no real advantages using ld.gold with lto.
_________________
Sorry for my English. I'm still learning this language.

NTU · Apprentice Joined: 17 Jul 2015 Posts: 187

trubicoid · Tux's lil' helper Joined: 04 Aug 2009 Posts: 79

CaptainBlood · Advocate Joined: 24 Jan 2010 Posts: 3586

ls -la /bin/gzip
-rwxr-xr-x 1 root root 85752 10 janv. 18:56 /bin/gzip

gcc-5.4 + ld.gold / fat-lto, openmp & graphite!!!

for

trubicoid · Tux's lil' helper Joined: 04 Aug 2009 Posts: 79

to see the difference, you need to try linkers and lto by yourself

I have gcc 4.9.4 and -march=native, which makes the reproduction of exactly the same numbers difficult

CaptainBlood · Advocate Joined: 24 Jan 2010 Posts: 3586

IvanMajhen · Guru Joined: 10 Jun 2006 Posts: 392 Location: Croatia

costel78 · Guru Joined: 20 Apr 2007 Posts: 402

Ok, ok, I surrender. There is, at least, the benefit of size when using gold, with or without lto. Nice catch.
Regarding binaries sizes I was impressed by musl. Too bad it does not support all usual desktop packages and it is a bit slow than glibc.
I do not know... first was icc, and, in time gcc surpass it (except when using their math libs), then llvm which now is on pair with gcc and slower during compile time (sic). Maybe is time to not believe in every wheel reinvent.

For compile chromium, or anything else, with pgo there is one method: https://forums.gentoo.org/viewtopic-t-938468.html.
_________________
Sorry for my English. I'm still learning this language.