Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Portage config for building with -O3, Graphite, and LTO
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
InBetweenNames
n00b
n00b


Joined: 11 Sep 2017
Posts: 7
Location: Windsor, Ontario

PostPosted: Mon Sep 11, 2017 6:28 pm    Post subject: Portage config for building with -O3, Graphite, and LTO Reply with quote

Hi everyone,

I'm new to the Gentoo Forums, but I have received some positive feedback about a project I've started with Gentoo and I figured I would share it with the Gentoo community as a whole.

I've been silently working on a Portage configuration for building with O3, Graphite, and LTO by default since about the start of the year. My configuration and results are in this GitHub repo: https://github.com/InBetweenNames/gentooLTO

My goals of this project are as follows:

* Identify packages which do not play nice with LTO and fix them if possible
* LTO, O3 and Graphite optimizations will help reveal the use of Undefined Behaviour in C and C++ programs (see https://en.wikipedia.org/wiki/Undefined_behavior )
* Perhaps, in the best case, even get a performance improvement on the whole system. Benchmarks are sorely needed for this.
* Identify cases where -O3 and Graphite optimizations actually have a performance regression versus O2. These would make for good GCC bug reports. Especially with Graphite!
* In all cases, let the compiler decide which transformations to apply. Never override the compiler's cost function for applying a certain transformation (no -funroll-all-loops -- ever).

This is definitely advised for expert users only. Breakages do happen and you will probably need a strong programming and administration background in order to resolve them.

I have a lot more information in the README about this which is accessible via the GitHub link. If you are interested, contributions are very welcome!
Back to top
View user's profile Send private message
duby2291
Guru
Guru


Joined: 17 Oct 2004
Posts: 583

PostPosted: Tue Sep 12, 2017 2:28 pm    Post subject: Reply with quote

I can see a lot of the configuration you've chosen is specific to your personal preferences. But all in all it does give a great reference to start figuring it out. Thanks for your effort!
Back to top
View user's profile Send private message
Maitreya
Guru
Guru


Joined: 11 Jan 2006
Posts: 441

PostPosted: Tue Sep 12, 2017 6:27 pm    Post subject: Reply with quote

I have to agree with duby2291 here.
For example:

gentooLTO/env/addncurses.conf

in the repo is a fix (which I use too) but is not related to gentooLTO, so if you want people to be able to easily fork your project make it only about gentooLTO and put the fix for ncurses in a seperate/personal overlay/repo.
Back to top
View user's profile Send private message
InBetweenNames
n00b
n00b


Joined: 11 Sep 2017
Posts: 7
Location: Windsor, Ontario

PostPosted: Wed Sep 13, 2017 3:00 pm    Post subject: Reply with quote

Thanks for the feedback! Would you consider contributing to the issue about this in the gentooLTO github? Someone suggested that things be abstracted out into an actual Gentoo profile such that people could more easily try out these settings for themselves. It sounded like a great idea to me, but I'm not sure about what technical details are involved.

I have heard from a few people that a more generic repository would be ideal. It's unlikely that I will change this main repository to be more generic one as this is actually synced from my own /etc/portage, but I fully support the idea of a generic configuration springing from this one, and I'd be willing to help with that. If the profile idea works, I'd even switch to that profile myself and contribute my relevant configuration updates there.
Back to top
View user's profile Send private message
Maitreya
Guru
Guru


Joined: 11 Jan 2006
Posts: 441

PostPosted: Wed Sep 13, 2017 5:33 pm    Post subject: Reply with quote

Yes, I would. And I think many would.
You do the kickoff and you'll see people are suddenly interested.
This project is not very old and it already has quite some followers and stars.
I don't think that is because of reputation but more "finally someone is doing it"-mentality

But for that idea to succeed you need a clear "goal" and/or "target".
So if you would remove that fix for ncurses, it becomes more generic and useable for a broader audience without conflicting other overlays/builds.

So I would really recommend either starting a new repo for your "personal" stuff or the "new idea" stuff. Actually, strongly recommend because I do not see it succeed if it's a mingle of functionality and random system patches.

But do be careful, because with LTO + O3 + graphite you are stacking optimizations and other new stuff at the same time. Which could put people off.
If you look at the forums, you can see that there is a lot of stuff happening around LTO + graphite for years. So I bet when you remove O3 (or put it in a different overlay) those people in those topics will join you really quickly.
(now everybody has their own collection of profile.env and overrides)
Back to top
View user's profile Send private message
InBetweenNames
n00b
n00b


Joined: 11 Sep 2017
Posts: 7
Location: Windsor, Ontario

PostPosted: Thu Sep 14, 2017 2:49 pm    Post subject: Reply with quote

What would the Gentoo community think of an "lto-overlay" with just the LTO-relevant parts of this configuration, packaged as profile? Is it possible to put package.env overrides in profiles? I'm unfortunately a bit time limited today, but if someone could point me in the right direction/give feedback on this idea, I'd be glad to work on it and make such an overlay.
Back to top
View user's profile Send private message
Maitreya
Guru
Guru


Joined: 11 Jan 2006
Posts: 441

PostPosted: Thu Sep 14, 2017 4:23 pm    Post subject: Reply with quote

I use them in my overlay. And as a workaround I just make a symlink /var/lib/layman/REPONAME/profiles/PATH/env in /etc/portage/env/REPONAME and for /etc/profile/package.env likewise.
Not pretty but it gets the job done until it is supported. (I have no idea btw, why that is, I'll ask around on irc)
Back to top
View user's profile Send private message
InBetweenNames
n00b
n00b


Joined: 11 Sep 2017
Posts: 7
Location: Windsor, Ontario

PostPosted: Thu Sep 14, 2017 8:25 pm    Post subject: Reply with quote

I've got an issue on the issues page that discusses a potential "LTOize" script to help users migrate to an LTO configuration. I don't have time to work on it today but I found the idea promising. It would be a simple lightweight script that you could use to retrieve the latest LTO workarounds and configurations, and advise you how to modify your make.conf to have LTO working well. I would have to make a small number of changes to the repo for it to work nicely, but those are changes I would be willing to do. If someday "env" support is added to Portage, we could make it a profile very easily that way. Thoughts/suggestions? I figure we can discuss it for a bit to make sure we get a good solution, then when I get some time I'll try it out.
Back to top
View user's profile Send private message
InBetweenNames
n00b
n00b


Joined: 11 Sep 2017
Posts: 7
Location: Windsor, Ontario

PostPosted: Sat Sep 16, 2017 7:58 pm    Post subject: Reply with quote

For anyone interested: I have updated the repo and converted it into an overlay. I also have a minimal 'ltoize' script that helps set things up for you. Check it out and lets get bug hunting!
Back to top
View user's profile Send private message
tranquilcool
Veteran
Veteran


Joined: 25 Mar 2005
Posts: 1179

PostPosted: Tue Sep 19, 2017 7:44 am    Post subject: Reply with quote

InBetweenNames wrote:
For anyone interested: I have updated the repo and converted it into an overlay. I also have a minimal 'ltoize' script that helps set things up for you. Check it out and lets get bug hunting!


errors:
1) Error during set creation: Redefinition of set 'toolchain' (sections: 'usersets', 'lto-overlay sets')

2) you have sys-devel/binutils-libs instead of sys-libs/binutils-libs

careful, you could break portage!
_________________
this is a strange strange world.
Back to top
View user's profile Send private message
InBetweenNames
n00b
n00b


Joined: 11 Sep 2017
Posts: 7
Location: Windsor, Ontario

PostPosted: Tue Sep 19, 2017 4:22 pm    Post subject: Reply with quote

Thanks--I'll file an issue about this in the issue tracker.

EDIT: just a quick question, can you elaborate on the sys-libs/binutils-libs? I can't find any references to it in my repo except in @ltoknownworking

EDIT 2: Nevermind, I found it. Issue filed.
Back to top
View user's profile Send private message
InBetweenNames
n00b
n00b


Joined: 11 Sep 2017
Posts: 7
Location: Windsor, Ontario

PostPosted: Tue Sep 19, 2017 8:11 pm    Post subject: Reply with quote

Fixed as of HEAD. Thanks for bringing that to my attention!
Back to top
View user's profile Send private message
42n4
n00b
n00b


Joined: 10 Feb 2015
Posts: 9

PostPosted: Tue Jul 17, 2018 9:23 am    Post subject: Reply with quote

Thanks a lot!
It works as I describe in other threads, but compiled with gcc 8.1.0 and flags -march=native some programs e.g. opencl R packages crashes.
Thus, I compiled with -march=core2 and lto graphite all my opencl programs, R packages based on RCPP, even RInside C++ files (with embedded R).

And after:
Code:
 emerge -e world

there were all 1500 packages compiled in 500min on four core i7 cpu (only php, adplug, qtscript, avidemux-core, gnome-keyring without lto, O3).
A few packages have to be switched to clang e.g. libcli
https://forums.gentoo.org/viewtopic-p-8241956.html#8241956
_________________
gentoo gcc 8.3.0 lto graphite, arm rpi3 7.4.0,
http://bit.ly/gen2ls


Last edited by 42n4 on Fri Jul 20, 2018 10:58 pm; edited 2 times in total
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Tue Jul 17, 2018 1:46 pm    Post subject: Re: Portage config for building with -O3, Graphite, and LTO Reply with quote

InBetweenNames wrote:
I've been silently working on a Portage configuration for building with O3, Graphite, and LTO by default since about the start of the year. My configuration and results are in this GitHub repo: https://github.com/InBetweenNames/gentooLTO
This is excellent work.
Please consider using -O2 instead of -O3, which is for turning on all experimental optimisations in gcc as a test of the latest draft set, and is explicitly intended to interact badly, rather than for use in production.

You can confirm this for yourself in #gcc on irc.oftc.net (I think it is.)
You will also like ##workingset on chat.freenode.net or .org, especially if you want to learn autotools professionally.

Well done, and keep it up :-)
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum