Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Howto compile packages with PGO
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
xming
Guru
Guru


Joined: 02 Jul 2002
Posts: 437

PostPosted: Wed Oct 03, 2012 1:51 pm    Post subject: Howto compile packages with PGO Reply with quote

Profile guided optimization can sometimes yield better performance, I am going to explain howto to use the portage system to do that without hacking portage itself, so PGO maybe become more popular.

1. mkdir /var/tmp/pgo
2. put the following in /etc/portage/env/cflags.pgo
Code:
PROFILE_DIR=/var/tmp/pgo/${PN}
CFLAGS_PROFILE_GEN="-fprofile-generate=${PROFILE_DIR} -Wno-error=coverage-mismatch -fprofile-arcs -fvpt"
CFLAGS_PROFILE_USE="-fprofile-use=${PROFILE_DIR} -Wno-error=coverage-mismatch -fprofile-correction"
LDFLAGS_PROFILE_GEN="-fprofile-arcs"

CFLAGS_CUSTOM=""
LDFLAGS_CUSTOM=""

if [ -d ${PROFILE_DIR} ]
then
  CFLAGS="${CFLAGS} ${CFLAGS_CUSTOM} ${CFLAGS_PROFILE_USE}"
  LDFLAGS="${LDFLAGS} ${LDFLAGS_CUSTOM}"
else
  CFLAGS="${CFLAGS} ${CFLAGS_CUSTOM} ${CFLAGS_PROFILE_GEN}"
  LDFLAGS="${LDFLAGS} ${LDFLAGS_CUSTOM} ${LDFLAGS_PROFILE_GEN}"
fi

CXXFLAGS="${CFLAGS}"

3. for packages which you want to compile with pgo, you have to use cflags.pgo, see here for instructions.
4. emerge <pkgs>
5. you need to chown/chmod /var/tmp/pgo/${PN} so that your normal user can write to the files/dirs
6. use the pkgs as you normally would
7. emerge <pkgs> (again)
8. now you have PGO builds

Caveats:

- Well you will have to compile twice (for the first time) and use the application in between compiles, after that it will use the existing profile stats.
- Step 5, sometimes during the compilation, some exec are run (conftest, ...) so there are files generated in /var/tmp/pgo/${PN}, you will have to manually delete those or fix the permissions.
- if you want to regenerate the profile stats then you need to manually delete /var/tmp/pgo/${PN} and start over

It's not fully automatic method, but this is as far as I could get to reduce the manual steps, any suggestions are welcome.
_________________
http://wojia.be
Back to top
View user's profile Send private message
avx
Advocate
Advocate


Joined: 21 Jun 2004
Posts: 2150

PostPosted: Wed Oct 03, 2012 1:56 pm    Post subject: Reply with quote

Quote:
use the application in between compiles
How? Trigger any possible action in the application, only what is used mostly?
_________________
++++++++++[>+++++++>++++++++++>+++>+<<<<-]>++.>+.+++++++..+++.>++.<<+++++++++++++++.>.+++.------.--------.>+.>.
Back to top
View user's profile Send private message
xming
Guru
Guru


Joined: 02 Jul 2002
Posts: 437

PostPosted: Wed Oct 03, 2012 1:59 pm    Post subject: Reply with quote

avx wrote:
Quote:
use the application in between compiles
How? Trigger any possible action in the application, only what is used mostly?

I would say use it normally.
_________________
http://wojia.be
Back to top
View user's profile Send private message
i13m
n00b
n00b


Joined: 26 Jun 2003
Posts: 46
Location: UK / China PR

PostPosted: Sun Oct 07, 2012 8:24 am    Post subject: Reply with quote

Hi, I followed this guide, but get an error

Code:

!!! Problem in 'app-emulation/qemu-kvm' dependencies.
!!! "/etc/portage/env/cflags.pgo", line 10: Invalid token '[' (not '=') portage.exception
... done!
"/etc/portage/env/cflags.pgo", line 10: Invalid token '[' (not '=')


As far as I know, /etc/portage/env/cflags.pgo cannot do bash expression, or am I wrong
Back to top
View user's profile Send private message
xming
Guru
Guru


Joined: 02 Jul 2002
Posts: 437

PostPosted: Sun Oct 07, 2012 8:00 pm    Post subject: Reply with quote

I am using symlinks from /etc/portage/env/<pkg> to /etc/portage/env/cflags.pgo and not using the /etc/portage/package.env .
And I am using portage-2.2.0_alpha* .

I don't know which of the two would allow bash expressions (I am guessing the second), but it certainly works here.
_________________
http://wojia.be
Back to top
View user's profile Send private message
bergelin
n00b
n00b


Joined: 12 Dec 2006
Posts: 6

PostPosted: Tue Oct 09, 2012 1:58 pm    Post subject: Reply with quote

avx wrote:
Quote:
use the application in between compiles
How? Trigger any possible action in the application, only what is used mostly?

Wikipedia says
Quote:
The caveat, however, is that the sample of data fed to the program during the profiling stage must be statistically representative of the typical usage scenarios; otherwise, profile-guided feedback has the potential to harm the overall performance of the final build instead of improve it.

Found a lecture transcript (p. 34 and forwards) with some more info.


I'm gonna give this a try on a few applications. Ran into the same problem i13m did, but solved it by using symlinks as mentioned above, after which everything compiled fine.
It would be interesting to see benchmarks, or any other information, as to the possible performance benefits in some use cases, so one can determine which (if any) applications are worth the while.
Back to top
View user's profile Send private message
papu
Guru
Guru


Joined: 25 Jan 2008
Posts: 452
Location: Sota algun pi o una alzina...

PostPosted: Wed Oct 31, 2012 3:20 pm    Post subject: Reply with quote

xming wrote:
I am using symlinks from /etc/portage/env/<pkg> to /etc/portage/env/cflags.pgo and not using the /etc/portage/package.env .
.


i don't undestand this, etc/portage/env/ pkg???, i don't have pkg here... write a real example please.

i have same error
Code:
sudo USE="pgo" emerge -a firefox

These are the packages that would be merged, in order:

Calculating dependencies      y

!!! Problem in 'www-client/firefox' dependencies.
!!! "/etc/portage/env/cflags.pgo", line 9: Invalid token '[' (not '=') portage.exception
... done!
"/etc/portage/env/cflags.pgo", line 9: Invalid token '[' (not '=')



thanks :)
_________________
--Intel i5 3570k --Asrock z77 ext. 4 --Skill 16GB dd3 --Radeon r9 270 -- Dual boot: Gentoo ~amd64 & Windows 8.1
Back to top
View user's profile Send private message
xming
Guru
Guru


Joined: 02 Jul 2002
Posts: 437

PostPosted: Thu Nov 01, 2012 9:37 pm    Post subject: Reply with quote

like
Code:
ln -s /etc/portage/env/cflags.pgo /etc/portage/env/media-libs/mesa


Or even better (relative instead of absolute)

Code:
mkdir /etc/portage/env/media-lib && cd /etc/portage/env/media-lib && ln -s ../cflags.pgo mesa

_________________
http://wojia.be
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 5307

PostPosted: Sun Nov 04, 2012 6:16 am    Post subject: Reply with quote

If you use portage-bashrc-mv (available e.g. from the mv overlay), you can simply put into /etc/portage/package.cflags (or, if you prefer, into a file in there):
/etc/portage/package.cflags/pgo wrote:
category/package PGO=true

(if you have installed eix then instead of category/package you can also specify versions/version ranges/slots/overlays as usual in /etc/portage/package.* files).

Then after the first emerge of the correponding packages run them and then re-emerge them a second time. (The mechanism used is quite similar than in this thread and is described in more detail in the README).

You need >=portage-bashrc-mv-12.0 for the correct name PGO (in earlier versions, this was the misnomer GPO which I realized only after reading this thread...).
Back to top
View user's profile Send private message
ryszardzonk
Apprentice
Apprentice


Joined: 18 Dec 2003
Posts: 197
Location: Rzeszów, POLAND

PostPosted: Fri Apr 15, 2016 9:14 pm    Post subject: Reply with quote

I just read that using PGO for python should yield extra 10% perfomance boost - http://komodoide.com/blog/2014-06/python-pgo-on-linux/ That having said the only way I found searching for using PGO in Gentoo other than some packages having use flag for it is to use this script. I did have to change location to the PROFILE_DIR=/tmp/portage/${P} as with original one I had some write permissions problems and from ${PN} to ${P} IMHO should be more appropriate as packages like python have more than one version that may be present in the system at the time

Now to questions
- what to do so I would not have to emerge package twice as first "emerge python" emerged package with just profile generated and only second run used generated profile

-march-native -O3 -fprofile-generate
dev-lang/python-3.5.1-r2: 5975 files, 192 non-files, 108627,379 KiB
-march-native -O3 -fprofile-use
dev-lang/python-3.5.1-r2: 5975 files, 192 non-files, 96819,41 KiB

BTW If you wanted to enable lto on python with GCC 5.3 think again ;) It balooned in size for me

-march-native -O3 -flto=4 -ffat-lto-objects -fprofile-generate
dev-lang/python-3.5.1-r2: 5975 files, 192 non-files, 132411,422 KiB
-march-native -O3 -flto=4 -ffat-lto-objects -fprofile-use
dev-lang/python-3.5.1-r2: 5975 files, 192 non-files, 111267,736 KiB

- After having it done in just one run it would be nice for portage to delete that profile as it should not be needed any more right?
_________________
Sky is not the limit...
Back to top
View user's profile Send private message
steveL
Advocate
Advocate


Joined: 13 Sep 2006
Posts: 4520
Location: The Peanut Gallery

PostPosted: Sat Apr 16, 2016 12:55 pm    Post subject: Reply with quote

That is a lovely post, xming.
ryszardzonk wrote:
After having it done in just one run it would be nice for portage to delete that profile as it should not be needed any more right?

Just DIY, at step 8; script the process if you prefer.

And thanks for the thread necromancy ;) I'd never have seen this topic otherwise.
Back to top
View user's profile Send private message
ryszardzonk
Apprentice
Apprentice


Joined: 18 Dec 2003
Posts: 197
Location: Rzeszów, POLAND

PostPosted: Sat Apr 16, 2016 9:18 pm    Post subject: Reply with quote

I would love to write such a script to cover that but it s over my head stuff. Anyways I believe that by now there should be some kind of eclass that one could add to their ebuild to get PGO enabled for ebuilds. Emerging stuff twice is not what tigers like most ;)
_________________
Sky is not the limit...
Back to top
View user's profile Send private message
trippels
Tux's lil' helper
Tux's lil' helper


Joined: 24 Nov 2010
Posts: 102
Location: Berlin

PostPosted: Sun Apr 17, 2016 8:03 am    Post subject: Reply with quote

For python you can simply edit the ebuild and use the built-in profile-opt target:

Code:

diff --git a/dev-lang/python/python-2.7.11-r2.ebuild b/dev-lang/python/python-2.7.11-r2.ebuild
index 5e7b2b6..4e708de 100644
--- a/dev-lang/python/python-2.7.11-r2.ebuild
+++ b/dev-lang/python/python-2.7.11-r2.ebuild
@@ -217,7 +217,7 @@ src_compile() {
        touch Include/graminit.h Python/graminit.c
 
        cd "${BUILD_DIR}" || die
-       emake
+       emake profile-opt
 
        # Work around bug 329499. See also bug 413751 and 457194.
        if has_version dev-libs/libffi[pax_kernel]; then
diff --git a/dev-lang/python/python-3.4.3-r7.ebuild b/dev-lang/python/python-3.4.3-r7.ebuild
index e281de4..54c7e37 100644
--- a/dev-lang/python/python-3.4.3-r7.ebuild
+++ b/dev-lang/python/python-3.4.3-r7.ebuild
@@ -176,7 +176,7 @@ src_compile() {
 
        cd "${BUILD_DIR}" || die
 
-       emake CPPFLAGS= CFLAGS= LDFLAGS=
+       emake profile-opt
 
        # Work around bug 329499. See also bug 413751 and 457194.
        if has_version dev-libs/libffi[pax_kernel]; then
Back to top
View user's profile Send private message
steveL
Advocate
Advocate


Joined: 13 Sep 2006
Posts: 4520
Location: The Peanut Gallery

PostPosted: Sun Apr 17, 2016 12:59 pm    Post subject: Reply with quote

ryszardzonk wrote:
Emerging stuff twice is not what tigers like most ;)

Yeah, but you have to build at least twice to profile at all.

Given that, it's an admin-level decision when to profile, since it is best done on typical usage scenarios, and it implies a lack of performance during profiling. Both require human input.
Back to top
View user's profile Send private message
steveL
Advocate
Advocate


Joined: 13 Sep 2006
Posts: 4520
Location: The Peanut Gallery

PostPosted: Sun Apr 17, 2016 1:00 pm    Post subject: Reply with quote

trippels wrote:
For python you can simply edit the ebuild and use the built-in profile-opt target

Hmm nice patch; nicer USE-flag dependent? ie:
Code:
if use profile; then ..

(already a global USE) which does throw up the option of making it into a more generic thing, at the FEATURE level perhaps.

If someone's already doing that, please share.
Back to top
View user's profile Send private message
ryszardzonk
Apprentice
Apprentice


Joined: 18 Dec 2003
Posts: 197
Location: Rzeszów, POLAND

PostPosted: Sat Apr 23, 2016 8:23 pm    Post subject: Reply with quote

steveL wrote:
ryszardzonk wrote:
Emerging stuff twice is not what tigers like most ;)

Yeah, but you have to build at least twice to profile at all.

Yeah I know that. But emerging twice takes even longer than just compiling twice hence the idea of having more generic approach to the build process. I wish one of the devs would add it to the src_compile routines adding lets say "pgo_src_compile" section in one or the other eclass so it would be easier and faster to use for users and package maintainers to use.
_________________
Sky is not the limit...
Back to top
View user's profile Send private message
steveL
Advocate
Advocate


Joined: 13 Sep 2006
Posts: 4520
Location: The Peanut Gallery

PostPosted: Fri Apr 29, 2016 6:59 pm    Post subject: Reply with quote

ryszardzonk wrote:
Emerging stuff twice is not what tigers like most ;)

steveL wrote:
Yeah, but you have to build at least twice to profile at all.

ryszardzonk wrote:
Yeah I know that. But emerging twice takes even longer than just compiling twice

No, it does not. Manually compiling software is much more tedious, and far less useful, than using an ebuild.
Quote:
hence the idea of having more generic approach to the build process. I wish one of the devs would add it to the src_compile routines adding lets say "pgo_src_compile" section in one or the other eclass so it would be easier and faster to use for users and package maintainers to use.

I'm fine with a more generic approach, though I think that's a bad notion of one.

The thing you cannot get away from is that there MUST be a time lag between the builds, for the profile data to be generated, in order to be used in the subsequent production build.

As such it means a next ebuild run, and nothing should be changed at the ebuild level, imo.

I'd knock up a script in bash, using eix and portage-utils, to begin with.
#bash are your friends.
Back to top
View user's profile Send private message
ryszardzonk
Apprentice
Apprentice


Joined: 18 Dec 2003
Posts: 197
Location: Rzeszów, POLAND

PostPosted: Tue May 10, 2016 4:36 am    Post subject: Reply with quote

ryszardzonk wrote:
Yeah I know that. But emerging twice takes even longer than just compiling twice

steveL wrote:
No, it does not. Manually compiling software is much more tedious, and far less useful, than using an ebuild.

When I said about compiling twice it would be faster I meant that when using ebuild to do
- unpack & prepare & pgo_compile (2xcompile) & install & qmerge will be faster than 2x ( unpack & prepare & compile & install & qmerge).

steveL wrote:
The thing you cannot get away from is that there MUST be a time lag between the builds, for the profile data to be generated, in order to be used in the subsequent production build.
What lag? If first compile finishes, than second may start. Otherwise those ebuilds in portage like firefox would not have pgo flag.
_________________
Sky is not the limit...
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum