Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Howto compile packages with PGO
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks
View previous topic :: View next topic  
Author Message
xming
Guru
Guru


Joined: 02 Jul 2002
Posts: 434

PostPosted: Wed Oct 03, 2012 1:51 pm    Post subject: Howto compile packages with PGO Reply with quote

Profile guided optimization can sometimes yield better performance, I am going to explain howto to use the portage system to do that without hacking portage itself, so PGO maybe become more popular.

1. mkdir /var/tmp/pgo
2. put the following in /etc/portage/env/cflags.pgo
Code:
PROFILE_DIR=/var/tmp/pgo/${PN}
CFLAGS_PROFILE_GEN="-fprofile-generate=${PROFILE_DIR} -Wno-error=coverage-mismatch -fprofile-arcs -fvpt"
CFLAGS_PROFILE_USE="-fprofile-use=${PROFILE_DIR} -Wno-error=coverage-mismatch -fprofile-correction"
LDFLAGS_PROFILE_GEN="-fprofile-arcs"

CFLAGS_CUSTOM=""
LDFLAGS_CUSTOM=""

if [ -d ${PROFILE_DIR} ]
then
  CFLAGS="${CFLAGS} ${CFLAGS_CUSTOM} ${CFLAGS_PROFILE_USE}"
  LDFLAGS="${LDFLAGS} ${LDFLAGS_CUSTOM}"
else
  CFLAGS="${CFLAGS} ${CFLAGS_CUSTOM} ${CFLAGS_PROFILE_GEN}"
  LDFLAGS="${LDFLAGS} ${LDFLAGS_CUSTOM} ${LDFLAGS_PROFILE_GEN}"
fi

CXXFLAGS="${CFLAGS}"

3. for packages which you want to compile with pgo, you have to use cflags.pgo, see here for instructions.
4. emerge <pkgs>
5. you need to chown/chmod /var/tmp/pgo/${PN} so that your normal user can write to the files/dirs
6. use the pkgs as you normally would
7. emerge <pkgs> (again)
8. now you have PGO builds

Caveats:

- Well you will have to compile twice (for the first time) and use the application in between compiles, after that it will use the existing profile stats.
- Step 5, sometimes during the compilation, some exec are run (conftest, ...) so there are files generated in /var/tmp/pgo/${PN}, you will have to manually delete those or fix the permissions.
- if you want to regenerate the profile stats then you need to manually delete /var/tmp/pgo/${PN} and start over

It's not fully automatic method, but this is as far as I could get to reduce the manual steps, any suggestions are welcome.
_________________
http://wojia.be
Back to top
View user's profile Send private message
avx
Advocate
Advocate


Joined: 21 Jun 2004
Posts: 2111

PostPosted: Wed Oct 03, 2012 1:56 pm    Post subject: Reply with quote

Quote:
use the application in between compiles
How? Trigger any possible action in the application, only what is used mostly?
_________________
++++++++++[>+++++++>++++++++++>+++>+<<<<-]>++.>+.+++++++..+++.>++.<<+++++++++++++++.>.+++.------.--------.>+.>.
Back to top
View user's profile Send private message
xming
Guru
Guru


Joined: 02 Jul 2002
Posts: 434

PostPosted: Wed Oct 03, 2012 1:59 pm    Post subject: Reply with quote

avx wrote:
Quote:
use the application in between compiles
How? Trigger any possible action in the application, only what is used mostly?

I would say use it normally.
_________________
http://wojia.be
Back to top
View user's profile Send private message
i13m
n00b
n00b


Joined: 26 Jun 2003
Posts: 46
Location: UK / China PR

PostPosted: Sun Oct 07, 2012 8:24 am    Post subject: Reply with quote

Hi, I followed this guide, but get an error

Code:

!!! Problem in 'app-emulation/qemu-kvm' dependencies.
!!! "/etc/portage/env/cflags.pgo", line 10: Invalid token '[' (not '=') portage.exception
... done!
"/etc/portage/env/cflags.pgo", line 10: Invalid token '[' (not '=')


As far as I know, /etc/portage/env/cflags.pgo cannot do bash expression, or am I wrong
Back to top
View user's profile Send private message
xming
Guru
Guru


Joined: 02 Jul 2002
Posts: 434

PostPosted: Sun Oct 07, 2012 8:00 pm    Post subject: Reply with quote

I am using symlinks from /etc/portage/env/<pkg> to /etc/portage/env/cflags.pgo and not using the /etc/portage/package.env .
And I am using portage-2.2.0_alpha* .

I don't know which of the two would allow bash expressions (I am guessing the second), but it certainly works here.
_________________
http://wojia.be
Back to top
View user's profile Send private message
bergelin
n00b
n00b


Joined: 12 Dec 2006
Posts: 6

PostPosted: Tue Oct 09, 2012 1:58 pm    Post subject: Reply with quote

avx wrote:
Quote:
use the application in between compiles
How? Trigger any possible action in the application, only what is used mostly?

Wikipedia says
Quote:
The caveat, however, is that the sample of data fed to the program during the profiling stage must be statistically representative of the typical usage scenarios; otherwise, profile-guided feedback has the potential to harm the overall performance of the final build instead of improve it.

Found a lecture transcript (p. 34 and forwards) with some more info.


I'm gonna give this a try on a few applications. Ran into the same problem i13m did, but solved it by using symlinks as mentioned above, after which everything compiled fine.
It would be interesting to see benchmarks, or any other information, as to the possible performance benefits in some use cases, so one can determine which (if any) applications are worth the while.
Back to top
View user's profile Send private message
papu
Guru
Guru


Joined: 25 Jan 2008
Posts: 390
Location: Sota algun pi de l'empordà

PostPosted: Wed Oct 31, 2012 3:20 pm    Post subject: Reply with quote

xming wrote:
I am using symlinks from /etc/portage/env/<pkg> to /etc/portage/env/cflags.pgo and not using the /etc/portage/package.env .
.


i don't undestand this, etc/portage/env/ pkg???, i don't have pkg here... write a real example please.

i have same error
Code:
sudo USE="pgo" emerge -a firefox

These are the packages that would be merged, in order:

Calculating dependencies      y

!!! Problem in 'www-client/firefox' dependencies.
!!! "/etc/portage/env/cflags.pgo", line 9: Invalid token '[' (not '=') portage.exception
... done!
"/etc/portage/env/cflags.pgo", line 9: Invalid token '[' (not '=')



thanks :)
_________________
--Intel i5 3570k --Asrock z77 ext. 4 --Skill 32GB dd3 --Radeon r9 270 --S.O Gentoo & Win 64bits
Back to top
View user's profile Send private message
xming
Guru
Guru


Joined: 02 Jul 2002
Posts: 434

PostPosted: Thu Nov 01, 2012 9:37 pm    Post subject: Reply with quote

like
Code:
ln -s /etc/portage/env/cflags.pgo /etc/portage/env/media-libs/mesa


Or even better (relative instead of absolute)

Code:
mkdir /etc/portage/env/media-lib && cd /etc/portage/env/media-lib && ln -s ../cflags.pgo mesa

_________________
http://wojia.be
Back to top
View user's profile Send private message
mv
Advocate
Advocate


Joined: 20 Apr 2005
Posts: 4308

PostPosted: Sun Nov 04, 2012 6:16 am    Post subject: Reply with quote

If you use portage-bashrc-mv (available e.g. from the mv overlay), you can simply put into /etc/portage/package.cflags (or, if you prefer, into a file in there):
/etc/portage/package.cflags/pgo wrote:
category/package PGO=true

(if you have installed eix then instead of category/package you can also specify versions/version ranges/slots/overlays as usual in /etc/portage/package.* files).

Then after the first emerge of the correponding packages run them and then re-emerge them a second time. (The mechanism used is quite similar than in this thread and is described in more detail in the README).

You need >=portage-bashrc-mv-12.0 for the correct name PGO (in earlier versions, this was the misnomer GPO which I realized only after reading this thread...).
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Documentation, Tips & Tricks All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum