| View previous topic :: View next topic |
| Author |
Message |
xming
Guru
Joined: 02 Jul 2002
Posts: 434
|
 Posted: Wed Oct 03, 2012 1:51 pm Post subject: Howto compile packages with PGO |
 |
|
Profile guided optimization can sometimes yield better performance, I am going to explain howto to use the portage system to do that without hacking portage itself, so PGO maybe become more popular.
1. mkdir /var/tmp/pgo
2. put the following in /etc/portage/env/cflags.pgo
| Code: | PROFILE_DIR=/var/tmp/pgo/${PN}
CFLAGS_PROFILE_GEN="-fprofile-generate=${PROFILE_DIR} -Wno-error=coverage-mismatch -fprofile-arcs -fvpt"
CFLAGS_PROFILE_USE="-fprofile-use=${PROFILE_DIR} -Wno-error=coverage-mismatch -fprofile-correction"
LDFLAGS_PROFILE_GEN="-fprofile-arcs"
CFLAGS_CUSTOM=""
LDFLAGS_CUSTOM=""
if [ -d ${PROFILE_DIR} ]
then
CFLAGS="${CFLAGS} ${CFLAGS_CUSTOM} ${CFLAGS_PROFILE_USE}"
LDFLAGS="${LDFLAGS} ${LDFLAGS_CUSTOM}"
else
CFLAGS="${CFLAGS} ${CFLAGS_CUSTOM} ${CFLAGS_PROFILE_GEN}"
LDFLAGS="${LDFLAGS} ${LDFLAGS_CUSTOM} ${LDFLAGS_PROFILE_GEN}"
fi
CXXFLAGS="${CFLAGS}"
|
3. for packages which you want to compile with pgo, you have to use cflags.pgo, see here for instructions.
4. emerge <pkgs>
5. you need to chown/chmod /var/tmp/pgo/${PN} so that your normal user can write to the files/dirs
6. use the pkgs as you normally would
7. emerge <pkgs> (again)
8. now you have PGO builds
Caveats:
- Well you will have to compile twice (for the first time) and use the application in between compiles, after that it will use the existing profile stats.
- Step 5, sometimes during the compilation, some exec are run (conftest, ...) so there are files generated in /var/tmp/pgo/${PN}, you will have to manually delete those or fix the permissions.
- if you want to regenerate the profile stats then you need to manually delete /var/tmp/pgo/${PN} and start over
It's not fully automatic method, but this is as far as I could get to reduce the manual steps, any suggestions are welcome.
_________________
http://wojia.be |
|
| Back to top |
|
 |
avx
Advocate
Joined: 21 Jun 2004
Posts: 2063
|
| Posted: Wed Oct 03, 2012 1:56 pm Post subject: |
|
|
| Quote: | | use the application in between compiles |
How? Trigger any possible action in the application, only what is used mostly?
_________________
++++++++++[>+++++++>++++++++++>+++>+<<<<-]>++.>+.+++++++..+++.>++.<<+++++++++++++++.>.+++.------.--------.>+.>. |
|
| Back to top |
|
|
xming
Guru
Joined: 02 Jul 2002
Posts: 434
|
| Posted: Wed Oct 03, 2012 1:59 pm Post subject: |
|
|
| avx wrote: | | Quote: | | use the application in between compiles |
How? Trigger any possible action in the application, only what is used mostly? |
I would say use it normally.
_________________
http://wojia.be |
|
| Back to top |
|
|
i13m
n00b
Joined: 26 Jun 2003
Posts: 46
Location: UK / China PR
|
| Posted: Sun Oct 07, 2012 8:24 am Post subject: |
|
|
Hi, I followed this guide, but get an error
| Code: |
!!! Problem in 'app-emulation/qemu-kvm' dependencies.
!!! "/etc/portage/env/cflags.pgo", line 10: Invalid token '[' (not '=') portage.exception
... done!
"/etc/portage/env/cflags.pgo", line 10: Invalid token '[' (not '=')
|
As far as I know, /etc/portage/env/cflags.pgo cannot do bash expression, or am I wrong |
|
| Back to top |
|
|
xming
Guru
Joined: 02 Jul 2002
Posts: 434
|
| Posted: Sun Oct 07, 2012 8:00 pm Post subject: |
|
|
I am using symlinks from /etc/portage/env/<pkg> to /etc/portage/env/cflags.pgo and not using the /etc/portage/package.env .
And I am using portage-2.2.0_alpha* .
I don't know which of the two would allow bash expressions (I am guessing the second), but it certainly works here.
_________________
http://wojia.be |
|
| Back to top |
|
|
bergelin
n00b
Joined: 12 Dec 2006
Posts: 6
|
| Posted: Tue Oct 09, 2012 1:58 pm Post subject: |
|
|
| avx wrote: | | Quote: | | use the application in between compiles |
How? Trigger any possible action in the application, only what is used mostly? |
Wikipedia says
| Quote: | | The caveat, however, is that the sample of data fed to the program during the profiling stage must be statistically representative of the typical usage scenarios; otherwise, profile-guided feedback has the potential to harm the overall performance of the final build instead of improve it. |
Found a lecture transcript (p. 34 and forwards) with some more info.
I'm gonna give this a try on a few applications. Ran into the same problem i13m did, but solved it by using symlinks as mentioned above, after which everything compiled fine.
It would be interesting to see benchmarks, or any other information, as to the possible performance benefits in some use cases, so one can determine which (if any) applications are worth the while. |
|
| Back to top |
|
|
papu
Guru
Joined: 25 Jan 2008
Posts: 365
Location: Sota algun pi de l'empordà
|
| Posted: Wed Oct 31, 2012 3:20 pm Post subject: |
|
|
| xming wrote: | I am using symlinks from /etc/portage/env/<pkg> to /etc/portage/env/cflags.pgo and not using the /etc/portage/package.env .
. |
i don't undestand this, etc/portage/env/ pkg???, i don't have pkg here... write a real example please.
i have same error
| Code: | sudo USE="pgo" emerge -a firefox
These are the packages that would be merged, in order:
Calculating dependencies y
!!! Problem in 'www-client/firefox' dependencies.
!!! "/etc/portage/env/cflags.pgo", line 9: Invalid token '[' (not '=') portage.exception
... done!
"/etc/portage/env/cflags.pgo", line 9: Invalid token '[' (not '=') |
thanks 
_________________
--Intel i5 3570k --Asrock z77 ext. 4 --Skill 4x4GB dd3 --Ati 5850 --S.O Gentoo & Win8 64bits |
|
| Back to top |
|
|
xming
Guru
Joined: 02 Jul 2002
Posts: 434
|
| Posted: Thu Nov 01, 2012 9:37 pm Post subject: |
|
|
like
| Code: | ln -s /etc/portage/env/cflags.pgo /etc/portage/env/media-libs/mesa
|
Or even better (relative instead of absolute)
| Code: | mkdir /etc/portage/env/media-lib && cd /etc/portage/env/media-lib && ln -s ../cflags.pgo mesa
|
_________________
http://wojia.be |
|
| Back to top |
|
|
mv
Advocate
Joined: 20 Apr 2005
Posts: 3144
|
| Posted: Sun Nov 04, 2012 6:16 am Post subject: |
|
|
If you use portage-bashrc-mv (available e.g. from the mv overlay), you can simply put into /etc/portage/package.cflags (or, if you prefer, into a file in there):
| /etc/portage/package.cflags/pgo wrote: | | category/package PGO=true |
(if you have installed eix then instead of category/package you can also specify versions/version ranges/slots/overlays as usual in /etc/portage/package.* files).
Then after the first emerge of the correponding packages run them and then re-emerge them a second time. (The mechanism used is quite similar than in this thread and is described in more detail in the README).
You need >=portage-bashrc-mv-12.0 for the correct name PGO (in earlier versions, this was the misnomer GPO which I realized only after reading this thread...). |
|
| Back to top |
|
|
|
Documentation, Tips & Tricks
