Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
firefox 73.0 gcc+lto+pgo fails, lonely case?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
CaptainBlood
Veteran
Veteran


Joined: 24 Jan 2010
Posts: 1366

PostPosted: Tue Feb 11, 2020 12:23 pm    Post subject: firefox 73.0 gcc+lto+pgo fails, lonely case? Reply with quote

Code:
equery u  firefox
[ Legend : U - final flag setting for installation]
[        : I - package is installed with flag     ]
[ Colors : set, unset                             ]
 * Found these USE flags for www-client/firefox-73.0:
 U I
 - - bindist              : Disable official Firefox branding (icons, name) which are not binary-redistributable according to upstream.
 - - clang                : Use Clang compiler instead of GCC
 + + cpu_flags_x86_avx2   : Adds support for Advanced Vector Extensions 2 instructions
 + + custom-cflags        : Build with user-specified CFLAGS (unsupported)
 - - custom-optimization  : Build with user-specified compiler optimizations (-Os, -O0, -O1, -O2, -O3) from CFLAGS (unsupported)
 - - debug                : Enable extra debug codepaths, like asserts and extra output. If you want to get meaningful backtraces see
                            https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Backtraces
 - - eme-free             : Disable EME (DRM plugin) cabability at build time
 - - geckodriver          : Enable WebDriver support
 - - gmp-autoupdate       : Allow Gecko Media Plugins (binary blobs) to be automatically downloaded and kept up-to-date in user profiles
 - - hardened             : Activate default security enhancements for toolchain (gcc, glibc, binutils)
 - - hwaccel              : Force-enable hardware-accelerated rendering (Mozilla bug 594876)
 + + jack                 : Add support for the JACK Audio Connection Kit
 - - l10n_ach             : Acoli
 - - l10n_af              : Afrikaans
 - - l10n_an              : Aragonese
 - - l10n_ar              : Arabic
 - - l10n_ast             : Asturian
 - - l10n_az              : Azerbaijani
 - - l10n_be              : Belarusian
 - - l10n_bg              : Bulgarian
 - - l10n_bn              : Bengali
 - - l10n_br              : Breton
 - - l10n_bs              : Bosnian
 - - l10n_ca              : Catalan
 - - l10n_cak             : Kaqchikel
 - - l10n_cs              : Czech
 - - l10n_cy              : Welsh
 - - l10n_da              : Danish
 - - l10n_de              : German
 - - l10n_dsb             : Lower Sorbian
 - - l10n_el              : Modern Greek
 - - l10n_en-CA           : English (Canada)
 - - l10n_en-GB           : English (United Kingdom)
 - - l10n_eo              : Esperanto
 - - l10n_es-AR           : Spanish (Argentina)
 - - l10n_es-CL           : Spanish (Chile)
 - - l10n_es-ES           : Spanish (Spain)
 - - l10n_es-MX           : Spanish (Mexico)
 - - l10n_et              : Estonian
 - - l10n_eu              : Basque
 - - l10n_fa              : Persian
 - - l10n_ff              : Fulah
 - - l10n_fi              : Finnish
 + + l10n_fr              : French
 - - l10n_fy              : Western Frisian
 - - l10n_ga              : Irish
 - - l10n_gd              : Scottish Gaelic
 - - l10n_gl              : Galician
 - - l10n_gn              : Guarani
 - - l10n_gu              : Gujarati
 - - l10n_he              : Hebrew
 - - l10n_hi              : Hindi
 - - l10n_hr              : Croatian
 - - l10n_hsb             : Upper Sorbian
 - - l10n_hu              : Hungarian
 - - l10n_hy              : Armenian
 - - l10n_ia              : Interlingua
 - - l10n_id              : Indonesian
 - - l10n_is              : Icelandic
 - - l10n_it              : Italian
 - - l10n_ja              : Japanese
 - - l10n_ka              : Georgian
 - - l10n_kab             : Kabyle
 - - l10n_kk              : Kazakh
 - - l10n_km              : Central Khmer
 - - l10n_kn              : Kannada
 - - l10n_ko              : Korean
 - - l10n_lij             : Ligurian
 - - l10n_lt              : Lithuanian
 - - l10n_lv              : Latvian
 - - l10n_mk              : Macedonian
 - - l10n_mr              : Marathi
 - - l10n_ms              : Malay (macrolanguage)
 - - l10n_my              : Burmese
 - - l10n_nb              : Norwegian Bokmål
 - - l10n_nl              : Dutch
 - - l10n_nn              : Norwegian Nynorsk
 - - l10n_oc              : Occitan
 - - l10n_pa              : Punjabi
 - - l10n_pl              : Polish
 - - l10n_pt-BR           : Portuguese (Brazil)
 - - l10n_pt-PT           : Portuguese (Portugal)
 - - l10n_rm              : Romansh
 - - l10n_ro              : Romanian
 - - l10n_ru              : Russian
 - - l10n_si              : Sinhala
 - - l10n_sk              : Slovak
 - - l10n_sl              : Slovenian
 - - l10n_son             : Songhai languages
 - - l10n_sq              : Albanian
 - - l10n_sr              : Serbian
 - - l10n_sv              : Swedish
 - - l10n_ta              : Tamil
 - - l10n_te              : Telugu
 - - l10n_th              : Thai
 - - l10n_tr              : Turkish
 - - l10n_uk              : Ukrainian
 - - l10n_ur              : Urdu
 - - l10n_uz              : Uzbek
 - - l10n_vi              : Vietnamese
 - - l10n_xh              : Xhosa
 - - l10n_zh-CN           : Chinese (China)
 - - l10n_zh-TW           : Chinese (Taiwan)
 + - lto                  : Enable Link Time Optimization (LTO). Requires Gold linker when using GCC or LDD linker when using Clang
 + - pgo                  : Add support for profile-guided optimization using gcc-4.5, for faster binaries. This option will double the
                            compile time.
 - - pulseaudio           : Add support for PulseAudio sound server
 - - screenshot           : Allow to disable screenshot extension in global profile
 + + startup-notification : Enable application startup event feedback mechanism
 + + system-av1           : Use the system-wide media-libs/dav1d and media-libs/libaom library instead of bundled.
 + + system-harfbuzz      : Use the system-wide media-libs/harfbuzz and media-gfx/graphite2 instead of bundled.
 + + system-icu           : Use the system-wide dev-libs/icu instead of bundled.
 + + system-jpeg          : Use the system-wide media-libs/libjpeg-turbo instead of bundled.
 + + system-libevent      : Use the system-wide dev-libs/libevent instead of bundled.
 + + system-libvpx        : Use the system-wide media-libs/libvpx instead of bundled.
 + + system-sqlite        : Use the system-wide dev-db/sqlite installation with secure-delete enabled
 + + system-webp          : Use the system-wide media-libs/libwebp instead of bundled.
 - - test                 : Enable dependencies and/or preparations necessary to run tests (usually controlled by FEATURES=test but can be
                            toggled independently)
 + + wayland              : Enable dev-libs/wayland backend
 - - wifi                 : Enable wireless network functions
USE="-lto -pgo" works fine though.
Report
Am I the only one?
Thks 4 ur attention.


Last edited by CaptainBlood on Fri Feb 14, 2020 10:43 am; edited 2 times in total
Back to top
View user's profile Send private message
altblitz
n00b
n00b


Joined: 25 Oct 2016
Posts: 6

PostPosted: Fri Feb 14, 2020 8:00 am    Post subject: Reply with quote

Quote:
... Requires Gold linker when using GCC or LDD linker when using Clang

Code:
LDFLAGS="-Wl,-O1 [b]-Wl,-fuse-ld=bfd[/b] -Wl,--enable-new-dtags -march=skylake -mtune=skylake -pipe -Wl,-rpath=/usr/lib64/firefox,--enable-new-dtags -Wl,--compress-debug-sections=zlib"

ld.bfd is not a Gold linker.

Either this way:
Code:
% ls ld*
lrwxrwxrwx 1 root root       7 Feb 14 08:52 ld -> ld.gold*
-rwxr-xr-x 1 root root 1703072 Feb  2 10:02 ld.bfd*
-rwxr-xr-x 1 root root 2287560 Feb  2 10:02 ld.gold*
lrwxrwxrwx 1 root root       3 Dec 30 00:12 ld.lld -> lld*
lrwxrwxrwx 1 root root       3 Dec 30 00:12 ld64.lld -> lld*

or another:
Code:
  LDFLAGS="-march=${MARCH} -mtune=${MARCH} -${OPT_LDFLAGS} -Wl,-${OPT_LDFLAGS},--sort-common,--as-needed,-z,relro,-z,now"
  if [ "${CLANG}" == true ]; then
    if [ "${LLD}" == true ]; then
    LDFLAGS="${LDFLAGS} -fuse-ld=lld"
    else
    LDFLAGS="${LDFLAGS} -Wl,-plugin-opt=${OPT_LDFLAGS} -fuse-ld=gold"
    fi
  else
    LDFLAGS="${LDFLAGS} -fuse-ld=gold"
  fi
Back to top
View user's profile Send private message
CaptainBlood
Veteran
Veteran


Joined: 24 Jan 2010
Posts: 1366

PostPosted: Fri Feb 14, 2020 10:28 am    Post subject: Reply with quote

I've noticed the gold linker requirement, installed here.
Didn't force the use of gold linker, though, as metadata doesn't express such a requirement.
It doesn't mean it shouldn't.

As clang is an option only, one might expect ebuild to be gcc ready or close by default.

Did you succeeded forcing gold linker?
Thks 4 ur attention, interest & support.
Back to top
View user's profile Send private message
Anarchy
Developer
Developer


Joined: 29 Jun 2005
Posts: 136

PostPosted: Fri Feb 14, 2020 5:01 pm    Post subject: Reply with quote

CaptainBlood wrote:
I've noticed the gold linker requirement, installed here.
Didn't force the use of gold linker, though, as metadata doesn't express such a requirement.
It doesn't mean it shouldn't.

As clang is an option only, one might expect ebuild to be gcc ready or close by default.

Did you succeeded forcing gold linker?
Thks 4 ur attention, interest & support.


You do not need to force anything, the ebuild had a known issue with pgo and lto, you should check bugzilla before you assume your the only one with a problem.

I have already addressed the build issues, people are still reporting lto builds are crashing, refer to bugzilla for the full details on the bug.
Back to top
View user's profile Send private message
Ionen
Guru
Guru


Joined: 06 Dec 2018
Posts: 395

PostPosted: Fri Feb 14, 2020 9:16 pm    Post subject: Reply with quote

If I was a maintainer, at this point I'd be tempted to just drop gcc support, or at least for lto+pgo, and make clang default. It's not that I prefer clang over gcc but in the real world mozilla is just testing everything with clang and leaving gcc issues for downstream to deal with (often been hard to diagnose runtime issues too). If the maintainers have the patience to keep dealing with this great for those who'd rather avoid both clang and firefox-bin though :) but well...
Back to top
View user's profile Send private message
CaptainBlood
Veteran
Veteran


Joined: 24 Jan 2010
Posts: 1366

PostPosted: Fri Feb 14, 2020 9:57 pm    Post subject: Reply with quote

Anarchy wrote:
you should check bugzilla before you assume your the only one with a problem
......
I have already addressed the build issues, people are still reporting lto builds are crashing, refer to bugzilla for the full details on the bug.
I wasn't assuming I was the only one with the problem, ain't english native though.

Things where fine on previous 72.0.2, at least here.

When I reported Bug on the (2020-02-11-09:11-UTC) it was first about 73 in bugzilla.

New patched ebuild claims to fix:
Code:
https://bugs.gentoo.org/706772=>73 Beta is only beta
https://bugs.gentoo.org/709440 =>2020-02-12
https://bugs.gentoo.org/709594=>2020-02-14
My bug report has been cancelled as duplicate of the above beta one, funnily enough as mine contains a 500k xz build log file, whereas the beta has a 25k gz build log file. How to conclude for duplication is undoubtably out of my scope of comprehension.

Anarchy wrote:
... people are still reporting lto builds are crashing ...
Indeed a new Bug has been opened on the 2020-02-14-18:45-UTC which fails on the same kind of issue I initially reported almost 3 days & 1/2 earlier, i.e. an 1/8th of the average lifetime of an firefox release.

Just finished trying to rebuild with that patched ebuild: It failed, in an identical way to what I initially reported. Updated my bug report accordingly.

Thks 4 ur attention, interest & support.


Last edited by CaptainBlood on Fri Feb 14, 2020 10:38 pm; edited 1 time in total
Back to top
View user's profile Send private message
CaptainBlood
Veteran
Veteran


Joined: 24 Jan 2010
Posts: 1366

PostPosted: Fri Feb 14, 2020 10:37 pm    Post subject: Reply with quote

Ionen wrote:
If I was a maintainer, at this point I'd be tempted to just drop gcc support, or at least for lto+pgo, and make clang default. It's not that I prefer clang over gcc but in the real world mozilla is just testing everything with clang and leaving gcc issues for downstream to deal with (often been hard to diagnose runtime issues too). If the maintainers have the patience to keep dealing with this great for those who'd rather avoid both clang and firefox-bin though :) but well...
I understand your point.
Iirc clang lto is thin only, and on an ealier release, i.e. the one were gcc broke at first, although I don't have the numbers anymore the gcc binpkg size was quite smaller than clang here. Can't tell about memory comsumption though.

For years now Phoronix has been pointing out how llvm is catching up with gcc.

Firefox is one of the flagship for such a challenge.

clang 10 & gcc 10 are not so far to be released.
It would be for the best if gcc keeps playing his part and remains playable on Gentoo, imho.

Thks 4 ur attention, interest & support.
Back to top
View user's profile Send private message
altblitz
n00b
n00b


Joined: 25 Oct 2016
Posts: 6

PostPosted: Sat Feb 15, 2020 10:22 am    Post subject: Reply with quote

CaptainBlood wrote:
I've noticed the gold linker requirement, installed here.
Didn't force the use of gold linker, though, as metadata doesn't express such a requirement.
It doesn't mean it shouldn't.


It means LTOptimization would left inoperable when linking with ld.fd for that matter.
What is stated in `metadata' is just irrelevant and wrong.

Code:
MARCH=native

OPT_CFLAGS=O3
OPT_LDFLAGS="${OPT_CFLAGS}"
...
CLANG_LTO_CFLAGS="-flto=full"
CLANG_LTO_CXXFLAGS="${CLANG_LTO_CFLAGS}"
if [ "${LLD}" == true ]; then
  CLANG_LLD_LTO_LDFLAGS="${CLANG_LTO_CFLAGS} -Wl,-lto-${OPT_LDFLAGS}"
else
  CLANG_LD_LTO_LDFLAGS="${CLANG_LTO_CFLAGS} -Wl,-plugin-opt=${OPT_LDFLAGS}"
fi

GCC_LTO_CFLAGS="-flto=$(echo $($(which getconf) _NPROCESSORS_ONLN)+2 | bc) -fno-fat-lto-objects"
GCC_LTO_CXXFLAGS="${GCC_LTO_CFLAGS}"
GCC_LTO_LDFLAGS="${GCC_LTO_CFLAGS} -fuse-linker-plugin -Wl,-plugin-opt=-${OPT_LDFLAGS}"
...
  if [ "${LTO}" == true ]; then
    msg2 "Configuring LTO build flags..."
    if [ "${CLANG}" == true ]; then
      CFLAGS="${CFLAGS} ${CLANG_LTO_CFLAGS}"
      CXXFLAGS="${CXXFLAGS} ${CLANG_LTO_CXXFLAGS}"
      LDFLAGS="${LDFLAGS} ${CLANG_LTO_LDFLAGS}"
    else
      CFLAGS="${CFLAGS} ${GCC_LTO_CFLAGS}"
      CXXFLAGS="${CXXFLAGS} ${GCC_LTO_CXXFLAGS}"
      LDFLAGS="${LDFLAGS} ${GCC_LTO_LDFLAGS}"
    fi
  fi


Edit:
Executing LTO+PGO on massive packages such as Firefox, it's not necessary to do LTO on both PGO stages, i.e. instrumentation and generation.
It's sufficient to execute LTO on the second and last stage.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum