Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Firefox will default to Clang+LTO compiler?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
wrc1944
Advocate
Advocate


Joined: 15 Aug 2002
Posts: 3182
Location: Gainesville, Florida

PostPosted: Wed Sep 19, 2018 6:57 pm    Post subject: Firefox will default to Clang+LTO compiler? Reply with quote

Just saw this post on Phoronix, which prompted a few questions: https://www.phoronix.com/scan.php?page=news_item&px=Firefox-Clang-LTO-All-Platforms
1. Since Firefox will now default to Clang, does this relate or affect Gentoo users using GCC?
2. If your entire system is built with global -PIE, would it matter? I assume not, but am not sure.
3. Is building only Firefox with Clang on a Gentoo GCC system advisible (or not), or even plausible? In other words, is building specific packages with either Clang or GCC supported on Gentoo, or other distros?
4. Would all the firefox deps need to be compiled with Clang?
5. Does Gentoo plan on any special Firefox patches to address any conflicts, or am I overly concerned with the change?
Quote:
Firefox Is Now Built With Clang+LTO Everywhere, Sizable Performance Wins For Linux
Written by Michael Larabel in Mozilla on 12 September 2018 at 05:33 AM EDT.

Firefox nightly builds are now built with the LLVM Clang compiler on all major platforms and the Linux build in particular is also now utilizing PGO optimizations too. Faster Firefox is coming thanks to this compiler work.

All of Mozilla's tier-one platforms are now building the newest Firefox browser code under the Clang compiler and having LTO (Link Time Optimizations) enabled. That includes Linux, Mac, Android, Windows across ARM / AArch64 / x86 relying upon this open-source compiler. For now only the Linux builds also have PGO (Profile Guided Optimizations) enabled.

With the compiler re-tooling on Linux they are seeing about 5% better performance and upwards of 18% in select tests compared to their previous GCC (6.4-based) compiler with PGO. The current Clang stack they are using is based on LLVM6 while when they transition to the upcoming LLVM/Clang 7.0 they expect 2~5% better performance on top of that.

Mozilla developers did try GCC LTO as well as switching to the newest GCC8 release, but ran into problems. These new release binaries should go into effect for the Firefox 64 release.

As not a performance win but a security win, Firefox Linux builds are also finally being built as Position Intependent Executables (PIE).

More details on these exciting compiler advancements for Firefox via this blog post.
https://glandium.org/blog/?p=3888
Quote:
Firefox is now built with clang LTO on all* platforms

You might have read that Mozilla recently switched Windows builds to clang-cl. More recently, those Windows builds have seen both PGO (Profile-Guided Optimization) and LTO (Link-Time Optimization) enabled.

As of next nightly (as of writing, obviously), all tier-1 platforms are now built with clang with LTO enabled. Yes, this means Linux, Mac and Android arm, aarch64 and x86. Linux builds also have PGO enabled.

Mac and Android builds were already using clang, so the only difference is LTO being enabled, which brought some performance improvements.

The most impressive difference, though, was on Linux, where we’re getting more than 5% performance improvements on most Talos tests (up to 18% (!) on some tests) compared to GCC 6.4 with PGO. I must say I wasn’t expecting switching from GCC to clang would make such a difference. And that is with clang 6. A quick test with upcoming clang 7 suggests we’d additionally get between 2 and 5% performance improvement from an upgrade, but our static analysis plugin doesn’t like it.

This doesn’t mean GCC is being unsupported. As a matter of fact, we still have automated jobs using GCC for some static analysis, and we also have jobs ensuring everything still builds with a baseline of GCC 6.x.

You might wonder if we tried LTO with GCC, or tried upgrading to GCC 8.x. As a matter of fact, I did. Enabling LTO turned up linker errors, and upgrading to GCC 7.x turned up breaking binary compatibility with older systems, and if I remember correctly had some problems with our test suite. GCC 8.1 was barely out when I was looking into this, and we all know to stay away from any new major GCC version until one or two minor updates. Considering the expected future advantages from using clang (cross-language inlining with Rust, consistency between platforms), it seemed a better deal to switch to clang than to try to address those issues.

Update: As there’s been some interest on reddit and HN, and I failed to mention it originally, it’s worth noting that comparing GCC+PGO vs. clang+LTO or GCC+PGO vs. clang+PGO was a win for clang overall in both cases, although GCC was winning on a few benchmarks. If I remember correctly, clang without PGO/LTO was also winning against GCC without PGO.

Anyways, what led me on this quest was a casual conversation at our last All Hands, where we were discussing possibly turning on LTO on Mac, and how that should roughly just be about turning a switch.

Famous last words.

At least, that’s a somehow reasonable assumption. But when you have a codebase the size of Firefox, you’re up for “interesting” discoveries.

This involved compiler bugs, linker bugs (with a special mention for a bug in ld64 that Apple has apparently fixed in Xcode 9 but hasn’t released the source of), build system problems, elfhack issues, crash report problems, clang plugin problems (would you have guessed that __attribute__((annotate("foo"))) can affect the generated machine code?), sccache issues, inline assembly bugs (getting inputs, outputs and clobbers correctly is hard), binutils bugs, and more.

I won’t bother you with all the details, but here we are, 3 months later with it all, finally, mostly done. Counting only the bugs assigned to me, there are 77 bugs on bugzilla (so, leaving out anything in other bug trackers, like LLVM’s). Some of them relied on work from other people (most notably, Nathan Froyd’s work to switch to clang and then non-NDK clang on Android). This spread over about 150 commits on mozilla-central, 20 of which were backouts. Not everything went according to plan, obviously, although some of those backouts were on purpose as a taskcluster trick.

Hopefully, this sticks, and Firefox 64 will ship built with clang with LTO on all tier-1 platforms as well as PGO on some. Downstreams are encouraged to do the same if they can. The build system will soon choose clang by default on all builds, but won’t enable PGO/LTO.

As a bonus, as of a few days ago, Linux builds are also finally using Position Independent Executables, which improves Address Space Layout Randomization for the few things that are in the executables instead of some library (most notably, mozglue and the allocator). This was actually necessary for LTO, because clang doesn’t build position independent code in executables that are not PIE (but GCC does), and that causes other problems.

Work is not entirely over, though, as more inline assembly bugs might be remaining only not causing visible problems by sheer luck, so I’m now working on a systematic analysis of inline assembly blocks with our clang plugin.

2018-09-12 17:10:49+0900

_________________
Main box- AsRock x370 Gaming K4
Ryzen 1700, 3.0GHz, 16GB GSkill Flare DDR4 3200mhz
Samsung SATA 1000GB, Radeon HD R7 350 2GB DDR5
Gentoo ~amd64 plasma, glibc-2.27-r6, gcc-8.2.0-r3 kernel-4.18.15-gentoo USE=experimental


Last edited by wrc1944 on Wed Sep 19, 2018 7:30 pm; edited 1 time in total
Back to top
View user's profile Send private message
mike155
l33t
l33t


Joined: 17 Sep 2010
Posts: 625
Location: Frankfurt, Germany

PostPosted: Wed Sep 19, 2018 7:19 pm    Post subject: Reply with quote

You should also quote the paragraph below from https://glandium.org/blog/?p=3888:
Quote:
This doesn’t mean GCC is being unsupported. As a matter of fact, we still have automated jobs using GCC for some static analysis, and we also have jobs ensuring everything still builds with a baseline of GCC 6.x.
Back to top
View user's profile Send private message
wrc1944
Advocate
Advocate


Joined: 15 Aug 2002
Posts: 3182
Location: Gainesville, Florida

PostPosted: Wed Sep 19, 2018 11:36 pm    Post subject: Reply with quote

https://wiki.gentoo.org/wiki/Clang#GCC_fallback_environments gives good info, and apparently you can:
Quote:
Clang environments

Now that we've set up a safe fallback we can proceed to enable the usage of Clang in Gentoo. There are two ways to do this: System wide using /etc/portage/make.conf or via environmental variables like the one(s) we created for the GCC fallback.

We'll use the same process as we did earlier in the article for setting up GCC fallbacks.
FILE /etc/portage/env/compiler-clangEnvironment variable named compiler-clang

CC="clang"
CXX="clang++"

You can now use Clang on a per package basis by invoking the compiler-clang environmental variable you created.
FILE /etc/portage/package.envUsing the Clang compiler for app-foo/bar and app-bar/baz


app-foo/bar compiler-clang
app-bar/baz compiler-clang

Of course, that's about as basic as it gets. You can create more complex environments if you wish. It's a good idea to have a variable for link-time optimized Clang and one without if you want to take advantage of LTO as detailed later in the article. Simply create two environments, one with the flag -flto and one with -fno-lto and apply each to packages as needed.


I guess the idea would be on a GCC compiled system to setup Firefox (and all its deps ???) for a Clang environment. Probably more complex, especially if some Firefox deps were also used by other GCC compiled apps? Now wondering if it's worth the trouble.
_________________
Main box- AsRock x370 Gaming K4
Ryzen 1700, 3.0GHz, 16GB GSkill Flare DDR4 3200mhz
Samsung SATA 1000GB, Radeon HD R7 350 2GB DDR5
Gentoo ~amd64 plasma, glibc-2.27-r6, gcc-8.2.0-r3 kernel-4.18.15-gentoo USE=experimental
Back to top
View user's profile Send private message
Juippisi
Apprentice
Apprentice


Joined: 30 Sep 2005
Posts: 288
Location: /home

PostPosted: Thu Sep 20, 2018 6:13 am    Post subject: Reply with quote

I was under the impression that firefox has been building itself with clang since 57 release? At least it's a hard dep ever since.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum