[firefox 10] difficult to build

WvR · Posted: Tue Feb 28, 2012 1:44 pm Post subject: [firefox 10] difficult to build

Laptop computer, Compaq, 2GB mem, 4GB swap, 2 CPUs, AMD64.

While compiling Firefox-10, I noticed that my laptop became very slow, up to the point of being almost unresponsive. I was able to go into a text terminal from Gnome (Ctrl-Alt-F1) and log in as root. I used "top" to diagnose the problem and noticed that the linker ("ld") was using a very large amount of memory: 1.4 GB (!). As a result the RAM was overflowing and my computer was swapping a lot and became very slow. I killed ld and let the emerge die.

I rebooted the computer to have a clean RAM, and turned off XDM to reduce memory use. I compiled firefox-10 from a text terminal. At the end of the compilation phase, ld used up to 1500 MB of memory (!). I think this is a very large amount. If you run anything else at the same time, the amount of swapping becomes quite large. Any other users with similar issues? Should this be filed as a bug and if so, where? It is very inconvenient that this ebuild requires so much memory.

franzf · Advocate Joined: 29 Mar 2005 Posts: 4565

https://bugs.gentoo.org/show_bug.cgi?id=402359
Does "MAKEOPTS=-j1" help?

John R. Graham · Posted: Tue Feb 28, 2012 2:56 pm Post subject:

No it doesn't because MAKEOPTS only controls the number of instance of the compiler that are launched in parallel. The link step is a choke point and seeing very large RAM usage on the major packages is normal. LibreOffice is the worst. :wink:

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.

franzf · Advocate Joined: 29 Mar 2005 Posts: 4565

Gusar · Posted: Tue Feb 28, 2012 3:18 pm Post subject:

franzf · Advocate Joined: 29 Mar 2005 Posts: 4565

Etal · Veteran Joined: 15 Jul 2005 Posts: 1931

franzf is right. -jx controls the number of jobs it's doing - it does not matter at all what those jobs are as long as they don't interfere with each other. So if there are 3 different things to link, it will easily use up a lot of RAM by linking them at the same time.
_________________
“And even in authoritarian countries, information networks are helping people discover new facts and making governments more accountable.”– Hillary Clinton, Jan. 21, 2010

John R. Graham · Posted: Tue Feb 28, 2012 4:50 pm Post subject:

Arg. Me spouteth nonsense. Sorry. You're technically exactly correct, but the firefox build does not, in fact, launch multiple instances of ld, thus -j1 can't fix the issue of ld using a lot of memory. The system may have been on the edge of a memory precipice but that would appear to be coincidental.

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.

Gusar · Posted: Tue Feb 28, 2012 5:55 pm Post subject:

kriz · Apprentice Joined: 22 Jan 2003 Posts: 231

i've got a 4 gigs tmpfs partition for /var/tmp/portage and gcc will snatches 2,7 gigs of these during firefox compile

Chiitoo · Posted: Tue Feb 28, 2012 10:09 pm Post subject:

Yep, it started at hmm, Firefox 4? I think.

Around that time any ways.
I had 2 sticks with 2GB each, other one went bad, building stuff like firefox and friends with MAKEOPTS="-j7" would make the system running KDE and whatnots at the same time very slow with just that one stick. I think eventually the build(s) would just die.

I managed to make it more tolerable by reducing the MAKEOPTS="-j7" to around "-j3" and actually build those biggies.

If I remember right, I didn't really try any other solutions, like for example creating more swap space, duh (I still use the first swap partition I created at 0,51 GiB as per handbook

). I did read about 'niceness' and other things, but it was really temporary for me so I didn't bother with more in-depth analysis.

Now I have 7,8 GiB of Memory, so meh, no issues!
Unfortunately I don't remember how much I have managed to have used from that yet but firefox and friends certainly don't do it, and I've tried building them at the same time with other friends, too!
_________________
Kindest of regardses.

WvR · Posted: Tue Feb 28, 2012 11:38 pm Post subject:

WvR · Posted: Tue Feb 28, 2012 11:42 pm Post subject:

WvR · Posted: Tue Feb 28, 2012 11:44 pm Post subject:

John R. Graham · Posted: Wed Feb 29, 2012 12:55 am Post subject:

I didn't say the memory use was coincidental (or I didn't mean that). What I said was that changing MAKEOPTS wouldn't make much of a difference. I too only see a single instance of ld and I too see very large memory consumption.

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.

Gusar · Posted: Wed Feb 29, 2012 2:07 am Post subject:

WvR · Posted: Wed Feb 29, 2012 11:36 pm Post subject:

I don't know what a "benchmark pissing contest" is and I think I don't want to know, either

If the developers have deliberately chosen for the present setup, i.e. with a very large memory requirement, then that means that the developers can also choose otherwise. Let's hope that in a future release of Firefox, there will be an option to select a "not quite so memory intensive compilation"

Gusar · Posted: Wed Feb 29, 2012 11:55 pm Post subject:

You're not familiar with the phrase "pissing contest"? Google it then.

As for your "option"... Why exactly would Mozilla do that? You want them to complicate their build system, which also increases the QA burden and everything... for what exactly?

chaseguard · Tux's lil' helper Joined: 25 Jun 2004 Posts: 140

Are you compiling with pgo use flag? If so, unset it.

John R. Graham · Posted: Thu Mar 01, 2012 5:12 am Post subject:

pgo doesn't affect the link phase.

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.

WvR · Posted: Thu Mar 01, 2012 5:12 am Post subject:

@Gusar: yes, I know what a 'pissing contest' is and yes, I am smart enough to use Google if I don't. I was hoping that the smiley face would convey the mild irony in my message

. Just as it does now

.

You ask why I think the Firefox developers should make an option for a less resource intensive compilation. First of all, in the trade off between putting a burden on the developer(s) or on the users, I think that the burden should be on the developers. At present, the compilation of Firefox is so resource intensive that even on a modern laptop (*) it requires that the user takes special steps to ensure proper memory space etc. And why? Well, I may just switch to something that compiles without needing 2 GB of RAM. And yes, I do realize that the Firefox don't care much about that.

@Chaseguard. I have -pgo, so unfortunately that will not make a difference.

(*) I realize that hardware specifications improve every day. I realize that my laptop is almost 3 years old, and even when I bought it three years ago it was a cheap computer. So maybe 2 GB of RAM is very small by modern standards. I don't know, I don't keep track of developments of computer hardware. But I am a professional user of many, very large numerical codes. I deal with calculations running for months on tens of CPUs with many GB of memory on a frequent basis. I was simply surprised that software like Firefox, which is merely a web browser, more or less a toy, would need so many resources to compile.

dmpogo · Advocate Joined: 02 Sep 2004 Posts: 3266 Location: Canada

dmpogo · Advocate Joined: 02 Sep 2004 Posts: 3266 Location: Canada

xiber · Posted: Thu Mar 01, 2012 5:59 am Post subject:

Wow, thought it was only me having this ld issue. Been happening since firefox 9 for me. System grinds to a halt with 1GB of ram. After ram and swap are exausted, emerge fails. The only way I've been able to compile firefox is to shut everything down, including the xserver, and use a vt. I use tmux and let it compile while I'm at work. I've started to dread those little firefox point releases.
_________________
Athlon XP-M 2600 @ 2.3 GHz OC | Abit NF7-S r2.0 | 2x512MB PC3200 | 6600GT OC | Audigy 2 | Gentoo | 2005

Athlon 64 X2 4600 @ 2.4 GHz | Asus M2N-SLI DLX | 4x1GB PC6400 | 7600GT KO | 7HD @ 3.1TB | OpenSolaris SXCE | 2007

WvR · Posted: Thu Mar 01, 2012 7:16 am Post subject: