View previous topic :: View next topic |
Author |
Message |
WvR Apprentice
Joined: 03 Mar 2011 Posts: 200 Location: Tsuruga, Japan
|
Posted: Tue Feb 28, 2012 1:44 pm Post subject: [firefox 10] difficult to build |
|
|
Laptop computer, Compaq, 2GB mem, 4GB swap, 2 CPUs, AMD64.
While compiling Firefox-10, I noticed that my laptop became very slow, up to the point of being almost unresponsive. I was able to go into a text terminal from Gnome (Ctrl-Alt-F1) and log in as root. I used "top" to diagnose the problem and noticed that the linker ("ld") was using a very large amount of memory: 1.4 GB (!). As a result the RAM was overflowing and my computer was swapping a lot and became very slow. I killed ld and let the emerge die.
I rebooted the computer to have a clean RAM, and turned off XDM to reduce memory use. I compiled firefox-10 from a text terminal. At the end of the compilation phase, ld used up to 1500 MB of memory (!). I think this is a very large amount. If you run anything else at the same time, the amount of swapping becomes quite large. Any other users with similar issues? Should this be filed as a bug and if so, where? It is very inconvenient that this ebuild requires so much memory. |
|
Back to top |
|
|
franzf Advocate
Joined: 29 Mar 2005 Posts: 4565
|
|
Back to top |
|
|
John R. Graham Administrator
Joined: 08 Mar 2005 Posts: 10590 Location: Somewhere over Atlanta, Georgia
|
Posted: Tue Feb 28, 2012 2:56 pm Post subject: |
|
|
No it doesn't because MAKEOPTS only controls the number of instance of the compiler that are launched in parallel. The link step is a choke point and seeing very large RAM usage on the major packages is normal. LibreOffice is the worst.
- John _________________ I can confirm that I have received between 0 and 499 National Security Letters. |
|
Back to top |
|
|
franzf Advocate
Joined: 29 Mar 2005 Posts: 4565
|
Posted: Tue Feb 28, 2012 3:05 pm Post subject: |
|
|
John R. Graham wrote: | No it doesn't because MAKEOPTS only controls the number of instance of the compiler that are launched in parallel. The link step is a choke point and seeing very large RAM usage on the major packages is normal. LibreOffice is the worst.
- John |
I know, but the bug did not appear for the reporter, when he switched to -j1, and so the bug got closed. And libreoffice does not make any problems here - no extrem RAM-usage, System stays usable all the time.
firefox-10.0 was the first time this problem appeared to me, it never did before. When you read the report, you see I experienced a high RAM-Usage for python. The update to 10.0.1 also made PC-usage impossible. So this probably is a real issue. Not being able to build firefox with 4GB of RAM - a joke |
|
Back to top |
|
|
Gusar Advocate
Joined: 09 Apr 2005 Posts: 2665 Location: Slovenia
|
Posted: Tue Feb 28, 2012 3:18 pm Post subject: |
|
|
franzf wrote: | Not being able to build firefox with 4GB of RAM - a joke |
The OP has 2GB of RAM. On 64bit, that's just not enough for big compilations. Yeah, compiling Firefox is resource intensive, but the modern web and the JS benchmark pissing contests demand it. |
|
Back to top |
|
|
franzf Advocate
Joined: 29 Mar 2005 Posts: 4565
|
Posted: Tue Feb 28, 2012 3:21 pm Post subject: |
|
|
Gusar wrote: | franzf wrote: | Not being able to build firefox with 4GB of RAM - a joke |
The OP has 2GB of RAM. On 64bit, that's just not enough for big compilations. |
I have 4GB, and it swapped like hell! I have 4GB of swap, so it could finish... |
|
Back to top |
|
|
Etal Veteran
Joined: 15 Jul 2005 Posts: 1931
|
Posted: Tue Feb 28, 2012 3:24 pm Post subject: |
|
|
franzf is right. -jx controls the number of jobs it's doing - it does not matter at all what those jobs are as long as they don't interfere with each other. So if there are 3 different things to link, it will easily use up a lot of RAM by linking them at the same time. _________________ “And even in authoritarian countries, information networks are helping people discover new facts and making governments more accountable.”– Hillary Clinton, Jan. 21, 2010 |
|
Back to top |
|
|
John R. Graham Administrator
Joined: 08 Mar 2005 Posts: 10590 Location: Somewhere over Atlanta, Georgia
|
Posted: Tue Feb 28, 2012 4:50 pm Post subject: |
|
|
Arg. Me spouteth nonsense. Sorry. You're technically exactly correct, but the firefox build does not, in fact, launch multiple instances of ld, thus -j1 can't fix the issue of ld using a lot of memory. The system may have been on the edge of a memory precipice but that would appear to be coincidental.
- John _________________ I can confirm that I have received between 0 and 499 National Security Letters. |
|
Back to top |
|
|
Gusar Advocate
Joined: 09 Apr 2005 Posts: 2665 Location: Slovenia
|
Posted: Tue Feb 28, 2012 5:55 pm Post subject: |
|
|
franzf wrote: | I have 4GB, and it swapped like hell! I have 4GB of swap, so it could finish... |
Well, 64bit sucks then . I use 32bit and have 2GB of RAM, swap isn't even touched during a Firefox compile. |
|
Back to top |
|
|
kriz Apprentice
Joined: 22 Jan 2003 Posts: 231
|
Posted: Tue Feb 28, 2012 8:31 pm Post subject: |
|
|
i've got a 4 gigs tmpfs partition for /var/tmp/portage and gcc will snatches 2,7 gigs of these during firefox compile |
|
Back to top |
|
|
Chiitoo Administrator
Joined: 28 Feb 2010 Posts: 2596 Location: Here and Away Again
|
Posted: Tue Feb 28, 2012 10:09 pm Post subject: |
|
|
Yep, it started at hmm, Firefox 4? I think.
Around that time any ways.
I had 2 sticks with 2GB each, other one went bad, building stuff like firefox and friends with MAKEOPTS="-j7" would make the system running KDE and whatnots at the same time very slow with just that one stick. I think eventually the build(s) would just die.
I managed to make it more tolerable by reducing the MAKEOPTS="-j7" to around "-j3" and actually build those biggies.
If I remember right, I didn't really try any other solutions, like for example creating more swap space, duh (I still use the first swap partition I created at 0,51 GiB as per handbook ). I did read about 'niceness' and other things, but it was really temporary for me so I didn't bother with more in-depth analysis.
Now I have 7,8 GiB of Memory, so meh, no issues!
Unfortunately I don't remember how much I have managed to have used from that yet but firefox and friends certainly don't do it, and I've tried building them at the same time with other friends, too! _________________ Kindest of regardses. |
|
Back to top |
|
|
WvR Apprentice
Joined: 03 Mar 2011 Posts: 200 Location: Tsuruga, Japan
|
Posted: Tue Feb 28, 2012 11:38 pm Post subject: |
|
|
Gusar wrote: | franzf wrote: | Not being able to build firefox with 4GB of RAM - a joke |
The OP has 2GB of RAM. On 64bit, that's just not enough for big compilations. Yeah, compiling Firefox is resource intensive, but the modern web and the JS benchmark pissing contests demand it. |
With 2 GB of RAM I have not experienced a lack of RAM during a compilation before. OpenOffice, LibreOffice, Gnome, I have compiled many packages on this machine and never any problems. But yesterday with Firefox, 100% RAM use, extreme swapping, a load of 25.00+ for more than 30 minutes.... extremely resource intensive. I have a feeling that there is an error in a Makefile or a memory leak somewhere.
I have MAKEOPTS=-j3 (dual core CPU); I could try MAKEOPTS=-j1 to see if it makes any difference. As far as I could tell from "top", there was only one "ld" process (i.e. only one PID) so I don't expect much difference
Packages requiring more than 2 GB of RAM to compile - that indicates to me an error by the developer. I am sure some optimization can be done in the Makefiles to make linking steps less resource intensive. |
|
Back to top |
|
|
WvR Apprentice
Joined: 03 Mar 2011 Posts: 200 Location: Tsuruga, Japan
|
Posted: Tue Feb 28, 2012 11:42 pm Post subject: |
|
|
John R. Graham wrote: | Arg. Me spouteth nonsense. Sorry. You're technically exactly correct, but the firefox build does not, in fact, launch multiple instances of ld, thus -j1 can't fix the issue of ld using a lot of memory. The system may have been on the edge of a memory precipice but that would appear to be coincidental.
- John |
I compiled Firefox three times. The first two attempts I aborted during the linking step. In all cases, ld memory use went up to near 1500 MB. That's why I decided to completely reboot: make the RAM as empty as possible, and stop all unnecessary processes. So the memory use issue is certainly not coincidental. |
|
Back to top |
|
|
WvR Apprentice
Joined: 03 Mar 2011 Posts: 200 Location: Tsuruga, Japan
|
Posted: Tue Feb 28, 2012 11:44 pm Post subject: |
|
|
Etal wrote: | franzf is right. -jx controls the number of jobs it's doing - it does not matter at all what those jobs are as long as they don't interfere with each other. So if there are 3 different things to link, it will easily use up a lot of RAM by linking them at the same time. |
If there are three "ld" processes simultaneously, then if each instance uses 10% of RAM, you'd indeed be using 30% of RAM. But in my case, there was only one "ld" process, and it used nearly 80% of RAM..... |
|
Back to top |
|
|
John R. Graham Administrator
Joined: 08 Mar 2005 Posts: 10590 Location: Somewhere over Atlanta, Georgia
|
Posted: Wed Feb 29, 2012 12:55 am Post subject: |
|
|
I didn't say the memory use was coincidental (or I didn't mean that). What I said was that changing MAKEOPTS wouldn't make much of a difference. I too only see a single instance of ld and I too see very large memory consumption.
- John _________________ I can confirm that I have received between 0 and 499 National Security Letters. |
|
Back to top |
|
|
Gusar Advocate
Joined: 09 Apr 2005 Posts: 2665 Location: Slovenia
|
Posted: Wed Feb 29, 2012 2:07 am Post subject: |
|
|
WvR wrote: | Packages requiring more than 2 GB of RAM to compile - that indicates to me an error by the developer. |
Like I said, it's mandated by the JS benchmark pissing contests between browsers. The Mozilla folks had big issues on their Windows build boxes recently, running out of memory during compile. A temporary fix was exactly what you propose - "Makefile optimizations" - in particular moving some stuff out of libxul into separate libraries. Keyword *temporary*. So what you attribute to "developer error" is actually very deliberate.
The developers deliberately prioritize pissing contests over "Makefile optimizations". You can thank the masses for that, who are wooed by pretty benchmark numbers, oblivious to how little these actually affect their daily browsing. But placebo is a powerful thing, and so Mozilla has to cater to those masses, even if it means ridiculous RAM requirements for compiles. |
|
Back to top |
|
|
WvR Apprentice
Joined: 03 Mar 2011 Posts: 200 Location: Tsuruga, Japan
|
Posted: Wed Feb 29, 2012 11:36 pm Post subject: |
|
|
I don't know what a "benchmark pissing contest" is and I think I don't want to know, either If the developers have deliberately chosen for the present setup, i.e. with a very large memory requirement, then that means that the developers can also choose otherwise. Let's hope that in a future release of Firefox, there will be an option to select a "not quite so memory intensive compilation" |
|
Back to top |
|
|
Gusar Advocate
Joined: 09 Apr 2005 Posts: 2665 Location: Slovenia
|
Posted: Wed Feb 29, 2012 11:55 pm Post subject: |
|
|
You're not familiar with the phrase "pissing contest"? Google it then.
As for your "option"... Why exactly would Mozilla do that? You want them to complicate their build system, which also increases the QA burden and everything... for what exactly? |
|
Back to top |
|
|
chaseguard Tux's lil' helper
Joined: 25 Jun 2004 Posts: 140
|
Posted: Thu Mar 01, 2012 4:36 am Post subject: |
|
|
Are you compiling with pgo use flag? If so, unset it. |
|
Back to top |
|
|
John R. Graham Administrator
Joined: 08 Mar 2005 Posts: 10590 Location: Somewhere over Atlanta, Georgia
|
Posted: Thu Mar 01, 2012 5:12 am Post subject: |
|
|
pgo doesn't affect the link phase.
- John _________________ I can confirm that I have received between 0 and 499 National Security Letters. |
|
Back to top |
|
|
WvR Apprentice
Joined: 03 Mar 2011 Posts: 200 Location: Tsuruga, Japan
|
Posted: Thu Mar 01, 2012 5:12 am Post subject: |
|
|
@Gusar: yes, I know what a 'pissing contest' is and yes, I am smart enough to use Google if I don't. I was hoping that the smiley face would convey the mild irony in my message . Just as it does now .
You ask why I think the Firefox developers should make an option for a less resource intensive compilation. First of all, in the trade off between putting a burden on the developer(s) or on the users, I think that the burden should be on the developers. At present, the compilation of Firefox is so resource intensive that even on a modern laptop (*) it requires that the user takes special steps to ensure proper memory space etc. And why? Well, I may just switch to something that compiles without needing 2 GB of RAM. And yes, I do realize that the Firefox don't care much about that.
@Chaseguard. I have -pgo, so unfortunately that will not make a difference.
(*) I realize that hardware specifications improve every day. I realize that my laptop is almost 3 years old, and even when I bought it three years ago it was a cheap computer. So maybe 2 GB of RAM is very small by modern standards. I don't know, I don't keep track of developments of computer hardware. But I am a professional user of many, very large numerical codes. I deal with calculations running for months on tens of CPUs with many GB of memory on a frequent basis. I was simply surprised that software like Firefox, which is merely a web browser, more or less a toy, would need so many resources to compile. |
|
Back to top |
|
|
dmpogo Advocate
Joined: 02 Sep 2004 Posts: 3269 Location: Canada
|
Posted: Thu Mar 01, 2012 5:38 am Post subject: |
|
|
Gusar wrote: | franzf wrote: | Not being able to build firefox with 4GB of RAM - a joke |
The OP has 2GB of RAM. On 64bit, that's just not enough for big compilations. Yeah, compiling Firefox is resource intensive, but the modern web and the JS benchmark pissing contests demand it. |
Yes, but 64-bit firefox-9.0 compiled succesfully in 1.5 GB ramdisk for me, without touching swap. And firefox-10 wants 4 Gb somehow. Somebody got inventive with the build process. |
|
Back to top |
|
|
dmpogo Advocate
Joined: 02 Sep 2004 Posts: 3269 Location: Canada
|
Posted: Thu Mar 01, 2012 5:53 am Post subject: |
|
|
WvR wrote: |
You ask why I think the Firefox developers should make an option for a less resource intensive compilation. First of all, in the trade off between putting a burden on the developer(s) or on the users, I think that the burden should be on the developers. At present, the compilation of Firefox is so resource intensive that even on a modern laptop (*) it requires that the user takes special steps to ensure proper memory space etc. And why? Well, I may just switch to something that compiles without needing 2 GB of RAM. And yes, I do realize that the Firefox don't care much about that.
|
We do have such an option. It is firefox-bin and thunderbird-bin. There the developers carried the burden of compilation. I switched to that starting with unwieldy version 10. |
|
Back to top |
|
|
xiber Apprentice
Joined: 28 Oct 2003 Posts: 245 Location: Fremont, CA
|
Posted: Thu Mar 01, 2012 5:59 am Post subject: |
|
|
Wow, thought it was only me having this ld issue. Been happening since firefox 9 for me. System grinds to a halt with 1GB of ram. After ram and swap are exausted, emerge fails. The only way I've been able to compile firefox is to shut everything down, including the xserver, and use a vt. I use tmux and let it compile while I'm at work. I've started to dread those little firefox point releases. _________________ Athlon XP-M 2600 @ 2.3 GHz OC | Abit NF7-S r2.0 | 2x512MB PC3200 | 6600GT OC | Audigy 2 | Gentoo | 2005
Athlon 64 X2 4600 @ 2.4 GHz | Asus M2N-SLI DLX | 4x1GB PC6400 | 7600GT KO | 7HD @ 3.1TB | OpenSolaris SXCE | 2007 |
|
Back to top |
|
|
WvR Apprentice
Joined: 03 Mar 2011 Posts: 200 Location: Tsuruga, Japan
|
Posted: Thu Mar 01, 2012 7:16 am Post subject: |
|
|
dmpogo wrote: | We do have such an option. It is firefox-bin and thunderbird-bin. There the developers carried the burden of compilation. I switched to that starting with unwieldy version 10. |
I know there are binary versions available for some of the packages which are considered a burden to compile. But let's be honest, those packages run counter to the Gentoo philosophy. If you start using binary packages, you might as well take the plunge and move entirely to Ubuntu .
And once again I would like to point out that I am __not__ complaining about Firefox behavior. I wanted to point out my problem to see if I was the only one. Apparently, I am not the only, which means that the issue is not related to my hardware or my gentoo installation . |
|
Back to top |
|
|
|