| View previous topic :: View next topic |
| Author |
Message |
sphakka
Tux's lil' helper
Joined: 24 Jun 2003
Posts: 79
|
 Posted: Mon May 21, 2012 9:25 am Post subject: emerging SeaMonkey: ld takes ages and eats tons of memory |
 |
|
Hi there,
I noticed that emerging SeaMonkey takes longer and longer (since rel. 2.4 to current 2.9). While monitoring last builds, I could see ld running long (~30-45' or more) and eating a lot of memory (up to 1.4GiB), to the point that my laptop starts swapping (I have 2.3GiB)... I can stand long linking times as it's mostly IO, but the huge memory footprint worries me a bit.
I know it's a big package with lots of deps, though I wonder if there's anything wrong with it or with my set-up. Apparently, no other build causes the same.
Here's a genlop extract:
| Code: |
Fri Oct 28 16:34:36 2011 >>> www-client/seamonkey-2.3.1
merge time: 1 hour, 8 minutes and 38 seconds.
Fri Oct 28 20:26:14 2011 >>> www-client/seamonkey-2.4.1-r1
merge time: 1 hour, 11 minutes and 47 seconds.
Wed Jan 11 20:48:31 2012 >>> www-client/seamonkey-2.6.1
merge time: 1 hour, 45 minutes and 33 seconds.
Fri Feb 17 21:58:39 2012 >>> www-client/seamonkey-2.7.1
merge time: 1 hour, 38 minutes and 36 seconds.
Thu Mar 1 21:17:14 2012 >>> www-client/seamonkey-2.7.1-r1
merge time: 1 hour, 37 minutes and 46 seconds.
Thu Mar 22 21:19:23 2012 >>> www-client/seamonkey-2.8
merge time: 1 hour, 55 minutes and 19 seconds.
Mon Apr 30 20:18:40 2012 >>> www-client/seamonkey-2.9
merge time: 1 hour, 59 minutes and 41 seconds.
Sun May 20 22:47:24 2012 >>> www-client/seamonkey-2.9
merge time: 1 hour, 54 minutes.
|
Note that at the first sign of swap-out I close stuff to free memory, thus the merge times are good approximates of wall times (the OS might be trashing for a couple of minutes though).
Any idea?
Cheers,
^s |
|
| Back to top |
|
 |
Polynomial-C
Retired Dev
Joined: 01 Jun 2003
Posts: 1432
Location: Germany
|
|
| Back to top |
|
|
sphakka
Tux's lil' helper
Joined: 24 Jun 2003
Posts: 79
|
| Posted: Mon May 21, 2012 10:03 am Post subject: |
|
|
Thanks Polynomial-C.
Given that core stuff is shared among Mozilla's apps, I guess it's the same issue. |
|
| Back to top |
|
|
Ant P.
Watchman
Joined: 18 Apr 2009
Posts: 6920
|
| Posted: Mon May 21, 2012 2:33 pm Post subject: |
|
|
| Hopefully it won't get any worse at this point, since some of the servers in Mozilla's own(!) build farm can't cope with the bloat any more. |
|
| Back to top |
|
|
sphakka
Tux's lil' helper
Joined: 24 Jun 2003
Posts: 79
|
| Posted: Mon May 21, 2012 8:57 pm Post subject: |
|
|
I wrote a Perl script to trace emerge time stats for all the version of SeaMonkey I built on my laptop (2 x AMD Turion 1.6, 2.3GiB RAM). Here's the result:
| Code: |
$ ./emergetstat seamonkey
Emerge time stats for 'seamonkey':
------------------------------------------------------
version : h m s : tot m
------------------------------------------------------
1.1.1 : 1h 0m 13s : 60.22m ***********
1.1.10 : 0h 24m 24s : 24.40m *****
1.1.11 : 0h 26m 0s : 26.00m *****
1.1.12 : 0h 25m 53s : 25.88m *****
1.1.13 : 0h 22m 44s : 22.73m ****
1.1.14 : 0h 23m 43s : 23.72m ****
1.1.15 : 0h 31m 44s : 31.73m ******
1.1.17 : 0h 24m 29s : 24.48m *****
1.1.18 : 0h 39m 7s : 39.12m *******
1.1.2 : 1h 1m 27s : 61.45m ***********
1.1.3 : 0h 28m 46s : 28.77m *****
1.1.4 : 0h 26m 45s : 26.75m *****
1.1.5 : 0h 24m 12s : 24.20m *****
1.1.6 : 0h 30m 41s : 30.68m ******
1.1.7 : 0h 25m 10s : 25.17m *****
1.1.8 : 0h 25m 46s : 25.77m *****
1.1.9 : 0h 24m 59s : 24.98m *****
1.1.9-r1 : 0h 23m 21s : 23.35m ****
2.0.10 : 0h 48m 24s : 48.40m *********
2.0.11 : 0h 47m 44s : 47.73m ********
2.0.11-r1 : 0h 46m 31s : 46.52m ********
2.0.11-r2 : 0h 59m 19s : 59.32m **********
2.0.12 : 0h 59m 8s : 59.13m **********
2.0.13 : 0h 53m 51s : 53.85m *********
2.0.14 : 0h 48m 20s : 48.33m *********
2.0.14-r1 : 0h 46m 8s : 46.13m ********
2.0.4-r1 : 0h 46m 16s : 46.27m ********
2.0.4-r2 : 0h 46m 18s : 46.30m ********
2.0.5 : 0h 46m 30s : 46.50m ********
2.0.6 : 0h 45m 50s : 45.83m ********
2.0.7 : 0h 44m 1s : 44.02m ********
2.0.8 : 0h 45m 24s : 45.40m ********
2.0.8-r1 : 0h 44m 48s : 44.80m ********
2.0.9 : 0h 50m 3s : 50.05m *********
2.1 : 0h 47m 58s : 47.97m *********
2.2-r2 : 1h 7m 24s : 67.40m ************
2.3.1 : 1h 8m 38s : 68.63m ************
2.3.2 : 0h 56m 35s : 56.58m **********
2.3.3 : 1h 1m 3s : 61.05m ***********
2.4.1-r1 : 1h 11m 47s : 71.78m ************
2.6.1 : 1h 45m 33s : 105.55m ******************
2.7.1 : 1h 38m 36s : 98.60m *****************
2.7.1-r1 : 1h 37m 46s : 97.77m *****************
2.8 : 1h 55m 19s : 115.32m ********************
2.9 : 1h 54m 0s : 114.00m ********************
|
apart from some spourious samples, the trend is clear: build time ~doubled between 1.1 and 2.0 branches, and again starting from 2.6. I didn't check the dates, though I guess that's worse than Moore's law... Worrysome, to say the least 
Of course, I'll post the plot on mozillazine.
And here's the script (based on app-portage/genlop)
| Code: |
#!/usr/bin/perl
################################################################################
# Copyright 2012 'sphakka' @ <forums.gentoo.org>
#
# This is emergetstat.
#
# emergetstat is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# emergetstat is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with emergetstat. If not, see <http://www.gnu.org/licenses/>.
################################################################################
use warnings;
use POSIX;
my $ebuild = $ARGV[0];
my @genlop_info=qx(genlop -nt $ebuild);
my %stats = (
# ver => ( h, m, s, totm ),
);
# max time stat, moving average, #samples, time stat sum
my ($maxt, $tavg, $nsmp, $tsum) = (0, 0, 1, 0);
# "suspicious" threshold factor
my $stfac = 10;
while (my $line = shift @genlop_info) {
chomp $line;
if ((my $ver) = ($line =~ /^\s*[MTWFS].+${ebuild}-(\S+)$/)) {
$stat = shift @genlop_info;
$stat =~ /^.+?(?:(\d+)\s+hour.*?)?(?:(\d+)\s+minute.*?)?(?:(\d+)\s+second.*?)?$/ && do {
$h = ($1 || 0); $m = ($2 || 0); $s = ($3 || 0);
$totm = ($h*60+$m+$s/60);
# moving average initialized here to allow filtering in the first sample
$tavg = $totm if ($nsmp == 1);
# discard suspicious (i.e. too big) samples
if ($totm <= $tavg*$stfac) {
$nsmp++;
$tsum += $totm;
$tavg = $tsum/$nsmp;
$maxt = $totm if ($totm > $maxt);
@{$stats{$ver}} = ($h, $m, $s, $totm);
}
}
}
}
# stats are normalized to $maxt and represented as bars in [0, $maxs]
my $hfmt = "%-20s: %3s %3s %3s : %6s\n";
my $lfmt = "%-20s: %2dh %2dm %2ds : %6.2fm %-30s\n";
my $hsep = ('-' x 64) . "\n";
print "Emerge time stats for '$ebuild':\n";
print $hsep;
printf $hfmt, 'version', 'h', 'm', 's', 'tot m';
print $hsep;
my $maxs = 20;
for my $v (sort keys %stats) {
my ($h, $m, $s, $t) = @{$stats{$v}};
my $bar = '*' x ceil($t/$maxt * $maxs);
printf $lfmt, $v, $h, $m, $s, $t, $bar;
}
exit 0;
|
|
|
| Back to top |
|
|
Princess Nell
l33t
Joined: 15 Apr 2005
Posts: 918
|
| Posted: Tue May 22, 2012 8:09 am Post subject: |
|
|
Alternatively, you could have emerged genlop 
| Code: |
~ $ eix genlop
[I] app-portage/genlop
Available versions: 0.30.5 0.30.7 ~0.30.8 0.30.8-r1 0.30.8-r2 {bash-completion}
Installed versions: 0.30.8-r2(04:29:14 04/10/10)(bash-completion)
Homepage: http://www.gentoo.org/proj/en/perl
Description: A nice emerge.log parser
~ $
|
I recently added another 2GB of RAM because 2GB wasn't any longer enough to compile ff and friends without massive swapping. |
|
| Back to top |
|
|
sphakka
Tux's lil' helper
Joined: 24 Jun 2003
Posts: 79
|
| Posted: Tue May 22, 2012 8:40 am Post subject: |
|
|
| Princess Nell wrote: | Alternatively, you could have emerged genlop
|
Indeed, that's what my script uses to get relevant info, which then it properly mangles to get some more machine-crunchable numbers.
| Quote: |
I recently added another 2GB of RAM because 2GB wasn't any longer enough to compile ff and friends without massive swapping.
|
I did it *years before* FF & Co. got bloated, and never thought that compiling that stuff would become my swap partition _killer_ app. BTW, I'm worried about the road to Bloat City which Mozilla is now on... as for building heavy stuff, I'm OK switching to console mode and let the box crunch code all the night long. |
|
| Back to top |
|
|
Gusar
Advocate
Joined: 09 Apr 2005
Posts: 2665
Location: Slovenia
|
| Posted: Tue May 22, 2012 10:27 am Post subject: |
|
|
It seems with all this criticism of Mozilla, you guys haven't noticed that Chrome/Chromium packs *even more* stuff in to a single binary and is therefore just as resource intensive to link. Really, compare the on-disk footprint for chromium vs. Mozilla's libxul. But somehow no one calls Chrome bloated.
And about the doubling of compile time... it's pretty much about the new fancy JIT javascript engines. On my Core i3, compiling Firefox 3.6 vs 15.0a1 (latest nightly), it went from 15 minutes to almost 20. Though one option I noticed makes a big impact, that I think is not in ebuilds (I've since forever compile Firefox outside of portage), is --disable-debug-symbols. Before I added that, it took up to 25 minutes to compile. |
|
| Back to top |
|
|
sphakka
Tux's lil' helper
Joined: 24 Jun 2003
Posts: 79
|
| Posted: Tue May 22, 2012 1:33 pm Post subject: |
|
|
| Gusar wrote: | It seems with all this criticism of Mozilla, you guys haven't noticed that Chrome/Chromium packs *even more* stuff in to a single binary and is therefore just as resource intensive to link. Really, compare the on-disk footprint for chromium vs. Mozilla's libxul. But somehow no one calls Chrome bloated.
|
You're perfectly right, Gusar! That's why I concentrate on Mozilla's products: just having a look at Chromium's tarball size and deps made me stick to SeaMonkey, as with it I have a full integrated suite -- and not an user-level reimplementation of the OS as Chromium folks are doing. Ironically, someone made the same commentary about FireFox elsewhere, though IMHO it doesn't apply at all... it was a Windows user I guess
Morale... don't even talk about Chromium... And as far as I'm concerned Chr*me (censored) is TABOO as it's not free software |
|
| Back to top |
|
|
Ant P.
Watchman
Joined: 18 Apr 2009
Posts: 6920
|
| Posted: Tue May 22, 2012 6:10 pm Post subject: |
|
|
I don't know about Chromium being bloated - they clean up bundled libraries when they can, they work with distros instead of pretending only GnomeOS exists, and it will still compile on my netbook. Firefox no longer does.
Which is kind of ironic given the removal of bloat is its original reason for existing. |
|
| Back to top |
|
|
Gusar
Advocate
Joined: 09 Apr 2005
Posts: 2665
Location: Slovenia
|
| Posted: Tue May 22, 2012 7:47 pm Post subject: |
|
|
Err, are you implying that Firefox, a DE-agnostic app, is pretending only GnomeOS exists? Interesting that Chromium still compiles for you. Last time I compiled it (about a week ago), linking used up all my RAM, just like compiling Firefox does.
But my point wasn't that Chromium is bloated. It's that Firefox isn't! A resource intensive linking stage isn't bloat. It's a design decision to squeeze out the last possible bit of performance and to reduce start-up time. Most Firefox users (who are on Windows) are welcoming such improvements. Which makes sense, start-up time used to be a long-time criticism of Firefox. It's only here that this is called "bloat". |
|
| Back to top |
|
|
Gusar
Advocate
Joined: 09 Apr 2005
Posts: 2665
Location: Slovenia
|
| Posted: Thu May 24, 2012 12:46 pm Post subject: |
|
|
Just for fun, I started a Firefox compile (latest nightly) on my netbook. 1GB of RAM, no swap (the ssd in this thing is so slow, swap would hurt a lot more than it'd help). So of course I expected the linking stage to bail out. But you know what? It didn't! It linked. Didn't even take that long. By that I mean the linking didn't take long, the entire process was a bit over two and a half hours. You guys actually using Gentoo on netbooks? Whoa.
How is it possible that I can compile Firefox in 1GB of RAM, while you guys can't in 2GB + swap? Now I do deactivate some stuff, but I can't imagine this making much of a difference, this stuff is little compared to the really big stuff (html and js engines). PGO is the only thing I see could make a difference, I don't use that. |
|
| Back to top |
|
|
Polynomial-C
Retired Dev
Joined: 01 Jun 2003
Posts: 1432
Location: Germany
|
| Posted: Thu May 24, 2012 12:52 pm Post subject: |
|
|
| Gusar wrote: | | How is it possible that I can compile Firefox in 1GB of RAM, while you guys can't in 2GB + swap? Now I do deactivate some stuff, but I can't imagine this making much of a difference, this stuff is little compared to the really big stuff (html and js engines). PGO is the only thing I see could make a difference, I don't use that. |
Seamonkey combines firefox and thunderbird in one application. So it's obvious that seamonkey needs more ressources for the linking stage.
_________________
The manual said "Requires Windows10 or better" so I installed GNU/Linux...
my portage overlay
Need a stage1 tarball? (Unofficial builds) |
|
| Back to top |
|
|
Gusar
Advocate
Joined: 09 Apr 2005
Posts: 2665
Location: Slovenia
|
| Posted: Thu May 24, 2012 12:57 pm Post subject: |
|
|
| Polynomial-C wrote: | | Seamonkey combines firefox and thunderbird in one application. So it's obvious that seamonkey needs more ressources for the linking stage. |
That's not it. One, Seamonkey will use the same html and js engine for both the browser and mail part, so the crucial linking stage (libxul) isn't much different. And two, if you read this and the other thread, it's not just Seamonkey, people have problems compiling Firefox too. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Portage & Programming
