webkit-gtk took so long to compile?

[truongan]

Well I don't know if this is normal or not but it struck me how come a package only 7MB in size took over 3 hours to compile. My computer can compile larger package (in size) in a more timely manner. Firefox worth 70MB of download but took only 1 hour to compile. Does webkit-gtk took that long for you guys as well? May be you guys could take a look at my emerge --info to notice if there is anything wrong?

Code: Select all

 emerge --info webkit-gtk
Portage 2.1.10.49 (default/linux/amd64/10.0/desktop/gnome, gcc-4.5.3, glibc-2.14.1-r3, 3.2.12-gentoo x86_64)
=================================================================
                        System Settings
=================================================================
System uname: Linux-3.2.12-gentoo-x86_64-AMD_Athlon-tm-_64_X2_Dual_Core_Processor_4400+-with-gentoo-2.0.3
Timestamp of tree: Wed, 02 May 2012 10:35:01 +0000
ccache version 3.1.6 [enabled]
app-shells/bash:          4.2_p20
dev-java/java-config:     2.1.11-r3
dev-lang/python:          2.7.2-r3, 3.2.2
dev-util/ccache:          3.1.6
dev-util/cmake:           2.8.6-r4
dev-util/pkgconfig:       0.26
sys-apps/baselayout:      2.0.3
sys-apps/openrc:          0.9.8.4
sys-apps/sandbox:         2.5
sys-devel/autoconf:       2.13, 2.68
sys-devel/automake:       1.11.1
sys-devel/binutils:       2.21.1-r1
sys-devel/gcc:            4.5.3-r2
sys-devel/gcc-config:     1.5-r2
sys-devel/libtool:        2.4-r1
sys-devel/make:           3.82-r1
sys-kernel/linux-headers: 3.1 (virtual/os-headers)
sys-libs/glibc:           2.14.1-r3
Repositories: gentoo x-portage
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=k8-sse3 -m3dnow -msse3 -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt /var/lib/hsqldb"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=k8-sse3 -m3dnow -msse3 -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="assume-digests binpkg-logs ccache distlocks ebuild-locks fixlafiles news parallel-fetch protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch"
FFLAGS=""
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LANG="en_US.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="en vi"
MAKEOPTS="-j3"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="3dnow 3dnowext X a52 aac acl acpi aim alsa amd64 anthy apng aspell audio bash-completion berkdb bittorrent branding brasero bzip2 cairo cdda cdr cjk cleartype cli colord consolekit cracklib crypt cxx dbus dhcp dri dts dvd dvdr eds emboss enca encode evo exif fam ffmpeg firefox flac fortran gdbm gdu gif gmplayer gnome gnome-keyring gnome-online-accounts gpm gstreamer gtk gtk3 hal iconv immqt-bc introspection ipv6 jpeg lame lcms ldap libnotify m17n-lib mad matroska midi mmx mng modules mp3 mp4 mpeg mudflap multilib nautilus ncurses networkmanager nls nptl nptlonly ogg opengl openmp pam pango pcre pdf png policykit ppds pppd pulseaudio python qt3support quicktime readline sdl session socialweb sound spell sqlite sse sse2 ssl startup-notification svg sysfs tcpd theora tiff truetype udev unicode usb vdpau vorbis wxwidgets x264 xcb xml xorg xulrunner xv xvid zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en vi" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="nvidia vesa" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON

=================================================================
                        Package Settings
=================================================================

net-libs/webkit-gtk-1.8.0-r200 was built with the following:
USE="geoloc gstreamer introspection jit (multilib) spell webgl (-aqua) -coverage -debug -test"


net-libs/webkit-gtk-1.8.1-r300 was built with the following:
USE="geoloc gstreamer introspection jit (multilib) spell webgl (-aqua) -coverage -debug -doc -test -webkit2"

[The Unabeefer]

I don't know if it takes 3 hours or not... but it does take quite a while for me. Whenever I see webkit-gtk, I normally wait to update until I am in bed asleep because it takes so long.

So if there's something wrong on your end, there probably is on my end. OR, it just takes that long for everyone.

[ulenrich]

change c(xx)flags from
"-O2" to "-O0"
and use gcc-4.6.3 instead of previous compilers in case you havent already. This will fasten your compiles!

[Gusar]

There's a bug in webkit-gtk's build system, so make -j1 is used to compile it. That's why it takes so long. Edit: Actually, the bug seems to be in GNU make . Either way, this is the reason for the long compile.

[salahx]

change c(xx)flags from
"-O2" to "-O0"
and use gcc-4.6.3 instead of previous compilers in case you havent already. This will fasten your compiles!

No, never use -O0 ! Features like FORTIFY_SOURCE actually require optimization to work!

[Hypnos]

C++ code making extensive use of templates and smart pointers, as WebKit does, takes a long time to compile.

I'm not expert enough in compiler technology to explain why exactly, other than the complexity involved in shoehorning generics and "pseudo"-dynamic memory technology into a static-binding language. Objective-C has a similar feature set but compiles much, much, much faster because it is dynamic binding.

[broken_chaos]

It's pretty big. Seems my average is around 2 hours on my netbook (dual core, 64-bit, hyper-threaded Atom).

If you haven't already, be sure to put /var/tmp/portage on a big tmpfs (most things compile with 1G of tmpfs -- occasionally something with need more space, like gcc).

[truongan]

It's pretty big. Seems my average is around 2 hours on my netbook (dual core, 64-bit, hyper-threaded Atom).

If you haven't already, be sure to put /var/tmp/portage on a big tmpfs (most things compile with 1G of tmpfs -- occasionally something with need more space, like gcc).

Well I really haven't Perhaps put PORTAGE_TMPDIR on tmpfs can put a dent in that horrible 3 hours waiting of mine. But still I think it a problem, at least an upstream problem. Seem that we will have to bear with it for now.

[Gusar]

The latest version (1.8.1-r201 and 1.8.1-r301) contain a hack to allow parallel building.

[Angrychile]

There's a bug in webkit-gtk's build system, so make -j1 is used to compile it. That's why it takes so long. Edit: Actually, the bug seems to be in GNU make . Either way, this is the reason for the long compile.

Calling from the future, this surely cannot still be true! Webkit is still a pain to compile.

[Rion]

looks like it compiles already 5 or so hours on ramdisk. i7-3610QM

[jochenf]

Bad enough that it takes so long, but it will install TWO versions of webkit-gtk (2.4.4 and 2.6.1) on a fresh installation with "emerge gnome firefox".
My pc is now compiling since 18 hours for gnome and firefox, and no end in sight.

[Velnias75]

Just my two cents:

• According to the build log dependency tracking seems to be enabled. This may could be fixed easily by the ebuild maintainers by adding

  Code: Select all

  --disable-dependency-tracking

  to the configure call
• there are tons of surely unneccesary search include paths, but that may be a task for upstream

I started my build 8 p.m. last evening, letting it run all the night and currently (9.45 a.m.) it continues to build, but in a virtual machine which has been suspended around 1 hour because I needed my PC for other tasks.

[sbdmmg]

Just my two cents:
the compilation of

Code: Select all

net-libs/webkit-gtk-2.4.8

+

Code: Select all

net-libs/webkit-gtk-2.4.8-r200

took >12h on a i686 i5 @ 1.80GHz with 4GB of ram.
Using gcc 4.8.3 and

Code: Select all

tmp/portage

is on SSD with enough room.
It would be nice if it were a bit faster

Cheers,
Davide

[Velnias75]

Tip: since one night wasn't enough to compile I used

Code: Select all

env FEATURES=keepwork emerge ...

So I could interrupt it at the morning and continue it the night.

[ballsystemlord]

Hmm, I was about to ask the same question, I have webkit 2.4.8 and then I also need to compile another version, webkit-gtk 2.4.8 r200. I've got make 4.0-r1 and gcc 4.8.3.
It takes 3.5 hours for one and 4.3 for the other (I don't remember which is which). I'm also most curious about this.
Maybe I'll write the devs...
Not that I'm an expert, but if you actually take the time to watch it it includes lots of various headers... this will slow down compilation, so this is at least part of the problem.

[ballsystemlord]

I've done the research, here are the resaults:
The devs agnolage that it take a while to compile but higher then 3.5 hours is unreasonable so you might want to see whats going on unless you have a really slow box. As for me, I descovered that for many, but not all packages, my MAKE_OPTS setting ignored. webkit-gtk builds in a single threaded enviroment as does php and several others (I have not made a list). This is probablty a topic for another thread though so I, or you, might consider starting one.

[svartulf]

Build on my laptop (AMD x86_64, 4GB RAM, /var/tmp on SSD) has taken >8hr for the past couple nights (had forgot about FEATURES=keepwork - been shutting the box down in the morning to take it to work). I seem to recall a warning about parallel builds (-jN make option, where N > 1) - some packages are sensitive to sequencing and there can be issues during compilation/linking so that's disabled in some cases. Perhaps webkit has that problem.

But what about a binary release? LibreOffice has a binary version because it takes ages to build. I'm not doing anything special with use flags and would be fine with doing a binary install rather than getting that sinking feeling I get when I see webkit in my `emerge -pv' output.

[ballsystemlord]

My mistake, when I first set MAKE_OPTS I put an _ in there and it does not belong (everyone check your spelling). I'm shocked that emerge and portage missed this invalid variable.
If you want to know a better way to enter in these values I recommend using nano and then hitting ^R (read file), select ^X (execute command), and type into the bar
man make.conf } grep (put the variable name in question here)
You can then edit out any additional text that grep brought in and save the file.

[lavacano]

FWIW from http://www.linuxfromscratch.org/blfs/vi ... tgtk2.html

Code: Select all

Estimated build time: 44 SBU (using parallelism=4, both built, webkitgtk-3.0 takes a little longer than webkitgtk-1.0)

SBU is the amount of time to build toolchain or binutils I forget if its all 3 glibc/gcc/binutils or just binutils, anyways its normal for this 9.4MB package that compiles to 1.5GB to take a "while"

Sorry for resurrecting this. Also It'd be cool to have SBU for portage

So I did it https://bugs.gentoo.org/show_bug.cgi?id=546246 please click vote/chime in your support if you'd like to know how long it might take to do your next emerge -NuD(e) world

[kernelOfTruth]

https://bugs.gentoo.org/show_bug.cgi?id=546336

This can be solved by moving from Unix Makefile Generator to ninja. Didn't believe it, but it cut off all the 30 extra minutes!
It is as simple as putting
CMAKE_MAKEFILE_GENERATOR="ninja"
in the ebuild.
I then enabled it globally in the cmake-utils.eclass (just to see how it behaves) and did the kde-frameworks update from 5.8.0 to 5.9.0, and everything went fine. So this might really be an option to enable by default...

[Eusaphious]

FWIW from http://www.linuxfromscratch.org/blfs/vi ... tgtk2.html

Code: Select all

Estimated build time: 44 SBU (using parallelism=4, both built, webkitgtk-3.0 takes a little longer than webkitgtk-1.0)

SBU is the amount of time to build toolchain or binutils I forget if its all 3 glibc/gcc/binutils or just binutils, anyways its normal for this 9.4MB package that compiles to 1.5GB to take a "while"

Sorry for resurrecting this. Also It'd be cool to have SBU for portage

So I did it https://bugs.gentoo.org/show_bug.cgi?id=546246 please click vote/chime in your support if you'd like to know how long it might take to do your next emerge -NuD(e) world

Check out app-portage/genlop.

Im currently recompiling webkit-gtk and it can provide a constant ETA update:

Code: Select all

genlop -unc                                                                                        Fri Apr 24 22:09:56 2015

 Currently merging 1 out of 1

 * net-libs/webkit-gtk-2.4.8-r200

       current merge time: 58 minutes and 4 seconds.
       ETA: 33 minutes and 32 seconds.

If you wanted to estimate how long a package (or world, etc..) would take:

Code: Select all

emerge -p firefox | genlop --pretend

[ebuild   R    ] dev-lang/python-2.7.9-r1  USE="sqlite*" 
[ebuild   R    ] www-client/firefox-31.5.3 

The following USE changes are necessary to proceed:
 (see "package.use" in the portage(5) man page for more details)
# required by www-client/firefox-31.5.3
# required by @selected
# required by @world (argument)
=dev-lang/python-2.7.9-r1 sqlite


Estimated update time: 45 minutes.

[rudregues]

Build time yet sucks for this package...

[kernelOfTruth]

@Eusaphious:

no you can't !

you only can because you already compiled it before

without that you don't know.


Those SBU units would be nice - e.g. webkit-gtk compared to firefox with USE-flags X, Y, Z (without dependencies building),

etc. etc.

Build time yet sucks for this package...

yes, it does

Mon Apr 20 23:21:06 2015 >>> net-libs/webkit-gtk-2.8.1
merge time: 50 minutes and 39 seconds.

[ISHAIM]

This isn't a problem confined to Gentoo, per se, but across the board it seems as WebKit maintainers know this: https://trac.webkit.org/wiki/WebKitGTK/SpeedUpBuild

I wonder what the potential is for a binary release? It doesn't seem like this would only be a problem for Gentoo users. You would think it would make webkit-gtk maintenance a pain in and of itself. Come to think of that, I almost feel sorry for the maintainers if they have to deal with this, too. Probably they've got some decent processing power in-house, but still, that spells trouble for the way open-source works, with developers with limited hardware constraints trying to contribute. Who knows?

After Googling "webkit build time", meeting minutes from April 2011: http://trac.webkit.org/wiki/April%20201 ... %20control
so they have known about this since then.

WebKit maintainers are the ones who have to provide a downloadable binary, just like libreoffice-bin and firefox-bin, AFAIK.