Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
webkit-gtk took so long to compile?
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
truongan
n00b
n00b


Joined: 08 Mar 2006
Posts: 38
Location: Việt Nam

PostPosted: Wed May 02, 2012 4:00 pm    Post subject: webkit-gtk took so long to compile? Reply with quote

Well I don't know if this is normal or not but it struck me how come a package only 7MB in size took over 3 hours to compile. My computer can compile larger package (in size) in a more timely manner. Firefox worth 70MB of download but took only 1 hour to compile. Does webkit-gtk took that long for you guys as well? May be you guys could take a look at my emerge --info to notice if there is anything wrong?

Code:
 emerge --info webkit-gtk
Portage 2.1.10.49 (default/linux/amd64/10.0/desktop/gnome, gcc-4.5.3, glibc-2.14.1-r3, 3.2.12-gentoo x86_64)
=================================================================
                        System Settings
=================================================================
System uname: Linux-3.2.12-gentoo-x86_64-AMD_Athlon-tm-_64_X2_Dual_Core_Processor_4400+-with-gentoo-2.0.3
Timestamp of tree: Wed, 02 May 2012 10:35:01 +0000
ccache version 3.1.6 [enabled]
app-shells/bash:          4.2_p20
dev-java/java-config:     2.1.11-r3
dev-lang/python:          2.7.2-r3, 3.2.2
dev-util/ccache:          3.1.6
dev-util/cmake:           2.8.6-r4
dev-util/pkgconfig:       0.26
sys-apps/baselayout:      2.0.3
sys-apps/openrc:          0.9.8.4
sys-apps/sandbox:         2.5
sys-devel/autoconf:       2.13, 2.68
sys-devel/automake:       1.11.1
sys-devel/binutils:       2.21.1-r1
sys-devel/gcc:            4.5.3-r2
sys-devel/gcc-config:     1.5-r2
sys-devel/libtool:        2.4-r1
sys-devel/make:           3.82-r1
sys-kernel/linux-headers: 3.1 (virtual/os-headers)
sys-libs/glibc:           2.14.1-r3
Repositories: gentoo x-portage
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=k8-sse3 -m3dnow -msse3 -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt /var/lib/hsqldb"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=k8-sse3 -m3dnow -msse3 -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="assume-digests binpkg-logs ccache distlocks ebuild-locks fixlafiles news parallel-fetch protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch"
FFLAGS=""
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LANG="en_US.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="en vi"
MAKEOPTS="-j3"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/usr/local/portage"
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="3dnow 3dnowext X a52 aac acl acpi aim alsa amd64 anthy apng aspell audio bash-completion berkdb bittorrent branding brasero bzip2 cairo cdda cdr cjk cleartype cli colord consolekit cracklib crypt cxx dbus dhcp dri dts dvd dvdr eds emboss enca encode evo exif fam ffmpeg firefox flac fortran gdbm gdu gif gmplayer gnome gnome-keyring gnome-online-accounts gpm gstreamer gtk gtk3 hal iconv immqt-bc introspection ipv6 jpeg lame lcms ldap libnotify m17n-lib mad matroska midi mmx mng modules mp3 mp4 mpeg mudflap multilib nautilus ncurses networkmanager nls nptl nptlonly ogg opengl openmp pam pango pcre pdf png policykit ppds pppd pulseaudio python qt3support quicktime readline sdl session socialweb sound spell sqlite sse sse2 ssl startup-notification svg sysfs tcpd theora tiff truetype udev unicode usb vdpau vorbis wxwidgets x264 xcb xml xorg xulrunner xv xvid zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en vi" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="nvidia vesa" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON

=================================================================
                        Package Settings
=================================================================

net-libs/webkit-gtk-1.8.0-r200 was built with the following:
USE="geoloc gstreamer introspection jit (multilib) spell webgl (-aqua) -coverage -debug -test"


net-libs/webkit-gtk-1.8.1-r300 was built with the following:
USE="geoloc gstreamer introspection jit (multilib) spell webgl (-aqua) -coverage -debug -doc -test -webkit2"
Back to top
View user's profile Send private message
The Unabeefer
n00b
n00b


Joined: 23 Dec 2008
Posts: 60
Location: Chicago, IL

PostPosted: Wed May 02, 2012 4:41 pm    Post subject: Reply with quote

I don't know if it takes 3 hours or not... but it does take quite a while for me. Whenever I see webkit-gtk, I normally wait to update until I am in bed asleep because it takes so long.

So if there's something wrong on your end, there probably is on my end. OR, it just takes that long for everyone.
_________________
I am a flying cow... worship me or DIE!!!

...and there he was, reigning supreme at number two...
The One... The Only... The Unabeefer
Back to top
View user's profile Send private message
ulenrich
Veteran
Veteran


Joined: 10 Oct 2010
Posts: 1358

PostPosted: Wed May 02, 2012 4:52 pm    Post subject: Reply with quote

change c(xx)flags from
"-O2" to "-O0"
and use gcc-4.6.3 instead of previous compilers in case you havent already. This will fasten your compiles!
_________________
fun2gen2
Back to top
View user's profile Send private message
Gusar
Advocate
Advocate


Joined: 09 Apr 2005
Posts: 2656
Location: Slovenia

PostPosted: Wed May 02, 2012 5:17 pm    Post subject: Reply with quote

There's a bug in webkit-gtk's build system, so make -j1 is used to compile it. That's why it takes so long. Edit: Actually, the bug seems to be in GNU make :). Either way, this is the reason for the long compile.
Back to top
View user's profile Send private message
salahx
Guru
Guru


Joined: 12 Mar 2005
Posts: 432

PostPosted: Wed May 02, 2012 5:30 pm    Post subject: Reply with quote

ulenrich wrote:
change c(xx)flags from
"-O2" to "-O0"
and use gcc-4.6.3 instead of previous compilers in case you havent already. This will fasten your compiles!


No, never use -O0 ! Features like FORTIFY_SOURCE actually require optimization to work!
Back to top
View user's profile Send private message
Hypnos
Advocate
Advocate


Joined: 18 Jul 2002
Posts: 2887
Location: Omnipresent

PostPosted: Wed May 02, 2012 5:47 pm    Post subject: Reply with quote

C++ code making extensive use of templates and smart pointers, as WebKit does, takes a long time to compile.

I'm not expert enough in compiler technology to explain why exactly, other than the complexity involved in shoehorning generics and "pseudo"-dynamic memory technology into a static-binding language. Objective-C has a similar feature set but compiles much, much, much faster because it is dynamic binding.
_________________
Personal overlay | Simple backup scheme
Back to top
View user's profile Send private message
broken_chaos
Guru
Guru


Joined: 18 Jan 2006
Posts: 370
Location: Ontario, Canada

PostPosted: Wed May 02, 2012 6:55 pm    Post subject: Reply with quote

It's pretty big. Seems my average is around 2 hours on my netbook (dual core, 64-bit, hyper-threaded Atom).

If you haven't already, be sure to put /var/tmp/portage on a big tmpfs (most things compile with 1G of tmpfs -- occasionally something with need more space, like gcc).
Back to top
View user's profile Send private message
truongan
n00b
n00b


Joined: 08 Mar 2006
Posts: 38
Location: Việt Nam

PostPosted: Fri May 04, 2012 4:09 pm    Post subject: Reply with quote

broken_chaos wrote:
It's pretty big. Seems my average is around 2 hours on my netbook (dual core, 64-bit, hyper-threaded Atom).

If you haven't already, be sure to put /var/tmp/portage on a big tmpfs (most things compile with 1G of tmpfs -- occasionally something with need more space, like gcc).


Well I really haven't :oops: Perhaps put PORTAGE_TMPDIR on tmpfs can put a dent in that horrible 3 hours waiting of mine. But still I think it a problem, at least an upstream problem. Seem that we will have to bear with it for now.
Back to top
View user's profile Send private message
Gusar
Advocate
Advocate


Joined: 09 Apr 2005
Posts: 2656
Location: Slovenia

PostPosted: Fri May 04, 2012 6:32 pm    Post subject: Reply with quote

The latest version (1.8.1-r201 and 1.8.1-r301) contain a hack to allow parallel building.
Back to top
View user's profile Send private message
Angrychile
Apprentice
Apprentice


Joined: 27 Oct 2009
Posts: 231

PostPosted: Thu Jun 19, 2014 4:46 am    Post subject: Reply with quote

Gusar wrote:
There's a bug in webkit-gtk's build system, so make -j1 is used to compile it. That's why it takes so long. Edit: Actually, the bug seems to be in GNU make :). Either way, this is the reason for the long compile.


Calling from the future, this surely cannot still be true! Webkit is still a pain to compile.
_________________
hola
Back to top
View user's profile Send private message
Rion
Guru
Guru


Joined: 12 Oct 2006
Posts: 374
Location: Ekaterinburg, Russia

PostPosted: Wed Aug 27, 2014 7:17 pm    Post subject: Reply with quote

looks like it compiles already 5 or so hours on ramdisk. i7-3610QM
_________________
~amd64
rion-overlay
Back to top
View user's profile Send private message
jochenf
n00b
n00b


Joined: 05 Nov 2014
Posts: 9

PostPosted: Sun Nov 16, 2014 1:02 pm    Post subject: Reply with quote

Bad enough that it takes so long, but it will install TWO versions of webkit-gtk (2.4.4 and 2.6.1) on a fresh installation with "emerge gnome firefox". :evil:
My pc is now compiling since 18 hours for gnome and firefox, and no end in sight. :evil:
Back to top
View user's profile Send private message
Velnias75
n00b
n00b


Joined: 30 Nov 2014
Posts: 17
Location: Germany

PostPosted: Sun Dec 21, 2014 8:47 am    Post subject: Reply with quote

Just my two cents:
  • According to the build log dependency tracking seems to be enabled. This may could be fixed easily by the ebuild maintainers by adding
    Code:
    --disable-dependency-tracking
    to the configure call
  • there are tons of surely unneccesary search include paths, but that may be a task for upstream

I started my build 8 p.m. last evening, letting it run all the night and currently (9.45 a.m.) it continues to build, but in a virtual machine which has been suspended around 1 hour because I needed my PC for other tasks.
Back to top
View user's profile Send private message
sbdmmg
n00b
n00b


Joined: 01 Jul 2010
Posts: 34
Location: NJ, USA

PostPosted: Sat Jan 31, 2015 4:52 pm    Post subject: Reply with quote

Just my two cents:
the compilation of
Code:
net-libs/webkit-gtk-2.4.8
+
Code:
net-libs/webkit-gtk-2.4.8-r200

took >12h on a i686 i5 @ 1.80GHz with 4GB of ram.
Using gcc 4.8.3 and
Code:
tmp/portage
is on SSD with enough room.
It would be nice if it were a bit faster :(

Cheers,
Davide
Back to top
View user's profile Send private message
Velnias75
n00b
n00b


Joined: 30 Nov 2014
Posts: 17
Location: Germany

PostPosted: Sat Jan 31, 2015 4:58 pm    Post subject: Reply with quote

Tip: since one night wasn't enough to compile I used
Code:
env FEATURES=keepwork emerge ...

So I could interrupt it at the morning and continue it the night.
Back to top
View user's profile Send private message
ballsystemlord
n00b
n00b


Joined: 26 Feb 2013
Posts: 66

PostPosted: Tue Feb 10, 2015 1:23 am    Post subject: Reply with quote

Hmm, I was about to ask the same question, I have webkit 2.4.8 and then I also need to compile another version, webkit-gtk 2.4.8 r200. I've got make 4.0-r1 and gcc 4.8.3.
It takes 3.5 hours for one and 4.3 for the other (I don't remember which is which). I'm also most curious about this.
Maybe I'll write the devs...
Not that I'm an expert, but if you actually take the time to watch it it includes lots of various headers... this will slow down compilation, so this is at least part of the problem.
Back to top
View user's profile Send private message
ballsystemlord
n00b
n00b


Joined: 26 Feb 2013
Posts: 66

PostPosted: Mon Feb 16, 2015 8:29 pm    Post subject: Reply with quote

I've done the research, here are the resaults:
The devs agnolage that it take a while to compile but higher then 3.5 hours is unreasonable so you might want to see whats going on unless you have a really slow box. As for me, I descovered that for many, but not all packages, my MAKE_OPTS setting ignored. webkit-gtk builds in a single threaded enviroment as does php and several others (I have not made a list). This is probablty a topic for another thread though so I, or you, might consider starting one.
Back to top
View user's profile Send private message
svartulf
n00b
n00b


Joined: 24 Nov 2003
Posts: 11
Location: Orange CT

PostPosted: Wed Feb 25, 2015 5:38 pm    Post subject: Reply with quote

Build on my laptop (AMD x86_64, 4GB RAM, /var/tmp on SSD) has taken >8hr for the past couple nights (had forgot about FEATURES=keepwork - been shutting the box down in the morning to take it to work). I seem to recall a warning about parallel builds (-jN make option, where N > 1) - some packages are sensitive to sequencing and there can be issues during compilation/linking so that's disabled in some cases. Perhaps webkit has that problem.

But what about a binary release? LibreOffice has a binary version because it takes ages to build. I'm not doing anything special with use flags and would be fine with doing a binary install rather than getting that sinking feeling I get when I see webkit in my `emerge -pv' output.
Back to top
View user's profile Send private message
ballsystemlord
n00b
n00b


Joined: 26 Feb 2013
Posts: 66

PostPosted: Wed Feb 25, 2015 10:41 pm    Post subject: Reply with quote

My mistake, when I first set MAKE_OPTS I put an _ in there and it does not belong (everyone check your spelling). I'm shocked that emerge and portage missed this invalid variable.
If you want to know a better way to enter in these values I recommend using nano and then hitting ^R (read file), select ^X (execute command), and type into the bar
man make.conf } grep (put the variable name in question here)
You can then edit out any additional text that grep brought in and save the file.
Back to top
View user's profile Send private message
lavacano
Apprentice
Apprentice


Joined: 29 May 2005
Posts: 190
Location: Poulsbo, WA

PostPosted: Sat Apr 11, 2015 8:16 am    Post subject: Reply with quote

FWIW from http://www.linuxfromscratch.org/blfs/view/svn/x/webkitgtk2.html

Code:
Estimated build time: 44 SBU (using parallelism=4, both built, webkitgtk-3.0 takes a little longer than webkitgtk-1.0)


SBU is the amount of time to build toolchain or binutils I forget if its all 3 glibc/gcc/binutils or just binutils, anyways its normal for this 9.4MB package that compiles to 1.5GB to take a "while"

Sorry for resurrecting this. Also It'd be cool to have SBU for portage :)

So I did it https://bugs.gentoo.org/show_bug.cgi?id=546246 please click vote/chime in your support if you'd like to know how long it might take to do your next emerge -NuD(e) world
_________________
Sincerely,

Chadwick Ferguson
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6103
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Mon Apr 20, 2015 8:31 pm    Post subject: Reply with quote

https://bugs.gentoo.org/show_bug.cgi?id=546336

Franz Fellner wrote:
This can be solved by moving from Unix Makefile Generator to ninja. Didn't believe it, but it cut off all the 30 extra minutes!
It is as simple as putting
CMAKE_MAKEFILE_GENERATOR="ninja"
in the ebuild.
I then enabled it globally in the cmake-utils.eclass (just to see how it behaves) and did the kde-frameworks update from 5.8.0 to 5.9.0, and everything went fine. So this might really be an option to enable by default...

_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
Eusaphious
n00b
n00b


Joined: 12 Dec 2006
Posts: 17

PostPosted: Sat Apr 25, 2015 3:25 am    Post subject: Reply with quote

lavacano wrote:
FWIW from http://www.linuxfromscratch.org/blfs/view/svn/x/webkitgtk2.html

Code:
Estimated build time: 44 SBU (using parallelism=4, both built, webkitgtk-3.0 takes a little longer than webkitgtk-1.0)


SBU is the amount of time to build toolchain or binutils I forget if its all 3 glibc/gcc/binutils or just binutils, anyways its normal for this 9.4MB package that compiles to 1.5GB to take a "while"

Sorry for resurrecting this. Also It'd be cool to have SBU for portage :)

So I did it https://bugs.gentoo.org/show_bug.cgi?id=546246 please click vote/chime in your support if you'd like to know how long it might take to do your next emerge -NuD(e) world



Check out app-portage/genlop.

Im currently recompiling webkit-gtk and it can provide a constant ETA update:
Code:

genlop -unc                                                                                        Fri Apr 24 22:09:56 2015

 Currently merging 1 out of 1

 * net-libs/webkit-gtk-2.4.8-r200

       current merge time: 58 minutes and 4 seconds.
       ETA: 33 minutes and 32 seconds.



If you wanted to estimate how long a package (or world, etc..) would take:
Code:

emerge -p firefox | genlop --pretend

[ebuild   R    ] dev-lang/python-2.7.9-r1  USE="sqlite*"
[ebuild   R    ] www-client/firefox-31.5.3

The following USE changes are necessary to proceed:
 (see "package.use" in the portage(5) man page for more details)
# required by www-client/firefox-31.5.3
# required by @selected
# required by @world (argument)
=dev-lang/python-2.7.9-r1 sqlite


Estimated update time: 45 minutes.
Back to top
View user's profile Send private message
rudregues
Apprentice
Apprentice


Joined: 29 Jan 2013
Posts: 217
Location: Brazil

PostPosted: Mon May 11, 2015 9:54 pm    Post subject: Reply with quote

Build time yet sucks for this package...
_________________
Emerging en gentoo
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6103
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Mon May 11, 2015 10:39 pm    Post subject: Reply with quote

@Eusaphious:

no you can't ! :P

you only can because you already compiled it before

without that you don't know.


Those SBU units would be nice - e.g. webkit-gtk compared to firefox with USE-flags X, Y, Z (without dependencies building),

etc. etc.



rudregues wrote:
Build time yet sucks for this package...


yes, it does :cry:

genlop -l wrote:
Mon Apr 20 23:21:06 2015 >>> net-libs/webkit-gtk-2.8.1
merge time: 50 minutes and 39 seconds.

_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
ISHAIM
Tux's lil' helper
Tux's lil' helper


Joined: 08 Oct 2006
Posts: 148
Location: Chicago, IL

PostPosted: Thu Sep 10, 2015 5:45 pm    Post subject: Reply with quote

This isn't a problem confined to Gentoo, per se, but across the board it seems as WebKit maintainers know this: https://trac.webkit.org/wiki/WebKitGTK/SpeedUpBuild

I wonder what the potential is for a binary release? It doesn't seem like this would only be a problem for Gentoo users. You would think it would make webkit-gtk maintenance a pain in and of itself. Come to think of that, I almost feel sorry for the maintainers if they have to deal with this, too. Probably they've got some decent processing power in-house, but still, that spells trouble for the way open-source works, with developers with limited hardware constraints trying to contribute. Who knows?

After Googling "webkit build time", meeting minutes from April 2011: http://trac.webkit.org/wiki/April%202011%20Meeting/Getting%20compile%20time%20under%20control
so they have known about this since then.

WebKit maintainers are the ones who have to provide a downloadable binary, just like libreoffice-bin and firefox-bin, AFAIK.
_________________
http://isaiassifuentes.net
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum