| View previous topic :: View next topic |
| Author |
Message |
The_Document
Apprentice
Joined: 03 Feb 2018
Posts: 275
|
 Posted: Thu Sep 20, 2018 1:38 am Post subject: qt-webkit taking too long |
 |
|
it normally takes no more than 35mins to build currently it doesent build after many hours.
I believe jit flag extends duration, is jit absolutely needed? |
|
| Back to top |
|
 |
spidark
Tux's lil' helper
Joined: 01 Sep 2011
Posts: 147
|
| Posted: Thu Sep 20, 2018 12:26 pm Post subject: Re: qt-webkit taking too long |
|
|
| The_Document wrote: | it normally takes no more than 35mins to build currently it doesent build after many hours.
I believe jit flag extends duration, is jit absolutely needed? |
I curious about this one. 
The first run i thought it was stuck in a loop or whatever, and i stopped the process after 2 hours.
Second run Qtwekkit took about 1 1/2 hours to compile, this with /var/portage on tmpfs (6GB). On a Intel(R) Core(TM) i7-3632QM -2.20GHz CPU
I gave up, and living on xfce4 webkit free for the moment.
So yes very curious about this one.
_________________
Laptop HP Pavilion G6 2310-SD Intel(R) Core(TM) i7-3632QM CPU @ 2.20GHz |
|
| Back to top |
|
|
Juippisi
Developer
Joined: 30 Sep 2005
Posts: 724
Location: /home
|
| Posted: Thu Sep 20, 2018 12:53 pm Post subject: |
|
|
Use tools to see if CPU is being used, or if thre's any I/O activity.
Sometimes the processes just hang. |
|
| Back to top |
|
|
astroe
n00b
Joined: 12 Aug 2004
Posts: 14
Location: Bucharest, Romania
|
| Posted: Fri Sep 21, 2018 6:42 am Post subject: |
|
|
I've had this happen on two machines in the past couple of days now (one left to compile over night). There is something definitely amiss, it's not just a duration thing. Both had to be cold-restarted, as the system had frozen solid.
Strangely enough, it's trying to jump from 5.9.1 to 5.212.0_pre20180120 and it's marked as deprecated in favor of qtwebengine, although there are still packages that depend on it.
LE: I got it to work by running it outside of the X window system, in a console of its own. |
|
| Back to top |
|
|
spidark
Tux's lil' helper
Joined: 01 Sep 2011
Posts: 147
|
| Posted: Fri Sep 21, 2018 12:42 pm Post subject: |
|
|
Still way up there.
| Code: |
* dev-qt/qtwebkit
Fri Sep 21 10:54:25 2018 >>> dev-qt/qtwebkit-5.212.0_pre20180120
merge time: 1 hour and 48 seconds. |
_________________
Laptop HP Pavilion G6 2310-SD Intel(R) Core(TM) i7-3632QM CPU @ 2.20GHz |
|
| Back to top |
|
|
The_Document
Apprentice
Joined: 03 Feb 2018
Posts: 275
|
| Posted: Sat Sep 22, 2018 10:24 pm Post subject: |
|
|
| spidark wrote: | Still way up there.
| Code: |
* dev-qt/qtwebkit
Fri Sep 21 10:54:25 2018 >>> dev-qt/qtwebkit-5.212.0_pre20180120
merge time: 1 hour and 48 seconds. |
|
I meant version | Code: | [ebuild U ] dev-qt/qtwebkit-5.212.0_pre20180120 [5.9.1] USE="X%* gstreamer* hyphen%* -nsplugin%"
|
is taking too long |
|
| Back to top |
|
|
astroe
n00b
Joined: 12 Aug 2004
Posts: 14
Location: Bucharest, Romania
|
| Posted: Sun Sep 23, 2018 8:34 am Post subject: |
|
|
| astroe wrote: | I've had this happen on two machines in the past couple of days now.
LE: I got it to work by running it outside of the X window system, in a console of its own. |
Actually, this only worked on one of the two machines, the other one still stalls. I don't think it's a processor thing (I run emerge with nice -n 10 so it's low priority). I rather think there is some kind of a memory leak. I also noticed this build has some freakishly long command lines. |
|
| Back to top |
|
|
spidark
Tux's lil' helper
Joined: 01 Sep 2011
Posts: 147
|
| Posted: Sun Sep 23, 2018 1:10 pm Post subject: |
|
|
| The_Document wrote: | | spidark wrote: | Still way up there.
| Code: |
* dev-qt/qtwebkit
Fri Sep 21 10:54:25 2018 >>> dev-qt/qtwebkit-5.212.0_pre20180120
merge time: 1 hour and 48 seconds. |
|
I meant version | Code: | [ebuild U ] dev-qt/qtwebkit-5.212.0_pre20180120 [5.9.1] USE="X%* gstreamer* hyphen%* -nsplugin%"
|
is taking too long |
Yes i believed that's what you meant  earlier version ( 5.9.1 ) took longer on my Machine, 2 1/2 hour.
dev-qt/qtwebkit-5.212.0_pre20180120 took only 1 hour and 48 seconds (with standard use flags enabled), which for me,a big improvement.
| astroe wrote: | | Actually, this only worked on one of the two machines, the other one still stalls. I don't think it's a processor thing (I run emerge with nice -n 10 so it's low priority). I rather think there is some kind of a memory leak. I also noticed this build has some freakishly long command lines. |
Maybe, don't know, but had some stalls or in a loop hangs experiences myself, always in tmpfs , wonder if it that has anything to do with it
Still curious 
_________________
Laptop HP Pavilion G6 2310-SD Intel(R) Core(TM) i7-3632QM CPU @ 2.20GHz
Last edited by spidark on Sat Oct 20, 2018 8:23 am; edited 1 time in total |
|
| Back to top |
|
|
tld
Veteran
Joined: 09 Dec 2003
Posts: 1816
|
| Posted: Sat Oct 20, 2018 4:52 am Post subject: |
|
|
Wow...Unfortunately I need qtwebkit for MythTV, though I have a feeling that nothing I do with it actually uses it. I have older x86 frontend/backend machines, one with 2 GB of RAM and one with 1.5. I've been able to compile 5.9.1 though it took like 5-6 hours. I ran into this configure error...apparently using -hyphen must be broken?:
https://forums.gentoo.org/viewtopic-t-1088162-highlight-.html
...but Holy crap...the stuff I'm reading makes me seriously doubt whether I'll even be able to compile this new version. I hunted down 5.9.1 and all the patches and put it in my overlay. There are no words for the hatred I have for this package.
The comments in bug I linked in that post is especially scary. I currently have MAKEOPTS="-j2" it appears. Do I stand a chance of being able to compile this?
Tom |
|
| Back to top |
|
|
CaptainBlood
Advocate
Joined: 24 Jan 2010
Posts: 3627
|
| Posted: Sat Oct 20, 2018 6:14 am Post subject: |
|
|
Here there is:
| Code: | lscpu; eix qtwebkit; qlop -tH qtwebkit; grep MAKEOPTS /etc/portage/make.conf
------------------------------------------------------------------------
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 2
Core(s) per socket: 2
Socket(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 58
Model name: Intel(R) Core(TM) i3-3217U CPU @ 1.80GHz
Stepping: 9
CPU MHz: 1737.487
CPU max MHz: 1800,0000
CPU min MHz: 800,0000
BogoMIPS: 3593.58
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 3072K
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer xsave avx f16c lahf_lm epb ssbd ibrs ibpb stibp kaiser tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts flush_l1d
------------------------------------------------------------------------
[I] dev-qt/qtwebkit
Available versions: (5) 5.212.0_pre20180120(5/5.212)
{X geolocation gles2 +gstreamer +hyphen +jit multimedia nsplugin opengl orientation +printsupport qml webp}
Installed versions: 5.212.0_pre20180120(5/5.212)(03:27:13 17/10/2018)(-X -geolocation -gles2 -gstreamer -hyphen -jit -multimedia -nsplugin -opengl -orientation -printsupport -qml -webp)
Homepage: https://www.qt.io/
Description: WebKit rendering library for the Qt5 framework (deprecated)
------------------------------------------------------------------------
qtwebkit: 1 hour, 36 minutes, 45 seconds for 6 merges
------------------------------------------------------------------------
MAKEOPTS="-j3 -l3" |
Thks 4 ur attention. |
|
| Back to top |
|
|
spidark
Tux's lil' helper
Joined: 01 Sep 2011
Posts: 147
|
| Posted: Sat Oct 20, 2018 9:22 am Post subject: |
|
|
| tld wrote: | | The comments in bug I linked in that post is especially scary. I currently have MAKEOPTS="-j2" it appears. Do I stand a chance of being able to compile this? |
I know, i'm not being helpfull, because i chickened out on qtwebkit, for the time being.
I'm pretty sure you will get it to compile though.
Good luck.
_________________
Laptop HP Pavilion G6 2310-SD Intel(R) Core(TM) i7-3632QM CPU @ 2.20GHz |
|
| Back to top |
|
|
proteusx
Guru
Joined: 21 Jan 2008
Posts: 338
|
| Posted: Sat Oct 20, 2018 12:06 pm Post subject: |
|
|
qtwebengine is even worse:
| Code: | genlop -t qtwebengine
Sat Apr 14 08:57:57 2018 >>> dev-qt/qtwebengine-5.9.5
merge time: 3 hours, 53 minutes and 55 seconds.
Fri Jun 15 19:38:14 2018 >>> dev-qt/qtwebengine-5.9.6
merge time: 3 hours, 28 minutes and 54 seconds.
Sat Oct 20 00:07:36 2018 >>> dev-qt/qtwebengine-5.11.2
merge time: 8 hours, 53 minutes and 36 seconds. |
If I USE="jumbo-build" it runs out memory. |
|
| Back to top |
|
|
tld
Veteran
Joined: 09 Dec 2003
Posts: 1816
|
| Posted: Sat Oct 20, 2018 1:56 pm Post subject: |
|
|
Thanks for the replies. The -hyphen bug I ran into was apparently just fixed. When I try this I'm going to at least use MAKEOPTS="-j1" as per this:
https://wiki.gentoo.org/wiki//etc/portage/package.env
Horrible...and I'm almost sure nothing I do in MythTV ever even uses it.
Tom |
|
| Back to top |
|
|
spidark
Tux's lil' helper
Joined: 01 Sep 2011
Posts: 147
|
| Posted: Sat Oct 20, 2018 2:48 pm Post subject: |
|
|
| tld wrote: | Thanks for the replies. The -hyphen bug I ran into was apparently just fixed. When I try this I'm going to at least use MAKEOPTS="-j1" as per this:
https://wiki.gentoo.org/wiki//etc/portage/package.env
Horrible...and I'm almost sure nothing I do in MythTV ever even uses it.
Tom |
The newest Firefox uses rust,i was a happy Man with rust-bin ( just lazy and protective , and i know firefox-bin exist ), rust is also a big package which on my system, reached above the hour compile time,( witch is not a big deal if you have new Hardware).
And i hope i'm not stepping on toes here asking this ( just protecting my old hardware ).
My point :There is a rust-bin, why not qtwebkit-bin and qtwebengine-bin ?
| proteusx: | Sat Oct 20 00:07:36 2018 >>> dev-qt/qtwebengine-5.11.2
merge time: 8 hours, 53 minutes and 36 seconds. |
But 8 hours Compile time Ill pass. or get me a few Power servers.
_________________
Laptop HP Pavilion G6 2310-SD Intel(R) Core(TM) i7-3632QM CPU @ 2.20GHz |
|
| Back to top |
|
|
asturm
Developer
Joined: 05 Apr 2007
Posts: 8936
|
| Posted: Sat Oct 20, 2018 2:52 pm Post subject: |
|
|
| spidark wrote: | | My point :There is a rust-bin, why not qtwebkit-bin and qtwebengine-bin ? |
Well, someone needs to do it. And binary packages are a world of pain when they need to be rebuilt on library updates, in case of qtwebkit: icu, libpng, webp, Qt. |
|
| Back to top |
|
|
tld
Veteran
Joined: 09 Dec 2003
Posts: 1816
|
| Posted: Sat Oct 20, 2018 3:08 pm Post subject: |
|
|
Wow...I just discovered something pretty painful. So I synced in part to get the qtwebkit -hyphen fix to the ebuild, and I see that qt 5.11 is now stable. I also see that the the dev-qt/qtwebkit-5.212.0_pre20180120 has this change as compared to 5.9.1:
| Code: | | >=dev-qt/qtwidgets-${QT_MIN_VER}= |
...where 5.9.1 had: | Code: | | >=dev-qt/qtwidgets-${QT_MIN_VER} |
I take it that now means that this monstrosity needs to be recompiled on every minor version change of qtwidgets? Beyond painful.
Until I find out whether or not I can compile the new qtwebkit at all, I don't think I dare upgrade from 5.9.1 to 5.11...unless my existing dev-qt/qtwebkit-5.9.1 will work with the other 5.11 packages(???). I think that means that...assuming I can compile that at all, I'll have to do it all over again as part of the upgrade to 5.11. This package is a cancer I swear.
Tom |
|
| Back to top |
|
|
asturm
Developer
Joined: 05 Apr 2007
Posts: 8936
|
| Posted: Sat Oct 20, 2018 3:11 pm Post subject: |
|
|
| tld wrote: | | I take it that now means that this monstrosity needs to be recompiled on every minor version change of qtwidgets? Beyond painful. |
It's always been like that: QtWebkit had been updated for every little Qt 5.x.x point release until 5.9.1.
Now you rebuild from 5.9 to 5.10, from 5.10 to 5.11.
So you're actually saving rebuilds. |
|
| Back to top |
|
|
spidark
Tux's lil' helper
Joined: 01 Sep 2011
Posts: 147
|
| Posted: Sat Oct 20, 2018 3:34 pm Post subject: |
|
|
| asturm wrote: |
Well, someone needs to do it. And binary packages are a world of pain when they need to be rebuilt on library updates, in case of qtwebkit: icu, libpng, webp, Qt. |
Ok Astrum , I understand the pain ( messed with LFS for a while )
For now it is what it is, I need new hardware for these new packages.
_________________
Laptop HP Pavilion G6 2310-SD Intel(R) Core(TM) i7-3632QM CPU @ 2.20GHz |
|
| Back to top |
|
|
tld
Veteran
Joined: 09 Dec 2003
Posts: 1816
|
| Posted: Sat Oct 20, 2018 3:36 pm Post subject: |
|
|
| asturm wrote: | | It's always been like that: QtWebkit had been updated for every little Qt 5.x.x point release until 5.9.1. |
Ahh...ok. I see that now. For example the update from qtwidgets-5.9.4-r1 to qtwidgets-5.9.6-r1 didn't require an update or recompile of qtwebkit.
So since my qtwebkit 5.9.1 obviously can't be used with qt 5.11, I think my only safe option is to:
a) Temporarily mask qt 5.11,
b) Compile (hopefully) the new qtwebkit-5.212.0_pre20180120 with qt 5.9.1 installed, and
c) assuming that works, update to qt 5.11 which will recompile qtwebkit-5.212.0_pre20180120(?)
Does that sound right? I don't see that I have any other safe options.
Tom |
|
| Back to top |
|
|
tld
Veteran
Joined: 09 Dec 2003
Posts: 1816
|
| Posted: Mon Oct 22, 2018 9:58 pm Post subject: |
|
|
As it turned out I got lucky on this one. After doing some research I found that it IS actually possible to compile MythTV (which was my only dependency for that) without qtewbkit at all, but only if you're not using mythbrowser or any of the other plugins that require it (which I'm not). What I did was this (using the ebuild I already had in my overlay for other reasons):
1. Removed the qtwebkit dependency from the ebuild.
2. Added a user patch that modified the mythtv configure script so as to not require qtwebkit.
3. As a safeguard / backup I used quickpkg to make a binary package from the current installed qtwebkit.
4. Unmerged qtwebkit (regardless of anything in the mythtv configure script, qtwebkit DOES get linked in as long as it's installed).
5. Rebuilt mythtv from my ebuild.
All seems to be working great . Hopefully I'm done with qtwebkit forever...which will also mean I can get rid of that absurd build-time-only requirement it has for ruby.
Tom |
|
| Back to top |
|
|
pourpier
Apprentice
Joined: 27 Sep 2017
Posts: 166
|
| Posted: Tue Oct 23, 2018 8:38 am Post subject: |
|
|
| proteusx wrote: | qtwebengine is even worse:
| Code: | genlop -t qtwebengine
Sat Apr 14 08:57:57 2018 >>> dev-qt/qtwebengine-5.9.5
merge time: 3 hours, 53 minutes and 55 seconds.
Fri Jun 15 19:38:14 2018 >>> dev-qt/qtwebengine-5.9.6
merge time: 3 hours, 28 minutes and 54 seconds.
Sat Oct 20 00:07:36 2018 >>> dev-qt/qtwebengine-5.11.2
merge time: 8 hours, 53 minutes and 36 seconds. |
If I USE="jumbo-build" it runs out memory. |
Hello proteusx,
You can have jumbo-build as well.
I have been following qtwebkit and qtwebengine carefully with htop in a second tab of a terminal and I've seen that some steps take almost 3GB per job. So you need to adjust the situation like this: the number N of jobs for both qtwebkit and qtwebengine must respect this rule M -3N > 1 where M is the number of GB of RAM
Let us suppose you have 8GB of RAM => N can't be greater than 2.
I created the env directory under /etc/portage and I created in the env directory a file named qtweb.conf with this line:
MAKEOPTS="-j2" (adjust to your case)
I created a file package.env under /etc/portage/ with these lines:
dev-qt/qtwebkit qtweb.conf
dev-qt/qtwebengine qtweb.conf
And it worked perfectly. Previously I had ninja -j4 l0 failed
Now if you only have 4GB of RAM you need MAKEOPTS="-j1" and let it compile during the night!
Cheers |
|
| Back to top |
|
|
asturm
Developer
Joined: 05 Apr 2007
Posts: 8936
|
| Posted: Tue Oct 23, 2018 10:12 am Post subject: |
|
|
| tld wrote: | 1. Removed the qtwebkit dependency from the ebuild.
2. Added a user patch that modified the mythtv configure script so as to not require qtwebkit. |
Do you think you could make the patch so that QtWebkit is optional / the revdepending plugins are installed if QtWebkit is found, and submit these changes to Gentoo? I'm sure other mythtv users would appreciate, and it currently has no dedicated maintainer at all. |
|
| Back to top |
|
|
tld
Veteran
Joined: 09 Dec 2003
Posts: 1816
|
| Posted: Sun Oct 28, 2018 12:57 pm Post subject: |
|
|
| asturm wrote: | | tld wrote: | 1. Removed the qtwebkit dependency from the ebuild.
2. Added a user patch that modified the mythtv configure script so as to not require qtwebkit. |
Do you think you could make the patch so that QtWebkit is optional / the revdepending plugins are installed if QtWebkit is found, and submit these changes to Gentoo? I'm sure other mythtv users would appreciate, and it currently has no dedicated maintainer at all. |
Somehow I didn't see until today. I'd actually really like to be able to do that but I think that's a bit out of my wheelhouse. I don't know a lot about that build process in general to begin with. In addition to that, I'm unclear on the process for building and installing plugins. In the past Gentoo has handled those as a separate ebuild (and I've never used any of them). What I also don't totally get is that setting "disable qtwebkit" in their current configure script prevents QtWebkit from getting linked into their libmythui, but other things (including mythfrontend itself) pull it in as long as it's there. Here's the original discussion I had on the MythTV mailing list about making it optional:
https://lists.gt.net/mythtv/users/603960
I'm frankly still a little confused as to it's requirements...I just know the basic mythtv works without it. I'd love to be able to do the same in regard to QtDBUS which I'm not using either. Just as with QtWebkit the build just uses it as long as it's installed, which is why it's not yet possible to base that on a USE flag:
https://bugs.gentoo.org/580856
I agree though, it's be great to have all that optional in the configure.
Tom |
|
| Back to top |
|
|
Hu
Moderator
Joined: 06 Mar 2007
Posts: 21635
|
| Posted: Sun Oct 28, 2018 4:44 pm Post subject: |
|
|
If the build system auto-detects an optional supporting library, uses the library if it is present, and skips using it if the library is absent, that is considered an "automatic" (or, more pejoratively, "automagic") dependency. Distribution maintainers despise automagic dependencies, and with good reason. They cause the results of the final build to depend on state not recorded by the package manager (whether the supporting library was present). A common workaround for this in Gentoo is to set the ebuild to require the optional package, so that the package is always present and always used. The alternative is to exclude the dependency, but then the user will be able to depclean the supporting package and Portage will not warn that you are removing a package that is in use, since it evaluates "in use" based on ebuild dependencies.
The proper way to handle optional features is for them to have a build system switch, for the build to respect that switch in all cases, and for the build to reliably fail if the feature is set to enabled and the supporting library is absent. From your description, MythTV does not fully satisfy those requirements. This is not unusual. Many build systems get this wrong, in part because it is so easy to get wrong.
You may need to patch their build system so that all its components respect the configure switch. In my opinion, the first priority would be getting an ebuild that has a USE flag to control this, that the build succeeds with both USE=+ and USE=-, and that in the USE=- case, MythTV does not use qtwebkit at all, even if it was installed at build time. Once you have it working at a technical level, then you can evaluate the negative side effects of excluding the qtwebkit-based functionality and document accordingly. |
|
| Back to top |
|
|
curmudgeon
Veteran
Joined: 08 Aug 2003
Posts: 1741
|
| Posted: Mon Oct 29, 2018 8:02 am Post subject: qtweb(engine|kit) have become absolute monsters |
|
|
Definitely needed. I came today to complain (did search first :) ) about what monsters qtweb(engine|kit) have become. Only have 12 GB of RAM, and I shut down all of the big users of RAM (browsers) before starting the recent qt and kde updates. Went away for a bit, and came back to find machine completely unresponsive and hard drive thrashing. Took me over a half-hour to "break into" the machine remotely (had locked it, and could not unlock it due to updates). The load (shown by top) was over forty, and there were five "cc1plus" processes each consuming over 2.5 GB, all of them locked in uninterruptible sleep, and less than fifty MB of memory available. It turns out that there had not been a single line added to the compile log (in /var/log/portage) for more than seventy five minutes! I managed to kill what few remaining programs I was running (password manager), and eventually everything became unstuck, and the compile began to crawl forward again. Well over twelve hours to finish (I remember when "emerge -e world" would complete in less than six hours :( ). I just hate even seeing qt updates now.
Don't understand why each of these processes needs so much memory, and I hate the whole idea (which I assume is a cause of this) of trying to turn browsers into operating systems that do everything (that's not the Unix way - I want to watch video in a media player, not a browser). Also wish there were an easy way to just say "don't start any more cc1plus processes if the current ones are using more than 8 GB," but at least I can restrict these two packages to -j2 (and will in the future). |
|
| Back to top |
|
|
|
Portage & Programming
