Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
qtwebengine taking more than a day to emerge
View unanswered posts
View posts from last 24 hours

Goto page 1, 2, 3  Next  
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
urcindalo
Guru
Guru


Joined: 08 Feb 2005
Posts: 594
Location: Almeria, Spain

PostPosted: Mon Jan 14, 2019 2:38 pm    Post subject: qtwebengine taking more than a day to emerge Reply with quote

Hi!

Whenever portage decides to emerge qtwebengine, it takes almost forever 8O :
Code:
$ sudo splat qtwebengine
 * dev-qt/qtwebengine-5.11.1

        Emerged at: dom oct 21 00:01:10 2018
        Build time: 1 day, 15 hours, 15 minutes, and 20 seconds

        Emerged at: lun dic 10 13:02:54 2018
        Build time: 18 hours, 30 minutes, and 8 seconds

        Emerged at: lun ene  7 06:06:01 2019
        Build time: 1 day, 6 hours, 24 minutes, and 44 seconds

 * dev-qt/qtwebengine-5.11.3

        Emerged at: vie ene 11 09:41:04 2019
        Build time: 1 day, 9 hours, 54 minutes, and 29 seconds


The particular box where this happens is old, I know, but equivalent packages take only a few hours:
Code:
$ sudo splat webkit-gtk
 * net-libs/webkit-gtk-2.22.2

        Emerged at: dom oct 14 12:46:40 2018
        Build time: 7 hours, 9 minutes, and 29 seconds

        Emerged at: vie oct 26 17:38:55 2018
        Build time: 9 hours, 3 minutes, and 1 second

        Emerged at: lun nov 26 03:13:26 2018
        Build time: 6 hours, 43 minutes, and 48 seconds

        Emerged at: lun ene  7 00:51:34 2019
        Build time: 5 hours, 14 minutes, and 8 seconds

 * net-libs/webkit-gtk-2.22.5

        Emerged at: jue ene 10 23:16:22 2019
        Build time: 9 hours, 5 minutes, and 5 seconds

.....

$ sudo splat libreoffice
 * app-office/libreoffice-6.1.4.2

        Emerged at: mar ene  8 12:46:00 2019
        Build time: 4 hours, 31 minutes, and 41 seconds



Any ideas why? I have no clue.
Here is the hardware info:
Code:
$ sudo lshw
lab                         
    description: Desktop Computer
    product: System Product Name (To Be Filled By O.E.M.)
    vendor: System manufacturer
    version: System Version
    serial: System Serial Number
    width: 4294967295 bits
    capabilities: smbios-2.5 dmi-2.5 smp vsyscall32
    configuration: boot=normal chassis=desktop family=To Be Filled By O.E.M. sku=To Be Filled By O.E.M. uuid=609A0D5B-3CE4-DE11-8276-E0CB4E39193A
  *-core
       description: Motherboard
       product: P5QL/EPU
       vendor: ASUSTeK Computer INC.
       physical id: 0
       version: Rev X.0x
       serial: 102963730001615
       slot: To Be Filled By O.E.M.
     *-firmware
          description: BIOS
          vendor: American Megatrends Inc.
          physical id: 0
          version: 0408
          date: 07/20/2009
          size: 64KiB
          capacity: 960KiB
          capabilities: isa pci pnp apm upgrade shadowing escd cdboot bootselect socketedrom edd int13floppy1200 int13floppy720 int13floppy2880 int5printscreen int9keyboard int14serial int17printer int10video acpi usb ls120boot zipboot biosbootspecification
     *-cpu
          description: CPU
          product: Intel(R) Core(TM)2 Quad CPU    Q8300  @ 2.50GHz
          vendor: Intel Corp.
          physical id: 4
          bus info: cpu@0
          version: Intel(R) Core(TM)2 Quad CPU Q8300 @ 2.50GHz
          serial: To Be Filled By O.E.M.
          slot: LGA775
          size: 2500MHz
          capacity: 3800MHz
          width: 64 bits
          clock: 333MHz
          capabilities: x86-64 fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx constant_tsc arch_perfmon pebs bts rep_good nopl cpuid aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 xsave lahf_lm pti tpr_shadow vnmi flexpriority dtherm
          configuration: cores=4 enabledcores=4 threads=4
        *-cache:0
             description: L1 cache
             physical id: 5
             slot: L1-Cache
             size: 128KiB
             capacity: 128KiB
             capabilities: internal write-back data
             configuration: level=1
        *-cache:1
             description: L2 cache
             physical id: 6
             slot: L2-Cache
             size: 4MiB
             capacity: 4MiB
             capabilities: internal write-back unified
             configuration: level=2
     *-memory
          description: System Memory
          physical id: 33
          slot: System board or motherboard
          size: 4GiB
        *-bank:0
             description: DIMM DDR2 Synchronous 800 MHz (1,2 ns)
             product: PartNum0
             vendor: Manufacturer0
             physical id: 0
             serial: SerNum0
             slot: DIMM0
             size: 2GiB
             width: 64 bits
             clock: 800MHz (1.2ns)
        *-bank:1
             description: DIMM [empty]
             product: PartNum1
             vendor: Manufacturer1
             physical id: 1
             serial: SerNum1
             slot: DIMM1
        *-bank:2
             description: DIMM DDR2 Synchronous 800 MHz (1,2 ns)
             product: PartNum2
             vendor: Manufacturer2
             physical id: 2
             serial: SerNum2
             slot: DIMM2
             size: 2GiB
             width: 64 bits
             clock: 800MHz (1.2ns)
        *-bank:3
             description: DIMM [empty]
             product: PartNum3
             vendor: Manufacturer3
             physical id: 3
             serial: SerNum3
             slot: DIMM3
     *-pci
          description: Host bridge
          product: 4 Series Chipset DRAM Controller
          vendor: Intel Corporation
          physical id: 100
          bus info: pci@0000:00:00.0
          version: 03
          width: 32 bits
          clock: 33MHz
        *-pci:0
             description: PCI bridge
             product: 4 Series Chipset PCI Express Root Port
             vendor: Intel Corporation
             physical id: 1
             bus info: pci@0000:00:01.0
             version: 03
             width: 32 bits
             clock: 33MHz
             capabilities: pci pm msi pciexpress normal_decode bus_master cap_list
             configuration: driver=pcieport
             resources: irq:16 ioport:c000(size=4096) memory:fa000000-fe9fffff ioport:d0000000(size=268435456)
           *-display
                description: VGA compatible controller
                product: G98 [GeForce 8400 GS Rev. 2]
                vendor: NVIDIA Corporation
                physical id: 0
                bus info: pci@0000:01:00.0
                version: a1
                width: 64 bits
                clock: 33MHz
                capabilities: pm msi pciexpress vga_controller bus_master cap_list rom
                configuration: driver=nouveau latency=0
                resources: irq:24 memory:fd000000-fdffffff memory:d0000000-dfffffff memory:fa000000-fbffffff ioport:cc00(size=128) memory:c0000-dffff
        *-usb:0
             description: USB controller
             product: 82801JI (ICH10 Family) USB UHCI Controller #4
             vendor: Intel Corporation
             physical id: 1a
             bus info: pci@0000:00:1a.0
             version: 00
             width: 32 bits
             clock: 33MHz
             capabilities: uhci bus_master cap_list
             configuration: driver=uhci_hcd latency=0
             resources: irq:16 ioport:b800(size=32)
        *-usb:1
             description: USB controller
             product: 82801JI (ICH10 Family) USB UHCI Controller #5
             vendor: Intel Corporation
             physical id: 1a.1
             bus info: pci@0000:00:1a.1
             version: 00
             width: 32 bits
             clock: 33MHz
             capabilities: uhci bus_master cap_list
             configuration: driver=uhci_hcd latency=0
             resources: irq:21 ioport:b880(size=32)
        *-usb:2
             description: USB controller
             product: 82801JI (ICH10 Family) USB UHCI Controller #6
             vendor: Intel Corporation
             physical id: 1a.2
             bus info: pci@0000:00:1a.2
             version: 00
             width: 32 bits
             clock: 33MHz
             capabilities: uhci bus_master cap_list
             configuration: driver=uhci_hcd latency=0
             resources: irq:18 ioport:bc00(size=32)
        *-usb:3
             description: USB controller
             product: 82801JI (ICH10 Family) USB2 EHCI Controller #2
             vendor: Intel Corporation
             physical id: 1a.7
             bus info: pci@0000:00:1a.7
             version: 00
             width: 32 bits
             clock: 33MHz
             capabilities: pm debug ehci bus_master cap_list
             configuration: driver=ehci-pci latency=0
             resources: irq:18 memory:f9fffc00-f9ffffff
        *-multimedia
             description: Audio device
             product: 82801JI (ICH10 Family) HD Audio Controller
             vendor: Intel Corporation
             physical id: 1b
             bus info: pci@0000:00:1b.0
             version: 00
             width: 64 bits
             clock: 33MHz
             capabilities: pm msi pciexpress bus_master cap_list
             configuration: driver=snd_hda_intel latency=0
             resources: irq:25 memory:f9ff8000-f9ffbfff
        *-pci:1
             description: PCI bridge
             product: 82801JI (ICH10 Family) PCI Express Root Port 1
             vendor: Intel Corporation
             physical id: 1c
             bus info: pci@0000:00:1c.0
             version: 00
             width: 32 bits
             clock: 33MHz
             capabilities: pci pciexpress msi pm normal_decode bus_master cap_list
             configuration: driver=pcieport
             resources: irq:17 ioport:1000(size=4096) memory:f0000000-f03fffff ioport:f8f00000(size=1048576)
        *-pci:2
             description: PCI bridge
             product: 82801JI (ICH10 Family) PCI Express Port 2
             vendor: Intel Corporation
             physical id: 1c.1
             bus info: pci@0000:00:1c.1
             version: 00
             width: 32 bits
             clock: 33MHz
             capabilities: pci pciexpress msi pm normal_decode bus_master cap_list
             configuration: driver=pcieport
             resources: irq:16 ioport:e000(size=4096) memory:feb00000-febfffff ioport:f0400000(size=2097152)
           *-ide UNCLAIMED
                description: IDE interface
                product: JMB368 IDE controller
                vendor: JMicron Technology Corp.
                physical id: 0
                bus info: pci@0000:03:00.0
                version: 00
                width: 32 bits
                clock: 33MHz
                capabilities: ide pm pciexpress msi bus_master cap_list
                configuration: latency=0
                resources: ioport:ec00(size=8) ioport:e880(size=4) ioport:e800(size=8) ioport:e480(size=4) ioport:e400(size=16) memory:febf0000-febfffff
        *-pci:3
             description: PCI bridge
             product: 82801JI (ICH10 Family) PCI Express Root Port 3
             vendor: Intel Corporation
             physical id: 1c.2
             bus info: pci@0000:00:1c.2
             version: 00
             width: 32 bits
             clock: 33MHz
             capabilities: pci pciexpress msi pm normal_decode bus_master cap_list
             configuration: driver=pcieport
             resources: irq:18 ioport:d000(size=4096) memory:fea00000-feafffff ioport:f8e00000(size=1048576)
           *-network
                description: Ethernet interface
                product: RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
                vendor: Realtek Semiconductor Co., Ltd.
                physical id: 0
                bus info: pci@0000:02:00.0
                logical name: eth0
                version: 02
                serial: e0:cb:4e:39:19:3a
                size: 100Mbit/s
                capacity: 1Gbit/s
                width: 64 bits
                clock: 33MHz
                capabilities: pm msi pciexpress msix vpd bus_master cap_list rom ethernet physical tp mii 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation
                configuration: autonegotiation=on broadcast=yes driver=r8169 driverversion=2.3LK-NAPI duplex=full ip=192.168.212.11 latency=0 link=yes multicast=yes port=MII speed=100Mbit/s
                resources: irq:26 ioport:d800(size=256) memory:f8eff000-f8efffff memory:f8ee0000-f8eeffff memory:feaf0000-feafffff
        *-usb:4
             description: USB controller
             product: 82801JI (ICH10 Family) USB UHCI Controller #1
             vendor: Intel Corporation
             physical id: 1d
             bus info: pci@0000:00:1d.0
             version: 00
             width: 32 bits
             clock: 33MHz
             capabilities: uhci bus_master cap_list
             configuration: driver=uhci_hcd latency=0
             resources: irq:23 ioport:b080(size=32)
        *-usb:5
             description: USB controller
             product: 82801JI (ICH10 Family) USB UHCI Controller #2
             vendor: Intel Corporation
             physical id: 1d.1
             bus info: pci@0000:00:1d.1
             version: 00
             width: 32 bits
             clock: 33MHz
             capabilities: uhci bus_master cap_list
             configuration: driver=uhci_hcd latency=0
             resources: irq:19 ioport:b400(size=32)
        *-usb:6
             description: USB controller
             product: 82801JI (ICH10 Family) USB UHCI Controller #3
             vendor: Intel Corporation
             physical id: 1d.2
             bus info: pci@0000:00:1d.2
             version: 00
             width: 32 bits
             clock: 33MHz
             capabilities: uhci bus_master cap_list
             configuration: driver=uhci_hcd latency=0
             resources: irq:18 ioport:b480(size=32)
        *-usb:7
             description: USB controller
             product: 82801JI (ICH10 Family) USB2 EHCI Controller #1
             vendor: Intel Corporation
             physical id: 1d.7
             bus info: pci@0000:00:1d.7
             version: 00
             width: 32 bits
             clock: 33MHz
             capabilities: pm debug ehci bus_master cap_list
             configuration: driver=ehci-pci latency=0
             resources: irq:23 memory:f9fff800-f9fffbff
        *-pci:4
             description: PCI bridge
             product: 82801 PCI Bridge
             vendor: Intel Corporation
             physical id: 1e
             bus info: pci@0000:00:1e.0
             version: 90
             width: 32 bits
             clock: 33MHz
             capabilities: pci subtractive_decode bus_master cap_list
        *-isa
             description: ISA bridge
             product: 82801JIB (ICH10) LPC Interface Controller
             vendor: Intel Corporation
             physical id: 1f
             bus info: pci@0000:00:1f.0
             version: 00
             width: 32 bits
             clock: 33MHz
             capabilities: isa bus_master cap_list
             configuration: driver=lpc_ich latency=0
             resources: irq:0
        *-ide:0
             description: IDE interface
             product: 82801JI (ICH10 Family) 4 port SATA IDE Controller #1
             vendor: Intel Corporation
             physical id: 1f.2
             bus info: pci@0000:00:1f.2
             version: 00
             width: 32 bits
             clock: 66MHz
             capabilities: ide pm bus_master cap_list
             configuration: driver=ata_piix latency=0
             resources: irq:19 ioport:a000(size=8) ioport:9c00(size=4) ioport:9880(size=8) ioport:9800(size=4) ioport:9480(size=16) ioport:9400(size=16)
        *-serial UNCLAIMED
             description: SMBus
             product: 82801JI (ICH10 Family) SMBus Controller
             vendor: Intel Corporation
             physical id: 1f.3
             bus info: pci@0000:00:1f.3
             version: 00
             width: 64 bits
             clock: 33MHz
             configuration: latency=0
             resources: memory:f9fff400-f9fff4ff ioport:400(size=32)
        *-ide:1
             description: IDE interface
             product: 82801JI (ICH10 Family) 2 port SATA IDE Controller #2
             vendor: Intel Corporation
             physical id: 1f.5
             bus info: pci@0000:00:1f.5
             version: 00
             width: 32 bits
             clock: 66MHz
             capabilities: ide pm bus_master cap_list
             configuration: driver=ata_piix latency=0
             resources: irq:19 ioport:b000(size=8) ioport:ac00(size=4) ioport:a880(size=8) ioport:a800(size=4) ioport:a480(size=16) ioport:a400(size=16)


These are the HD partitions:
Code:
$ df -h
S.ficheros     Tamaño Usados  Disp Uso% Montado en
/dev/sda2         40G    28G  9,9G  74% /
devtmpfs         2,0G      0  2,0G   0% /dev
tmpfs            2,0G      0  2,0G   0% /dev/shm
tmpfs            2,0G   1,2M  2,0G   1% /run
tmpfs            2,0G      0  2,0G   0% /sys/fs/cgroup
tmpfs            2,0G    12K  2,0G   1% /tmp
/dev/sda6         22G    42M   22G   1% /var/tmp
/dev/sda3        168G   140G   20G  88% /home
tmpfs            395M   8,0K  395M   1% /run/user/1000
/dev/sda7        208G   183G   26G  88% /run/media/ramiro/Trabajo
Back to top
View user's profile Send private message
fedeliallalinea
Bodhisattva
Bodhisattva


Joined: 08 Mar 2003
Posts: 22699
Location: here

PostPosted: Mon Jan 14, 2019 2:55 pm    Post subject: Reply with quote

qtwebengine include chromium code, and no, webkit-gtk isn't equivalent to qtwebengine in term of compilation time.
In my i7-4930K_CPU_@_3.40GHz
Code:
     Thu Jan  3 10:36:09 2019 >>> dev-qt/qtwebengine-5.11.3
       merge time: 1 hour, 4 minutes and 5 seconds.

     Thu Sep  8 08:05:35 2016 >>> net-libs/webkit-gtk-2.4.11-r200
       merge time: 25 minutes and 15 seconds.

_________________
Questions are guaranteed in life; Answers aren't.
Back to top
View user's profile Send private message
urcindalo
Guru
Guru


Joined: 08 Feb 2005
Posts: 594
Location: Almeria, Spain

PostPosted: Mon Jan 14, 2019 3:11 pm    Post subject: Reply with quote

Thanks for the info.

By "equivalent" I mean source code size: I also show libreoffice compilation time.

In your case, qtwebengine compilation time doubles that of webkit-gtk, but in mine it is up to six or seven times larger because, as a side note, I must say the compilation times showed by splat are not real: last one took a little bit more than two days, not 1.4 as reported.
So, I find it strange it takes that long.

For instance, in a more powerful box I have the compilation times for qtwebengine and webkit-gtk are:
Code:
$ sudo splat qtwebengine
 * dev-qt/qtwebengine-5.11.3

        Emerged at: jue ene 10 10:11:42 2019
        Build time: 3 hours, 33 minutes, and 42 seconds

.....

$ sudo splat webkit-gtk
net-libs/webkit-gtk-2.22.5

        Emerged at: jue ene 10 19:42:00 2019
        Build time: 1 hour, 28 minutes, and 6 seconds


As you can see, about two times more, as in your case, not six or seven.


Last edited by urcindalo on Tue Jan 15, 2019 7:55 am; edited 1 time in total
Back to top
View user's profile Send private message
Chiitoo
Administrator
Administrator


Joined: 28 Feb 2010
Posts: 1886
Location: Here and Away Again

PostPosted: Mon Jan 14, 2019 9:50 pm    Post subject: ><)))°€ Reply with quote

For a bit of a comparison, I have a log of 6 hours, 36 minutes, and 6 seconds on qtwebengine-5.11.1, on an AMD Athlon 7750 dual core processor with around 3 GiBs of RAM. I do forget if that build made use of distcc or not, but I'd still probably expect yours to be able to do better.

The Qt WebEngine build really likes RAM. A lot. Which you seem to have around 4 GiB, and my first suspect therefore would be major swapping during the build.

If you have been building with USE="jumbo-build", I might try without it, and/or lowering the MAKEOPTS value.

Or/and add more RAM, if possible. :]
_________________
Kind regards,
Chiitoo.

You might remember me from Gentoo projects such as Forums, LXQt, Qt, and Wine.
Back to top
View user's profile Send private message
urcindalo
Guru
Guru


Joined: 08 Feb 2005
Posts: 594
Location: Almeria, Spain

PostPosted: Tue Jan 15, 2019 7:59 am    Post subject: Re: ><)))°€ Reply with quote

Chiitoo wrote:
The Qt WebEngine build really likes RAM. A lot. Which you seem to have around 4 GiB, and my first suspect therefore would be major swapping during the build.


Maybe that's the reason. My fast box has 8Gb RAM. I wonder if 8Gb is the bare minimum for qtwebengine to compile "normally" (as other packages do).
Back to top
View user's profile Send private message
mael
n00b
n00b


Joined: 28 Feb 2018
Posts: 37
Location: Germany

PostPosted: Tue Jan 15, 2019 9:28 am    Post subject: Reply with quote

Code:


dev-qt/qtwebengine-5.11.3

Emerged at: Mi Jan  2 13:37:06 2019
Build time: 1 hour, 29 minutes, and 49 seconds



compiled with ggdb debug symbols, -j16 makeopts and in my ram disk. it eats up sometimes 22 gb of my systems memory (just only the build files in ram disk) but also there where some other heavy tasks running so it could be a lot faster next time.

i concur with my colleagues here: swapping is a b***h that extends SERIOUSLY compile time and please dont forget: qtwebengine is one of the biggest packages in gentoos repository so it takes its time
_________________
Gigabyte X99-SLI
Intel Core i7-5960X 16 Threads @ 3 Ghz
64 GB DDR4 RAM @ 2,13 Ghz
AMD Radeon 290X 8 GB RAM
Creative Soundblaster Z
Areca ARC-1883ix-16
16 Samsung 860 EVO 250 GB SSDs @ 3,5 TB RAID 6
Hauppauge HVR-5525
Firmwares are up-to-date
Back to top
View user's profile Send private message
asturm
Developer
Developer


Joined: 05 Apr 2007
Posts: 7214
Location: Austria

PostPosted: Tue Jan 15, 2019 11:42 am    Post subject: Re: ><)))°€ Reply with quote

urcindalo wrote:
Maybe that's the reason. My fast box has 8Gb RAM. I wonder if 8Gb is the bare minimum for qtwebengine to compile "normally" (as other packages do).

No, it builds fine on my boxes with only 4 GB. But when I do that, I let it build overnight or while doing nothing else on the system.
_________________
backend.cpp:92:2: warning: #warning TODO - this error message is about as useful as a cooling unit in the arctic
Back to top
View user's profile Send private message
urcindalo
Guru
Guru


Joined: 08 Feb 2005
Posts: 594
Location: Almeria, Spain

PostPosted: Tue Jan 15, 2019 11:57 am    Post subject: Re: ><)))°€ Reply with quote

asturm wrote:
urcindalo wrote:
Maybe that's the reason. My fast box has 8Gb RAM. I wonder if 8Gb is the bare minimum for qtwebengine to compile "normally" (as other packages do).

No, it builds fine on my boxes with only 4 GB. But when I do that, I let it build overnight or while doing nothing else on the system.


This is what I do whenever I see qtwebengine is to be compiled:
1) I reboot the computer
2) I ssh to it from another box
3) I run a screen session to upgrade
4) I quit and, every now and then, I ssh again to look at the progress.

I noticed that, if I don't proceed this way, qtwebengine even fails to compile (emerging is interrupted).

The thing that you can do it overnight with just 4Gb RAM makes me wonder what's the issue in my case.
Back to top
View user's profile Send private message
mael
n00b
n00b


Joined: 28 Feb 2018
Posts: 37
Location: Germany

PostPosted: Tue Jan 15, 2019 2:21 pm    Post subject: Reply with quote

you should also check if your cpu or memory is temporary clocked down because longtime compilings like qtwebengine can stress out bad cooling systems in some boxes or notebooks and therefore activate some slowing safety measures inside your online cpus
_________________
Gigabyte X99-SLI
Intel Core i7-5960X 16 Threads @ 3 Ghz
64 GB DDR4 RAM @ 2,13 Ghz
AMD Radeon 290X 8 GB RAM
Creative Soundblaster Z
Areca ARC-1883ix-16
16 Samsung 860 EVO 250 GB SSDs @ 3,5 TB RAID 6
Hauppauge HVR-5525
Firmwares are up-to-date
Back to top
View user's profile Send private message
urcindalo
Guru
Guru


Joined: 08 Feb 2005
Posts: 594
Location: Almeria, Spain

PostPosted: Wed Jan 16, 2019 8:02 am    Post subject: Reply with quote

mael wrote:
you should also check if your cpu or memory is temporary clocked down because longtime compilings like qtwebengine can stress out bad cooling systems in some boxes or notebooks and therefore activate some slowing safety measures inside your online cpus


Good catch! It seems quite logical. I'll take a look.
Back to top
View user's profile Send private message
brokendreams
n00b
n00b


Joined: 11 Nov 2018
Posts: 22
Location: clinton,ia US

PostPosted: Wed Jan 16, 2019 5:37 pm    Post subject: Reply with quote

Adding --quiet to the emerge command will speed this up some. As well as adding -w to cflags and cxxflags (disables printing of most warning messages). Shaved 30mins off my build time (Intel 7700hq cpu). Started at 2 hrs...dropped it down to just under 90mins and thats with all use flags enabled. And portage tmpdir on tmpfs.
Back to top
View user's profile Send private message
curmudgeon
Veteran
Veteran


Joined: 08 Aug 2003
Posts: 1546

PostPosted: Wed Jan 16, 2019 6:04 pm    Post subject: Reply with quote

I hate having to emerge qtwebengine (and recently spent the day emerging it because of an dev-libs/icu update, and then had to emerge it a couple of days later when a newer version of qtwebengine itself became stable (GRRRRR!).

It takes between five and fifteen hours on my machines, and used to crash more often than not. The biggest problem (which has only appeared in the most recent versions) is that each thread takes up about 2.5 GB of RAM, so if you have lots of cores, but not a corresponding amount of RAM, you will thrash the hard drive by swapping everything at best, or exhaust memory and crash.

I can at least get it to compile by limiting threads to one on machines with 4 GB of RAM, and two on machines with 8 GB of RAM (instead of the usual -j5). That actually seems faster (haven't done any empirical tests), because the processes aren't always stalled waiting for RAM to become available).

The easiest way to do that is:

Code:

# echo dev-qt/qtwebengine makeopts-j1 >> /etc/portage/package.env
# mkdir /etc/portage/env
# echo "MAKEOPTS='-j1'" >> /etc/portage/env/makeopts-j1
Back to top
View user's profile Send private message
mike155
Veteran
Veteran


Joined: 17 Sep 2010
Posts: 1728
Location: Frankfurt, Germany

PostPosted: Wed Jan 16, 2019 6:22 pm    Post subject: Reply with quote

Quote:
I hate having to emerge qtwebengine

I fully agree. But I found a solution: I changed a few USE Flags and qtwebengine is not getting pulled in anymore.

Really, look at the packages that pull in qtwebengine
Code:
equery d qtwebengine

If you're lucky, you can get rid of qtwebengine by changing a few USE flags. Maybe you will also have to drop a few packages that pull in qtwebengine unconditionally. But it's better to drop a few packages than having to compile qtwebengine!
Back to top
View user's profile Send private message
urcindalo
Guru
Guru


Joined: 08 Feb 2005
Posts: 594
Location: Almeria, Spain

PostPosted: Thu Jan 17, 2019 9:54 am    Post subject: Reply with quote

curmudgeon wrote:
I hate having to emerge qtwebengine (and recently spent the day emerging it because of an dev-libs/icu update, and then had to emerge it a couple of days later when a newer version of qtwebengine itself became stable (GRRRRR!).

It takes between five and fifteen hours on my machines, and used to crash more often than not. The biggest problem (which has only appeared in the most recent versions) is that each thread takes up about 2.5 GB of RAM, so if you have lots of cores, but not a corresponding amount of RAM, you will thrash the hard drive by swapping everything at best, or exhaust memory and crash.

I can at least get it to compile by limiting threads to one on machines with 4 GB of RAM, and two on machines with 8 GB of RAM (instead of the usual -j5). That actually seems faster (haven't done any empirical tests), because the processes aren't always stalled waiting for RAM to become available).

The easiest way to do that is:

Code:

# echo dev-qt/qtwebengine makeopts-j1 >> /etc/portage/package.env
# mkdir /etc/portage/env
# echo "MAKEOPTS='-j1'" >> /etc/portage/env/makeopts-j1


You have described my situation to the letter and your suggestion seems to me like the easiest solution. Thanks!!
I just configured my system as per your instructions.
Back to top
View user's profile Send private message
sao98021
Tux's lil' helper
Tux's lil' helper


Joined: 03 Apr 2012
Posts: 144
Location: Michigan

PostPosted: Thu Jan 17, 2019 11:22 am    Post subject: Reply with quote

Code:

     Sat Jun  9 12:13:43 2018 >>> dev-qt/qtwebengine-5.9.4
       merge time: 2 hours, 33 minutes and 39 seconds.

     Sun Sep 23 04:57:08 2018 >>> dev-qt/qtwebengine-5.9.6-r1
       merge time: 2 hours, 50 minutes.

    Fri Jan  4 18:53:01 2019 >>> dev-qt/qtwebengine-5.11.1
       merge time: 2 hours, 30 minutes and 47 seconds.

     Thu Jan 17 02:47:47 2019 >>> dev-qt/qtwebengine-5.11.3
       merge time: 2 hours, 33 minutes and 18 seconds.


Code:
[?] dev-qt/qtwebengine
     Available versions:  (5) 5.9.6-r1(5/5.9)^t 5.11.1(5/5.11)^t ~5.11.2(5/5.11)^t
       {alsa bindist debug designer geolocation +jumbo-build pax_kernel pulseaudio +system-ffmpeg +system-icu test widgets}
     Installed versions:  5.11.3(5/5.11)^t(02:35:03 AM 01/17/2019)(alsa jumbo-build system-ffmpeg system-icu widgets -bindist -debug -designer -geolocation -pax_kernel -pulseaudio -test)
     Homepage:            https://www.qt.io/
     Description:         Library for rendering dynamic web content in Qt5 C++ and QML applications


jumbo-build and 8gb ram, makeopts -j7
Back to top
View user's profile Send private message
urcindalo
Guru
Guru


Joined: 08 Feb 2005
Posts: 594
Location: Almeria, Spain

PostPosted: Thu Jan 17, 2019 3:06 pm    Post subject: Reply with quote

mike155 wrote:
Quote:
I hate having to emerge qtwebengine

I fully agree. But I found a solution: I changed a few USE Flags and qtwebengine is not getting pulled in anymore.

Really, look at the packages that pull in qtwebengine
Code:
equery d qtwebengine

If you're lucky, you can get rid of qtwebengine by changing a few USE flags. Maybe you will also have to drop a few packages that pull in qtwebengine unconditionally. But it's better to drop a few packages than having to compile qtwebengine!


Apart from following curmudgeon's directions, just in case, I tried also yours... and it was easier than expected. It turned out the only packages depending on qtwebengine were: knotes -which I don't use-; luminance-hdr -which I can replace with its appimage- and mendeleydesktop, which I can install through flatpak. I also had to remove the webengine USE flag for kdecore-meta and konqueror and the gui USE flag for gpsbabel. No big deal, after all, to get rid of this monster :lol:
Back to top
View user's profile Send private message
dmpogo
Advocate
Advocate


Joined: 02 Sep 2004
Posts: 2542
Location: Canada

PostPosted: Thu Jan 17, 2019 8:01 pm    Post subject: Reply with quote

mike155 wrote:
Quote:
I hate having to emerge qtwebengine

I fully agree. But I found a solution: I changed a few USE Flags and qtwebengine is not getting pulled in anymore.

Really, look at the packages that pull in qtwebengine
Code:
equery d qtwebengine

If you're lucky, you can get rid of qtwebengine by changing a few USE flags. Maybe you will also have to drop a few packages that pull in qtwebengine unconditionally. But it's better to drop a few packages than having to compile qtwebengine!


The point is that one probably needs at least one of the huge packages for web browsing - firefox, chromium or qtwebengine (webkit is a lighter option - but is there good browser based on it left ?) However, you perhapds do not need more than one. So make your pick judicially and try to get rid of the rest.
Back to top
View user's profile Send private message
sao98021
Tux's lil' helper
Tux's lil' helper


Joined: 03 Apr 2012
Posts: 144
Location: Michigan

PostPosted: Thu Jan 17, 2019 8:09 pm    Post subject: Reply with quote

dmpogo wrote:
mike155 wrote:
Quote:
I hate having to emerge qtwebengine

I fully agree. But I found a solution: I changed a few USE Flags and qtwebengine is not getting pulled in anymore.

Really, look at the packages that pull in qtwebengine
Code:
equery d qtwebengine

If you're lucky, you can get rid of qtwebengine by changing a few USE flags. Maybe you will also have to drop a few packages that pull in qtwebengine unconditionally. But it's better to drop a few packages than having to compile qtwebengine!


The point is that one probably needs at least one of the huge packages for web browsing - firefox, chromium or qtwebengine (webkit is a lighter option - but is there good browser based on it left ?) However, you perhapds do not need more than one. So make your pick judicially and try to get rid of the rest.


https://linux.palemoon.org/ forked from firefox, like a faster firefox. its deps https://github.com/gentoo-mirror/palemoon/blob/master/www-client/palemoon/palemoon-28.3.0.ebuild
Back to top
View user's profile Send private message
mael
n00b
n00b


Joined: 28 Feb 2018
Posts: 37
Location: Germany

PostPosted: Fri Jan 18, 2019 3:25 pm    Post subject: Reply with quote

he is like me probably a qt-5/kde-5 user, so migrating to palemoon and other gtk+ based stuff is not what he wants, right? are you using falkon by any chance?
_________________
Gigabyte X99-SLI
Intel Core i7-5960X 16 Threads @ 3 Ghz
64 GB DDR4 RAM @ 2,13 Ghz
AMD Radeon 290X 8 GB RAM
Creative Soundblaster Z
Areca ARC-1883ix-16
16 Samsung 860 EVO 250 GB SSDs @ 3,5 TB RAID 6
Hauppauge HVR-5525
Firmwares are up-to-date
Back to top
View user's profile Send private message
dmpogo
Advocate
Advocate


Joined: 02 Sep 2004
Posts: 2542
Location: Canada

PostPosted: Fri Jan 18, 2019 5:39 pm    Post subject: Reply with quote

mael wrote:
he is like me probably a qt-5/kde-5 user, so migrating to palemoon and other gtk+ based stuff is not what he wants, right? are you using falkon by any chance?


I am using falkon, indeed :) gave up on firefox when it went to 60+ version with rust/clang requirements. Now suffer with qtwebengine recompilations :)
Back to top
View user's profile Send private message
dmpogo
Advocate
Advocate


Joined: 02 Sep 2004
Posts: 2542
Location: Canada

PostPosted: Fri Jan 18, 2019 11:10 pm    Post subject: Reply with quote

Honestly speaking, it would go long way in aleviating problems with long compilations, if there was some coordination between releases of large packages and their dependencies which cause recompilation. icu is one of the main culprits. All the time there is update to icu, recompilation, and then couple of days later there is update to qtwebengine or libreoffice, and you start all over again. Or in reverse order.
Back to top
View user's profile Send private message
xor_
n00b
n00b


Joined: 09 Aug 2009
Posts: 10

PostPosted: Sat Jan 19, 2019 11:01 pm    Post subject: Reply with quote

Quote:

The point is that one probably needs at least one of the huge packages for web browsing - firefox, chromium or qtwebengine (webkit is a lighter option - but is there good browser based on it left ?) However, you perhapds do not need more than one. So make your pick judicially and try to get rid of the rest.


I've resolved to locally updating the current firefox-binaries outside of portage.

No doubt even qtwekbit has become a monster. It brought my mom's perfectly good i5 t430 with 4GB of RAM to its knees, and it overheats my t440p with 8 GB. (Which got me looking into micro-managing the intel p-states driver).

I've also been trying to eliminate it, but so far my need for teamviewer to remote-administrate my mom keeps me keeping it around.
Back to top
View user's profile Send private message
dmpogo
Advocate
Advocate


Joined: 02 Sep 2004
Posts: 2542
Location: Canada

PostPosted: Wed Jan 23, 2019 12:20 am    Post subject: Reply with quote

dmpogo wrote:
Honestly speaking, it would go long way in aleviating problems with long compilations, if there was some coordination between releases of large packages and their dependencies which cause recompilation. icu is one of the main culprits. All the time there is update to icu, recompilation, and then couple of days later there is update to qtwebengine or libreoffice, and you start all over again. Or in reverse order.


And just few days as I wrote my post, the icu is again updated and qtwebengine and libreoffice need to be recompiled :(
Back to top
View user's profile Send private message
Jaglover
Watchman
Watchman


Joined: 29 May 2005
Posts: 7328
Location: Saint Amant, Acadiana

PostPosted: Wed Jan 23, 2019 12:34 am    Post subject: Reply with quote

ccache may be evil, but it helps in such cases.
_________________
Please learn how to denote units correctly!
Back to top
View user's profile Send private message
mike155
Veteran
Veteran


Joined: 17 Sep 2010
Posts: 1728
Location: Frankfurt, Germany

PostPosted: Wed Jan 23, 2019 12:45 am    Post subject: Reply with quote

Quote:
And just few days as I wrote my post, the icu is again updated and qtwebengine and libreoffice need to be recompiled

Come on, mask the new version in /etc/portage/package.mask. Keep the mask until you really want to update qtwebengine or Libreoffice.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Goto page 1, 2, 3  Next
Page 1 of 3

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum