QtWebEngine - compile time

[DarkAudit]

Something's changed in the r1 package, hasn't it? There's no other way to explain how a package that takes 12 hours or more to build on an A10-8700P laptop suddenly is taking just over an hour.

[asturm]

It just means you have ccache enabled.

[DarkAudit]

It just means you have ccache enabled.

I've had ccache enabled for months, and the previous dev-qt/qtwebengine-5.15.2_p20210824 still took 17 hours to compile.

[asturm]

exactly.

[Hu]

Can I avoid building qtwebengine 3 times, by building it once and then make a binary with

Code: Select all

quickpkg qtwebengine

and copy it to the other 2 installs to their /usr/portage/packages/dev-qt/ directories, and then emerge it with

Code: Select all

emerge -K qtwebengine

whenever portage wants to rebuild or update it on an @world update?

That is an unusually manual way of implementing a binhost server, but yes, that should work. The only caveat is that your binary package needs to meet all of Portage's criteria for the secondary machines: correct version, acceptable keywords, matching USE flags and subslots, etc. I expect the realistic failure modes to be that Portage shows an error message and refuses to proceed, so this should be a safe experiment even if it takes you a few tries to get right.

If I were doing this, I would use FEATURES=buildpkg on the primary instead of using quickpkg. If I were doing it much, I would consider using either an NFS mount of the packages directory from primary onto secondaries, or setting up the Portage binhost variables for use by --getbinpkg.

Something's changed in the r1 package, hasn't it? There's no other way to explain how a package that takes 12 hours or more to build on an A10-8700P laptop suddenly is taking just over an hour.

Quite the opposite, actually. If much had changed, ccache would have failed to get many/any cache hits, and you would have rebuilt almost everything. As it is, you got lucky that -r1 made very few changes to central files, so most of the source files were eligible to be retrieved from cache instead of being rebuilt.

[Tony0945]

I forget what -K means.
I use this method to build packages for amdfam10 on ryzen.
I compile with --buildpkgonly and on the target I extract with --usepkgonly
I also switch directories.

Of course when using a different architecture you must change march also.
I automate this with some scripts.

If you are gong to change flags and profiles, you are better off with cross-compiling.

[pd1986]

Everytime i complied qtwebengine, I had to reduce the thread from -j12 to -j6. otherwise, it would fail. Does anyone know why? Much appreciated.

[Tony0945]

Everytime i complied qtwebengine, I had to reduce the thread from -j12 to -j6. otherwise, it would fail. Does anyone know why? Much appreciated.

I leave my 2700X at -j16 I have 32G memory, no swap.
Maybe you need more memory.

[ulenrich]

Everytime i complied qtwebengine, I had to reduce the thread from -j12 to -j6. otherwise, it would fail. Does anyone know why? Much appreciated.

I leave my 2700X at -j16 I have 32G memory, no swap.
Maybe you need more memory.

Yes, qtwebengine needs very much memory to compile: With 4gByte RAM I need another 4bB on swap and I cannot run anything else during compilation. Another malfunction could be due to bad memory at regions normally not used that much.

[firephoto]

Everytime i complied qtwebengine, I had to reduce the thread from -j12 to -j6. otherwise, it would fail. Does anyone know why? Much appreciated.

If you run out of memory for it to build it will fail so by reducing the parallel builds you use less memory. Most of it would probably build at the higher -j but there are few parts that eat up the memory.

I have MAKEOPTS="-j13 -l23" for qtwebengine with 24 cores and 32G ram and tmp/portage on tmpfs up to 16G. Higher -j just had me hitting swap and I still get the same compile times. It did fail the other day but I had like 20 tabs open in firefox and some in chrome, and firefox and wine ended up building at the same time, and I wasn't paying attention to all the memory going away until it went oom when I was restarting firefox because I had been trying to fix emojis (critical things). Waited a few minutes to be able to do anything, qtwebengine failed, firefox and wine were done building. Using zswap too if it matters. If tmp wasn't on ram it wouldn't happen and it might be slower but it would also just be thrashing the ssd more than is necessary.


This is on a 5900X I upgraded from a 2700X.

Code: Select all

     Sun Oct 10 22:09:39 2021 >>> dev-qt/qtwebengine-5.15.2_p20210824-r1
       merge time: 43 minutes and 50 seconds.

[wrc1944]

IIRC, I've had it fail on huge long compiles if I forget to make sure /var/tmp/portage wasn't full of failed packages clutter, which can amount to many Gb's.

This would only happen if the root partition was small, and tends to run out of disk space, which is true on 2 of my Gentoo installs.

I know it's ridiculous, but I still have 2 gentoos, and 6-7 other distros I fool with, plus 2 100GB data partitions on one 4+ years old 500GB hard disk. Guess my luck is running out, and I better clean it all up and get a new and bigger disk, and/or re-allocate data and partition arrangements to reflect a little sanity.

[Fitzcarraldo]

wrc1944,

You could automatically empty /var/tmp/portage/ when it gets too full. I created a file /etc/local.d/99delete_tmp_files_from_failed_merges.stop containing the following code:

Code: Select all

#!/bin/bash
# If root partition is more than 50% full, delete any temporary directories and
# files that were left in /var/tmp/portage/ instead of being deleted.
#
# My root partition is on /dev/sda6 and the emerge command must not be running.
#
if [ `pgrep -c emerge` -eq 0 ] && [ `df | awk '/sda6/ {print $5}' | awk -F% '{print $1}'` -gt 50 ]; then
    rm -rf /usr/tmp/portage/*
fi

If the root partition is more than 50% full when I shut down the laptop, the script will automatically empty that directory. One less thing to think about.

[nikolis]

Everytime i complied qtwebengine, I had to reduce the thread from -j12 to -j6. otherwise, it would fail. Does anyone know why? Much appreciated.

Unfortunately, you need more memory.

[Hu]

If the root partition is more than 50% full when I shut down the laptop, the script will automatically empty that directory. One less thing to think about.

If you would prefer rapid cleanup, there is FEATURES=fail-clean.

Code: Select all

              fail-clean
                     Clean  up  temporary files after a build failure. This is
                     particularly useful if you have PORTAGE_TMPDIR on  tmpfs.
                     If this feature is enabled, you probably also want to en‐
                     able PORTAGE_LOGDIR in order to save the build log.  Both
                     the  ebuild(1)  command and the noclean feature cause the
                     fail-clean feature to be automatically disabled.

This does have the drawback that it is immediate, so there is no opportunity to inspect or attempt to repair a failed build.

[pd1986]

Everytime i complied qtwebengine, I had to reduce the thread from -j12 to -j6. otherwise, it would fail. Does anyone know why? Much appreciated.

I leave my 2700X at -j16 I have 32G memory, no swap.
Maybe you need more memory.

Everytime i complied qtwebengine, I had to reduce the thread from -j12 to -j6. otherwise, it would fail. Does anyone know why? Much appreciated.

I leave my 2700X at -j16 I have 32G memory, no swap.
Maybe you need more memory.

Yes, qtwebengine needs very much memory to compile: With 4gByte RAM I need another 4bB on swap and I cannot run anything else during compilation. Another malfunction could be due to bad memory at regions normally not used that much.

Everytime i complied qtwebengine, I had to reduce the thread from -j12 to -j6. otherwise, it would fail. Does anyone know why? Much appreciated.

If you run out of memory for it to build it will fail so by reducing the parallel builds you use less memory. Most of it would probably build at the higher -j but there are few parts that eat up the memory.

I have MAKEOPTS="-j13 -l23" for qtwebengine with 24 cores and 32G ram and tmp/portage on tmpfs up to 16G. Higher -j just had me hitting swap and I still get the same compile times. It did fail the other day but I had like 20 tabs open in firefox and some in chrome, and firefox and wine ended up building at the same time, and I wasn't paying attention to all the memory going away until it went oom when I was restarting firefox because I had been trying to fix emojis (critical things). Waited a few minutes to be able to do anything, qtwebengine failed, firefox and wine were done building. Using zswap too if it matters. If tmp wasn't on ram it wouldn't happen and it might be slower but it would also just be thrashing the ssd more than is necessary.


This is on a 5900X I upgraded from a 2700X.

Code: Select all

     Sun Oct 10 22:09:39 2021 >>> dev-qt/qtwebengine-5.15.2_p20210824-r1
       merge time: 43 minutes and 50 seconds.

Understood. I never paid attention to the memory usage. I have 16G memory without setting swap. Thanks for pointing it out

[Hund]

Does anyone know anything about the jumbo-build feature, if it's coming back or if we're doomed at long compile times forever now?

[Ionen]

Does anyone know anything about the jumbo-build feature, if it's coming back or if we're doomed at long compile times forever now?

But it's already there? qtwebengine has USE=jumbo-build and it's enabled by default.

[Hund]

Does anyone know anything about the jumbo-build feature, if it's coming back or if we're doomed at long compile times forever now?

But it's already there? qtwebengine has USE=jumbo-build and it's enabled by default.

Compile fails when jumbo-build is enabled, it doesn't matter how low the value is. It's been like this since last year, it used to work just fine before.

[Hu]

Your first question reads to me like:

• USE=jumbo-build previously existed.
• It no longer exists, but I want to use it.
• Does anyone know if it will come back so that I can use it?

If it is present, but does not work, then that is a different problem. Since Ionen says it is enabled by default, I expect that it works by default for whoever tested it. If it does not work for you, we should understand why.

[Hund]

Your first question reads to me like:

• USE=jumbo-build previously existed.
• It no longer exists, but I want to use it.
• Does anyone know if it will come back so that I can use it?

If it is present, but does not work, then that is a different problem. Since Ionen says it is enabled by default, I expect that it works by default for whoever tested it. If it does not work for you, we should understand why.

It used to work fine. It took me 'only' 2 hours to compile it then. Unfortunately, it was later completely removed and the time went up to 4-5 hours for me. It was then later added back, but it has never worked for me since it came back.

[asturm]

So, when did it last not work, and how did it fail?

[Hund]

So, when did it last not work, and how did it fail?

I can't remember now, but I'll show you a log the next time there's an update and I'll try to update it with jumbo-build enabled. :)

[Hund]

So, when did it last not work, and how did it fail?

Here's the emerge log for it: https://cloud.operationtulip.com/s/x99Aw5mY3sZnZYd

[Chiitoo]

So, when did it last not work, and how did it fail?

Here's the emerge log for it: https://cloud.operationtulip.com/s/x99Aw5mY3sZnZYd

Does it really fail that way for you when using the default 'jumbo_file_merge_limit' as well?

[Hund]

So, when did it last not work, and how did it fail?

Here's the emerge log for it: https://cloud.operationtulip.com/s/x99Aw5mY3sZnZYd

Does it really fail that way for you when using the default 'jumbo_file_merge_limit' as well?

What's the default? I have tried setting it to 100 and 50. I can't remember if I tried as low as 25.