| View previous topic :: View next topic |
| Author |
Message |
DarkAudit
n00b
Joined: 27 Mar 2021
Posts: 27
|
 Posted: Mon Oct 11, 2021 9:34 am Post subject: |
 |
|
| Something's changed in the r1 package, hasn't it? There's no other way to explain how a package that takes 12 hours or more to build on an A10-8700P laptop suddenly is taking just over an hour. |
|
| Back to top |
|
 |
asturm
Developer
Joined: 05 Apr 2007
Posts: 8936
|
| Posted: Mon Oct 11, 2021 9:47 am Post subject: |
|
|
| It just means you have ccache enabled. |
|
| Back to top |
|
|
DarkAudit
n00b
Joined: 27 Mar 2021
Posts: 27
|
| Posted: Mon Oct 11, 2021 12:37 pm Post subject: |
|
|
| asturm wrote: | | It just means you have ccache enabled. |
I've had ccache enabled for months, and the previous dev-qt/qtwebengine-5.15.2_p20210824 still took 17 hours to compile. |
|
| Back to top |
|
|
asturm
Developer
Joined: 05 Apr 2007
Posts: 8936
|
| Posted: Mon Oct 11, 2021 1:07 pm Post subject: |
|
|
| exactly. |
|
| Back to top |
|
|
Hu
Moderator
Joined: 06 Mar 2007
Posts: 21631
|
| Posted: Mon Oct 11, 2021 3:42 pm Post subject: |
|
|
| wrc1944 wrote: | Can I avoid building qtwebengine 3 times, by building it once and then make a binary with | Code: | | quickpkg qtwebengine |
and copy it to the other 2 installs to their /usr/portage/packages/dev-qt/ directories, and then emerge it with | Code: | | emerge -K qtwebengine |
whenever portage wants to rebuild or update it on an @world update? |
That is an unusually manual way of implementing a binhost server, but yes, that should work. The only caveat is that your binary package needs to meet all of Portage's criteria for the secondary machines: correct version, acceptable keywords, matching USE flags and subslots, etc. I expect the realistic failure modes to be that Portage shows an error message and refuses to proceed, so this should be a safe experiment even if it takes you a few tries to get right.
If I were doing this, I would use FEATURES=buildpkg on the primary instead of using quickpkg. If I were doing it much, I would consider using either an NFS mount of the packages directory from primary onto secondaries, or setting up the Portage binhost variables for use by --getbinpkg. | DarkAudit wrote: | | Something's changed in the r1 package, hasn't it? There's no other way to explain how a package that takes 12 hours or more to build on an A10-8700P laptop suddenly is taking just over an hour. |
Quite the opposite, actually. If much had changed, ccache would have failed to get many/any cache hits, and you would have rebuilt almost everything. As it is, you got lucky that -r1 made very few changes to central files, so most of the source files were eligible to be retrieved from cache instead of being rebuilt. |
|
| Back to top |
|
|
Tony0945
Watchman
Joined: 25 Jul 2006
Posts: 5127
Location: Illinois, USA
|
| Posted: Mon Oct 11, 2021 5:09 pm Post subject: |
|
|
I forget what -K means.
I use this method to build packages for amdfam10 on ryzen.
I compile with --buildpkgonly and on the target I extract with --usepkgonly
I also switch directories.
Of course when using a different architecture you must change march also.
I automate this with some scripts.
If you are gong to change flags and profiles, you are better off with cross-compiling. |
|
| Back to top |
|
|
pd1986
Guru
Joined: 19 Feb 2012
Posts: 404
Location: Paris
|
| Posted: Fri Oct 15, 2021 1:07 pm Post subject: |
|
|
| Everytime i complied qtwebengine, I had to reduce the thread from -j12 to -j6. otherwise, it would fail. Does anyone know why? Much appreciated. |
|
| Back to top |
|
|
Tony0945
Watchman
Joined: 25 Jul 2006
Posts: 5127
Location: Illinois, USA
|
| Posted: Fri Oct 15, 2021 2:15 pm Post subject: |
|
|
| pd1986 wrote: | | Everytime i complied qtwebengine, I had to reduce the thread from -j12 to -j6. otherwise, it would fail. Does anyone know why? Much appreciated. |
I leave my 2700X at -j16 I have 32G memory, no swap.
Maybe you need more memory. |
|
| Back to top |
|
|
ulenrich
Veteran
Joined: 10 Oct 2010
Posts: 1480
|
| Posted: Fri Oct 15, 2021 5:39 pm Post subject: |
|
|
| Tony0945 wrote: | | pd1986 wrote: | | Everytime i complied qtwebengine, I had to reduce the thread from -j12 to -j6. otherwise, it would fail. Does anyone know why? Much appreciated. |
I leave my 2700X at -j16 I have 32G memory, no swap.
Maybe you need more memory. |
Yes, qtwebengine needs very much memory to compile: With 4gByte RAM I need another 4bB on swap and I cannot run anything else during compilation. Another malfunction could be due to bad memory at regions normally not used that much. |
|
| Back to top |
|
|
firephoto
Veteran
Joined: 29 Oct 2003
Posts: 1612
Location: +48° 5' 23.40", -119° 48' 30.00"
|
| Posted: Fri Oct 15, 2021 6:18 pm Post subject: |
|
|
| pd1986 wrote: | | Everytime i complied qtwebengine, I had to reduce the thread from -j12 to -j6. otherwise, it would fail. Does anyone know why? Much appreciated. |
If you run out of memory for it to build it will fail so by reducing the parallel builds you use less memory. Most of it would probably build at the higher -j but there are few parts that eat up the memory.
I have MAKEOPTS="-j13 -l23" for qtwebengine with 24 cores and 32G ram and tmp/portage on tmpfs up to 16G. Higher -j just had me hitting swap and I still get the same compile times. It did fail the other day but I had like 20 tabs open in firefox and some in chrome, and firefox and wine ended up building at the same time, and I wasn't paying attention to all the memory going away until it went oom when I was restarting firefox because I had been trying to fix emojis (critical things). Waited a few minutes to be able to do anything, qtwebengine failed, firefox and wine were done building. Using zswap too if it matters. If tmp wasn't on ram it wouldn't happen and it might be slower but it would also just be thrashing the ssd more than is necessary.
This is on a 5900X I upgraded from a 2700X.
| Code: |
Sun Oct 10 22:09:39 2021 >>> dev-qt/qtwebengine-5.15.2_p20210824-r1
merge time: 43 minutes and 50 seconds.
|
_________________
#gentoo-kde on freenode |
|
| Back to top |
|
|
wrc1944
Advocate
Joined: 15 Aug 2002
Posts: 3435
Location: Gainesville, Florida
|
| Posted: Fri Oct 15, 2021 7:43 pm Post subject: |
|
|
IIRC, I've had it fail on huge long compiles if I forget to make sure /var/tmp/portage wasn't full of failed packages clutter, which can amount to many Gb's.
This would only happen if the root partition was small, and tends to run out of disk space, which is true on 2 of my Gentoo installs.
I know it's ridiculous, but I still have 2 gentoos, and 6-7 other distros I fool with, plus 2 100GB data partitions on one 4+ years old 500GB hard disk. Guess my luck is running out, and I better clean it all up and get a new and bigger disk, and/or re-allocate data and partition arrangements to reflect a little sanity. 
_________________
Main box- AsRock x370 Gaming K4
Ryzen 7 3700x, 3.6GHz, 16GB GSkill Flare DDR4 3200mhz
Samsung SATA 1000GB, Radeon HD R7 350 2GB DDR5
OpenRC Gentoo ~amd64 plasma, glibc-2.36-r7, gcc-13.2.1_p20230304
kernel-6.8.4 USE=experimental python3_11 |
|
| Back to top |
|
|
Fitzcarraldo
Advocate
Joined: 30 Aug 2008
Posts: 2034
Location: United Kingdom
|
| Posted: Fri Oct 15, 2021 8:26 pm Post subject: |
|
|
wrc1944,
You could automatically empty /var/tmp/portage/ when it gets too full. I created a file /etc/local.d/99delete_tmp_files_from_failed_merges.stop containing the following code:
| Code: | #!/bin/bash
# If root partition is more than 50% full, delete any temporary directories and
# files that were left in /var/tmp/portage/ instead of being deleted.
#
# My root partition is on /dev/sda6 and the emerge command must not be running.
#
if [ `pgrep -c emerge` -eq 0 ] && [ `df | awk '/sda6/ {print $5}' | awk -F% '{print $1}'` -gt 50 ]; then
rm -rf /usr/tmp/portage/*
fi |
If the root partition is more than 50% full when I shut down the laptop, the script will automatically empty that directory. One less thing to think about.
_________________
Clevo W230SS: amd64, VIDEO_CARDS="intel modesetting nvidia".
Compal NBLB2: ~amd64, xf86-video-ati. Dual boot Win 7 Pro 64-bit.
OpenRC udev elogind & KDE on both.
Fitzcarraldo's blog |
|
| Back to top |
|
|
nikolis
Apprentice
Joined: 21 Aug 2003
Posts: 199
Location: Athens
|
| Posted: Fri Oct 15, 2021 9:20 pm Post subject: |
|
|
| pd1986 wrote: | | Everytime i complied qtwebengine, I had to reduce the thread from -j12 to -j6. otherwise, it would fail. Does anyone know why? Much appreciated. |
Unfortunately, you need more memory. |
|
| Back to top |
|
|
Hu
Moderator
Joined: 06 Mar 2007
Posts: 21631
|
| Posted: Sat Oct 16, 2021 12:42 am Post subject: |
|
|
| Fitzcarraldo wrote: | | If the root partition is more than 50% full when I shut down the laptop, the script will automatically empty that directory. One less thing to think about. |
If you would prefer rapid cleanup, there is FEATURES=fail-clean. | man make.conf: | fail-clean
Clean up temporary files after a build failure. This is
particularly useful if you have PORTAGE_TMPDIR on tmpfs.
If this feature is enabled, you probably also want to en‐
able PORTAGE_LOGDIR in order to save the build log. Both
the ebuild(1) command and the noclean feature cause the
fail-clean feature to be automatically disabled. |
This does have the drawback that it is immediate, so there is no opportunity to inspect or attempt to repair a failed build. |
|
| Back to top |
|
|
pd1986
Guru
Joined: 19 Feb 2012
Posts: 404
Location: Paris
|
| Posted: Sat Oct 16, 2021 1:11 am Post subject: |
|
|
| Tony0945 wrote: | | pd1986 wrote: | | Everytime i complied qtwebengine, I had to reduce the thread from -j12 to -j6. otherwise, it would fail. Does anyone know why? Much appreciated. |
I leave my 2700X at -j16 I have 32G memory, no swap.
Maybe you need more memory. |
| ulenrich wrote: | | Tony0945 wrote: | | pd1986 wrote: | | Everytime i complied qtwebengine, I had to reduce the thread from -j12 to -j6. otherwise, it would fail. Does anyone know why? Much appreciated. |
I leave my 2700X at -j16 I have 32G memory, no swap.
Maybe you need more memory. |
Yes, qtwebengine needs very much memory to compile: With 4gByte RAM I need another 4bB on swap and I cannot run anything else during compilation. Another malfunction could be due to bad memory at regions normally not used that much. |
| firephoto wrote: | | pd1986 wrote: | | Everytime i complied qtwebengine, I had to reduce the thread from -j12 to -j6. otherwise, it would fail. Does anyone know why? Much appreciated. |
If you run out of memory for it to build it will fail so by reducing the parallel builds you use less memory. Most of it would probably build at the higher -j but there are few parts that eat up the memory.
I have MAKEOPTS="-j13 -l23" for qtwebengine with 24 cores and 32G ram and tmp/portage on tmpfs up to 16G. Higher -j just had me hitting swap and I still get the same compile times. It did fail the other day but I had like 20 tabs open in firefox and some in chrome, and firefox and wine ended up building at the same time, and I wasn't paying attention to all the memory going away until it went oom when I was restarting firefox because I had been trying to fix emojis (critical things). Waited a few minutes to be able to do anything, qtwebengine failed, firefox and wine were done building. Using zswap too if it matters. If tmp wasn't on ram it wouldn't happen and it might be slower but it would also just be thrashing the ssd more than is necessary.
This is on a 5900X I upgraded from a 2700X.
| Code: |
Sun Oct 10 22:09:39 2021 >>> dev-qt/qtwebengine-5.15.2_p20210824-r1
merge time: 43 minutes and 50 seconds.
|
|
Understood. I never paid attention to the memory usage. I have 16G memory without setting swap. Thanks for pointing it out |
|
| Back to top |
|
|
Hund
Apprentice
Joined: 18 Jul 2016
Posts: 218
Location: Sweden
|
| Posted: Sun Feb 06, 2022 1:38 pm Post subject: |
|
|
Does anyone know anything about the jumbo-build feature, if it's coming back or if we're doomed at long compile times forever now?
_________________
Collect memories, not things. |
|
| Back to top |
|
|
Ionen
Developer
Joined: 06 Dec 2018
Posts: 2719
|
| Posted: Sun Feb 06, 2022 2:54 pm Post subject: |
|
|
| Hund wrote: | | Does anyone know anything about the jumbo-build feature, if it's coming back or if we're doomed at long compile times forever now? |
But it's already there? qtwebengine has USE=jumbo-build and it's enabled by default. |
|
| Back to top |
|
|
Hund
Apprentice
Joined: 18 Jul 2016
Posts: 218
Location: Sweden
|
| Posted: Sun Feb 06, 2022 3:03 pm Post subject: |
|
|
| Ionen wrote: | | Hund wrote: | | Does anyone know anything about the jumbo-build feature, if it's coming back or if we're doomed at long compile times forever now? |
But it's already there? qtwebengine has USE=jumbo-build and it's enabled by default. |
Compile fails when jumbo-build is enabled, it doesn't matter how low the value is. It's been like this since last year, it used to work just fine before.
_________________
Collect memories, not things. |
|
| Back to top |
|
|
Hu
Moderator
Joined: 06 Mar 2007
Posts: 21631
|
| Posted: Sun Feb 06, 2022 4:47 pm Post subject: |
|
|
Your first question reads to me like:- USE=jumbo-build previously existed.
- It no longer exists, but I want to use it.
- Does anyone know if it will come back so that I can use it?
If it is present, but does not work, then that is a different problem. Since Ionen says it is enabled by default, I expect that it works by default for whoever tested it. If it does not work for you, we should understand why. |
|
| Back to top |
|
|
Hund
Apprentice
Joined: 18 Jul 2016
Posts: 218
Location: Sweden
|
| Posted: Sun Feb 06, 2022 8:15 pm Post subject: |
|
|
| Hu wrote: | Your first question reads to me like:- USE=jumbo-build previously existed.
- It no longer exists, but I want to use it.
- Does anyone know if it will come back so that I can use it?
If it is present, but does not work, then that is a different problem. Since Ionen says it is enabled by default, I expect that it works by default for whoever tested it. If it does not work for you, we should understand why. |
It used to work fine. It took me 'only' 2 hours to compile it then. Unfortunately, it was later completely removed and the time went up to 4-5 hours for me. It was then later added back, but it has never worked for me since it came back.
_________________
Collect memories, not things. |
|
| Back to top |
|
|
asturm
Developer
Joined: 05 Apr 2007
Posts: 8936
|
| Posted: Sun Feb 06, 2022 8:36 pm Post subject: |
|
|
| So, when did it last not work, and how did it fail? |
|
| Back to top |
|
|
Hund
Apprentice
Joined: 18 Jul 2016
Posts: 218
Location: Sweden
|
| Posted: Mon Feb 07, 2022 6:44 pm Post subject: |
|
|
| asturm wrote: | | So, when did it last not work, and how did it fail? |
I can't remember now, but I'll show you a log the next time there's an update and I'll try to update it with jumbo-build enabled. :)
_________________
Collect memories, not things. |
|
| Back to top |
|
|
Hund
Apprentice
Joined: 18 Jul 2016
Posts: 218
Location: Sweden
|
|
| Back to top |
|
|
Chiitoo
Administrator
Joined: 28 Feb 2010
Posts: 2573
Location: Here and Away Again
|
| Posted: Fri Feb 11, 2022 11:39 pm Post subject: |
|
|
Does it really fail that way for you when using the default 'jumbo_file_merge_limit' as well?
_________________
Kindest of regardses. |
|
| Back to top |
|
|
Hund
Apprentice
Joined: 18 Jul 2016
Posts: 218
Location: Sweden
|
| Posted: Sat Feb 12, 2022 4:25 am Post subject: |
|
|
| Chiitoo wrote: |
Does it really fail that way for you when using the default 'jumbo_file_merge_limit' as well? |
What's the default? I have tried setting it to 100 and 50. I can't remember if I tried as low as 25.
_________________
Collect memories, not things. |
|
| Back to top |
|
|
|
Desktop Environments
