View previous topic :: View next topic |
Author |
Message |
pjp Administrator
![Administrator Administrator](/images/ranks/rank-admin.gif)
![](images/avatars/1154772887439692d88303b.jpg)
Joined: 16 Apr 2002 Posts: 20067
|
Posted: Thu May 26, 2022 8:42 pm Post subject: Normal build times for clang 13? |
|
|
Code: | clang-11.0.0 1h 17m 34s
clang-11.0.1 1h 20m 10s
clang-11.1.0 1h 17m 56s
clang-11.1.0 3h 37m 54s
clang-12.0.0-r1 3h 57m 12s
clang-12.0.1 3h 56m 9s
clang-12.0.1 3h 58m 51s
clang-13.0.0 11h 7m 1s
clang-13.0.1 11h 9m 5s |
In emerge -1 clang takes 14 hours? [SOLVED!!] the solution may have been LTO related? I'm not sure.
I don't do much with CFLAGS, although there are 2 related to Spectre, set long before 11.0.0.
When clang jumps to 3h for 11.1.0, that seems to coincide with configuring clang to use makeopts -j1 and not using tmpfs.
emerge --info: https://dpaste.com/BCWVCMZJB _________________ Quis separabit? Quo animo? |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
eccerr0r Watchman
![Watchman Watchman](/images/ranks/rank-G-2-watchman.gif)
Joined: 01 Jul 2004 Posts: 9679 Location: almost Mile High in the USA
|
Posted: Thu May 26, 2022 9:24 pm Post subject: |
|
|
Having no anonymous swap doesn't mean it won't swap, it just inefficiently swaps. You need anonymous swap.
TMPDIR on tmpfs does not significantly reduce build times except if you can't cache any of the build directory.
Spectre/Meltdown mitigation does not triple bulid times, but they do slow things down.
LTO uses a lot of RAM. Disabling LTO in limited memory situations will improve build times. Especially with 16GB RAM and no anonymous swap.
BTW I haven't seen a large increase in build times, however my timing data is contaminated with distcc... in fact build times have gone down with distcc. _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?
Last edited by eccerr0r on Thu May 26, 2022 9:27 pm; edited 1 time in total |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
mike155 Advocate
![Advocate Advocate](/images/ranks/rank-G-1-advocate.gif)
Joined: 17 Sep 2010 Posts: 4438 Location: Frankfurt, Germany
|
Posted: Thu May 26, 2022 9:36 pm Post subject: |
|
|
Do the settings below make sense?
Code: | EMERGE_DEFAULT_OPTS="--jobs=3 --load-average=3.0 --autounmask-keep-masks"
MAKEOPTS="-j4 -l2" |
What happens if you switch to
Code: | EMERGE_DEFAULT_OPTS="--autounmask-keep-masks"
MAKEOPTS="-j5" |
If I understand correctly, MAKEOPTS="-l2" limits the number of parallel compile jobs for clang to 'no more than 2'. It is no surprise that it takes hours. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
flysideways Guru
![Guru Guru](/images/ranks/rank_rect_3.gif)
Joined: 29 Jan 2005 Posts: 437
|
Posted: Thu May 26, 2022 10:56 pm Post subject: |
|
|
These are both Raspberry Pi's, the first is arm64 and the second is ~arm64. The 12'31" build was actually done on a Ryzen 9 3900X 12-Core Processor.
Code: | # qlop -mvt clang
2020-12-13T19:07:47 >>> sys-devel/clang-11.0.0: 12′31″
2021-01-16T01:33:24 >>> sys-devel/clang-11.0.0: 7:52:45
2021-03-06T15:51:57 >>> sys-devel/clang-11.0.1: 7:58:20
2021-04-08T03:36:33 >>> sys-devel/clang-11.1.0: 7:13:03
2021-06-17T00:40:49 >>> sys-devel/clang-11.1.0: 7:28:46
2021-06-22T17:15:57 >>> sys-devel/clang-12.0.0-r1: 8:40:42
2021-09-21T04:57:41 >>> sys-devel/clang-12.0.1: 8:06:07
2021-11-05T13:21:19 >>> sys-devel/clang-13.0.0: 8:50:52
2022-01-09T21:57:34 >>> sys-devel/clang-13.0.0: 8:51:36
2022-05-10T19:50:57 >>> sys-devel/clang-13.0.1: 9:17:47 |
Code: | # qlop -mvt clang
2020-12-13T21:29:06 >>> sys-devel/clang-11.0.0: 7:15:25
2021-01-04T12:29:33 >>> sys-devel/clang-11.0.0: 8:00:42
2021-01-07T14:51:58 >>> sys-devel/clang-11.0.1: 7:59:57
2021-01-16T01:39:55 >>> sys-devel/clang-11.0.1: 7:58:38
2021-03-07T05:00:08 >>> sys-devel/clang-11.1.0: 7:54:44
2021-04-27T16:41:36 >>> sys-devel/clang-12.0.0-r1: 8:06:48
2021-06-17T01:41:28 >>> sys-devel/clang-12.0.0-r1: 8:07:30
2021-06-17T09:56:02 >>> sys-devel/clang-11.1.0: 7:29:35
2021-07-10T14:03:35 >>> sys-devel/clang-12.0.1: 8:05:57
2021-10-03T13:52:05 >>> sys-devel/clang-13.0.0: 8:51:08
2021-10-17T19:33:02 >>> sys-devel/clang-12.0.1: 8:02:51
2022-01-09T22:22:27 >>> sys-devel/clang-13.0.0: 8:50:24
2022-02-06T17:35:32 >>> sys-devel/clang-13.0.1: 8:50:31
2022-05-10T14:18:47 >>> sys-devel/clang-14.0.3: 9:26:35 |
|
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
eccerr0r Watchman
![Watchman Watchman](/images/ranks/rank-G-2-watchman.gif)
Joined: 01 Jul 2004 Posts: 9679 Location: almost Mile High in the USA
|
Posted: Thu May 26, 2022 10:58 pm Post subject: |
|
|
Yes the EMERGE_DEFAULT_OPTS and MAKEOPTS need to be carefully balanced with each other... I don't see much use in limiting load average in EMERGE_DEFAULT_OPTS as it's not frequently checked - and only useful of MAKEOPTS="" ...
And to see how badly contaminated my build times are,
Code: | sys-devel/clang-6.0.1: 33′00″
sys-devel/clang-7.1.0: 2:06:16
sys-devel/clang-8.0.1: 2:24:25
sys-devel/clang-10.0.0: 2:18:59
sys-devel/clang-11.0.0: 58′08″
sys-devel/clang-11.0.1: 1:00:24
sys-devel/clang-11.1.0: 43′42″
sys-devel/clang-12.0.0-r1: 1:05:51
sys-devel/clang-12.0.1: 39′47″
sys-devel/clang-13.0.0: 51′14″
sys-devel/clang-13.0.1: 46′24″ |
I don't know when I fully started distcc but clearly something happened at clang11...and clang13 might have further been contaminated by distcc with compute farm upgrades or job tweaking. _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
pjp Administrator
![Administrator Administrator](/images/ranks/rank-admin.gif)
![](images/avatars/1154772887439692d88303b.jpg)
Joined: 16 Apr 2002 Posts: 20067
|
Posted: Fri May 27, 2022 12:24 am Post subject: |
|
|
eccerr0r wrote: | Having no anonymous swap doesn't mean it won't swap, it just inefficiently swaps. You need anonymous swap.
TMPDIR on tmpfs does not significantly reduce build times except if you can't cache any of the build directory.
Spectre/Meltdown mitigation does not triple bulid times, but they do slow things down.
LTO uses a lot of RAM. Disabling LTO in limited memory situations will improve build times. Especially with 16GB RAM and no anonymous swap.
BTW I haven't seen a large increase in build times, however my timing data is contaminated with distcc... in fact build times have gone down with distcc. | At first, I thought your swap comment was a general recommendation. To keep it short, I'm not intentionally running without swap. I apparently missed turning it on when doing a drive replacement / migration last year. That happened between clang-12.0.0-r1, 3h 57m 12s (Jul 1) and clang-12.0.1, 3h 56m 9s (Aug 19). I'll try to figure out what I intended to do with swap.
I use TMPDIR for consistency across devices (my laptop has an SSD, which is why I use it there). Except for accidentally not have swap, I don't think this is relevant to the 11 hour build times, is it?
I didn't figure the Spectre thing was relevant, but thought I'd mention it. I was primarily commenting that I don't mess with / "optimize" those settings.
Which brings me to LTO... I haven't enabled it, and I didn't notice it having been enabled. Did I miss that somewhere?
If the times doubled I might not have bothered to ask, but they almost quadrupled, so it definitely seemed "wrong."
Thanks. _________________ Quis separabit? Quo animo? |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
pjp Administrator
![Administrator Administrator](/images/ranks/rank-admin.gif)
![](images/avatars/1154772887439692d88303b.jpg)
Joined: 16 Apr 2002 Posts: 20067
|
Posted: Fri May 27, 2022 12:26 am Post subject: |
|
|
mike155 wrote: | Do the settings below make sense?
Code: | EMERGE_DEFAULT_OPTS="--jobs=3 --load-average=3.0 --autounmask-keep-masks"
MAKEOPTS="-j4 -l2" |
What happens if you switch to
Code: | EMERGE_DEFAULT_OPTS="--autounmask-keep-masks"
MAKEOPTS="-j5" |
If I understand correctly, MAKEOPTS="-l2" limits the number of parallel compile jobs for clang to 'no more than 2'. It is no surprise that it takes hours. | In what way do they not make sense? I had higher settings but have been trying to control for potential heat issues. clang itself should be working with -j1 (per package config). _________________ Quis separabit? Quo animo? |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
eccerr0r Watchman
![Watchman Watchman](/images/ranks/rank-G-2-watchman.gif)
Joined: 01 Jul 2004 Posts: 9679 Location: almost Mile High in the USA
|
Posted: Fri May 27, 2022 2:05 am Post subject: |
|
|
Is you computer "fully loaded" the whole time? Is it usable, as in, interactive sessions are still usable (i.e. not swapping)? The trouble is when you have no anonymous swap, text page swapping is the only swapping it can do, and this *really* kills performance, much worse than swapping heap pages to disk.
Note that TMPDIR=tmpfs competes with memory consumption of heap and text pages. This can really kill performance if you're on the edge of how much RAM you have versus how much you really need. Again with no anonymous swap, it can only swap text pages, and not heap (malloc) nor buffer (tmpfs) pages. tmpfs buffer pages is a very, very easy pick to swap pages and will greatly improve performance.
If you're trying to reduce core usage to keep heating issues down, that would be a serious detriment to build times. I'd focus on trying to fix the cooling issue instead of reducing jobs. BTW if anyone is reducing heating to reduce power consumption, this is counterproductive. Only reduce speed to extend battery life if it's on battery power or your power wires are too thin (not relevant in this case.)
IIRC clang will use all cores with -j as long as the max load average isn't hit. I mention this because some packages will not use all cores during build and having a multicore processor doesn't help unless you specify --jobs for the emerge command to run some other job at the same time.
It's worth monitoring the system while it's running to see what the nature of the issue is: what is it starving for: jobs (it's not using all cores - idle cores) or memory (it can't use the cores because it needs to swap - processes in D sleep), or is it simply cores (all cores are used 100% of the time and it's operating as fast as it can)? _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
pjp Administrator
![Administrator Administrator](/images/ranks/rank-admin.gif)
![](images/avatars/1154772887439692d88303b.jpg)
Joined: 16 Apr 2002 Posts: 20067
|
Posted: Fri May 27, 2022 3:19 am Post subject: |
|
|
It's a box I access remotely via ssh within my LAN. I haven't noticed any performance difference, so I doubt that is likely to be the problem. Also, whatever exists currently has existed since around the 12.0 install, so something would have had to change with 13 to make such a difference.
My first thought was noticing all of the new supported architectures and asking if it could have been that, but that's when I discovered that 13.0 also had the same issue. My understanding was the new architectures were forced for 13.0.1, so it would seem to not be related to that either.
The only "performance" issue I've had seems related to my wireless connection from my laptop to the system This occurs even when the system is doing mostly nothing. Occasionally ssh hangs. The connection may restore, it may not. Attempts to reestablish another ssh connection can usually bring it back (I'm talking multiple attempts with "no route to host"). ssh is the only connection I've noticed being impacted. Web use through the same router is not. Sometimes I reset the router, but not often. Two or more years ago that problem seemed "reliably fixable" by making 5 attempts to recreate a new ssh connection. The "5 attempts to fix" stopped being a reliable method a couple of years ago. Again, the system isn't usually doing much when that happens. The router is very old, so I blame that as possibly being the problem. It's annoying, but not enough that I've wanted to settle for new hardware I'd rather not buy.
I'll work on getting swap configured. Maybe the planets will align and that will solve the 11 hour clang issue.
Regarding cooling, yes, it'd be nice to address that. It could also be a CPU problem that occurs only rarely. Can CPUs die slowly? I've had some intermittent compile failures which I initially thought occurred at the same time as disk errors. I replaced multiple drives to resolve that, but that wasn't the problem, or not all of it.
After the disk replacements, clang failed again. That was why I forced -j1 for clang. Since making that change, I haven't I haven't had that compile failure *knock on wood*.
The error was something like "internal compiler error", which could indicate a hardware problem. I only noticed the errors occurring with gcc and clang (but not every time). The errors / build failures did seem to occur during long runs of compiling world updates. That led me to think heat could be a factor. When the errors occurred, attempting to rebuild usually worked. As I recall, second attempts were successful, and occurred the next day or more later.
One time, within some number of minutes after clang failed I retried. It failed again, somewhat reinforcing my heat suspicion. That was when I resorted to forcing -j1 on clang builds. But again, clang had been building within 3 hours. And since -j1, I've noticed no "internal compiler errors".
Thanks again. I'll report back when I've got swap working and have had a chance to retry a build. _________________ Quis separabit? Quo animo? |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
eccerr0r Watchman
![Watchman Watchman](/images/ranks/rank-G-2-watchman.gif)
Joined: 01 Jul 2004 Posts: 9679 Location: almost Mile High in the USA
|
Posted: Fri May 27, 2022 5:00 am Post subject: |
|
|
Okay, so now it sounds like the main reason for reducing speeds is starving the CPU, to prevent CPU failure - so using -j4 -l2 will get you 11 hour build times, but -j4 -l999 would result in internal compiler errors?
The marginal memory availability is still possible with TMPDIR=tmpfs. If the tarball just so happens to add more crap to fill your RAM to make it overflow, then you will end up swapping. However I see that this may not ultimately be the problem, though the clang-13 build tree is 150MB larger than clang-12. Also all the temporary files will need to be written, all of this is eating ram that's needed to build the C++. Technically the additional 150-200MB not a whole lot, but you never know, it could be just enough to cause problems for the build. _________________ Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
pjp Administrator
![Administrator Administrator](/images/ranks/rank-admin.gif)
![](images/avatars/1154772887439692d88303b.jpg)
Joined: 16 Apr 2002 Posts: 20067
|
Posted: Fri May 27, 2022 2:06 pm Post subject: |
|
|
eccerr0r wrote: | Okay, so now it sounds like the main reason for reducing speeds is starving the CPU, to prevent CPU failure - so using -j4 -l2 will get you 11 hour build times, but -j4 -l999 would result in internal compiler errors? | To hopefully prevent CPU failure / errors, yes. I'm not using -j4 or -l2 for clang. Clang is (supposed to be) using -j1. Prior builds of clang with -j1 did not have the excessive build times. I've configured that using portage/package.env/makeopts and portage/env/makeopts_j1 (different filenames in the chroot).
eccerr0r wrote: | The marginal memory availability is still possible with TMPDIR=tmpfs. If the tarball just so happens to add more crap to fill your RAM to make it overflow, then you will end up swapping. However I see that this may not ultimately be the problem, though the clang-13 build tree is 150MB larger than clang-12. Also all the temporary files will need to be written, all of this is eating ram that's needed to build the C++. Technically the additional 150-200MB not a whole lot, but you never know, it could be just enough to cause problems for the build. | I set up swap last night, so at least that has been corrected, but it doesn't seem to be relevant to the 11 hour build times. Nor does the use of TMPDIR, or any other physical characteristics.
On the same system in a chroot is where I build packages for my laptop. It also uses -j1 for clang, though it uses "generic" instead of x86_64 for CFLAGS.
13.0 was compiled November, 13.0.1 last night.
Code: | clang-13.0.0 0d 4h 16m 24s
clang-13.0.1 0d 4h 15m 30s |
Same physical properties, strongly suggesting those aren't the issue. I suppose I need to verify the portage configurations to force -j1 are working correctly.
chroot build times, primarily to show the consistency (10.0.0 was July 2020).
Code: | clang-10.0.0 0d 3h 38m 46s
clang-10.0.1 0d 3h 38m 52s
clang-11.0.0 0d 3h 24m 20s
clang-11.0.0 0d 3h 24m 19s
clang-11.1.0 0d 3h 25m 17s
clang-11.1.0 0d 3h 29m 55s
clang-12.0.0 0d 3h 47m 30s
clang-12.0.1 0d 3h 47m 57s
clang-12.0.1 0d 3h 50m 32s
clang-13.0.0 0d 4h 16m 24s
clang-12.0.1 0d 3h 50m 28s
clang-13.0.1 0d 4h 15m 30s |
_________________ Quis separabit? Quo animo? |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
Anon-E-moose Watchman
![Watchman Watchman](/images/ranks/rank-G-2-watchman.gif)
![](images/avatars/3064969284ad468b81342a.jpg)
Joined: 23 May 2008 Posts: 6098 Location: Dallas area
|
Posted: Fri May 27, 2022 3:57 pm Post subject: |
|
|
So when you start the emerge, flip to another console and make sure that the right number of threads/spawns are what you want
ps axf -o pid,args and look at the number of processes running.
Edit to add: you can also check memory usage "free -h" works as a broad brush lookat or if you have atop installed, that should show resources in real time (more or less) _________________ PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
elover Apprentice
![Apprentice Apprentice](/images/ranks/rank_rect_2.gif)
Joined: 20 Nov 2019 Posts: 159 Location: Spain
|
Posted: Fri May 27, 2022 10:41 pm Post subject: |
|
|
My compilation times are as follows
❯ sudo qlop -mvt clang
Contraseña:
2022-01-25T19:49:49 >>> sys-devel/clang-13.0.0: 34′17″
2022-01-30T14:02:35 >>> sys-devel/clang-12.0.1: 33′56″
2022-03-08T11:53:32 >>> sys-devel/clang-13.0.1: 32′41″
2022-04-19T19:53:56 >>> sys-devel/clang-14.0.1-r1: 34′22″
2022-05-02T14:26:00 >>> sys-devel/clang-14.0.3: 35′52″
AMD Ryzen 2700x 4.0Ghz + 16 GB RAM |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
pjp Administrator
![Administrator Administrator](/images/ranks/rank-admin.gif)
![](images/avatars/1154772887439692d88303b.jpg)
Joined: 16 Apr 2002 Posts: 20067
|
Posted: Fri May 27, 2022 11:05 pm Post subject: |
|
|
Anon-E-moose wrote: | So when you start the emerge, flip to another console and make sure that the right number of threads/spawns are what you want
ps axf -o pid,args and look at the number of processes running.
Edit to add: you can also check memory usage "free -h" works as a broad brush lookat or if you have atop installed, that should show resources in real time (more or less) | After my last post, I realized that I'm pretty sure the build is close to expected with -j1. The load displayed by the build process is "within expectations" at least.
This morning I finished this round of delayed updates. I'm going to try to at least evaluate updates weekly.
I'm guessing 'lookat' was a typo and not an actual command (I'm not seeing an obvious match). I don't have atop, but I can try it.
I'm not sure how to evaluate a "real" build, so i was thinking of seeing what the manual process offered using ebuild <step> commands. I'll put it on the schedule for next week.
@elover,
Interesting that your times dropped a bit from 13 to 13.01. But otherwise mostly consistent.
Based on the times others have offered, it certainly looks like mine is not to be expected. _________________ Quis separabit? Quo animo? |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
pjp Administrator
![Administrator Administrator](/images/ranks/rank-admin.gif)
![](images/avatars/1154772887439692d88303b.jpg)
Joined: 16 Apr 2002 Posts: 20067
|
Posted: Sat Jun 11, 2022 4:38 pm Post subject: |
|
|
I tried comparing the environment files from /var/db/pkg, but there really isn't much. Code: | environment:declare -x CFLAGS="-O2 -pipe -mindirect-branch=thunk -mfunction-return=thunk"
environment:declare -x CXXFLAGS="-O2 -pipe -mindirect-branch=thunk -mfunction-return=thunk"
environment:declare -x FCFLAGS="-O2 -pipe"
environment:declare -x FFLAGS="-O2 -pipe"
chroot_environment:declare -x CFLAGS="-mtune=generic -O2 -pipe"
chroot_environment:declare -x COMMON_FLAGS="-mtune=generic -O2 -pipe"
chroot_environment:declare -x CXXFLAGS="-mtune=generic -O2 -pipe"
chroot_environment:declare -x FCFLAGS="-mtune=generic -O2 -pipe"
chroot_environment:declare -x FFLAGS="-mtune=generic -O2 -pipe" |
Code: | environment:declare -- NINJA="ninja"
environment:declare -- NINJAOPTS="-j1 -l0"
environment:declare -- NINJA_DEPEND=">=dev-util/ninja-1.8.2"
chroot_environment:declare -- NINJAOPTS="-j1 -l0" |
The chroot has a few things different, but none seem like they ought to have a relevant impact: Code: | chroot_environment:declare -x MOZ_GMP_PATH="/usr/lib64/nsbrowser/plugins/gmp-gmpopenh264/system-installed"
environment:declare -x TINYSCHEMEINIT="/usr/share/tinyscheme/init.scm"
environment:declare -x MULTILIB_ABIS="amd64"
chroot_environment:declare -x MULTILIB_ABIS="amd64 x86"
chroot_environment:declare -x OPENGL_PROFILE="xorg-x11" | And the chroot had the xml USE flag. For both, MAKEOPTS="-j1".
For clang-14.0.4, there was a big improvement on the host, but still no indication of why it takes longer than the chroot. Maybe the difference is generic tuning? Code: | chroot: clang-14.0.4 4h 32m 16s
host: clang-14.0.4 5h 54m 47s | Nothing stood out with ps axf -o pid,args. swap was never touched. Using bpytop on the host, the cores were heavily engaged until clang-14, at which point it dropped as would be expected from -j1. I didn't think to run a top monitoring tool while it compiled in the chroot, so I'll try to remember for the next version. _________________ Quis separabit? Quo animo? |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
|