Emerge / gcc hang with -O1 through -O3 flags

atrix · n00b Joined: 25 Jun 2016 Posts: 30

Hello,

I'm running into some strange behavior with gcc while emerging certain packages. If I have -O1, -O2, or -O3 in my COMMON_FLAGS, the build hangs indefinitely at different points for each problematic package. I've been testing mainly against opus, but have seen this behavior when emerging libtool and gcc itself. Omitting the '-O' flag or using '-O0' allows these packages to build, and the majority of packages build without issue with '-O2.'

This behavior is occurring on an armv8 / aarch64 machine.

When emerging opus, compilation hangs on:

NeddySeagoon · Posted: Thu Mar 31, 2022 8:33 pm Post subject:

atrix,

Post the output of

NeddySeagoon · Posted: Thu Mar 31, 2022 8:35 pm Post subject:

Moved from Portage & Programming to Gentoo on ARM.

Its likely an arm64 problem, so fits better here.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

atrix · n00b Joined: 25 Jun 2016 Posts: 30

Hi NeddySeagoon,

Thanks for the reply. emerge --info output is below:

NeddySeagoon · Posted: Thu Mar 31, 2022 9:23 pm Post subject:

atrix,

I run -flto=6 on an X-Gene 1 based server.

I see several problems there,
1. Things fail to build because needed libraries are optimised away.
2. Things build but other packages that depend on them fail at run time.
3. Now and again, it appears to run out out of RAM and I have 128G

Swap space is only used for dynamically allocated RAM as it has no permanent home on disk
The kernel can also drop clean buffers and code.
Dirty buffers must be written to make them clean before they can be dropped.

Buffers and code will be reloaded from permanent storage as required.

Remove the -flto=6 for your test case.
Both

atrix · n00b Joined: 25 Jun 2016 Posts: 30

I see, thank you for those details. I didn't realize my -flto option was so aggressive.

I've lowered MAKEOPTS="-j6" to -j4 and -flto=6 to -flto=4 but am still hanging at the same place when emerging opus. I gave it a shot with -flto=0 as well, same result. I'll continue to play with these options and will start carving out package.env profiles.

The majority of the base packages were built in qemu chroot and I started noticing these problems when trying to emerge -e @world on this hardware with the correct optimizations and cpu flags, so may have some gremlins lying in wait.

Thanks for your help.

NeddySeagoon · Posted: Fri Apr 01, 2022 10:42 am Post subject:

atrix,

There are two constrains on MAKEOPTS.

The traditional one of cores/threads+1 and the more modern one of 2G RAM per make thread.
The first gives 5 for your CPU, the second gives 2 based on your RAM.

Its only big packages that would need MAKEOPTS="-j2" so on memory constrained systems, I would set MAKEOPTS="-j4" in make.conf and set a lower number on a per packages basis.
Portage can do that, you don't nee to do it at the command line.

-flto=0 may not do what you think it does. Its -fno-lto.
Sometimes passing =0 means go nuts. Unlimited. That's not what you want at all.

I'm in the middle of an arm64 server upgrade, It's suffering a faulty RAM stick, so I can't do a lot with it.

atrix · n00b Joined: 25 Jun 2016 Posts: 30

Thanks NeddySeagoon. I hadn't heard of the 2GB of RAM per thread recommendation. I thought I was playing it slightly safer by just using one thread per core than the standard/older recommendation. I will default to MAKEOPTS="-j4" and handle each package I run into issues with on an individual basis.

Ah, that's good to know. -flto=0 made just about everything worse, so I'm glad to hear that; I was a bit confused by that behavior. -fno-lto it is (when needed).

Due to my tardiness in replying (apologies) I can no longer access your make.conf, but your BINHOST is extremely helpful. I'll be leaning heavily on this as I work my way through these issues. My goal is to emerge -e @world without any hangs. I'm concerned that I can't build gcc and am wondering if this is related to my trouble building other packages. Tweaking MAKEOPTS="-j#" and -flto=# have a direct effect on where I hang while building gcc, but I still get stuck when using -j1 and -fno-lto. I'm still using opus as my test case since I can recreate the hang quicker, but am curious and concerned about not being able to build gcc.

NeddySeagoon · Posted: Tue Apr 05, 2022 8:07 pm Post subject:

atrix,

My /etc/portage/* is there somewhere too.
Most of the Raspberry Pi entries have been commented out but they should still be there.
Try looking here
Oops, it should not be in /gcc-10.x/ any more.

I'm in the process of moving from one arm64 build box to another, so the binhost is not getting updated as frequently as it used to.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

atrix · n00b Joined: 25 Jun 2016 Posts: 30

Ah, great, thank you Neddy; that has what I was looking for.

As a result of using different combinations of values for -flto, -O#, and occasionally using -fno-lto, I am able to make my way through most problematic packages by creating env profiles. However, I am at a complete impasse when it comes to rebuilding gcc. No combination of options seems to be doing the trick; trying to emerge different versions yield the same results. I either hang on "configuring source" or "STRIP FLAGS." I've removed all gcc flags individually until I had none left with no change in results.

I'll continue troubleshooting and will report back if I'm able to make any progress here. Your binhost has been extremely helpful as I work through these issues. You helped me on an ARM issue once long ago, and as always your help is greatly appreciated.

NeddySeagoon · Posted: Thu Apr 07, 2022 5:30 pm Post subject:

atrix,

gcc has a strange build system. Its done the way it is with good reason though.

1. enough of gcc is built with some random compiler on the system, so the gcc can build itself.
2. the gcc just built is used to build gcc again. This ensures that it is not contaminated with code from the host compiler, which is unknown.
3. the gcc built at stage 2 is used to build gcc again and the results between 2. and 3. are compared.
If the compare fails, its all over, the build terminates.

If all is well, the build system builds the optional language support, selected by your use flags and this gets installed.

A few things drop out of this. distcc can only help with stage 1.
There is only one copy of gcc as output from stage 1, so distcc cannot help.
If follows that the poor wee arm system is all on its own doing the building.

When the build fails, grep the build log for killed.

atrix · n00b Joined: 25 Jun 2016 Posts: 30

Hi NeddySeagoon,

This is really interesting; I was trying to get to the bottom of how gcc was actually built and these details clear that up. That said, I seem to be failing/hanging at stage 1.

I've posted the two different types of logs I get based on my USE flags, the first is a build with no special flags enabled - I make it past "Configuring source" then hang on the "strip-flags" portion.

http://dpaste.com/2BLAKLW2C

The second log is when attempting to build with additional USE flags - specifically custom-cflags, go, graphite, lto, zstd. I've tried this with just custom-cflags, same result.

http://dpaste.com/3NYEAMJNZ

With both builds, they hang at the above stages indefinitely until I kill the process. During these hangs, one core is at 100% CPU while the rest are idle. I've since disabled all custom USE flags and below is the process is the one at 100% CPU while the build is stopped at "strip-flags":

NeddySeagoon · Posted: Thu Apr 07, 2022 8:35 pm Post subject:

atrix,

My gcc is built with USE="cxx fortran graphite jit lto nls nptl openmp pie sanitize ssp"

USE=cxx is required to build a gcc that can built itself next time.
USE=pie and ssp come from your profile starting with the /17.0/ profiles.

Is your real time clock correct. Some configure scripts check that their output files are newer than the source files and run again if the test fails.
With the RTC set to 01-Jan-1970, with no RTC, and source code timestamped 2020, the check will take over 50 years to pass.
I've been bitten by that a few times :)

If the RTC is OK, then after you stop it, try the following

atrix · n00b Joined: 25 Jun 2016 Posts: 30

Good to know about cxx.

Interesting thought on the RTC - it is accurate, but sadly I receive the same results whether it is enabled or disabled.

The explicit ebuild configure command does hang in exactly the same spot, but at least without unpacking around a gig of files first. I also tried disabling each of the listed FEATURES individually, and all of them, same hang.

Your stage3 comment got me thinking.. I did build from stage 3, but in a qemu chroot, where I emerged many packages I wanted to have ready for this system (it hadn't arrived in the mail yet, I didn't have another arm64 host online).

I grabbed my stage3 tarball and unpacked it, chrooted into it, tested re-emerging gcc with default options, got past the hang point and killed it. I then moved my make.conf from my live system to the chroot, gcc got past the hang point, I killed it. I then copied the below dirs to my live system and tried emerging gcc, same hang:

NeddySeagoon · Posted: Fri Apr 08, 2022 4:38 pm Post subject:

atrix,

Run

atrix · n00b Joined: 25 Jun 2016 Posts: 30

The output of both is exactly the same:

atrix · n00b Joined: 25 Jun 2016 Posts: 30

Unfortunately, I haven't gotten any further with this and am going to pursue rebuilding in chroot. I'm not sure what went wrong with the initial install, but I can build gcc in stage 3 chroot and also am not seeing hangs on opus, libtool, and others with optimization flags enabled. This really is puzzling.

NeddySeagoon · Posted: Sun Apr 10, 2022 9:05 am Post subject:

atrix,

Does it really hang, that is the load average drops to zero, or very close, or does it stick in a loop?
The load average will be about one, if its doing a busy wait.

I'm not sure its worth the effort to find out. Build the package in the chroot and install the binary.
Keep the chroot around in case that's not the problem, so yo can build another package.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.

atrix · n00b Joined: 25 Jun 2016 Posts: 30

Hi NeddySeagoon,

"Hang" might not have been the best word to describe this behavior; the load average is slightly above one when this gets stuck. Strace output halts and I don't see any files being modified or IO activity when this happens.

Definitely, I've been leaning heavily on the chroot environment and have not had any issues building there, so will continue building problem packages there. If I can still recreate these hangs/stuck builds after some time I'll copy everything from the updated chroot to my system. Thanks very much for your assistance troubleshooting this strange issue.