Updating to glibc-2.17 causes python and gcc to segfault

dennis.jenkins.75 · n00b Joined: 17 Apr 2012 Posts: 6

Hello.

I have a Gentoo Linux system with glibc-2.16 installed. When attempting to update to glibc-2.17, the compile completes, but the system dies horribly during the install / removal of glibc-2.16. Python, gcc and other tools now segfault and I am unable to run commands like "emerge --info". Thankfully the system is a virtual machine and I was able to rollback to a snapshot. The underlying hardware is fine. I can move the VM to another system and get the same result. This is not a hardware problem (eg, bad RAM).

I have carefully updated everything on the system except for glibc, ran "python-updater", updated to kernel-3.10.25, etc... python and gcc seem to work just fine on glibc-2.16, and also on the previous kernel, 3.10.17. However, then I update glibc ("emerge -1v glibc" or "emerge -avuND world"), portage and the gcc tool chain segfault. I have duplicated this behavior multiple times over the past few weeks.

I would like to know how to update this system without breaking it. I have 40+ other Gentoo Linux virtual machines to update after this one, and I suspect that the others will suffer the same fate. They were all originally clones of a single Gentoo VM from years ago. Generally, Gentoo updates are applied weekly, so these systems are not stale.

In reading the output from the build of glibc-2.17, it looks like the system also trying to do something with glibc-2.16, as it encourages me to post the output of "emerge --info =glibc-2.16". I included the output of "emerge --info" for both glibc's below.

Also below I have included some portage diagnostic information from BEFORE it breaks. I will also show the breakage:

Thank you for your time and thoughts.

uraes · Posted: Wed Mar 26, 2014 7:42 pm Post subject:

Think I just hit the same problem. Live server ...

dennis.jenkins.75 · n00b Joined: 17 Apr 2012 Posts: 6

We still have not resolved this issue on our end. Some of our systems will accept the glibc update just fine, but other systems become broken. For the time being, we've been going "emerge -avuND world --exclude glibc", but sooner or later we must figure out why this is breaking and fix it.

I have no idea what is happening. I updated a test system, ran "ulimit -c; gcc --version", copied the core file to a working system and examined it with gdb. The stack trace was useless, holding only one frame.

Anon-E-moose · Posted: Wed Mar 26, 2014 7:54 pm Post subject:

Highly unlikely that it's a glibc problem per se.

What version of python is running "eselect python show"
_________________
PRIME x570-pro, 3700x, 6.1 zen kernel
gcc 13, profile 17.0 (custom bare multilib), openrc, wayland

dennis.jenkins.75 · n00b Joined: 17 Apr 2012 Posts: 6

# eselect python show
python2.7

# eselect python list
Available Python interpreters:
[1] python2.7 *
[2] python3.3

# equery l python
* Searching for python ...
[IP-] [ ] dev-lang/python-2.7.5-r3:2.7
[IP-] [ ] dev-lang/python-3.3.3:3.3

Note: Most of the GCC tool chain will also segfault: gcc, gdb (I have not tested other tools)

Anon-E-moose · Posted: Wed Mar 26, 2014 8:35 pm Post subject:

TomWij · Retired Dev Joined: 04 Jul 2012 Posts: 1553

These are the first reports I see for this; besides that, it has survived ~ just fine.

Given there is talked about "live server" and "VM"; could it be the case, that the both of you are using one or another specific (cloud) hosting? eg. Linode?

If that is the case, there might be something special about their processors or virtualization or so; and because of that, you might need to set something specific to fix that up.

dennis.jenkins.75 · n00b Joined: 17 Apr 2012 Posts: 6

Our virtual machines are running VMWare ESXi 5.1.0 on a mix of Sun 6270-M2 blades (Xeon CPUs) and Sun X2200-M2s (AMD CPUs) on our private hardware. They are not hosted with a cloud vendor. Although the hardware is 64-bit capable, the VMs themselves are 32-bit kernel and 32-bit user-space.

I will rebuilt glibc-2.17 and run "cmp -l" as suggested and publish results within a few days.

Lomendil · n00b Joined: 22 Jan 2003 Posts: 16

I just ran into this when I was updating a regular server. No cloud or VM or anything. This one was very outdated. I initially had trouble updating binutils with compilation failing because fallocate64 was not found. Then I tried updating glibc, but that failed. I updated gcc first (to 4.7.3), then tried glibc again. That's the upgrade that broke things.

I'm going to try unpacking some stuff from a stage3 tarball and see if that gets me a working toolchain.
_________________
This is my .sig

TomWij · Retired Dev Joined: 04 Jul 2012 Posts: 1553

Hmm, still odd to see this in 2.17 which works fine for most people; there were segfaults with 2.18, which were resolved in 2.19.

Can you try to do `emerge @preserved-rebuild` and `revdep-rebuild` to see if it is perhaps breakage due to linkage related incompatibility? If not, it might be worth trying glibc 2.19; though, I don't know how well that would run on a stable system and downgrading back from it is dangerous to impossible so take this last advice with a grain of salt. In other words; if glibc 2.19 breaks things worse, an extremely challenging downgrade or installing again might be necessary.

Lomendil · n00b Joined: 22 Jan 2003 Posts: 16

Python fails with a segmentation fault, so I can't run emerge. My attempt at fixing from stage3 failed as well; I'm going to have to fix it from an install CD. I'll try to keep track of what I can to see if I can come up with anything useful for debugging the problem.

I may have to wipe out that evidence to get the system running again, however.

EDIT: This has been resolved, it wasn't a problem with the toolchain per se. Somehow CHOST was changed from x86_64 to i686, so GCC was making libraries and executables that weren't compatible with the system. Using a livecd and unpacking some files from stage3 gave me a working system in chroot. Then it was a matter of re-emerging the broken packages.

Now to figure out who/what changed CHOST...
_________________
This is my .sig

dennis.jenkins.75 · n00b Joined: 17 Apr 2012 Posts: 6

The above mentioned problem with installing the glibc-2.17 update appears to have disappeared. I have no hard evidence as to why.

For the past few months, I've been doing this to update our systems (because updating glibc would brick gcc + python):

"emerge -qvuND @world --keep-going --exclude glibc"

Over the past few days, I have been making VMWare snapshots, updating systems one at a time, testing, and rolling back if needed. Not a single system has been broken.

I can only assume that some other set of packages or system changes applied in the past three months has somehow affected this issue.

IF there is interest, I can restore an older copy of a nearly-blank Gentoo VM that exhibited the problem, scrub private data off of it and make the disk image available to a Gentoo glibc maintainer for experimentation. Since this is a bit of work for me, I won't do this unless someone is interested. This VM would have glibc-2.16 (working), but upon update to glibc-2.17, would break the system in the manner described in this thread. The problem VM in mind is only used for fetching the Gentoo portage tree via rsync, and building "gcc" as a binary package for quick install on our other Gentoo systems. I would necessarily need to nuke "/var/log", edit "/etc/make.conf", remove ssh keys (to protect the guilty) and zero out all free disk space. After doing these steps, I would test to ensure that the VM still exhibits the problem. The oldest backup of this VM that I have is from 2014-03-17.

Any interest? (ps- I can port the disk image to QEMU if that would help).

EDIT: Damn! Right after I post this I find a VM that still breaks when updating glibc. I am going to roll it back, clone it and begin cleaning up the clone. Hopefully someone will want a copy to experiment on.