Python upgrades failed today [SOLVED]

[depontius]

Today I had upgrades to both python-2.7 and python 3.2. On most systems all went well, on one system both failed.

A few lines out of "/var/log/portage/dev-lang:python-2.7.3-r1:20120519-121536.log"

Code: Select all

Python build finished, but the necessary bits to build these modules were not found:
bsddb185           dl                 imageop         
sunaudiodev                                           
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


Failed to build these modules:
_elementtree       pyexpat                            

And from "/var/log/portage/dev-lang:python-3.2.3:20120519-122756.log"

Code: Select all

Failed to build these modules:
pyexpat                                               

I suspect that the second error is a subset of the first - I've really done nothing with python-3, so it's just there because Gentoo wants it to be. I've looked in setup.py as suggested, but it really doesn't help me. It appears that I'm obviously missing something, but I'm not sure what. I've got db installed, but it's db-4.8.something, so I presume the bsddb185 is some sort of virtual equivalence, and same for the rest.

Suggestions?

[SOLVED]
As suggested by Hu, I did indeed have some nasty stuff in /usr/local/lib*, and that was killing my python upgrades. I moved it temporarily aside so the upgrade could proceed, and for now have moved it back. Hu suggests that I could run the binary-only software that requires those libs in a partial chroot, to get around this problem in the future. I'll look at it, but for now am simply forewarned.

[teika]

Hi. Probably these messages have nothing to do with your build failure. My python-2.7 build log has these lines too, and it built successfully. You've got to spot the real error messages. (My 3.2 build log is lost. ;-)

[VoidMage]

depontius:, scroll a few lines back till the place pyexpat should have been built and search for the error there (_elementtree was most likely skipped, cause the former failed).

[depontius]

The _elementree/pyexpat message was in what I posted. Looking back up the log file:

Code: Select all

building 'pyexpat' extension
x86_64-pc-linux-gnu-gcc -pthread -fPIC -fno-strict-aliasing -O2 -march=native -pipe -fwrapv -DNDEBUG -I. -IInclude -I./Include -I/var/tmp/portage/dev-lang/python-2.7.3-r1/work/Python-2.7.3/Include -I/var/tmp/portage/dev-lang/python-2.7.3-r1/work/Python-2.7.3 -c /var/tmp/portage/dev-lang/python-2.7.3-r1/work/Python-2.7.3/Modules/pyexpat.c -o build/temp.linux-x86_64-2.7/var/tmp/portage/dev-lang/python-2.7.3-r1/work/Python-2.7.3/Modules/pyexpat.o
x86_64-pc-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,--as-needed -L. -Wl,-O1 -Wl,--as-needed -L. -fno-strict-aliasing -O2 -march=native -pipe -fwrapv -DNDEBUG -I. -IInclude -I./Include build/temp.linux-x86_64-2.7/var/tmp/portage/dev-lang/python-2.7.3-r1/work/Python-2.7.3/Modules/pyexpat.o -L/usr/local/lib -L. -lexpat -lpython2.7 -o build/lib.linux-x86_64-2.7/pyexpat.so
*** WARNING: renaming "pyexpat" since importing it failed: build/lib.linux-x86_64-2.7/pyexpat.so: undefined symbol: XML_SetHashSalt
building '_elementtree' extension
x86_64-pc-linux-gnu-gcc -pthread -fPIC -fno-strict-aliasing -O2 -march=native -pipe -fwrapv -DNDEBUG -DUSE_PYEXPAT_CAPI -I. -IInclude -I./Include -I/var/tmp/portage/dev-lang/python-2.7.3-r1/work/Python-2.7.3/Include -I/var/tmp/portage/dev-lang/python-2.7.3-r1/work/Python-2.7.3 -c /var/tmp/portage/dev-lang/python-2.7.3-r1/work/Python-2.7.3/Modules/_elementtree.c -o build/temp.linux-x86_64-2.7/var/tmp/portage/dev-lang/python-2.7.3-r1/work/Python-2.7.3/Modules/_elementtree.o
x86_64-pc-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,--as-needed -L. -Wl,-O1 -Wl,--as-needed -L. -fno-strict-aliasing -O2 -march=native -pipe -fwrapv -DNDEBUG -I. -IInclude -I./Include build/temp.linux-x86_64-2.7/var/tmp/portage/dev-lang/python-2.7.3-r1/work/Python-2.7.3/Modules/_elementtree.o -L/usr/local/lib -L. -lexpat -lpython2.7 -o build/lib.linux-x86_64-2.7/_elementtree.so
*** WARNING: renaming "_elementtree" since importing it failed: PyCapsule_Import could not import module "pyexpat"

So this tells me that the problem is building pyexpat, but that appears to be an internal part of python, not a package.

A quick search, and this seems to be a common problem. A couple of thoughts... I last upgraded expat on April 21, but I also upgraded dev-python/lxml later in this session. I suspect that tomorrow I should try this upgrade again - maybe this is a dependency order problem?

[VoidMage]

Not quite: XML_SetHashSalt was added in expat 2.1.0.
Thank you.

[depontius]

Turned out that the emerge failure was on a different system, one that runs overnight. (my mythbackend) So I kicked off the emerge last night before heading to bed. Failed again.

But that system does have expat-2.1.0 on it, and apparently that must have been back on April 19, so now I'm still not understanding why this fails.

[depontius]

I also tried manually installing elemettree, and python upgrades still fail.

I have a second system failing python upgrades now, an i686 at work, for the same reasons. The other day I tried installing dev-python/bsddb3, which didn't help. Today I installed elemettree here, and am rerunning. There are many upgrades happening, so I don't know when it will get back to the python packages. I seriously doubt that the combination of the 2 packages will do any good.

[VoidMage]

I'm getting a feeling the problem lies along the lines of /usr/local/lib pollution.

[depontius]

I'm getting a feeling the problem lies along the lines of /usr/local/lib pollution.

That's a very real possibility. I have had to keep a rather ugly /usr/local/lib on both systems, in order to run binaries from my employer. I'll move it out of the way temporarily and try a rebuild tonight.

[depontius]

That was it, at least for my system at home. I haven't move the /usr/local/lib* files back yet, but both pythons installed cleanly. My (ancient and slow i686) system at work was still building QT updates when I left today, but tomorrow I'll try the same fix, and mark this topic appropriately.

So I have to go off on a minor diatribe about "the platform". That's really the cause of my problems, because I had binary-only code from my employer, coded to "the platform", which so frequently means "in the most brittle fashion possible, may count of side-effects for proper operation, and has its best shot at working properly with only specific versions of every solib it uses." In the name of "low maintenance" they create a maintenance nightmare.

The philosophical opposite of Gentoo.

By the way, thanks for the suggestion.

[Hu]

If you only need to run a few programs with the custom libraries, you might be better served creating a partial chroot. Place the custom libraries in a non-standard location, such as /usr/local/platform-hack/usr/local/lib. Then, when you need to run those programs, run them in a chroot with bind mounts to provide /, /var, etc. This will avoid creating copies of any system managed libraries, let you run the fragile programs in a layout that they expect, and hide the custom libraries from the rest of the system.

[RAPHEAD]

Same Problem here:

86_64-pc-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,--as-needed -L. -Wl,-O1 -Wl,--as-needed -L. -fno-strict-aliasing -O2 -pipe -march=core2 -fomit-frame-pointer -$
building 'ossaudiodev' extension
x86_64-pc-linux-gnu-gcc -pthread -fPIC -fno-strict-aliasing -O2 -pipe -march=core2 -fomit-frame-pointer -fwrapv -DNDEBUG -I. -IInclude -I./Include -I/var/tmp/porta$
x86_64-pc-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,--as-needed -L. -Wl,-O1 -Wl,--as-needed -L. -fno-strict-aliasing -O2 -pipe -march=core2 -fomit-frame-pointer -$
building '_ctypes' extension
x86_64-pc-linux-gnu-gcc -pthread -fPIC -fno-strict-aliasing -O2 -pipe -march=core2 -fomit-frame-pointer -fwrapv -DNDEBUG -I/usr/lib64/libffi-3.0.11/include -I. -II$
x86_64-pc-linux-gnu-gcc -pthread -fPIC -fno-strict-aliasing -O2 -pipe -march=core2 -fomit-frame-pointer -fwrapv -DNDEBUG -I/usr/lib64/libffi-3.0.11/include -I. -II$
x86_64-pc-linux-gnu-gcc -pthread -fPIC -fno-strict-aliasing -O2 -pipe -march=core2 -fomit-frame-pointer -fwrapv -DNDEBUG -I/usr/lib64/libffi-3.0.11/include -I. -II$
x86_64-pc-linux-gnu-gcc -pthread -fPIC -fno-strict-aliasing -O2 -pipe -march=core2 -fomit-frame-pointer -fwrapv -DNDEBUG -I/usr/lib64/libffi-3.0.11/include -I. -II$
x86_64-pc-linux-gnu-gcc -pthread -fPIC -fno-strict-aliasing -O2 -pipe -march=core2 -fomit-frame-pointer -fwrapv -DNDEBUG -I/usr/lib64/libffi-3.0.11/include -I. -II$
x86_64-pc-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,--as-needed -L. -Wl,-O1 -Wl,--as-needed -L. -fno-strict-aliasing -O2 -pipe -march=core2 -fomit-frame-pointer -$

Python build finished, but the necessary bits to build these modules were not found:
bsddb185 dl imageop
sunaudiodev
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


Failed to build these modules:
_elementtree pyexpat

running build_scripts
creating build/scripts-2.7
copying and adjusting /var/tmp/portage/dev-lang/python-2.7.3-r1/work/Python-2.7.3/Tools/scripts/pydoc -> build/scripts-2.7
copying and adjusting /var/tmp/portage/dev-lang/python-2.7.3-r1/work/Python-2.7.3/Tools/scripts/idle -> build/scripts-2.7
copying and adjusting /var/tmp/portage/dev-lang/python-2.7.3-r1/work/Python-2.7.3/Tools/scripts/2to3 -> build/scripts-2.7
copying and adjusting /var/tmp/portage/dev-lang/python-2.7.3-r1/work/Python-2.7.3/Lib/smtpd.py -> build/scripts-2.7
changing mode of build/scripts-2.7/pydoc from 644 to 755
changing mode of build/scripts-2.7/idle from 644 to 755
changing mode of build/scripts-2.7/2to3 from 644 to 755
changing mode of build/scripts-2.7/smtpd.py from 644 to 755
make: *** [sharedmods] Fehler 1
emake failed



Shouldn't this be reported as a bug rather than fixing it with complicated workarounds?

[Hu]

Shouldn't this be reported as a bug rather than fixing it with complicated workarounds?

If there is a bug here, then by all means, please report it. The failure that depontius encountered is not a bug. It is a negative consequence of polluting the system with outdated and unmanaged files. He had a good reason for doing that, but since that is not an expected or supported configuration, there is nothing to be done there. It is an expected behavior that the compiler will search /usr/local. The only things Portage could do to protect you are to hide /usr/local from the build process or to warn you that you have files in /usr/local.

[shur]

I am having a similar problem with Python 2.7.3. I moved all files from /usr/local/lib and /usr/local/include
to my home directory, where hopefully they won't be used in the build. I also rebuilt all python dependencies (emerge -ev). Any suggestions?

Relevant cut and pastes follow :


[ebuild U ] dev-lang/python-2.7.3-r1 [2.7.2-r3] USE="gdbm ipv6 ncurses readline sqlite* ssl threads tk (wide-unicode) xml -berkdb -build -doc -examples -wininst" 0 kB

***** compiles normally up to here*******

Python build finished, but the necessary bits to build these modules were not found:
bsddb185 sunaudiodev
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


Failed to build these modules:
_elementtree pyexpat

running build_scripts
creating build/scripts-2.7
copying and adjusting /var/tmp/portage/dev-lang/python-2.7.3-r1/work/Python-2.7.3/Tools/scripts/pydoc -> build/scripts-2.7
copying and adjusting /var/tmp/portage/dev-lang/python-2.7.3-r1/work/Python-2.7.3/Tools/scripts/idle -> build/scripts-2.7
copying and adjusting /var/tmp/portage/dev-lang/python-2.7.3-r1/work/Python-2.7.3/Tools/scripts/2to3 -> build/scripts-2.7
copying and adjusting /var/tmp/portage/dev-lang/python-2.7.3-r1/work/Python-2.7.3/Lib/smtpd.py -> build/scripts-2.7
changing mode of build/scripts-2.7/pydoc from 644 to 755
changing mode of build/scripts-2.7/idle from 644 to 755
changing mode of build/scripts-2.7/2to3 from 644 to 755
changing mode of build/scripts-2.7/smtpd.py from 644 to 755
make: *** [sharedmods] Error 1
emake failed
* ERROR: dev-lang/python-2.7.3-r1 failed (compile phase):
* emake failed

[shur]

getting a bit deeper into the compile log I found this:

building 'pyexpat' extension
i686-pc-linux-gnu-gcc -pthread -fPIC -fno-strict-aliasing -O2 -march=i686 -pipe -fwrapv -DNDEBUG -I. -IInclude -I./Include -I/usr/local/include -I/var/tmp/portage/dev-lang/python-2.7.3-r1/work/Python-2.7.3/Include -I/var/tmp/portage/dev-lang/python-2.7.3-r1/work/Python-2.7.3 -c /var/tmp/portage/dev-lang/python-2.7.3-r1/work/Python-2.7.3/Modules/pyexpat.c -o build/temp.linux-i686-2.7/var/tmp/portage/dev-lang/python-2.7.3-r1/work/Python-2.7.3/Modules/pyexpat.o
i686-pc-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,--as-needed -L. -Wl,-O1 -Wl,--as-needed -L. -fno-strict-aliasing -O2 -march=i686 -pipe -fwrapv -DNDEBUG -I. -IInclude -I./Include build/temp.linux-i686-2.7/var/tmp/portage/dev-lang/python-2.7.3-r1/work/Python-2.7.3/Modules/pyexpat.o -L/usr/local/lib -L. -lexpat -lpython2.7 -o build/lib.linux-i686-2.7/pyexpat.so
*** WARNING: renaming "pyexpat" since importing it failed: build/lib.linux-i686-2.7/pyexpat.so: undefined symbol: XML_SetHashSalt
building '_elementtree' extension
i686-pc-linux-gnu-gcc -pthread -fPIC -fno-strict-aliasing -O2 -march=i686 -pipe -fwrapv -DNDEBUG -DUSE_PYEXPAT_CAPI -I. -IInclude -I./Include -I/usr/local/include -I/var/tmp/portage/dev-lang/python-2.7.3-r1/work/Python-2.7.3/Include -I/var/tmp/portage/dev-lang/python-2.7.3-r1/work/Python-2.7.3 -c /var/tmp/portage/dev-lang/python-2.7.3-r1/work/Python-2.7.3/Modules/_elementtree.c -o build/temp.linux-i686-2.7/var/tmp/portage/dev-lang/python-2.7.3-r1/work/Python-2.7.3/Modules/_elementtree.o
i686-pc-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,--as-needed -L. -Wl,-O1 -Wl,--as-needed -L. -fno-strict-aliasing -O2 -march=i686 -pipe -fwrapv -DNDEBUG -I. -IInclude -I./Include build/temp.linux-i686-2.7/var/tmp/portage/dev-lang/python-2.7.3-r1/work/Python-2.7.3/Modules/_elementtree.o -L/usr/local/lib -L. -lexpat -lpython2.7 -o build/lib.linux-i686-2.7/_elementtree.so
*** WARNING: renaming "_elementtree" since importing it failed: PyCapsule_Import could not import module "pyexpat"
***********

The problem seems to be related to undefined symbol: "XML_SetHashSalt" But I do have expat-2.1.0 installed. And no lib or include files in /usr/local.

One weird thing I noticed. The libexpat files in my  /usr/lib directory are version 1.6.0 and they were just installed last night, seemingly from the ebuild for expat-2.1.0! Indeed that is the case... I did a reinstall and that is exactly what it did.

Googling led to the discovery that a very similar problem has been reported as a bug (and purportedly fixed) on Debian
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=665346

[shur]

something interesting... there were expat.h and expat_extern.h from a previous version (installed in 2010) in the /include directory. The current version installs these files in the /usr/include directory, and the old ones were obviously not removed. Likewise there were libexpat 1.5.2 files in the /lib directory while the current version 1.6.0 installs to /usr/lib.

Removing these old files solved the problem.

The numbering discrepancy between the expat distribution and the libexpat files occurs in the SourceForge tarball. I don't know why they did it, but they did, and anyway this was not the problem.

[nc_jlm]

2/19/2019 - installing python3 in a Slackware 14.2 ...

Hit this issue. Rather than /usr/lib or /usr/local/lib pollution, I had LD_LIBRARY_PATH pollution, since I have Oracle libraries installed, and Oracle's older expat-1.5.2 was getting loaded ahead of the 1.6.0 or 1.6.8 in /usr/lib or /usr/local/lib . In my case, I updated my LD_LIBRARY_PATH var to include the standard library directories before Oracle's. I might try clearing it altogether.

Happy hacking!