Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
World update stuck as python3.4 can't compile itself
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
V10lator
Apprentice
Apprentice


Joined: 11 Jul 2004
Posts: 199

PostPosted: Fri Jan 27, 2017 10:43 am    Post subject: World update stuck as python3.4 can't compile itself Reply with quote

The problem is that python just takes forever (I was waiting for 50+ hours) to compile one file:
Code:
Compiling '/var/tmp/portage/dev-lang/python-3.4.5/image//usr/lib64/python3.4/test/test_buffer.py'...

This is what I see in from ps aux:
Code:
$ ps aux | grep python
v10lator 11581  0.0  0.0  19160  5592 pts/6    S+   11:36   0:00 grep --colour=auto python
root     23070 19.7  3.0 641416 503804 pts/7   S+   10:59   7:17 /usr/bin/python3.4 -b /usr/lib/python-exec/python3.4/emerge -avuD --newuse --with-bdeps=y --backtrack=90 world
root     26563  0.0  0.0  24388  7532 pts/7    S+   11:18   0:00 /bin/bash /var/tmp/portage/._portage_reinstall_.3eul5cgo/bin/ebuild-helpers/emake DESTDIR=/var/tmp/portage/dev-lang/python-3.4.5/image/ altinstall
root     26565  0.0  0.0  20984  8812 pts/7    S+   11:18   0:00 make -j8 DESTDIR=/var/tmp/portage/dev-lang/python-3.4.5/image/ altinstall
root     31904  0.0  0.0  12712  5540 pts/7    S+   11:19   0:00 /bin/sh -c PYTHONPATH=/var/tmp/portage/dev-lang/python-3.4.5/image//usr/lib64/python3.4  LD_LIBRARY_PATH=/var/tmp/portage/dev-lang/python-3.4.5/work/x86_64-pc-linux-gnu \ ?./python -E -Wi /var/tmp/portage/dev-lang/python-3.4.5/image//usr/lib64/python3.4/compileall.py \ ?-d /usr/lib64/python3.4 -f \ ?-x 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' \ ?/var/tmp/portage/dev-lang/python-3.4.5/image//usr/lib64/python3.4
root     31905 99.9  0.1  80396 23248 pts/7    R+   11:19  16:22 ./python -E -Wi /var/tmp/portage/dev-lang/python-3.4.5/image//usr/lib64/python3.4/compileall.py -d /usr/lib64/python3.4 -f -x bad_coding|badsyntax|site-packages|lib2to3/tests/data /var/tmp/portage/dev-lang/python-3.4.5/image//usr/lib64/python3.4

The last process in the list is eating 100% CPU untill I kill emerge. As you see this is a rather importand emerge command but I have on idea how to even debug this.

//EDIT: I was too fast with the last sentence. I can't even kill emerge! I have to kill python (pid 31905 in the example above).
Back to top
View user's profile Send private message
phajdan.jr
Developer
Developer


Joined: 23 Mar 2006
Posts: 1777
Location: Poland

PostPosted: Sun Feb 05, 2017 10:59 am    Post subject: Reply with quote

Try to see with strace/gdb what that process is trying to do.
_________________
http://phajdan-jr.blogspot.com/
Back to top
View user's profile Send private message
ross_anderson
n00b
n00b


Joined: 13 Feb 2017
Posts: 3

PostPosted: Mon Feb 13, 2017 5:04 pm    Post subject: Reply with quote

I can confirm this on numerous machines I have similar issues as reported. I tried to attach to the python call once is became stuck and strace provided zero info. Since it's not a compile error I'd welcome some guidance on how to capture the appropriate debug information.

Code:
strace -p 13753
strace: Process 13753 attached
strace: [ Process PID=13753 runs in x32 mode. ]

##Emerge --info
Portage 2.3.3 (python 3.4.5-final-0, default/linux/amd64/13.0, gcc-4.9.4, glibc-2.23-r3, 4.8.15-gentoo x86_64)
=================================================================
System uname: Linux-4.8.15-gentoo-x86_64-Intel-R-_Xeon-R-_CPU_X3470_@_2.93GHz-with-gentoo-2.3
KiB Mem:    24748356 total,  16308752 free
KiB Swap:    4145148 total,   4144576 free
Timestamp of repository gentoo: Mon, 13 Feb 2017 12:00:01 +0000
sh bash 4.3_p48-r1
ld GNU ld (Gentoo 2.25.1 p1.1) 2.25.1
app-shells/bash:          4.3_p48-r1::gentoo
dev-java/java-config:     2.2.0-r3::gentoo
dev-lang/perl:            5.22.3_rc4::gentoo
dev-lang/python:          2.7.12::gentoo, 3.4.5::gentoo
dev-util/cmake:           3.7.2::gentoo
dev-util/pkgconfig:       0.28-r2::gentoo
sys-apps/baselayout:      2.3::gentoo
sys-apps/openrc:          0.22.4::gentoo
sys-apps/sandbox:         2.10-r3::gentoo
sys-devel/autoconf:       2.13::gentoo, 2.69::gentoo
sys-devel/automake:       1.9.6-r4::gentoo, 1.11.6-r1::gentoo, 1.14.1::gentoo, 1.15::gentoo
sys-devel/binutils:       2.25.1-r1::gentoo
sys-devel/gcc:            4.9.3::gentoo, 4.9.4::gentoo
sys-devel/gcc-config:     1.7.3::gentoo
sys-devel/libtool:        2.4.6-r2::gentoo
sys-devel/make:           4.2.1::gentoo
sys-kernel/linux-headers: 4.9::gentoo (virtual/os-headers)
sys-libs/glibc:           2.23-r3::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -Ofast -pipe -fno-tree-loop-distribute-patterns"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt /var/bind /var/spool/munin-async/.ssh"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/php/apache2-php5.6/ext-active/ /etc/php/cgi-php5.6/ext-active/ /etc/php/cli-php5.6/ext-active/ /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=native -Ofast -pipe -fno-tree-loop-distribute-patterns"
DISTDIR="/usr/portage/distfiles"
EMERGE_DEFAULT_OPTS="--jobs=6 --load-average=8.1 --complete-graph"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j9"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="X acpi alsa amd64 apache2 at4 autoipd berkdb bzip2 cli cracklib crypt cups cxx dbus djvu dlz dri encode fortran fpx gd gdbm gudev hal hdri howl-compat hvm iconv ikev1 ikev2 innodb java jbig jpeg jpeg2k libav lm_sensors lzma mdnsresponder-compat mmx modules multilib mysql mysqli ncurses nfs nls noaudio nojoystick nptl opencl openexr openmp pam pcre php png popcnt python q16 q32 q64 q8 qt3support raw readline samba seccomp session sse sse2 sse4_1 sse4_2 ssl ssse3 tcpd tiff truetype udev unicode webp wmf xattr xml zlib" ABI_X86="64* 64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="actions alias auth_basic authn_core auth_digest authn_alias access_compat authn_anon authn_file authz_groupfile authz_core authz_host authz_owner authz_user autoindex cache cgi cgid deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack socache_shmcb vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="pc" INPUT_DEVICES="libinput keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_4" RUBY_TARGETS="ruby21" USERLAND="GNU" VIDEO_CARDS="vesa" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LANG, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON


###emerge log

https://pastebin.com/97Zyvx5d

[Moderator edit: added [code] tags to preserve output layout. -Hu]
Back to top
View user's profile Send private message
DeIM
Apprentice
Apprentice


Joined: 11 Apr 2006
Posts: 256

PostPosted: Wed Feb 22, 2017 4:08 pm    Post subject: Reply with quote

Have the same problem on ARM. Both 3.4.5 and ~3.4.6 the same problem.
Back to top
View user's profile Send private message
ross_anderson
n00b
n00b


Joined: 13 Feb 2017
Posts: 3

PostPosted: Sat Mar 04, 2017 11:02 pm    Post subject: Reply with quote

I continue to struggle with this on multiple systems. The python build just hangs indefinitely with 1 core locked 100%. I'd appreciate any guidance in getting this problem to a state of either resolution or more information to allow more knowledgeable members to assist.
Back to top
View user's profile Send private message
Irre
Guru
Guru


Joined: 09 Nov 2013
Posts: 316
Location: Stockholm

PostPosted: Sat Mar 04, 2017 11:17 pm    Post subject: Reply with quote

Try to kill only the looping process, not emerge!
Back to top
View user's profile Send private message
DeIM
Apprentice
Apprentice


Joined: 11 Apr 2006
Posts: 256

PostPosted: Mon Mar 06, 2017 7:48 am    Post subject: Reply with quote

When I kill only the hanging process, it restarts the part of compilation, where is the test_buffer.py

I tried patching the source to use installed python instead of the new compiled for compiling test_buffer.py and it passed and installed. So I tried to recompile package again but it hanged on the same file.
Back to top
View user's profile Send private message
Irre
Guru
Guru


Joined: 09 Nov 2013
Posts: 316
Location: Stockholm

PostPosted: Mon Mar 06, 2017 10:53 am    Post subject: Reply with quote

DeIM wrote:
When I kill only the hanging process, it restarts the part of compilation, where is the test_buffer.py

When I killed it, test continued and came to test_buffer.py one more time. When I killed also that process test continued and emerge was successful on Raspberry (arm6) and Cubieboard20 (arm7) :)
Back to top
View user's profile Send private message
Irre
Guru
Guru


Joined: 09 Nov 2013
Posts: 316
Location: Stockholm

PostPosted: Mon Mar 06, 2017 11:02 am    Post subject: Reply with quote

dev-lang/python-3.6.0 emerged successfully with no hanging process :)

Edit:
Did not build on Cubieboard 20 (ARMv7) :(

Code:
tail -f -n 1 /var/tmp/portage/dev-lang/python-3.6.0/temp/build.log
Compiling '/var/tmp/portage/dev-lang/python-3.6.0/image//usr/lib/python3.6/test/test_buffer.py'...

top or htop gives me PID for looping procedure to kill:
Code:
23971 root      39  19   21308  15272   5100 R  99.0  1.5  14:22.53 python

kill 23971

Repeat these steps once or twice?!

Now emerge continues! :D

Strange, strange but python emerged successfully after killing looping procedure three times!.
Back to top
View user's profile Send private message
ross_anderson
n00b
n00b


Joined: 13 Feb 2017
Posts: 3

PostPosted: Tue Mar 07, 2017 6:13 pm    Post subject: Reply with quote

On a hunch I made a custom ENV flag for the build.

Make file has

CFLAGS="-march=native -Ofast -pipe -fno-tree-loop-distribute-patterns"

For python I set

CFLAGS="-O2 -pipe -march=native"

And the build succeeded. Could other users comment on thier build flag?
Back to top
View user's profile Send private message
Syl20
Guru
Guru


Joined: 04 Aug 2005
Posts: 502
Location: France

PostPosted: Thu Mar 09, 2017 1:34 pm    Post subject: Reply with quote

https://wiki.gentoo.org/wiki/Safe_CFLAGS
https://wiki.gentoo.org/wiki/GCC_optimization

If you don't use the recommended safe flags, you will have problems.
Back to top
View user's profile Send private message
Leikor
Tux's lil' helper
Tux's lil' helper


Joined: 04 Sep 2003
Posts: 123

PostPosted: Mon Jun 12, 2017 5:01 am    Post subject: Reply with quote

Removing the "-Ofast" option fix this issue for me.
Back to top
View user's profile Send private message
abusse
n00b
n00b


Joined: 27 Oct 2017
Posts: 8

PostPosted: Fri Jan 26, 2018 12:00 pm    Post subject: Reply with quote

I have the same issue with the following cflags:
Code:
CFLAGS="-O2 -pipe -march=armv7-a -mtune=cortex-a7 -mfpu=neon-vfpv4 -mfloat-abi=hard -ffast-math"

As already suggested I defined an ENV configuration as a workaround for the Python package that avoids the issue:
Code:
nano-pi / # cat /etc/portage/env/python-cflags.conf
CFLAGS="-O2 -pipe -march=armv7-a"

nano-pi / # cat /etc/portage/package.env
dev-lang/python python-cflags.conf

However, if someone figured out the reason why certain cflags are a problem, please let me know as I would prefer a cleaner solution over this "hack". So far only the Python build seems to be affected.
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 6855

PostPosted: Fri Jan 26, 2018 1:53 pm    Post subject: Reply with quote

You should ask yourself: if -O3 is making everything faster and it is safe to do, why -O3 exist at all, gcc should have nothing and always use -O3 and do the best job.
Because some optimizations are dangerous, they are options, and the proper handling of something like that is that instead of disable -O3 in a package, you should do the contrary, which mean -O2 everywhere and -O3 on per package set ; packages you know are good with -O3 (not only good because they aren't buggy with it, but also good because -O3 do make them faster, which optimizations set by -O3 "should do", but "may not").

There's many reasons why optimizations may break code.
As a "raw" example: in computers shifting bit is something natural, and shifting a bit left is * 2, and shifting right is / 2.
So computer is able to calc : 100 SHR 1 = 50 faster than 100 / 2 = 50. Each time you see : X / 2, you should optimize that to do X SHR 1 and get a faster result.
But it's not always safe:
- Y = X / 2
- Z = 100 / Y
If X=1, with X / 2 you get Y=0.5, and with X SHR 1 you get Y=0, which mean Z = division per 0
I'm not saying gcc do that stupidity, i'm showing you a case where optimization lead to a failure in a program.
You shouldn't enable on all packages an optimization that is only present in -O3 set, gcc devs have put it in -O3 because they knows that optimization is tricky to apply and may lead to trouble.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 41572
Location: 56N 3W

PostPosted: Fri Jan 26, 2018 2:33 pm    Post subject: Reply with quote

abusse,

Look into
Code:
-ffast-math
on ARM.
Its certainly not safe on Intel/AMD hardware. It is faster but at the expense of precision.
You may not care about that in an image processing program where colours may be a bit or two out but its a really bad thing in your spreadsheets.

In short find out how it works on ARM and possibly don't use it globally. Keep it for where its safe.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 41572
Location: 56N 3W

PostPosted: Fri Jan 26, 2018 2:36 pm    Post subject: Reply with quote

krinn,

If X=1, with X / 2 you get Y=0.5 is true only if Y is some sort of floating paint number.
If its an integer, Y=0 is correct, however its calculated.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 6855

PostPosted: Fri Jan 26, 2018 3:14 pm    Post subject: Reply with quote

NeddySeagoon wrote:
krinn,

If X=1, with X / 2 you get Y=0.5 is true only if Y is some sort of floating paint number.
If its an integer, Y=0 is correct, however its calculated.

my point wasn't to show how to validate optimization with shr, which i have resume with a "I'm not saying gcc do that stupidity"
Back to top
View user's profile Send private message
s_j_newbury
n00b
n00b


Joined: 05 Apr 2007
Posts: 54
Location: UK

PostPosted: Tue Oct 16, 2018 9:30 pm    Post subject: Reply with quote

Optimisations *almost* always do things that are permitted by the language standards, but aren't *always* expected. Sometimes code relies on undefined behaviour which has produced the right result in the past, until the compiler does something different, but not wrong. The bug is in the code not the compiler.

Occasionally, optimisations do the *wrong* thing, which can break valid code, this is a bug in the compiler unless it is clearly indicated that it is the expected behaviour. GCC has a number of flags which are known to do the *wrong* thing on occasion and are named accordingly.

-ffast-math doesn't do the wrong thing conceptually, it's permitted by the C standard, but it does violate IEEE 754 which developers sometimes depend upon for predictable behaviour.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum