Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Python - and only Python - can't see the Internet, mostly...
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
minnmass
n00b
n00b


Joined: 29 Dec 2011
Posts: 16
Location: United States

PostPosted: Thu Dec 29, 2011 4:20 am    Post subject: Python - and only Python - can't see the Internet, mostly... Reply with quote

At least so far as I can tell, Python cannot see the Internet, though everything else can.

The most common error message I get from Python scripts is:
Quote:
ERROR: unable to download video webpage: <urlopen error [Errno -2] Name or service not known>


The only exception to this is Python Ping. This suggests to me that if Python would stick to IPv4, it would work, but that it's trying to use IPv6 for some reason. I have attempted to disable IPv6 on my system, as it just didn't work (virtually nothing would work without special configuration, about two years ago when I decided to give it a whirl on my new system).

System Information wrote:

$ python -V
Python 2.7.2

$ uname -a
Linux host 3.0.6-gentoo #3 SMP Mon Dec 26 10:35:33 CST 2011 x86_64 AMD Phenom(tm) II X4 810 Processor AuthenticAMD GNU/Linux

$ cat /etc/resolv.conf
# Generated by net-scripts for interface eth0
nameserver 192.168.2.1

$ cat /etc/hosts | cut -f 1 -d '#' | grep -E '..'
127.0.0.1 localhost adbrite.com unimatrix01.minnmass.dyndns.org unimatrix1 ad.doubleclick.net

$ /sbin/ifconfig
eth0 Link encap:Ethernet HWaddr 18:a9:05:1f:0a:dc
inet addr:192.168.2.10 Bcast:0.0.0.0 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:6216179 errors:0 dropped:180 overruns:0 frame:0
TX packets:7562327 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2262052670 (2.1 GiB) TX bytes:8897305115 (8.2 GiB)
Interrupt:42 Base address:0x4000

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:630305 errors:0 dropped:0 overruns:0 frame:0
TX packets:630305 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:189496047 (180.7 MiB) TX bytes:189496047 (180.7 MiB)


# ip route show
default via 192.168.2.1 dev eth0 metric 2
127.0.0.0/8 via 127.0.0.1 dev lo
192.168.2.0/24 dev eth0 proto kernel scope link src 192.168.2.10

# ip -6 route show
default via 192.168.2.1 dev eth0 metric 2
127.0.0.0/8 via 127.0.0.1 dev lo
192.168.2.0/24 dev eth0 proto kernel scope link src 192.168.2.10

$ cat /etc/make.conf

# These settings were set by the catalyst build script that automatically
# built this stage.
# Please consult /usr/share/portage/config/make.conf.example for a more
# detailed example.
# CFLAGS="-march=k8 -O2 -pipe"

# changed 2010/03/13 due to media-libs/lcms-1.19 's suggestion
# CFLAGS="-march=amdfam10 -O2 -pipe" -fPIC
#CFLAGS="-march=amdfam10 -O2 -pipe -fPIC"

# changed 2011/05/27: glibc 2.12.2 doesn't like something
# bug report led to suggestion to remove -fPIC
CFLAGS="-march=native -O2 -pipe"

CXXFLAGS="${CFLAGS}"
# WARNING: Changing your CHOST is not something that should be done lightly.
# Please consult http://www.gentoo.org/doc/en/change-chost.xml before changing.
CHOST="x86_64-pc-linux-gnu"
# These are the USE flags that were used in addition to what is provided by the
# profile used for building.
USE="mmx sse sse2 dvd alsa cdr -kde gnome qt3 bash-completion flac \
gnustep gtk hddtemp id3 id3tag imagemagick jpeg mplayer offensive \
ogg pdf png posix quicktime rar real realmedia sdl symlink \
theora threads tiff truetype usb vorbis win32codecs xv xvid \
opengl objc X 3dnow aac acpi aim apache2 apm bzip2 cdda cddb \
cdparanoia crypt css cups curl curlwrappers cxx dbus dga directfb \
dri dv dvdr emacs encode exif ffmpeg firefox foomaticdb ftp \
gif gimp gzip ipod jpeg2k lame libcaca latex libnotify \
matroska modules mp3 mp4 mpeg multilib ncurses nsplugin raw \
readline samba smp sockets spell ssl svg svga syslog unicode vcd v4l \
v4l2 wmf wxwidgets x264 xine xosd xscreensaver ati-drivers \
-lm_sensors sqlite fts3 -ipv6 a52 avi dvdread divx fame mjpeg \
subtitles extras openal gnutls mikmod xcb ancient icu \
javascript gcj gecko qt3support xml -vdpau tk devil maildir imap \
mysql nls"
# use number of cpus + 1 for compiling (4 cores = 4 cpus, right?)
MAKEOPTS="-j5"
LINGUAS="en en_US"
VIDEO_CARDS="radeon"

GENTOO_MIRRORS="ftp://lug.mtu.edu/gentoo/ "

ACCEPT_LICENSE="*"


Everything on my system should be up to date (or, at least, no more than a couple of days old).

I'm at a loss for where to look next; please point me to resources I may have overlooked or been unable to find; googling 'python internet' and 'urlopen error [Errno -2]' and the like haven't gotten me anywhere. I will, of course, post any other reasonable information that may be helpful.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 21490

PostPosted: Thu Dec 29, 2011 4:28 am    Post subject: Reply with quote

What is the output of emerge --info dev-lang/python:2.7 sys-libs/glibc ; zgrep IPV6 /proc/config.gz? What parameters did the failed Python script use when it received that error?
Back to top
View user's profile Send private message
minnmass
n00b
n00b


Joined: 29 Dec 2011
Posts: 16
Location: United States

PostPosted: Thu Dec 29, 2011 4:38 am    Post subject: Reply with quote

Quote:


$ emerge --info dev-lang/python:2.7 sys-libs/glibc ; zgrep IPV6 /proc/config.gz
Portage 2.1.10.41 (default/linux/amd64/10.0, gcc-4.5.3, glibc-2.13-r4, 3.0.6-gentoo x86_64)
=================================================================
System Settings
=================================================================
System uname: Linux-3.0.6-gentoo-x86_64-AMD_Phenom-tm-_II_X4_810_Processor-with-gentoo-2.0.3
Timestamp of tree: Wed, 28 Dec 2011 15:00:01 +0000
app-shells/bash: 4.1_p9
dev-java/java-config: 2.1.11-r3
dev-lang/python: 2.7.2-r3, 3.1.4-r3
dev-util/cmake: 2.8.6-r4
dev-util/pkgconfig: 0.26
sys-apps/baselayout: 2.0.3
sys-apps/openrc: 0.9.4
sys-apps/sandbox: 2.5
sys-devel/autoconf: 2.13, 2.68
sys-devel/automake: 1.9.6-r3, 1.11.1
sys-devel/binutils: 2.21.1-r1
sys-devel/gcc: 4.5.3-r1
sys-devel/gcc-config: 1.4.1-r1
sys-devel/libtool: 2.4-r1
sys-devel/make: 3.82-r1
sys-kernel/linux-headers: 2.6.39 (virtual/os-headers)
sys-libs/glibc: 2.13-r4
Repositories: gentoo
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt /var/lib/hsqldb"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=native -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
FEATURES="assume-digests binpkg-logs distlocks ebuild-locks fixlafiles news parallel-fetch protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch"
FFLAGS=""
GENTOO_MIRRORS="ftp://lug.mtu.edu/gentoo/ "
LANG="en_US.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LINGUAS="en en_US"
MAKEOPTS="-j5"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY=""
SYNC="rsync://rsync.gentoo.org/gentoo-portage"
USE="3dnow X a52 aac acl acpi aim alsa amd64 ancient apache2 apm ati-drivers avi bash-completion berkdb bzip2 cdda cddb cdparanoia cdr cli cracklib crypt css cups curl curlwrappers cxx dbus devil dga directfb divx dri dv dvd dvdr dvdread emacs encode exif extras fame ffmpeg firefox flac foomaticdb fortran ftp fts3 gcj gdbm gecko gif gimp gnome gnustep gnutls gpm gtk gzip hddtemp iconv icu id3 id3tag imagemagick imap ipod javascript jpeg jpeg2k lame latex libcaca libnotify maildir matroska mikmod mjpeg mmx modules mp3 mp4 mpeg mplayer mudflap multilib mysql ncurses nls nptl nptlonly nsplugin objc offensive ogg openal opengl openmp pam pcre pdf png posix pppd qt3 qt3support quicktime rar raw readline realmedia samba sdl session smp sockets spell sqlite sse sse2 ssl subtitles svg symlink sysfs syslog tcpd theora threads tiff tk truetype unicode usb v4l v4l2 vcd vorbis wmf wxwidgets x264 xcb xine xml xorg xosd xscreensaver xv xvid zlib" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan stage tables krita karbon braindump" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="keyboard mouse evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LINGUAS="en en_US" PHP_TARGETS="php5-3" RUBY_TARGETS="ruby18" USERLAND="GNU" VIDEO_CARDS="radeon" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

=================================================================
Package Settings
=================================================================

dev-lang/python-2.7.2-r3 was built with the following:
USE="gdbm (multilib) ncurses readline sqlite ssl threads tk (wide-unicode) xml -berkdb -build -doc -examples -ipv6 -wininst"
CFLAGS="-march=native -O2 -pipe -fwrapv"
CXXFLAGS="-march=native -O2 -pipe -fwrapv"
LDFLAGS="-Wl,-O1 -Wl,--as-needed -L."


sys-libs/glibc-2.13-r4 was built with the following:
USE="(multilib) nls -debug -gd -glibc-omitfp (-hardened) -profile (-selinux) -vanilla"
CFLAGS="-march=native -pipe -O2 -fno-strict-aliasing"
CXXFLAGS="-march=native -pipe -O2 -fno-strict-aliasing"

gzip: /proc/config.gz: No such file or directory
$ grep IPV6 /usr/src/linux/.config
# CONFIG_IPV6 is not set


(As a side note, the concept of /proc/config.gz is awesome, and will be available the next time I need to recompile my kernel anyway; for that alone, I thank you.)
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 21490

PostPosted: Thu Dec 29, 2011 11:19 pm    Post subject: Reply with quote

That output confirms that the system has been built consistent with your intent to disable IPv6. What about the Python script itself? I asked above about the urlopen parameters.
Back to top
View user's profile Send private message
minnmass
n00b
n00b


Joined: 29 Dec 2011
Posts: 16
Location: United States

PostPosted: Fri Dec 30, 2011 3:07 am    Post subject: Reply with quote

Sorry.

It appears to be any python script: app-portage/mirrorselect, for instance, doesn't work:
Code:

$ sudo !!
sudo mirrorselect -i
Password:
* Downloading a list of mirrors...
ERROR: Could not get mirror list. Check your internet connection.


Similarly, youtube-dl doesn't work:
Code:

$ youtube-dl -U
Updating to latest version...
Traceback (most recent call last):
  File "/home/minnmass/scripts/youtube-dl", line 4237, in <module>
    main()
  File "/home/minnmass/scripts/youtube-dl", line 4215, in main
    updateSelf(fd, sys.argv[0])
  File "/home/minnmass/scripts/youtube-dl", line 3838, in updateSelf
    urlh.close()
UnboundLocalError: local variable 'urlh' referenced before assignment

$ youtube-dl --version
2011.10.19

$ youtube-dl http://www.youtube.com/watch?v=$VIDEO
[youtube] Setting language
WARNING: unable to set language: <urlopen error [Errno -2] Name or service not known>
[youtube] $VIDEO: Downloading video webpage
ERROR: unable to download video webpage: <urlopen error [Errno -2] Name or service not known>

(In that last example, $VIDEO is a legitimate video ID; the particular video attempted doesn't matter.)

Manually updating to the most current version of youtube-dl (downloaded direct from the source) doesn't fix anything or change anything in the errors generated, save for some line numbers:
Code:


$ youtube-dl -U
Updating to latest version...
Traceback (most recent call last):
  File "/home/minnmass/scripts/youtube-dl", line 4514, in <module>
    main()
  File "/home/minnmass/scripts/youtube-dl", line 4505, in main
    _real_main()
  File "/home/minnmass/scripts/youtube-dl", line 4479, in _real_main
    updateSelf(fd, sys.argv[0])
  File "/home/minnmass/scripts/youtube-dl", line 4075, in updateSelf
    urlh.close()
UnboundLocalError: local variable 'urlh' referenced before assignment
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 21490

PostPosted: Fri Dec 30, 2011 4:04 am    Post subject: Reply with quote

I think the youtube-dl authors need to learn about modules. That file is huge!

After looking at the source, my guess is that urllib.urlopen fails for you, but does so in a way that does not raise an exception. What is the output of cat /etc/resolv.conf; ls -l /etc/resolv.conf? Also, please pastebin the output of strace -tt -s4096 youtube-dl -U.
Back to top
View user's profile Send private message
minnmass
n00b
n00b


Joined: 29 Dec 2011
Posts: 16
Location: United States

PostPosted: Fri Dec 30, 2011 4:17 am    Post subject: Reply with quote

Code:

$ cat /etc/resolv.conf ; ls -l /etc/resolv.conf
# Generated by net-scripts for interface eth0
nameserver 192.168.2.1
-rw-r--r-- 1 root root 69 Dec 26 10:42 /etc/resolv.conf


Also, in case it matters:
Code:

$ cat /etc/conf.d/net
# This blank configuration will automatically use DHCP for any net.*
# scripts in /etc/init.d.  To create a more complete configuration,
# please review /etc/conf.d/net.example and save your configuration
# in /etc/conf.d/net (this file :]!).
# config_eth0=( "dhcp" )

 config_eth0="192.168.2.10/24"
 routes_eth0="default via 192.168.2.1"
 dns_servers_eth0="192.168.2.1"
modules="wpa_supplicant"

That last line isn't necessary (it's a wired connection).

The 2.3M output (all 3970 lines of it) of strace are available here, via DropBox (Pastebin's limit is 500k for non-Pro users).
Code:
$ file strace_output.txt
strace_output.txt: ASCII English text, with very long lines, with CRLF, LF line terminators, with escape sequences
Back to top
View user's profile Send private message
calibumm
n00b
n00b


Joined: 27 Sep 2012
Posts: 1

PostPosted: Thu Sep 27, 2012 9:49 pm    Post subject: Reply with quote

I'd had the exact same issue, specifically with youtube-dl after I attempted to install a python package called buildbot but later removed it. After the removal,
It looks to have mis-configured something with certain python modules, namely urllib.

I added a few print statements in my youtube-dl script and it fails right before that call to urllib.urlopen and that module is found in
/usr/lib/python2.7/dist-packages
I wasn't sure if a resolution came about from this thread.
Back to top
View user's profile Send private message
chris...
Apprentice
Apprentice


Joined: 26 Sep 2006
Posts: 179
Location: Melbourne, AU

PostPosted: Fri Sep 28, 2012 2:13 am    Post subject: Reply with quote

Im getting the same issue as well
Doesnt work on groove shark either

It used to work before
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum