CrankyPenguin
Apprentice
Joined: 19 Jun 2003
Posts: 283
|
 Posted: Sat Feb 16, 2013 1:05 am Post subject: Optimizing Python and R. |
 |
|
I'm going to be running some large machine learning processes that use Python, and R. The Python code is being used primarily for data munging and then calls R via Rpy2. R, in turn, is built on fortran (whee!). Anyways. Since I have some time on my hands while these things run I was wondering if anyone has some suggestions for the best use flags and cflags to use on these tools for fast performance.
By way of exposition my current settings are:
| Code: |
Portage 2.1.11.31 (default/linux/amd64/10.0, gcc-4.5.4, glibc-2.15-r3, 3.6.11 x86_64)
=================================================================
System uname: Linux-3.6.11-x86_64-AMD_Athlon-tm-_64_Processor_3800+-with-gentoo-2.1
Timestamp of tree: Fri, 15 Feb 2013 22:15:01 +0000
ld GNU ld (GNU Binutils) 2.22
app-shells/bash: 4.2_p37
dev-java/java-config: 2.1.11-r3
dev-lang/python: 2.7.3-r2, 3.2.3
dev-util/cmake: 2.8.9
dev-util/pkgconfig: 0.27.1
sys-apps/baselayout: 2.1-r1
sys-apps/openrc: 0.11.8
sys-apps/sandbox: 2.5
sys-devel/autoconf: 2.13, 2.68
sys-devel/automake: 1.11.6
sys-devel/binutils: 2.22-r1
sys-devel/gcc: 4.5.4, 4.6.3
sys-devel/gcc-config: 1.7.3
sys-devel/libtool: 2.4-r1
sys-devel/make: 3.82-r4
sys-kernel/linux-headers: 3.6 (virtual/os-headers)
sys-libs/glibc: 2.15-r3
Repositories: gentoo science java-overlay
ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* -@EULA AdobeFlash-10.1 Oracle-BCLA-JavaSE AdobeFlash-10.3"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /var/lib/hsqldb"
CONFIG_PROTECT_MASK="${EPREFIX}/etc/gconf /etc/ca-certificates.conf /etc/env.d /etc/env.d/java/ /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c"
CXXFLAGS="-march=native -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://gentoo.osuosl.org/ http://gentoo.mirrors.pair.com/ rsync://mirrors.rit.edu/gentoo/ http://mirrors.rit.edu/gentoo/ http://ftp.ucsb.edu/pub/mirrors/linux/gentoo/"
LANG="en_US.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j2"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
PORTDIR="/usr/portage"
PORTDIR_OVERLAY="/var/lib/layman/science /var/lib/layman/java-overlay"
SYNC="rsync://rsync.us.gentoo.org/gentoo-portage"
USE="3dnow 3dnowext UTF8 X acl acpi alsa amd64 apache2 berkdb bzip2 cdparanoia cjk cli corefonts cracklib crypt cups cxx directfb dri dv dvd dvdread emacs f77 fortran gdbm ginac gpm hddtemp iconv ipv6 jpeg kpathsea libcaca libg++ loop-aes mmx mmxext modules moznoirc moznomail mudflap multilib ncurses network nls nptl nptlonly opengl openmp pam pcre png quicktime readline session spell sse sse2 ssl tcltk tcpd theora tiff truetype type1 unicode usb vorbis zlib" ABI_X86="64" ALSA_CARDS="via82xx" ALSA_PCM_PLUGINS="adpcm alaw asym copy dmix dshare dsnoop empty extplug file hooks iec958 ioplug ladspa lfloat linear meter mmap_emul mulaw multi null plug rate route share shm softvol" APACHE2_MODULES="access auth auth_dbm auth_anon auth_digest alias file-cache echo charset-lite cache disk-cache mem-cache filter ext-filter case_filter case-filter-in deflate mime-magic cern-meta expires headers usertrack unique-id proxy proxy-connect proxy-ftp proxy-http info include cgi cgid dav dav-fs vhost-alias speling rewrite log_config logio env setenvif mime status autoindex asis negotiation dir imap actions userdir so authz_host" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump" CAMERAS="canon ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ubx" INPUT_DEVICES="kbd keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="en en_US fr es es_ES zh_TW zh_CN de it ko" PHP_TARGETS="php5-3" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_2" RUBY_TARGETS="ruby18 ruby19" USERLAND="GNU" VIDEO_CARDS="vesa radeon" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset: CPPFLAGS, CTARGET, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON
|
With the following package-specific flags:
| Code: |
[I] dev-lang/python
2.7.3-r2(2.7)(02:05:20 AM 12/19/2012)
(gdbm ipv6 ncurses readline sqlite ssl threads tk wide-unicode xml
-berkdb -build -doc -elibc_uclibc -examples -wininst)
|
| Code: |
[U] dev-lang/R
Installed versions: 2.15.0(09:22:19 PM 12/19/2012)
(X cairo jpeg nls openmp png readline tiff
-bash-completion -doc -icu -java -lapack -minimal -perl -profile -static-libs -tk)
|
EDIT: As some will immediately note above I am running -O2 rather than -O3. I chose that because the code also requires high reliability for small floating point operations and I wasn't comfortable with -O3's lack of guarantees.
_________________
Linux, the OS for the obsessive-compulsive speed freak in all of us. |
|
Portage & Programming
