[HOWTO] Intel C++ Compiler and per-package CFLAGS

Message

Desperadoss · Post by **Desperadoss** » Thu Jun 19, 2008 7:19 am

|mattst88| wrote:I've rewritten Gentoo-Wiki's HOWTO on using the Intel C++ Compiler.

http://gentoo-wiki.com/HOWTO_Intel_C_Compiler

The old guide was terribly written and horribly messy.

It also made Portage compile with icc unless you explicitly told it to use gcc. This is bad. Far too many packages don't work with icc to use this behavior. My guide makes Portage compile only specified packages with icc and all others with gcc which will save a few headaches in the long run. It also will give your system the ability to use per-package CFLAGS for gcc and icc!

Need feedback on the guide. Suggestions are welcome.

Really Thanx !

Post by **mattst88** » Sun Jun 29, 2008 4:27 am

To fix the limits.h bug, update to icc-10.1.017. I've already filed a version bump request in bugzilla.

drakhain · Post by **drakhain** » Sun Jun 29, 2008 11:46 pm

|mattst88| wrote:To fix the limits.h bug, update to icc-10.1.017. I've already filed a version bump request in bugzilla.

Hello,
I've installed Gentoo 64 on my XPS M1710 with icc 10.1.17 and i'm still having problem with that limits.h. Since i'm not able to compile blender from the portage tree ( the method explained with bashrc and package.icc/gcc didn't worked for me, meaning that whatever is in those files it is being compiled with icc all the time) i've downloaded the source from blender.org.
Can someone help me compiling blender 2.46 with Icc please?

My Wiki page · Post by **mattst88** » Mon Jun 30, 2008 12:41 am

drakhain wrote:
|mattst88| wrote:To fix the limits.h bug, update to icc-10.1.017. I've already filed a version bump request in bugzilla.
Hello,
I've installed Gentoo 64 on my XPS M1710 with icc 10.1.17 and i'm still having problem with that limits.h. Since i'm not able to compile blender from the portage tree ( the method explained with bashrc and package.icc/gcc didn't worked for me, meaning that whatever is in those files it is being compiled with icc all the time) i've downloaded the source from blender.org.
Can someone help me compiling blender 2.46 with Icc please?

It looks to me like Blender's build system is hardcoded to use gcc.

It cannot be compiled by icc using this guide alone.

My Wiki page · Post by **mattst88** » Mon Jun 30, 2008 1:32 am

Well, crap. By updating to 10.1.017, it fixed the php build on my AMD64 system, but not on my Thinkpad Core2 Duo..

drakhain · Post by **drakhain** » Mon Jun 30, 2008 5:45 am

Yes by default blender is build with gcc. to enable the built to be done with icc i just edited the nan_complile file under the blender2.blabla/source directory. After that i compile it with the make command. I tried to use scons after editing it's files but i had to hardcode icc in it too (even if it has some compiler detection in Sconstruct i wasn't able to use it by doing things like: OCC=icc).

drakhain · Post by **drakhain** » Tue Jul 01, 2008 3:27 pm

So?, there is no workaround for using icc without being bored by a "can't open limits.h" or "limits.h trying to call himself"?

My Wiki page · Post by **mattst88** » Tue Jul 01, 2008 3:35 pm

I haven't found it yet if there is an existing fix.

The icc-10.1.017 readme says php now compiles on AMD64.

icc-10.1.015 will not compile php on either my desktop or my laptop due to the limits.h error.
icc-10.1.017 will compile php on my desktop, but still fails on my laptop due to the limits.h error.

Bizarre.

drakhain · Post by **drakhain** » Tue Jul 01, 2008 3:49 pm

Ok :s, but is it possible that includes from gcc and include from icc are interfering when icc is parsing the sources?

My Wiki page · Post by **mattst88** » Tue Jul 01, 2008 4:17 pm

drakhain wrote:Ok :s, but is it possible that includes from gcc and include from icc are interfering when icc is parsing the sources?

It is possible, but I think it is some kind of icc bug.

brian33x51 · Post by **brian33x51** » Thu Jul 10, 2008 4:49 pm

Yup I'll verify this is still a problem (10.0.017 x86_64). I can't compile since one of the headers in one of the libraries I depend on includes this file.

DDMask · Post by **DDMask** » Sun Jul 13, 2008 9:30 pm

So I did my own dumb benchmark thing, and bzipping a 2GB file on a Pentium-M (Celeron) went from 21 mins to 16.5 minutes with icc, so I'm feeling it's worthwhile for me to recompile a few more items in order to get this clunky laptop running faster. However my Core2Duo running at 2.6Ghz showed a lesser improvement by going from a bzip2 compression time of 30s on a different file to 26s.

Would any of you smarty-pants be willing to throw together a script which could modify /etc/portage/package.gcc based on failures during emerge? Maybe in the meantime a more comprehensive blacklist could be created on the wiki with not just programs that could bork a system, but also packages that will fail with icc 100% of the time.

The 2% or so gained in speed by using Gentoo is nice I guess, but a gain of >10% is substantial, even if it only applies to some programs.

My Wiki page · Post by **mattst88** » Mon Jul 14, 2008 3:15 am

DDMask wrote:So I did my own dumb benchmark thing, and bzipping a 2GB file on a Pentium-M (Celeron) went from 21 mins to 16.5 minutes with icc, so I'm feeling it's worthwhile for me to recompile a few more items in order to get this clunky laptop running faster. However my Core2Duo running at 2.6Ghz showed a lesser improvement by going from a bzip2 compression time of 30s on a different file to 26s.

21 minutes to 16.5 is a speed improvement of 21%. 30 seconds to 26 seconds is still a speed improvement of 13%. Not bad at all. Did you compile bzip2 with -xT -ip -O3 or similar on the Core 2 Duo?

DDMask · Post by **DDMask** » Mon Jul 14, 2008 11:38 am

|mattst88| wrote:Did you compile bzip2 with -xT -ip -O3 or similar on the Core 2 Duo?

I just used ICCCFLAGS="-O2 -xT -ip -gcc" maybe -O3 would be a better choice.

EDIT: I'm bored so here's my C2D with bzip2 compiled with -O2 -xT -ip -gcc:

real 3m11.639s
user 2m56.118s
sys 0m3.346s

with -O3 -xT -ipo -gcc:

real 3m0.241s
user 2m55.174s
sys 0m2.933s

with gcc, -march=nocona -02 -pipe

real 9m20.463s
user 9m17.525s
sys 0m2.969s

Ummm... that ain't right. 3 tries with gcc and they were all the same. I guess I screwed something up somewhere. Oh well, I'll just go back to an icc build.

Gregoire · Post by **Gregoire** » Tue Aug 26, 2008 1:23 pm

Is someone using icc and bashrcng ?

jdoe · Post by **jdoe** » Wed Aug 27, 2008 8:22 am

is there a way to use icc on a per package basis with paludis?

a.b. · Post by **a.b.** » Thu Aug 28, 2008 10:42 pm

jdoe wrote:is there a way to use icc on a per package basis with paludis?

I's amazing. I just have finished a bashrc for paludis and now I see your post

I've uploaded it together with some example config files here: http://ihatenicks.bplaced.net/paludis-bashrc/ Read the config files and the comments in the script for instructions.

I haven't done much testing yet, feedback is welcome.

Gregoire · Post by **Gregoire** » Tue Sep 02, 2008 7:46 am

Great, I'll emerge paludis this afternoon and I will try this new bashrc

In the meantime, I am for a few days under x11-base/xorg-server-1.4.2 compiled without debug (which fails with gcc) working great compiled with ICCCFLAGS="-O2 -xT -gcc -mtune=core2".

Also media-video/ffmpeg-20089999 from berkano overlay compil and run very well with -O3 -xT -gcc -mtune=core2.

Sofar I can't compil mplayer because of my system having freetype-1 and freetype-2 : http://forums.gentoo.org/viewtopic-t-50 ... t-475.html

Great !!!

Thank for all work that have been done

steveb · Post by **steveb** » Thu Sep 04, 2008 5:38 pm

I don't know when I installed ICC and used the unofficial Gentoo wiki to copy the script from there and use it. Anyway... after some time I started to modify the script from there. One bit there, another bit there. This is what I use now:

Code: Select all

##
## Set which compiler to favorize
##
primary_compiler="gcc"
secondary_compiler="icc"

###########################################################
##
## DO NOT CONFIGURE ANYTHING BELOW THIS LINE
##
###########################################################

##
## Try to exit as fast as needed
##
[ -z "${CATEGORY}" -o -z "${PN}" ] && return 0

[ "${primary_compiler}" == "gcc" -a -z "${secondary_compiler}" ] && secondary_compiler="icc"
[ "${primary_compiler}" == "icc" -a -z "${secondary_compiler}" ] && secondary_compiler="gcc"
[ "${secondary_compiler}" != "" -a ! -r "${ROOT}"/etc/portage/package.${secondary_compiler} ] && secondary_compiler=""
[ "${secondary_compiler}" == "" -a ! -r "${ROOT}"/etc/portage/package.${primary_compiler} ] && return 0
handled_by_primary=0
handled_by_secondary=0

##
## Function for exporting CFLAGS
##
export_CFLAGS() {
  local compiler="gcc"
  [ "${@}" == "icc" ] && compiler="icc"
  if [ -r "${ROOT}"/etc/portage/package.${compiler}-cflags ]
  then
    while read target cflags
    do
      if [ "${target}" == "${CATEGORY}/${PN}" ]
      then
        export CFLAGS="${cflags}"
        break
      fi
    done < "${ROOT}"/etc/portage/package.${compiler}-cflags
  fi
}

##
## Function for exporting CXXFLAGS
##
export_CXXFLAGS() {
  local compiler="gcc"
  [ "${@}" == "icc" ] && compiler="icc"
  if [ -r "${ROOT}"/etc/portage/package.${compiler}-cxxflags ]
  then
    while read target cxxflags
    do
      if [ "${target}" == "${CATEGORY}/${PN}" ]
      then
        export CXXFLAGS="${cxxflags}"
        break
      fi
    done < "${ROOT}"/etc/portage/package.${compiler}-cxxflags
  fi
}

##
## Function for exporting LDFLAGS
##
export_LDFLAGS() {
  local compiler="gcc"
  [ "${@}" == "icc" ] && compiler="icc"
  if [ -r "${ROOT}"/etc/portage/package.${compiler}-ldflags ]
  then
    while read target ldflags
    do
      if [ "${target}" == "${CATEGORY}/${PN}" ]
      then
        export LDFLAGS="${ldflags}"
        break
      fi
    done < "${ROOT}"/etc/portage/package.${compiler}-ldflags
  fi
}

##
## Check if primary compiler will handle the package
##

if [ -r "${ROOT}"/etc/portage/package.${primary_compiler} ]
then
  while read -a target
  do
    if [ "${target}" == "${CATEGORY}/${PN}" ]
    then
      handled_by_primary=1
      if [ "${primary_compiler}" == "icc" ]
      then
        export OCC="icc"
        export OCXX="icpc"
        #export F77="ifort"
        #export FC="ifort"
        export CFLAGS=${ICCCFLAGS}
        export CXXFLAGS=${ICCCXXFLAGS}
        export LDFLAGS=${ICCLDFLAGS}
      fi
      export_CFLAGS ${primary_compiler}
      export CXXFLAGS="${CFLAGS}"
      export_CXXFLAGS ${primary_compiler}
      export_LDFLAGS ${primary_compiler}
      break
    fi
  done < "${ROOT}"/etc/portage/package.${primary_compiler}
fi

##
## Check if secondary compiler should handle the package
##
if [ "${handled_by_primary}" == "0" -a "${secondary_compiler}" != "" -a -r "${ROOT}"/etc/portage/package.${secondary_compiler} ]
then
  while read -a target
  do
    if [ "${target}" == "${CATEGORY}/${PN}" ]
    then
      handled_by_secondary=1
      if [ "${secondary_compiler}" == "icc" ]
      then
        export OCC="icc"
        export OCXX="icpc"
        #export F77="ifort"
        #export FC="ifort"
        export CFLAGS=${ICCCFLAGS}
        export CXXFLAGS=${ICCCXXFLAGS}
        export LDFLAGS=${ICCLDFLAGS}
      fi
      export_CFLAGS ${secondary_compiler}
      export CXXFLAGS="${CFLAGS}"
      export_CXXFLAGS ${secondary_compiler}
      export_LDFLAGS ${secondary_compiler}
      break
    fi
  done < "${ROOT}"/etc/portage/package.${secondary_compiler}
fi

##
## System compiler will handle the package
##
[ "${handled_by_primary}" == "0" -a "${handled_by_secondary}" == "0" ] && return 0

##
## workaround gcc detection function in toolchain-funcs.eclass
##
if [ "${OCC}" != "" ]
then
  export CC_FOR_BUILD="${OCC}"
fi

I have another one but that one is not 100% finished and working the way I would like it to work. Anyway... maybe some one will find this one more flexible then the one in the unofficial Gentoo wiki? It works +/- the same way as the one in the unofficial Gentoo wiki except that it allows you as well to use different flags for the C++ compiler (/etc/portage/package.{gcc,icc}-cxxflags) and if needed LDFLAGS (/etc/portage/package.{gcc,icc}-ldflags). The CFLAGS can be specified in /etc/portage/package.{gcc,icc}-cflags. If no C++ flags are specified, then the script will use the one specified for the C compiler. Some stuff looks freaky but I tried to avoid running into problems by using external programs like sed, awk, grep, etc...

// SteveB

tranquilcool · Post by **tranquilcool** » Thu Sep 04, 2008 11:43 pm

steveb wrote:I don't know when I installed ICC and used the unofficial Gentoo wiki to copy the script from there and use it. Anyway... after some time I started to modify the script from there. One bit there, another bit there. This is what I use now:

Code: Select all

##
## Set which compiler to favorize
##
primary_compiler="gcc"
secondary_compiler="icc"

###########################################################
##
## DO NOT CONFIGURE ANYTHING BELOW THIS LINE
##
###########################################################

##
## Try to exit as fast as needed
##
[ -z "${CATEGORY}" -o -z "${PN}" ] && return 0

[ "${primary_compiler}" == "gcc" -a -z "${secondary_compiler}" ] && secondary_compiler="icc"
[ "${primary_compiler}" == "icc" -a -z "${secondary_compiler}" ] && secondary_compiler="gcc"
[ "${secondary_compiler}" != "" -a ! -r "${ROOT}"/etc/portage/package.${secondary_compiler} ] && secondary_compiler=""
[ "${secondary_compiler}" == "" -a ! -r "${ROOT}"/etc/portage/package.${primary_compiler} ] && return 0
handled_by_primary=0
handled_by_secondary=0

##
## Function for exporting CFLAGS
##
export_CFLAGS() {
  local compiler="gcc"
  [ "${@}" == "icc" ] && compiler="icc"
  if [ -r "${ROOT}"/etc/portage/package.${compiler}-cflags ]
  then
    while read target cflags
    do
      if [ "${target}" == "${CATEGORY}/${PN}" ]
      then
        export CFLAGS="${cflags}"
        break
      fi
    done < "${ROOT}"/etc/portage/package.${compiler}-cflags
  fi
}

##
## Function for exporting CXXFLAGS
##
export_CXXFLAGS() {
  local compiler="gcc"
  [ "${@}" == "icc" ] && compiler="icc"
  if [ -r "${ROOT}"/etc/portage/package.${compiler}-cxxflags ]
  then
    while read target cxxflags
    do
      if [ "${target}" == "${CATEGORY}/${PN}" ]
      then
        export CXXFLAGS="${cxxflags}"
        break
      fi
    done < "${ROOT}"/etc/portage/package.${compiler}-cxxflags
  fi
}

##
## Function for exporting LDFLAGS
##
export_LDFLAGS() {
  local compiler="gcc"
  [ "${@}" == "icc" ] && compiler="icc"
  if [ -r "${ROOT}"/etc/portage/package.${compiler}-ldflags ]
  then
    while read target ldflags
    do
      if [ "${target}" == "${CATEGORY}/${PN}" ]
      then
        export LDFLAGS="${ldflags}"
        break
      fi
    done < "${ROOT}"/etc/portage/package.${compiler}-ldflags
  fi
}

##
## Check if primary compiler will handle the package
##

if [ -r "${ROOT}"/etc/portage/package.${primary_compiler} ]
then
  while read -a target
  do
    if [ "${target}" == "${CATEGORY}/${PN}" ]
    then
      handled_by_primary=1
      if [ "${primary_compiler}" == "icc" ]
      then
        export OCC="icc"
        export OCXX="icpc"
        #export F77="ifort"
        #export FC="ifort"
        export CFLAGS=${ICCCFLAGS}
        export CXXFLAGS=${ICCCXXFLAGS}
        export LDFLAGS=${ICCLDFLAGS}
      fi
      export_CFLAGS ${primary_compiler}
      export CXXFLAGS="${CFLAGS}"
      export_CXXFLAGS ${primary_compiler}
      export_LDFLAGS ${primary_compiler}
      break
    fi
  done < "${ROOT}"/etc/portage/package.${primary_compiler}
fi

##
## Check if secondary compiler should handle the package
##
if [ "${handled_by_primary}" == "0" -a "${secondary_compiler}" != "" -a -r "${ROOT}"/etc/portage/package.${secondary_compiler} ]
then
  while read -a target
  do
    if [ "${target}" == "${CATEGORY}/${PN}" ]
    then
      handled_by_secondary=1
      if [ "${secondary_compiler}" == "icc" ]
      then
        export OCC="icc"
        export OCXX="icpc"
        #export F77="ifort"
        #export FC="ifort"
        export CFLAGS=${ICCCFLAGS}
        export CXXFLAGS=${ICCCXXFLAGS}
        export LDFLAGS=${ICCLDFLAGS}
      fi
      export_CFLAGS ${secondary_compiler}
      export CXXFLAGS="${CFLAGS}"
      export_CXXFLAGS ${secondary_compiler}
      export_LDFLAGS ${secondary_compiler}
      break
    fi
  done < "${ROOT}"/etc/portage/package.${secondary_compiler}
fi

##
## System compiler will handle the package
##
[ "${handled_by_primary}" == "0" -a "${handled_by_secondary}" == "0" ] && return 0

##
## workaround gcc detection function in toolchain-funcs.eclass
##
if [ "${OCC}" != "" ]
then
  export CC_FOR_BUILD="${OCC}"
fi

I have another one but that one is not 100% finished and working the way I would like it to work. Anyway... maybe some one will find this one more flexible then the one in the unofficial Gentoo wiki? It works +/- the same way as the one in the unofficial Gentoo wiki except that it allows you as well to use different flags for the C++ compiler (/etc/portage/package.{gcc,icc}-cxxflags) and if needed LDFLAGS (/etc/portage/package.{gcc,icc}-ldflags). The CFLAGS can be specified in /etc/portage/package.{gcc,icc}-cflags. If no C++ flags are specified, then the script will use the one specified for the C compiler. Some stuff looks freaky but I tried to avoid running into problems by using external programs like sed, awk, grep, etc...

// SteveB

what is this script? is it bashrc to put in /etc/portage?
i don't understand where you put or how you use this script.

steveb · Post by **steveb** » Thu Sep 04, 2008 11:52 pm

tranquilcool wrote:what is this script? is it bashrc to put in /etc/portage?

Yes. It is a bashrc for Portage.

tranquilcool wrote:i don't understand where you put or how you use this script.

Read the unofficial Gentoo wiki article Intel C++ Compiler how to use it.

// SteveB

a.b. · Post by **a.b.** » Thu Sep 04, 2008 11:53 pm

Gregoire wrote:Great, I'll emerge paludis this afternoon and I will try this new bashrc

It should actually work with portage as well, though it's kinda untested and portage complains about the fact I've used sed.

Thank for all work that have been done

My pleasure.

steveb · Post by **steveb** » Thu Sep 04, 2008 11:56 pm

a.b. wrote:... and portage complains about the fact I've used sed.

That's the reason I tried to avoid external programs in my bashrc above.

// SteveB

a.b. · Post by **a.b.** » Fri Sep 05, 2008 12:00 am

The fact portage complains about the usage of external programs in the bashrc seems like a bug to me. Ebuilds aren't supposed to use anything other than bash code but I see no actual reason to restrict that in a configuration file.

steveb · Post by **steveb** » Fri Sep 05, 2008 12:23 am

a.b. wrote:The fact portage complains about the usage of external programs in the bashrc seems like a bug to me. Ebuilds aren't supposed to use anything other than bash code but I see no actual reason to restrict that in a configuration file.

Wrong. If you ever have build Gentoo for an embedded system, then you will see that bash is not the number one choice for an shell. So stuff like:

Code: Select all

function debug(){
  if [[ -n "$debug" && "$debug" == "1" ]]; then
    echo "$(date +%T) $1" >> /tmp/pbshrc-test
  fi
}

Will not work. What is the reason you use "[[" ... "]]"? Would it not be better and more portable to use (untested but I think it should work):

Code: Select all

function debug(){
  if [ -n "${debug}" -a "${debug}" == "1" ]; then
    echo "$(date +%T) ${1}" >> /tmp/pbshrc-test
  fi
}

And stuff like this:

Code: Select all

  while read atom data; do
    data="$(echo $data | sed -e 's/#.*//')"   #Strip comments
      if matchatom "$atom"; then
        if [ $append -eq 0 ]; then       
          ret="$data"
        else
          ret="$ret $data"
        fi
      fi
  done < $1

Could be replaced by this (untested but I think it should work. It should be even faster since calling the replace (in your code it is a call to GNU sed) is only done when needed):

Code: Select all

  while read atom data; do
      if matchatom "${atom}"; then
        if [ ${append} -eq 0 ]; then       
          ret="${data%%#*}"
        else
          ret="${ret} ${data%%#*}"
        fi
      fi
  done < ${1}

// SteveB

[HOWTO] Intel C++ Compiler and per-package CFLAGS

Re: [HOWTO] Intel C++ Compiler and per-package CFLAGS