Different compilers and library

[big_gie]

Hi,

I'm using different set of compilers (gfortran, intel, pgi) and I would like to build libraries for all of them to use with my code.

Right now, I can't (or just don't see how) build multiple version of the library. I need to :
CC="gcc" CXX="g++" F77="gfortran" FC="gfortran" emerge -av sys-cluster/openmpi
to build using gnu, or :
CC="icc" CXX=$CC F77="ifc" FC="$F77" emerge -av sys-cluster/openmpi
to build with intel, etc. But after this I will get only one version, depending on the compiler used.

Is there a way to slot a single version of a package builded with different compiler? What I would like is having a version of that library (openmpi) for each compiler possible (gnu, intel and pgi) into different path like
/usr/lib/openmpi/gnu
/usr/lib/openmpi/intel
/usr/lib/openmpi/pgi

So I could compile my program with one of those compiler and link to the appropriate library....

Thanx

[Mad Merlin]

AFAIK, there's no easy way to just slot an arbitrary package. Probably the easiest way to do what you want would be to emerge the package once with each compiler and then copy the libraries to ~/lib/ or /usr/local/lib/. It would be relatively easy to script the entire process if you're intending to maintain the setup (ie, it's not just some quick benchmarking or something similar) through newer package versions.
_________________
Game! - Where the stick is mightier than the sword!

[Gentree]

It seems like you want to build the same package twice (using the different compilers) so you will need to separate the two installations. You could do this by the build prefix , eg install the intel based one in /opt/intel as is done with the compilers.

you would then probably need a short wrapper script to change the paths according to which one you want to run.

A quick and easy way if you have enough disk and dont have your entire system on one partition would be to clone your root partition and install the intel build in one and the gnu in the other then use chroot to when you want to use intel.

HTH
_________________
Linux, because I'd rather own a free OS than steal one that's not worth paying for.
Gentoo because I'm a masochist
AthlonXP-M on A7N8X. Portage ~x86

[big_gie]

Hi,

Sorry Gentree I didn't saw your post!!! I'm posting what I've did.

I normally use the PGI compilers for fortran simulation. I also compile with ifort and gfortran to make sure my code is ok. So I need to compile my libraries with every compilers to be able to use them. One of those libaries is OpenMPI, another one is netCDF4. So I ended up with:
/usr/lib/netcdf/4.0-pgi
/usr/lib/netcdf/4.0-intel
/usr/lib/netcdf/4.0-gcc
When I compile using pgi, I link to the right path manually using -L. I want to access my data using python and its interface to netCDF4. So I must have a netcdf4 version compiled with gcc to use it with python, and one compiled with pgi for my simulation... Anyway you see the problem. For scientific programs, I need libraries built with optimizing compiler, and the rest can be compiled with gcc.

The only way I got this working was to :
1) Compile manually.
I don't like this since you loose the power of portage (upgrade, clean, etc.) and you can't use dependencies in ebuilds (netcdf4 (may) depends on hdf5, etc.)
2) Modify ebuils to point installation to /usr/lib/${PN}/${PV}-<compiler>
I've done this for netcdf4, hdf5 and openmpi. So I need to explicitly specify the path for linking.

Does anyone ever got the same issue? Is it something that can be suggested to devs? I understand it's not the intended way of doing things, but I think scientific applications require this.

What do you think?

Thanx

[Gentree]

here's a wrapper that I use since I have gimp-2.2 and gimp-2.3.x installed concurrently.

gimp command runs that std package in std location , the following runs the latter with is gimp/lib elsewhere. This would seem similar to what I understand you are trying to do.

[big_gie]

[big_gie]

I've got an idea last nigh ( )

What about multiple use flags:
use="pgi intel gcc" emerge netcdf
which would compile a version for each of the compilers stated:
/usr/lib/gcc/netcdf/4.0 (or /usr/lib/netcdf/4.0)
/usr/lib/pgi/netcdf/4.0
/usr/lib/intel/netcdf/4.0

So:
-"emerge netcdf" would give:
/usr/lib/gcc/netcdf/4.0
-"use=pgi emerge netcdf" would give:
/usr/lib/gcc/netcdf/4.0
/usr/lib/pgi/netcdf/4.0

etc.

What do you think?

[big_gie]

Here's what I did:

I've made a generic ebuild for the library, with (almost) nothing in it. That ebuild depends on 3 other ebuilds conditionnally to use flags (gcc, pgi and intel). Those 3 ebuilds install the library with the correct compilers:
sci-libs/hdf5/hdf5-1.8.0_alpha5.ebuild
sci-libs/hdf5-gcc/hdf5-gcc-1.8.0_alpha5-r4.ebuild
sci-libs/hdf5-pgi/hdf5-pgi-1.8.0_alpha5-r4.ebuild
sci-libs/hdf5-intel/hdf5-intel-1.8.0_alpha5-r4.ebuild

The last 3 install their files in :
/usr/lib/gcc/hdf5/1.8.0-alpha5
/usr/lib/pgi/hdf5/1.8.0-alpha5
/usr/lib/intel/hdf5/1.8.0-alpha5

I've used that logic for netcdf4 also, which can depends on those:
sci-libs/netcdf/netcdf-4.0_alpha2007030802.ebuild
sci-libs/netcdf-gcc/netcdf-gcc-4.0_alpha2007030802.ebuild (files in /usr/lib/gcc/netcdf/4.0-alpha2007030802/)
sci-libs/netcdf-pgi/netcdf-pgi-4.0_alpha2007030802.ebuild (files in /usr/lib/pgi/netcdf/4.0-alpha2007030802/)
sci-libs/netcdf-intel/netcdf-intel-4.0_alpha2007030802.ebuild (files in /usr/lib/intel/netcdf/4.0-alpha2007030802/)

And finally a python module for netcdf4 reading, which depends on sci-libs/netcdf-gcc :
sci-libs/python-netcdf4/python-netcdf4-0.6.2-r1.ebuild

This help for scientific libraries for use by programs compiled with different compilers. I want to link to hdf5 and netcdf4 from my fortran code which can be built with gcc, pgi or intel and this is the "cleaner" solution I've found...

Any comments? Should that go directly to bugzilla?

[Genone]

The easiest work around would probably be to set up 3 different chroot environments (using $ROOT and $PORTAGE_CONFIGROOT), one for each compiler/toolchain, just requires a bit more CPU time and diskspace to install and update all three systems. Has also the benefit that you can eventually use different versions of a library for each compiler if that need should ever arise.

For a real solution you basically want a new variable similar to SLOT that's dynamically generated by some env variable (like CC). There are several issues right now that prevent such a thing, mainly the lack of SLOT deps (well, they exist but can't be used yet, and would probably have to be changed for this first) and the "metadata-constant" cache restriction that requires that dependency resolution can't use system specific information (like environment variables) but only constant data as it has to be serialized.
Your proposal of making multiple packages for each compiler has the nasty problem that it doesn't scale very well and requires pointless duplication of code and data. The solution with USE flags is a bit better but would need USE dependencies first to really work.

[hanni_ali]

How about using an eselect class to define which environment you want to use such that a symlink points to the desired library hence removing the reliance on using -L.

So by installing all the different versions compiled with the different compilers it would be possible to dynamically switch between them. This in combination with the use flags would cover the majority of the issues. Does anyone agree?

Hanni

[big_gie]

[Genone]

[hanni_ali]

Although using the chroots with multiple versions of compiled systems would provide a solution it would infact be less "simple" than you think. First if running MPI you would need to set-up the chroots on all the machines in your cluster and if running a heterogeneous cluster you would need to test the chroot across all the machines and maintaining the cluster could become a nightmare.

Then comes the issues when installing different versions of MPI https://bugs.gentoo.org/show_bug.cgi?id=44132 (I note big_gie has raised the issue in discussion here, there, as well) on the same system you would require even more chroot environments and in particular my preference for cluster/network operating systems is to minimise network traffic and if on selecting a different chroot and set that up before they can begin processing (all our clusters run diskless) this put's unecessary load on the network.

The point is there must be a more elegant and most importantly more maintainable method for managing different MPI implementations and compilers and their respective versions!

In my opinion a combination of some form of config selection (eselct or mpi-config), with appropriate use flags, virtual ebuilds which pull in the necessary stuff and may even run some sanity tests and perhaps some patches for ./configs of applications dependent on mpi systems to check that the currently selected mpi environment contains the necessary functionality, or alternatively if the application depends on a specific version only, then perhaps hard coding the path would be most sensible.

Keep the ideas coming.

Hanni

P.S. did you read this https://bugs.gentoo.org/show_bug.cgi?id=30453#c22 the ideas discussed there if considered could help in particular:

[big_gie]

[hanni_ali]

Have you put the ebuilds you are working on anywhere? I'd be interested in seeing them and I think your suggestion is an excellent way to deal with this.

A virtual for all libs which could be built with different compilers which uses a use flag during the build to select the compiler and an eselect class to select the lib at runtime.

Where the default configuration would install the standard gcc versions.

[big_gie]

[khpae]

i'm using ~amd64 profile (multilib) on a em64t (C2Q) system.
i got problem on installing openmpi-intel package.
here's what i did :
installed icc / ifc 10.0.026.
downloaded your openmpi-intell-1.1.5 ebuild and changed it to openmpi-intel-1.2.4.
edited the content of the ebuild accordingly, of course.
created /usr/local/portage/sys-cluster/openmpi-intel directory and placed it there.
added these lines in the /etc/make.conf file.

[khpae]

nevermind. i solved the problem. was my fault.
some CFLAGS were not filtered correctly.