OpenCL: New ebuild and eselect module.

[big_gie]

Hi all,

I have written a couple of ebuilds and programs that I think could be interesting for others.

First, I have written a library that abstract the initialization of OpenCL paltforms and devices. You choose which platform to use, and it will sort the available devices on it according to number of compute units (cores). It will try to create a context on the best device, and will fallback to the next one in the list if it does not work.
You can find oclutils here:
https://github.com/nbigaouette/oclutils/

Second, I have written an ebuild for the Intel OpenCL SDK 1.1. You'll find it here:
https://github.com/nbigaouette/ebuilds/tree/master/dev-util/intel-opencl-sdk

Third, I have updated the AMD Accelerated Parallel Processing (APP, formaly AMD Stream) ebuild to 2.5:
https://github.com/nbigaouette/ebuilds/tree/master/dev-util/amdstream

Fourth, I have forked eselect-opencl 0.0.2 from pentoo (http://dev.pentoo.ch/~grimmlin/distfiles/eselect-opencl-0.0.2.tar.bz2). I've updated it and cleaned it up a bit. The only issue I see is with the nvidia opencl file: x11-drivers/nvidia-drivers install its files to /usr/lib*/libOpenCL.so*. I have tried it on my box with intel, amd and nvidia's opencl and it works, but there might be some corner case I've haven't seen. There might be a better way to handle nvidia's case too. I'm open to suggestions.
You'll find the eselect module here:
https://github.com/nbigaouette/eselect-opencl/
and the ebuild here:
https://github.com/nbigaouette/ebuilds/tree/master/app-admin/eselect-opencl

I hope it will be useful to somebody!

Cheers

Edit: Bumped amdstream from 2.4 to 2.5

[i92guboj]

Moved from Gentoo Chat to Unsupported Software.

It fits better here.

[metapsyborg]

[yellowhat]

Some question for you:
What can opencl speed using GPU?
Can opencl help in compiling during emerge?

I am using a amd HD6870.

Thnaks

[big_gie]

@metapsyborg,
I'm glad you find it useful! I had to do this myself and I realized it could be useful to others. I will update to 2.5 when I'll found the time. You can always fork, modify and then send a pull request

@yellowhat,
OpenCL can speedup many different things. I use it to accelerate a molecular dynamics code and an quantum-fdtd code that I wrote for my thesis. To give you an example, one instance of the molecular dynamics' OpenMP implementation has been running since September 2nd on 24 cores. It's 31% done. I have launched the OpenCL implementation (on an nvidia GTX 590) on September 21st and it's at 69%. The time evolution is not linear so it's hard to evaluate how much faster it is, but a rough estimate is 40 times faster. And because on the 24 cores machine there is two video cards, the gpu version is basically 80 times faster

Apple developped OpenCL to accelerate it's own applications on a GPU, a completely different usage then mine. But because not every mac have a discrete video card, they needed a fallback to CPU. They inspired themselves from nvidia's cuda and created an open standard. Now AMD and Intel both have an OpenCL driver to expose the CPU as an accelerator. This is counter-intuitive, but can ease the development and even speed things up! (I do see a speedup on my physics large scale simulations when running with OpenCL as opposed to OpenMP, on the same machine).

OpenCL is not a just a useflag that, when enabled, speeds everything else. It's a programming language, similar to C. Instead of developing your calculations intensive programs in C, you move the heavy part to OpenCL (or cuda). To speed compilation, gcc would need to have some parts re-written in OpenCL. There is no way this is going to happen. And if I remember correctly, long compilations are often disk-bound. This is why you can launch more gcc instances then cores available (using make's -j flag). So it wouldn't be a good optimization anyway.

OpenCL (or any other GPU programming language) is relatively hard to learn. This is because a GPU is a completely different architecture then a CPU which we are used to. When learning it, I remember having the same feeling I had when learning MPI: you need to think differently.

[Trog Dog]

G'day big_gie

Thanks for your amdstream and eselect-opencl ebuilds I was previously using the ebuilds from pentoo overlay, just a couple of questions why does the amdstream ebuild want to pull in nvidia-cuda-toolkit and why doesn't it pull in eselect-opencl?

[big_gie]

Hi Trog Dog,

I just checked the ebuild. It does pull either nvidia-cuda-toolkit or dev-util/opencl-headers. Both of these packages are in conflict. dev-util/opencl-headers will provide only the opencl headers, necessary to compile opencl code. You'll have to unmask it first and install it if you don't want the nvidia stuff. (the cuda toolkit is probably stable, so this would be why emerge pulls it first before dev-util/opencl-headers).

As for the eselect-opencl, I could add a dependency. I'll have a look when bumping to 2.5.

[Trog Dog]

Thanks for the reply big_gie, I did unmask opencl-headers - I was just confused as to why I would need nvidia code with an ATI/AMD gpu, thanks for clearing that up. WRT eselect-opencl after installing your amdstream package the only way to get my existing binaries to see the new libopencl.so was to upgrade to your eselect-opencl package, the one from pentoo (which is pulled in by the pentoo amd-stream-sdk-bin) just wouldn't cut the mustard, even after env-update source /etc/profile.

[big_gie]

If I remember correctly, eselect-opencl looks in a specify place for the .so files. Pentoo's are probably installed somewhere else. If you look at the eselect module's source (https://github.com/nbigaouette/eselect-opencl/blob/master/opencl.eselect) you'll see neer the top:

[Trog Dog]

Once again, thank you big_gie.

[big_gie]

[alaaibrahim]

Everytime I try to install dev-util/amdstream I get with the error

[big_gie]

You mean when you try to compile/run one of your opencl program or when emerging amdstream?

On my machine gluPerspective is defined in /usr/include/GL/glu_mangle.h which is part of media-libs/mesa-7.11. For the linking, it is defined in /usr/lib/debug/usr/lib64/libGLU.so.1.3.071100.debug (actually /usr/lib64/libGLU.so.1.3.071100) which is also part of mesa...

Maybe it's just a linking problem?

[alaaibrahim]

When I emerge amdstream

I tried to install it from amd SDK right away, it worked fine
then

[big_gie]

Please open an issue here: https://github.com/nbigaouette/ebuilds/issues so I can track the problem and fix it without spamming the thread... Thanks!

[Pj0tr]

[big_gie]

I renamed the package since AMD had renamed it. So the valid one should be "dev-util/amdapp". I was trying to ease the transition by making =dev-util/amdstream-2.6 an empty ebuild that just depends on =dev-util/amdapp-2.6.

I think everything is in order, but won't be able to check right now, sorry, stomach flu... :S

[Pj0tr]

[big_gie]

I've updated dev-util/amdapp-2.6-r3. I've included a fix to the "../../../thread/semaphore.cpp:87: sem_wait() failed" issue faced when nvidia is eselect-ed as opengl provider (use the 'mesa' useflag to include the fix).

To use, make sure to remove "dev-util/amdstream" before (I renamed the package to dev-util/amdapp).

Of course, you will need the ebuild. Either enable my overlay (layman --add nbigaouette) or download the ebuild from either the bug report (https://bugs.gentoo.org/show_bug.cgi?id=257626) or my github (https://github.com/nbigaouette/ebuilds/tree/master/dev-util/amdapp).

Note that I did not test it with an ATI/AMD video card. I use it to run OpenCL on the CPU.