Science Sprint
Location, Date
When: 24 & 26 June 2012 (on the 25th there is a workshop on Debian)
- Where: European Synchrotron Radiation Facility, 6 rue Jules Horowitz, 38000 Grenoble (France)
OpenStreetMap GoogleMaps Access in 15 mn from the station with bus 34 (direction Sassenage) bus stop "Polygone Scientifique"
All participants need to be declared to be allowed to enter the site: Please register here
Accommodation
Some rooms are available at the local guest-house. The price is 29/35€/night (for a twin/single room including breakfast).
On-site restaurant offers meals for <10€
- Wifi (guest lan) is everywhere on site but no ethernet connection possible.
Participants
Coordination: Jérôme Kieffer: +33476882445 or jerome.kieffer@esrf.fr
Coordination (Saturday): Andrea Prodi: +33648521922 or prodi@esrf.fr
No |
Name |
Arrival at |
Departure at |
Traveling details |
|
1 |
Frédéric-Emmanuel Picca |
<picca> |
Sat 23.6.2012 |
Wed 27.6.2012 |
Train from Paris |
2 |
Andreas Tille |
<tille> |
Sat 23.6.2012 |
Tue 26.6.2012 |
Night train Hanover - Geneva -- feel free to join |
3 |
Filippo Rusconi |
|
|
|
|
4 |
Michael Hanke |
<mih> |
Sat 23.6.2012 |
Tue 26.6.2012 |
Flying VRN->LYS->LEJ |
5 |
Dominique Dumont |
|
|
|
|
6 |
Sylvestre Ledru |
|
Sun 24.6.2012 |
Mon 25.6.2012 |
|
7 |
Philippe Le Brouster |
Sun 24.6.2012 |
Tue 26.6.2012 |
|
|
8 |
Sébastien Villemot |
Sun 24.6.2012 |
Mon 25.6.2012 |
|
|
9 |
Julien Cristau |
Sun 24.6.2012 |
Tue 26.6.2012 |
|
Agenda
Topics to be addressed during the Sprint:
- Batch scheduler for HPC cluster:
- Package libraries of HPC:
- OpenCL (AMD/Intel/nVIdia, cohabitation of all of them ?)
- MPI
- OpenMPI 1.6
- Bug fixing in programming languages used by scientists:
- Octave
- Scilab
- Python: Numpy, Scipy, pyOpenCL, pyCuda, pyfft
- GDL ...
- General infrastructure
- More specific Blends
- Design of tasks/metapackages
- Citations
Reports
General report from the 3 days
More than 40 people attended to the workshop, it was great to realize how large the community was with people coming from all over Europe. We noticed we are facing the same problems in most of our facilities where HPC clusters are central point of the scientific activity. The input from P. Demichel from Hewlett-Packard gave a nice view on what the future of HPC is made of. The small workgroup around "OpenCL" had extremly valuable input from V. Danjean concerning the licencing issues and possible (forseeable) solutions. Systematic backports are a common need of most institutes as we all love the "rock-solid" debian stable and packager of scientific/HPC software should keep this in mind (i.e. not only unstable). M. Hanke exposed the solution he is working on for Neuro-debian (where they have similar issues). Last point concerning documention for debian packages and the deployment of HPC software on debian: the debian wiki is the best place to plut the documentation on (better indexed by google).
Some bugs were fixed during the Sprint itself on: OAR, OpenMPI, ?CodeAster, GMSH, Atlas ... and other.
Report from Andreas Tille
- Working on the bugs page of web sentinel
Discussing how NeuroDebian could make more use of Blends tools
- Teamanalysis for Neurodebian
- Introducing people into tasks files for Debian Science
MoM work
- Close some Debian Med bugs
- Sponsoring some Debian Science (astronomy) packages
- Creating new Blend files for "PAN Blend = Photon And Neutron Blend" together with Frédéric-Emmanuel Picca
- Upload Bugfix release of GNUmed to make sure it will enter Wheezy right in time
Packaging fdmnes
Thought on OpenCL by Vincent Danjean
Summary of the situation of OpenCL in Debian (06/2012)
===== OpenCL goals ======
- From wikipedia:
- Open Computing Language (OpenCL) is a framework for writing programs that execute across heterogeneous platforms consisting of central processing unit (CPUs), graphics processing unit (GPUs), and other processors.
- OpenCL includes a language (based on C99) for writing kernels (functions
- OpenCL is an open standard maintained by the non-profit technology
- Open Computing Language (OpenCL) is a framework for writing programs that execute across heterogeneous platforms consisting of central processing unit (CPUs), graphics processing unit (GPUs), and other processors.
What is important for Debian is that:
- OpenCL is a standard maintained by the Khronos Group
- OpenCL is a language (for target platforms) *and* an API.
- there will be several implementations (we can hope that, at least one for processors will be free)
- API of OpenCL is design to support several implementations at the same time
- API of OpenCL offert the possibility to choose the used platform (and OpenCL vendor) at runtime (instead of compilation/link-time as this is the case for MPI)
- API of OpenCL offers the possibility to use vendor extensions, looking for address function at runtime (here again by opposition to compile/runtime)
Mechanisms used to archieve all these goals
An application using OpenCL can compile with, and link directly at, a specific implementation (as in the MPI world). However, doing this will forbid most of the runtime choices presented in the previous paragraph.
The classical situation for OpenCL is that:
- each vendors (ie OpenCL implementor) must provide:
- an OpenCL ICD (Installable Client Driver) as a shared library with some convention/features/symbols (see later)
- the path of this library in a vendor-specific text file in /etc/OpenCL/vendors/ directory
- the application is compiled with
- some OpenCL headers. This can be:
- the official Kronos OpenCL headers (all official features and extensions,
- no private vendor extension)
- some vendors headers, generally based on the previous one but with
- some extensions (functions, constants, ...) added
- the official Kronos OpenCL headers (all official features and extensions,
- OpenCL ICD Loader which offer the OpenCL API to the applications. The
- ICD Loader is mainly a call dispatcher. Kronos group provides (nonfree) sources of one to its members.
- most vendors (amd, intel, nvidia at least) provide a ICD Loader
- ocl-icd provides a free one after some reverse engeenering of the
- previously cited ones.
- some OpenCL headers. This can be:
ICD Loaders and ICD
In therory (see below), any ICD Loader should be able to load and use any ICD. An ICD must implement some mandatory functions. One of these functions allows the ICD Loader to get the address of a structure which contains the address of all ICD implementation of OpenCL functions. Writing such an ICD Loader is easy once you know the order of the functions in the structure (list and order that the Khronos group restrict to its members and that the ocl-icd projet reverse engeenered)
- The Khronos group ensures that any functions added to this structure will
never be removed (so that their won't be incompatibility between to ICD Loaders).
In pratice, we can really use the ICD Loader from Intel to use the NVidia ICD. But, there is no support for the ICD to declare which version of the structure it provides. An ICD can offers (through the structure) more functions. It happens when the ICD provides an OpenCL version greater than the one provided by the ICD Loader. This is not really a problem but the an OpenCL application using the more recent OpenCL version will not be able to link to the ICD Loader. It will then fail at link time.
- But there is also the opposite: the ICD Loader supports a greater version of
OpenCL than the ICD. In this case, an OpenCL application using features from the new version will call the ICD Loader that will call the "function" at the address present in the memory after the end of the (old) structure provider by the ICD. This generally leads to a segfault.
- The ICD Loader cannot do anything about that: it does not have the
information. Even if ICDs declare a supported OpenCL version, it is the OpenCL version for which the support is full. At the time of writing, the Intel ICD declare to support OpenCL 1.1 but provides to the ICD Loader the structure of OpenCL 1.2 *with* some address of functions specific to OpenCL 1.2 (but not all of them, hence the no declaration of OpenCL 1.2 support).
- This situation is also a problem for application (PyCL ?) that detect at
compilation time which version of OpenCL is supported. They will get the OpenCL version of the ICD Loader, but not the one of the ICD itself that will be loaded at runtime.
- An ICD not implementing one function of the structure puts a NULL pointer.
But non-free ICD Loaders (cnot ocl-icd for now) do not do any checks for non-NULL value before dispatching the call. So it is again easy to get segfault with some combination of program/ICD.
- About compatibility, each ICD must also implement the support for the OpenCL
*language* (for the code dedicated to the target platforms). Of course, as the norm is not complete, or not fully implemented or implemented with extension, the portability of OpenCL code is not as good as one can hope. But there is little to do here with respect to Debian packaging.
===== Current packaging situation ======
Brief overview of sources packages/OpenCL implementations:
- AMD and NVidia OpenCL implementation are packaged into the non-free section.
- Intel is not packaged. It is not redistribuable, so its packaging would
- involve a 'installer' package. Intel ICD Loader support OpenCL 1.2 Intel ICD support OpenCL 1.1 and some functions of OpenCL 1.2
- Khronos OpenCL headers 1.2 are packaged in main
- Ocl-icd project is packaged in main
- Pocl, a free implementation of OpenCL, will be packaged in main. Its state
- does not seem yet good enough to be able to run non simple OpenCL applications. This should change in the future.
Overview of binary packages currently in sid (and probably testing) on amd64: ====== AMD =====
- amd-libopencl1: AMD ICD Loader (support OpenCL 1.2)
- install libOpenCL.so.1
- shared library
- soname libOpenCL.so.1 symbols versionned (with OpenCL version, but no conflicts on symbols)
- symlink to libOpenCL.so.1
- shared library
- Multi-Arch: same Provides: libopencl1 Conflicts: amd-app, nvidia-libopencl1
=> should Conflicts and Provides libopencl1. See #679038 (fixed in unstable)
- install libOpenCL.so.1
- amd-opencl-icd: AMD ICD (support OpenCL 1.2)
- Multi-Arch: same Depends: libopencl1 Enhances: libopencl1 Conflicts: amd-app
- amd-opencl-dev: AMD OpenCL development
- Metapackage
- Multi-Arch: same
Depends: amd-libopencl1 | libopencl1, opencl-headers (>= 1.2)
- amd-clinfo:
- No Multi-Arch Depends: amd-libopencl1
=> should depends on the virtual package libopencl1. See #679025 (fixed in unstable)
- No Multi-Arch Depends: amd-libopencl1
[NVidia]
- nvidia-libopencl1: NVidia ICD Loader (support OpenCL 1.1)
- install libOpenCL.so.1.0.0
- shared library
- soname libOpenCL.so.1 symbols not versionned
- symlink to libOpenCL.so.1.0.0
- symlink to libOpenCL.so.1
- shared library
- Multi-Arch: same Provides: libopencl1 Recommends: opencl-icd | nvidia-opencl-icd Conflicts: nvidia-libopencl1-dev
=> should Conflicts and Provides libopencl1. See #679047 (fixed in unstable)
- install libOpenCL.so.1.0.0
- nvidia-libopencl1-ia32: 32-bit NVidia ICD Loader (support OpenCL 1.1)
- install lib32/libOpenCL.so.1.0.0
- shared library
- soname libOpenCL.so.1 symbols not versionned
- symlink to libOpenCL.so.1.0.0
- symlink to libOpenCL.so.1
- shared library
- No Multi-Arch Provides: lib32opencl1 Recommends: opencl-icd | nvidia-opencl-icd-ia32
=> the Recommends to opencl-icd seems wrong to me => this package should disappear (replaced by nvidia-libopencl1:i386)
- install lib32/libOpenCL.so.1.0.0
- nvidia-opencl-common: common files for NVidia ICD (support OpenCL 1.1)
- Multi-Arch: foreign Recommends: nvidia-opencl-icd | nvidia-opencl-icd-ia32
- nvidia-opencl-icd: NVidia ICD (support OpenCL 1.1)
- Multi-Arch: same Provides: opencl-icd Depends: nvidia-opencl-common, libopencl1 Enhances: libopencl1
- nvidia-opencl-icd-ia32: 32-bit NVidia ICD (support OpenCL 1.1)
- No Multi-Arch Provides: opencl-icd Depends: nvidia-opencl-common, lib32opencl1 Enhances: lib32opencl1
=> the Provides opencl-icd seems wrong to me => this package should disappear (replaced by nvidia-opencl-icd:i386)
- No Multi-Arch Provides: opencl-icd Depends: nvidia-opencl-common, lib32opencl1 Enhances: lib32opencl1
- nvidia-opencl-dev: NVidia OpenCL development
- Metapackage
- No Multi-Arch Provides: opencl-dev
Depends: nvidia-libopencl1 | libopencl1, opencl-headers (>= 1.1)
===== [ocl-icd] (version 1.3-2 currently in NEW or in my own repo) ======
- ocl-icd-libopencl1: ocl-icd ICD Loader (support OpenCL 1.2)
- install libOpenCL.so.1.0.0
- shared library
- soname libOpenCL.so.1 symbols versionned (with OpenCL version, but no conflicts on symbols)
- symlink to libOpenCL.so.1.0.0
- symlink to libOpenCL.so.1.0.0
- shared library
- Multi-Arch: same Provides: libopencl1 Conflicts: libopencl1, amd-app, nvidia-libopencl1-dev Replaces: libopencl1, amd-app, nvidia-libopencl1-dev
- Note: amd-app, nvidia-libopencl1-dev seems to be old/external packages
- providing some of these files
- Note: would be promoted to a recommends once a free OpenCL implementation
- will be available
- Note: amd-app, nvidia-libopencl1-dev seems to be old/external packages
- install libOpenCL.so.1.0.0
- ocl-icd-opencl-dev: OpenCL development
- install OpenCL.pc
- pkg-config file for OpenCL
- Multi-Arch: same Provides: opencl-dev
Depends: opencl-headers (>= 1.2), libopencl1 Conflicts: amd-app, opencl-dev Replaces: amd-app, opencl-dev
- install OpenCL.pc
- ocl-icd-dev: devel package to develop an ICD
- install ocl_icd.h
- headers with the structure with all functions that an ICD must provide
- No Multi-Arch
- install ocl_icd.h
Intel OpenCL SDK
- ICD Loader (support OpenCL 1.2): can be installed in Multi-Arch paths
- install libOpenCL.so
- shared library
- soname libOpenCL.so symbols not versionned
=> problem with wrong soname => can still be used by programs that require libOpenCL.so.1 by creating
- a symlink (the dynamic linker do not check the soname)
- shared library
- install libOpenCL.so
- ICD (support OpenCL 1.1 + some 1.2 functions)
- OpenCL headers
- based on the Khronos ones. Intel's one seem a little bit outdated but include some constants/... that seems to be Intel specific...
======Virtual packages======
- opencl-icd: existing virtual package for OpenCL ICD
- ensure an OpenCL ICD is installed a package providing opencl-icd should (must?):
- Provides: opencl-icd
- Enhances: libopencl1
- opencl-dev: existing virtual package dor OpenCL development
- ensure headers and libraries are installed a package providing opencl-dev should (must?):
- Provides: opencl-dev
- install OpenCL headers
- can be done with a (versionned) dependency on the opencl-headers package
- Depends: libopencl1
- Recommends: libgl1-mesa-dev | libgl-dev
- GL headers are often required when compiling OpenCL program
=> not done in amd-opencl-dev
- GL headers are often required when compiling OpenCL program
- libopencl1: existing virtual package for OpenCL ICD Loaders
- ensure an OpenCL ICD Loader is installed a package providing libopencl1 should (must?):
- Conflicts/Replaces/Provides: libopencl1
- Recommends: opencl-icd
- install libOpenCL.so.1 in library search path
- install libOpenCL.so in library search path
- NOTE: this last one must not be in a -dev package if we want to support binaries compiled with the Intel SDK
- libOpenCL.so.1 must have libOpenCL.so.1 as soname
- [proposed, not right for the intel one]
- libOpenCL.so.1 must have its symbols versionned
- [proposed, not right for the intel and nvidia ones]
- libopencl1.1: proposed virtual package
- a package providing libopencl1.1 must:
- provides libopencl1
- implements and dispatch all functions from OpenCL 1.1 API
- libopencl1.2: proposed virtual package
- a package providing libopencl1.2 must:
- provides libopencl1
- implements and dispatch all functions from OpenCL 1.2 API
NOTE: libopencl1.0 is not proposed as a virtual package:
- the 1.0 API is not really designed to work with ICD Loaders
- most (all?) OpenCL implementations target at least OpenCL 1.1
A few remarks:
- due to the soname of Intel (libOpenCL.so instead of libOpenCL.so.1), it is
- better to install the libOpenCL.so link in the libopencl1 packages instead
of installing it in the dev packages -> Intel will fix that
- better to install the libOpenCL.so link in the libopencl1 packages instead
- is it really necessary to keep the three -dev packages?
- ocl-icd-opencl-dev is free and provides a OpenCL.pc file
- amd-opencl-dev is a metapackage that install by default the amd-libopencl1
- that itself install by default the amd-opencl-icd
- same for nvidia-opencl-dev
- if we keep the three -dev packages (to ease the installation of the
- full AMD and NVidia SDK), do we want to rename ocl-icd-opencl-dev into opencl-dev (ie use the virtual package name as the binary package name of the free alternative)
- do we want to keep the various *-libopencl1 (ie ICD loaders) packages in
- Debian? ocl-icd-libopencl1 provides a free alternative {amd,nvidia}-libopencl1 will be useful:
- when ocl-icd-libopencl1 is not yet updated to the latest version
- where as {amd,nvidia}-libopencl1 are
- if {amd,nvidia}-libopencl1 start to implement non standard things
- to ease the reverse engeenering required to improve ocl-icd-libopencl1
- when non-free ICD loader implements more functions
- at least, here is a summary of problems/warning for different combinaison of
- ICD *Loaders* used at compile-time (CT) and at run-time (RT) (problems with ICD compatibility themselves are not taken into account here)
CT\RT |
ocl-icd |
AMD |
NVidia |
Intel |
ocl-icd |
ok |
ok |
MVS/MS |
MVS/MSO |
AMD | ok | ok | MVS |
MVS |
|||
NVidia |
ok |
ok |
ok |
MSO |
Intel |
BSO |
BSO |
BSO/MS |
ok |
MVS: missing version on symbol => a warning from the linker at program start MS: missing symbols. The linker can fail to find some symbols (NVidia ICD
- Loader still only support the 1.1 API)
BSO: the libOpenCL.so link must be present at runtime.
- The requested soname is libOpenCL.so The soname in the loaded library (filename) is libOpenCL.so.1
MSO: wrong soname:
- The requested soname is libOpenCL.so.1 The soname in the loaded library (filename) is libOpenCL.so
This is the status around late June 2012 and things are likely to evolve
After a couple of exchanges with Intel about their OpenCL SDK; it turns out the software is re-distributable hence suitable to be packaged in "non-free".
Acknowledgments
the sprint has been possible thanks to: