This text is still in initial state. No deadline yet. To edit this page, follow through the [wiki:self:Teams/Publicity/DebianTimesTeam/Guidelines Debian Times team guidelines] for publishing workflow.
Debian GNU/Linux clusters at Max Planck draft title
The [http://gw.aei.mpg.de/ Gravitational Wave Astrophysics Research Group] is a team of scientists working at the [http://www.aei.mpg.de/hannover-en/66-contemporaryIssues/home/index.html Max Planck Institute for Gravitational Physics] ([http://www.aei.mpg.de/english/contemporaryIssues/home/index.html Albert Einstein Institute]) in [http://en.wikipedia.org/wiki/Hannover Hannover], [http://en.wikipedia.org/wiki/Germany Germany], and trying to detect [http://en.wikipedia.org/wiki/Gravitational_wave gravitational waves] directly, first [http://www.einstein-online.info/en/ predicted] by Albert Einstein. They are working with the friends and coleagues within the [http://www.ligo.org/ LIGO] Scientific Community.
The massive computing effort is done at the ATLAS Debian GNU / Linux cluster, achieving measured performance in terms of [http://www.top500.org/ top500.org] linpack of 32.8 TFlops, with a theoretical peak of 50 TFlops, using 10+ TB RAM, approximately 1.3 PB storage and a specialty network able to transfer almost 4 days of DVD movies per second (2880 Gb/s).
which consists of 1342 computer nodes (Intel Xeon 3220 quad-cores 2,4 GHz, 8 GB RAM, 500 GB Hitachi HDD, IPMI remote management) along with 31 data servers (2x Intel Xeon E5345 2,33 GHz, 16 GB RAM, Areca 1261ML, 16x750 GB Hitachi HDD) plus 4 similar head nodes with 4 x 750 GB HDD. Those are all running Debian GNU / Linux 4.0 Etch with a few modifications like custom kernel and Condor queuing system. Additional storage are supplied by 13 Sun Fire X4500 running Solaris 10. The system was built from off-the-shelf computers from a german mid-sized company, [http://www.pyramid.de/ Pyramid Computer GmbH].
Total disk space (gross): ~ 1.3 PB
Total RAM: bit more than 10 TB
Total possible network throughput: 2 x 144 x 10 Gb/s ~ 2880 Gb/s (assuming DVD quality video has 8 Mb/s) this means that we can transfer about 4 days worth of movies per second.
One of the many specialties they have is the network from [http://www.wovensystems.com/ Woven Systems] which is a hierarchical fully non-blocking network. The EFX 1000 core switch features 144 10 Gb/s CX4 ports and connects currently to 32 TRX100 edge switches which feature 48 1 Gb/s ports and 4x10 Gb/s uplinks. Also their Sun Fire X4500 are directly connected to the core switch.
Some benchmark numbers: Theoretical peak is beyong 50 TFlop/s, measured speed in terms of [http://www.top500.org/ top500.org] linpack is 32.8 TFlop/s. This would place us in spot #34 worldwide, #11 in Europe and #4 in Germany on the list from November 2007.
Another special point: The system was relatively cheap, i.e. less than EUR 1.8m (~ $2.8m).
> > 3- Why Debian was chosen? > The people designing, building and running the cluster (namely Dr Henning Fehrmann and Dr Carsten Aulbert) have been using Debian for years. Our brother and sister systems in [http://en.wikipedia.org/wiki/Potsdam Potsdam], [http://gw.aei.mpg.de/resources/computational-resources/merlin-morgane-dual-compute-cluster "Merlin" and "Morgane"] are running Debian for years (one converted from RH 7.x at some point) and the experience had been very, very good.
Debian features an extremely large set of packages, making it *the* distro of choice for us keeping us out of the hassle to package needed software ourselves.
Also Thomas Lange's FAI package is extremely useful for automatic deployment of Debian, for example without much tweaking using only two hosts, we were able to reinstall the cluster in about 2.5 hours and were only limited by those two servers' network connection.
(Note: > 2 weeks ago I would have written something about the very good security support, given that the reaction to the OpenSSL stuff was very good, I could still do, but in reality we don't need security updates except for the head nodes, everything else is exposed only internally.)
> 4- What are the benefits of using Debian?
Partly covered above, maybe, one should add
* the simplicity of creating own packages
* how repositories can be set-up easily (we use reprepro)
* using clean build environments (pbuilder et al.)
* and, of course, the superb packaging infrastructure in general (aka dpkg/apt/aptitude)
> > 5- How Debian enabled such success / feature / business model, > etc?
Hard point to say anything. our colleagues are mainly using CentOS 5 or older Fedora versions, thus the cluster would also run with those, however, possibly with more work for us.
Personally, I like community distros more since they offer more long-term stability than a distro which is governed by the need of releasing often to generate revenue. Although on the downside it would be better for us to have a more settled release plan and/or some kind of "stable and supported" backports.
[0a] http://en.wikipedia.org/wiki/Hannover [0b] http://en.wikipedia.org/wiki/Potsdam  http://www.ligo.org/  http://www.aei.mpg.de/hannover-en/66-contemporaryIssues/home/index.html  http://www.einstein-online.info/en/  http://www.pyramid.de/  http://www.wovensystems.com/  http://www.top500.org/  http://gw.aei.mpg.de/resources/computational-resources/merlin-morgane-dual-compute-cluster  http://www.informatik.uni-koeln.de/fai/
About the Debian Project
Debian GNU / Linux is [http://www.debian.org/ports/#nonlinux one] of the [http://www.debian.org/intro/free free libre] operating systems ([http://www.debian.org/ports/#released GNU/Linux], [http://www.debian.org/ports/hurd GNU/Hurd], [http://www.debian.org/ports/netbsd/ GNU/NetBSD], [http://www.debian.org/ports/kfreebsd-gnu/ GNU/kFreeBSD)], running 18733+ [http://qa.debian.org officially] maintained [http://packages.debian.org packages] on [http://www.debian.org/ports 15 hardware platforms], from [http://www.debian.org/ports/arm/ cell phones] and [http://www.linux-sh.org network devices] to [http://www.debian.org/ports/s390/ mainframes] and [http://wiki.debian.org/DebianBeowulf supercomputers], developed by more than [http://asdfasdf.debian.net/~tar/bugstats/?8 two thousand] volunteers from [http://www.debian.org/devel/developers.loc all over the world] who [http://www.debian.org/devel/ collaborate] via [http://www.us.debian.org/support the internet] on the [http://www.debian.org Debian Project].
Debian's dedication to [http://www.debian.org/intro/free Free Libre Open Source Software], its [http://www.debian.org/devel/constitution constitutional] non-profit nature, its [http://vote.debian.org/ open] and [http://en.wikipedia.org/wiki/Meritocracy meritocratic] development model, [http://www.debian.org/intro/organization organization] and social [http://www.techforce.com.br/index.php/news/linux_blog/scientific_study_about_debian_governance_and_organization governance] make it [http://www.debian.org/doc/manuals/project-history/ a first] among free libre operating system distributions.
The Debian project's key strengths are [http://www.debian.org/devel/people its volunteer base], its dedication to the [http://www.debian.org/social_contract Debian Social Contract] and the [http://www.debian.org/devel/constitution Debian Constitution], and its [http://wiki.debian.org/WhyDebianForDevelopers commitment] to [http://bugs.debian.org/release-critical/ provide the best] operating systems [http://release.debian.org/ attainable], following a strict quality [http://www.debian.org/doc/debian-policy policy], working with an established [http://qa.debian.org/ QA Team] and helpful [http://www.debian.org/users/ users] reporting [http://bugs.debian.org bugs, suggestions], [http://lists.debian.org exchanging ideas], and [http://wiki.debian.org registering experiences].
You can [http://www.debian.org/intro/help help] Debian Project without [http://www.debian.org/devel/join joining] it and [http://wiki.debian.org/DebianForNonCoderContributors even not being a ][http://wiki.debian.org/DebianForNonCoderContributors programmer], or being a development and or service [http://www.debian.org/partners/ partner] company or institution at the [http://www.debian.org/partners/partners Debian Partner Program], or simply making various [http://www.debian.org/donations donations] to the Debian Project.
Debian Project news, press releases and press coverage can be found from the official Debian wiki [http://wiki.debian.org/News page]. PR contact at [http://lists.debian.org/debian-publicity debian-publicity list].