Lustre

This pages should help you to get a running lustre cluster based on debian systems.

You can either build your own Lustre Debian packages from GIT or download these packages from

http://pkg-lustre.alioth.debian.org/backports/lustre-2.2.0-squeeze/

and

http://pkg-lustre.alioth.debian.org/backports/ldiskfsprogs-1.42.3/

If you download all packages incl. the kernel package you can continue with chapter Installation of Lustre 2.2 in Debian Squeeze.

If you download only the Lustre packages without the kernel and kernel module package, e.g. because you want to compile your own custom kernel, please continue with chapter Building Lustre kernel.

Building Lustre 2.2 for Debian Squeeze

Building Lustre packages from GIT

Change to a any Lustre building directory and clone the actual GIT tree of Debian Lustre project:

git clone http://anonscm.debian.org/git/pkg-lustre/lustre.git

Then you need the original source tarball of Lustre.

wget http://pkg-lustre.alioth.debian.org/backports/lustre-2.2.0-squeeze/lustre_2.2.0.orig.tar.gz

Finally you can build your packages like this:

cd lustre
dpkg-buildpackage

(If you don't want to sign your packages, you can add the parameters -us -uc to dpkg-buildpackage)

Now you'll find your packages in your Lustre building directory.

Building Lustre 2.2 kernel for Debian Squeeze

Install the linux-patch-lustre_2.2.0-5_all.deb package:

sudo dpkg -i linux-patch-lustre_2.2.0-5_all.deb

The Lustre patches for Linux kernel 2.6.32 will be placed in /usr/src/kernel-patches/lustre.

To go on please now install the kernel source package from Debian Squeeze:

apt-get source linux-2.6
cd linux-2.6-2.6.32
/usr/src/kernel-patches/lustre/scripts/apply

All the Lustre patches are being applied to the kernel sources. Now you can create your kernel. First prepare your sources:

fakeroot make-kpkg --config=menuconfig --revision deblust.1.0 --append-to-version "-lustre-2.2" configure

Please choose a meaningful revision name. Save your configuration as .config file and then initiate the building process:

fakeroot make-kpkg --initrd --revision deblust.1.0 --append-to-version "-lustre-2.2" kernel_image kernel_headers kernel_source

Depending on your system this can take a long time.

With the help of the script /usr/src/kernel-patches/lustre/scripts/unpatch you can remove the Lustre kernel patches from the sources.

Building Lustre 2.2 modules

Install the Lustre kernel image and Lustre source package:

sudo dpkg -i linux-image-2.6.32-lustre-2.2_deblust.1.0_amd64.deb
sudo dpkg -i linux-headers-2.6.32-lustre-2.2_deblust.1.0_amd64.deb
sudo dpkg -i linux-source-2.6.32-lustre-2.2_deblust.1.0_all.deb
sudo dpkg -i lustre-source_2.2.0-5_all.deb

Please reboot into the new Lustre kernel. Use the module-assistant to start the module building process:

sudo m-a auto-install lustre

Then the module kernel package should be there and the modules installed.

Installation of Lustre 2.2 in Debian Squeeze

For a minimal test cluster at least 4 machines are recommended:

The MGS stores configuration information for all the Lustre file systems in a cluster and provides this information to other Lustre components, whereas the MDT stores namespace metadata (filenames, directories, access permissions, file layout). The MGS/MDT is also an entry point. The OST stores the data.

Preparing the systems

Install on all machines:

sudo dpkg -i linux-image-2.6.32-lustre-2.2_deblust.1.0_amd64.deb
sudo dpkg -i lustre-modules-2.6.32-lustre-2.2_2.2.0-5_amd64.deb
sudo dpkg -i lustre-utils_2.2.0-3_amd64.deb
sudo dpkg -i ldiskfsprogs_1.42.3-1_amd64.deb

Now you are ready to boot the kernel an all machines. Then load the kernel modules with:

modprobe lnet
modprobe lustre
modprobe ldiskfs

If you're running ypbind or something else on your system which uses port 988 you should make sure that the ptctl module is loaded very early in the boot process (as lustre needs port 988 for communications), See the ?KnowledgeBase for details.

MGS/MDT

Create a new and free partition (as large as possible), e.g. /dev/sda4 and format it as MGS/MDT:

mkfs.lustre --fsname=temp --mgs --mdt /dev/sda4

temp is the name of our filesystem. This can be changed of course. The formatting process could take a long time.

Mount the new partition:

mount -t lustre /dev/sda4 /srv/mdt

You can choose another mountpoint of course.

OST

Create a new and free partition (as large as possible), e.g. /dev/sda4 and format it as MGS/MDT:

mkfs.lustre --ost --fsname=temp --mgsnode=192.168.80.100@tcp0 /dev/sda4

temp is the name of our filesystem. Please note that we have to set the IP adress or Hostname of the Lustre MGS node (here: 192.168.80.100).

Mount the new partition:

mount -t lustre /dev/sda4 /srv/ost1

You can choose another mountpoint of course.

Repeat for OST2.

Client

Go to the client machine and mount your new Lustre file system:

mount -t lustre 192.168.80.100@tcp0:/temp /mnt/lustre

Verify that the file system started and is working:

lfs df -h

Go to /mnt/lustre (or whatever your mount point is) and run a first basic test:

dd if=/dev/zero of=/lustre/zero.dat bs=4M count=20

Please check also your kernel logs for any error messages.

Using Quota

The binaries and drivers for lustre in debian are compiled with quota enabled. So if you like to use quota this is quite simple:

lfs quotacheck -ug $path/to/mounted/lustre/fs
lfs setquota -u $user 1000 2000 10000 20000 $path/to/mounted/lustre/fs
lfs quota -u $user $path/to/mounted/lustre/fs