Lustre
This pages should help you to get a running lustre cluster based on debian systems.
Contents
Installation of lustre servers on Debian
Building the kernel and the modules
- You've to download the linux-source-2.6.18|22 package, the linux-patch-lustre package and lustre-source. Unfortunately lustre is not supported to run on newer kernels as server, so you've to grab the kernel out of etch. In order to build the patched kernel automatically use this make-kpkg call (or something similiar). This will give you a linux-image, a linux-headers and a lustre-modules package afterwards.. so everything you need on your server:
make-kpkg --added-patches=lustre --initrd --added-modules=lustre binary-arch modules
- Install the kernel and the module package and reboot.
Preparing the system
After the system is running on the new kernel you've to install lustre-utils on every server as you need mkfs.lustre, tunefs.lustre & mount.lustre on every server:
apt-get install lustre-utils
If you're running ypbind or something else on your system which uses port 988 you should make sure that the ptctl module is loaded very early in the boot process (as lustre needs port 988 for communications), See the ?KnowledgeBase for details.
Creating partitions on the OST/MDT/MGS Servers
- Now you'll have to differentiate between the OST,MDT and MGS partitions We'll assume here that MDT and MGS are on the same partition:
- Create partitions:
MGS/MDT: mkfs.lustre --fsname testfs --mdt --mgs $device
OST: mkfs.lustre --fsname testfs --ost --mgsnode=$mgsnode@tcp0 $device
- There are plenty of other additional options for mkfs.lustre, for an overview please have a look into the manpage.
Preparing lustre clients
Building lustre client module
- According to the lustre servers lustre clients need a kernel module, however you can use here the default kernel which comes with debian, as the client kernel didn't need patches. lustre clients can be build on 2.6.18, 2.6.22, 2.6.24 and 2.6.26er kernels, so you can feel free to use the standard lenny (or etch) kernel for the client. To build the module you've to install lustre-source from the archive (or from backports.org):
testclient:~# m-a -t a-i lustre
Using the filesystem
- In order to use the Filesystem, you'll have to mount it on a client. Please note that this client also needs the lustre-modules to be installed. (You can install lustre-source and build the module as described in 1.)
mount -t lustre mgsnode@tcp0:/testfs /mnt
Using Quota
- The binaries and drivers for lustre in debian are compiled with enabled-quota. So if you like to use quota this is quite simple:
- - lfs quotacheck -ug $path/to/mounted/lustre/fs - lfs setquota -u $user 1000 2000 10000 20000 $path/to/mounted/lustre/fs - lfs quota -u $user $path/to/mounted/lustre/fs
- - lfs setquota -u $user 0 0 0 0 $path/to/mounted/lustre/fs - lfs setquota -u $user 2000 3000 3000 4000 $path/to/mounted/lustre/fs
This behaviour is reported as lustre bug: https://bugzilla.lustre.org/show_bug.cgi?id=14513