Translation(s): none
Linux "Huge page tables" (HugeTlb) is available in Debian since DebianLenny (actually, since 2.6.23). A good introduction to large pages is available from ibm.com.
Contents
Currently, there is no standard way to enable HugeTLBfs, mainly because the FHS has no provision for such kind of virtual file system, see 572733 . (Fedora mounts it in /dev/hugepages/, so don't be surprised if you find some example on the web that use this location)
Read the documentation for more information about hugetlbpage.
Enabling HugeTlbPage
Create a group for users of hugepages, and retrieve it's GID (is this example, 2021) then add yourself to the group.
Note: this should not be needed for libvirt (see /etc/libvirt/qemu.conf)% groupadd my-hugetlbfs % getent group my-hugetlbfs my-hugetlbfs:x:2021: % adduser franklin my-hugetlbfs Adding user `franklin' to group `my-hugetlbfs' ... Adding user franklin to group my-hugetlbfs Done.
Edit /etc/sysctl.conf and add this text to specify the number of pages you want to reserve (see pages-size)
# Allocate 256*2MiB for HugePageTables (YMMV) vm.nr_hugepages = 256 # Members of group my-hugetlbfs(2021) can allocate "huge" Shared memory segment vm.hugetlb_shm_group = 2021
Create a mount point for the file system
% mkdir /hugepages
Add this line in /etc/fstab (The mode of 1770 allows anyone in the group to create files but not unlink or rename each other's files.1)
hugetlbfs /hugepages hugetlbfs mode=1770,gid=2021 0 0
Reboot (This is the most reliable method of allocating huge pages before the memory gets fragmented. You don't necessarily have to reboot. You can try to run systclt -p to apply the changes. if grep "Huge" /proc/meminfo don't show all the pages, you can try to free the cache with sync ; echo 3 > /proc/sys/vm/drop_caches (where "3" stands for "purge pagecache, dentries and inodes") then try sysctl -p again. 2)
limits.conf
You should configure the amount of memory a user can lock, so an application can't crash your operating system by locking all the memory. Note that any page can be locked in RAM, not just huge pages. You should allow the process to lock a little bit more memory that just the the space for hugepages.
## Get huge-page size: % grep "Hugepagesize:" /proc/meminfo Hugepagesize: 4096 kB ## What's the current limit % ulimit -H -l 64 ## Just add them up... (how many pages do you want to allocate?)
See Limits (ulimit -l and memlock in /etc/security/limits.conf).
Multiple huge page size support
Some architectures (like ia64) can have multiple and/or configuration "huge" pages size.
(TODO)
See:
boot parameters and mount options in hugetlbpage.txt in documentations.
Getting informations
Well, typically you can retrieve the current available/used page from /proc/meminfo:
% grep Huge /proc/meminfo HugePages_Total: 256 HugePages_Free: 256 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 4096 kB
(read Documentation/vm/hugetlbpage.txt for more information about it)
Standard Debian Kernel have HUGETLB enabled (What about Lenny ? Xen ?):
% grep HUGETLB /boot/config-$(uname -r) CONFIG_HUGETLBFS=y CONFIG_HUGETLB_PAGE=y
Various runtime settings (see documentation)
% grep -R "" /sys/kernel/mm/hugepages/ /proc/sys/vm/*huge* /sys/kernel/mm/hugepages/hugepages-4096kB/nr_hugepages:256 /sys/kernel/mm/hugepages/hugepages-4096kB/nr_overcommit_hugepages:0 /sys/kernel/mm/hugepages/hugepages-4096kB/free_hugepages:256 /sys/kernel/mm/hugepages/hugepages-4096kB/resv_hugepages:0 /sys/kernel/mm/hugepages/hugepages-4096kB/surplus_hugepages:0 /proc/sys/vm/hugepages_treat_as_movable:0 /proc/sys/vm/hugetlb_shm_group:0 /proc/sys/vm/nr_hugepages:256 /proc/sys/vm/nr_overcommit_hugepages:0
Huge pages sizes
Tools
libhugetlbfs is a library which provides easy access to huge pages of memory. The library also comes with several userspace tools to help with huge page usability, environment setup, and control.
http://libhugetlbfs.ozlabs.org/
533708 - ITP: libhugetlbfs -- Initial package request
http://www.ibm.com/developerworks/wikis/display/LinuxP/libhuge+short+and+simpleHugeTLB for PostgreSQL (and other apps) non-official
http://oss.linbit.com/hugetlb/
Hugepage enabled applications
An application can allocate/use HugeTlbPage through two different means:
mmap system call require a mounted hugetlbfs, with appropriate permissions.
Shared memory segment (shmat/shmget system calls or mmap with MAP_HUGETLB), must be member of a group, configured in /proc/sys/vm/hugetlb_shm_group.
MySQL
(TODO), see:
Enabling Large Page Support
http://dev.mysql.com/doc/refman/5.1/en/large-page-support.html
Linux HugeTLBfs: Improve MySQL Database Application Performance
http://www.cyberciti.biz/tips/linux-hugetlbfs-and-mysql-performance.html (note tested)
Java (Sun, OpenJDK)
Sun and OpenJDK can use large pages.
sun-java6-bin - Sun Java(TM) Runtime Environment (JRE) 6 (architecture dependent files)
openjdk-6-jre-headless - OpenJDK Java runtime, using Hotspot JIT (headless)
(TODO)
Basically, it seems that one have to launch java with -XX:+UseLargePages. The maximum size of the Java heap (-Xmx) should fit in your reserved Huge pages ; same for ulimit -l and/or memlock in /etc/security/limits.conf.
See:
Java Support for Large Memory Pages - Sun
http://java.sun.com/javase/technologies/hotspot/largememory.jspConfiguring large page memory allocation - IBM User Guides for Java v6 on Linux
http://publib.boulder.ibm.com/infocenter/javasdk/v6r0/index.jsp?topic=/com.ibm.java.doc.user.lnx.60/user/alloc_large_page.html
Possible errors:
Insufficient /proc/sys/kernel/shmmax ?
Not in group defined in /proc/sys/vm/hugetlb_shm_group
% java -XX:+UseLargePages Java HotSpot(TM) Server VM warning: Failed to reserve shared memory (errno = 1). Java HotSpot(TM) Server VM warning: Failed to reserve shared memory (errno = 22).
Memcached
memcached can use huge pages, read the manpage :
- memcached -L
Try to use large memory pages (if available). Increasing the memory page size could reduce the number of TLB misses and improve the performance. In order to get large pages from the OS, memcached will allocate the total item-cache in one large chunk. Only available if supported on your OS.
PosgreSQL
Humm doesn't seems so... but see that hack: hugetlblib above.
Virtualisation
Some consideration about hugepages an virtualisation.
- Before enabling hugepages in a virtual machine, you should make sure the that your virtualization tool can handle it.
- Whether a virtualization tools supports hugepages for it's client or for itself are probably two different aspects.
KVM
(TODO), see:
http://fedoraproject.org/wiki/Features/KVM_Huge_Page_Backed_Memory
Get a performance boost by backing your KVM guest with hugetlbfs
http://www.linux-kvm.com/content/get-performance-boost-backing-your-kvm-guest-hugetlbfs (not tested)
Xen
(TODO)
It is unclear which version of Xen supports huge pages, and how it is used.
Documentation
Documentation http://git.kernel.org/....../Documentation/vm/hugetlbpage.txt
alloc_hugepages(2), free_hugepages(2) - allocate or free huge pages
See also
Article about Huge pages, on LWN.net
Introduction - Interfaces - Administration - Benchmarking
Credits: Russell Coker (1)
Credits for drop_caches: http://oss.linbit.com/hugetlb/ (2)