Differences between revisions 2 and 3
Revision 2 as of 2010-03-17 01:45:22
Size: 3265
Editor: GeoffSimmons
Comment: Formatting fix.
Revision 3 as of 2010-03-17 08:02:01
Size: 4834
Editor: FranklinPiat
Comment:
Deletions are marked like this. Additions are marked like this.
Line 7: Line 7:
## <<TableOfContents(2)>> <<TableOfContents(2)>>
Line 11: Line 11:
== Enabling hugetlbfs == Read the [[#documentation|documentation]] for more information about hugetlbpage.
Line 13: Line 13:
 1. Edit {{{/etc/sysctl.conf}}} and add this text to specify the number of page you want to reserve (in x86 architecture, the !HugePages are are 2MiB each) {{{#!text
# Allocate 256*2MiB for HugePageTables (YMMV)
vm.nr_hugepages = 256
}}}
 1. Create a mount point for the file system {{{
% mkdir /hugepages
}}}
 1. Create a group, and retrieve it's GID (is this example, 2021), and add yourself to the group. {{{
== Enabling HugeTlbPage ==

 1. Create a group for users of hugepages, and retrieve it's GID (is this example, 2021) then add yourself to the group.<<BR>>Note: this should not be needed for [[libvirt]] (see {{{/etc/libvirt/qemu.conf}}}){{{
Line 31: Line 26:
 1. Add this line in {{{/etc/fstab}}} (The mode of 1770 allows anyone in the group to create files but not unlink or rename each other's files.<<FootNote(Credits: [[http://etbe.coker.com.au/2010/03/17/starting-with-kvm/|Russell Coker]])>> {{{  1. Edit {{{/etc/sysctl.conf}}} and add this text to specify the number of page you want to reserve (the !HugePages are 4MiB on x86_32, 2MiB on x86_64 and x86_32 PAE) {{{#!text
# Allocate 256*2MiB for HugePageTables (YMMV)
vm.nr_hugepages = 256

# Members of group my-hugetlbfs(2021) are allowed to use
vm.hugetlb_shm_group = 2021
}}}
 1. Create a mount point for the file system {{{
% mkdir /hugepages
}}}
1. Add this line in {{{/etc/fstab}}} (The mode of 1770 allows anyone in the group to create files but not unlink or rename each other's files.<<FootNote(Credits: [[http://etbe.coker.com.au/2010/03/17/starting-with-kvm/|Russell Coker]])>>) {{{
Line 34: Line 39:
 1. Reboot (yes, you finally found an operation that actually requires a reboot in Linux ;)  1. Reboot <<FootNote(This is the most reliable method of allocating huge pages as memory has not yet become fragmented. Yes, you finally found an operation that actually recommend a reboot in Linux ;))>>
Line 37: Line 42:
Well, typically you can retrieve the current available/used page from {{{/proc/meminfo}}} (in this example !HugePages aren't enabled): Well, typically you can retrieve the current available/used page from {{{/proc/meminfo}}}:
Line 40: Line 45:
HugePages_Total:   0
HugePages_Free:   0
HugePages_Total: 256
HugePages_Free: 256
Line 46: Line 51:
Line 55: Line 61:
Various runtime settings (see [[#documentation|documentation]])
 {{{#!text
% grep -R "" /sys/kernel/mm/hugepages/ /proc/sys/vm/*huge*
/sys/kernel/mm/hugepages/hugepages-4096kB/nr_hugepages:256
/sys/kernel/mm/hugepages/hugepages-4096kB/nr_overcommit_hugepages:0
/sys/kernel/mm/hugepages/hugepages-4096kB/free_hugepages:256
/sys/kernel/mm/hugepages/hugepages-4096kB/resv_hugepages:0
/sys/kernel/mm/hugepages/hugepages-4096kB/surplus_hugepages:0
/proc/sys/vm/hugepages_treat_as_movable:0
/proc/sys/vm/hugetlb_shm_group:0
/proc/sys/vm/nr_hugepages:256
/proc/sys/vm/nr_overcommit_hugepages:0
}}}

== Hugepage enabled applications ==
An application can allocate/use !HugeTlbPage through two different means:
 1. {{{mmap}}} system call require a mounted {{{hugetlbfs}}}, with appropriate permissions.
 1. Shared memory segment (shmat/shmget system calls or mmap with MAP_HUGETLB), must be member of a group, configured in {{{/proc/sys/vm/hugetlb_shm_group}}}.
 {{{#!csv
Application ; hugetlbfs ; shared memory
QEMU/KVM ; No ; Yes
MySQM ; Yes ; No
}}}

=== KVM ===
 ''(TODO)'', see:
 * http://fedoraproject.org/wiki/Features/KVM_Huge_Page_Backed_Memory

=== MySQL ===
 ''(TODO)'', see:
 * Enabling Large Page Support ~-<<BR>>[[http://dev.mysql.com/doc/refman/5.1/en/large-page-support.html]]-~
Linux HugeTLBfs: Improve MySQL Database Application Performance ~-<<BR>>[[http://www.cyberciti.biz/tips/linux-hugetlbfs-and-mysql-performance.html]] (note tested)-~

<<Anchor(documentation)>>
Line 62: Line 102:
 * Linux HugeTLBfs: Improve MySQL Database Application Performance ~-<<BR>>[[http://www.cyberciti.biz/tips/linux-hugetlbfs-and-mysql-performance.html]] (note tested)<<BR>>[[http://dev.mysql.com/doc/refman/5.1/en/large-page-support.html]]-~
 * http://fedoraproject.org/wiki/Features/KVM_Huge_Page_Backed_Memory

Translation(s): none


Linux "Huge page tables" (HugeTlbPage) is available in Debian since DebianLenny (actually, since 2.6.23).

Currently, there is no standard way to enable HugeTLBfs, mainly because the FHS has no provision for such kind of virtual file system, see 572733 . (Fedora mounts it in /dev/hugepages/, so don't be surprised if you find some example on the web that use this location)

Read the documentation for more information about hugetlbpage.

Enabling HugeTlbPage

  1. Create a group for users of hugepages, and retrieve it's GID (is this example, 2021) then add yourself to the group.
    Note: this should not be needed for libvirt (see /etc/libvirt/qemu.conf)

    % groupadd my-hugetlbfs
    
    % getent group my-hugetlbfs
    my-hugetlbfs:x:2021:
    
    % adduser franklin my-hugetlbfs
    Adding user `franklin' to group `my-hugetlbfs' ...
    Adding user franklin to group my-hugetlbfs
    Done.
  2. Edit /etc/sysctl.conf and add this text to specify the number of page you want to reserve (the HugePages are 4MiB on x86_32, 2MiB on x86_64 and x86_32 PAE)

    # Allocate 256*2MiB for HugePageTables (YMMV)
    vm.nr_hugepages = 256
    
    # Members of group my-hugetlbfs(2021) are allowed to use 
    vm.hugetlb_shm_group = 2021
  3. Create a mount point for the file system

    % mkdir /hugepages
  4. Add this line in /etc/fstab (The mode of 1770 allows anyone in the group to create files but not unlink or rename each other's files.1)

    hugetlbfs /hugepages hugetlbfs mode=1770,gid=2021 0 0
  5. Reboot 2

Getting informations

Well, typically you can retrieve the current available/used page from /proc/meminfo:

  • % grep Huge /proc/meminfo 
    HugePages_Total:     256
    HugePages_Free:      256
    HugePages_Rsvd:        0
    HugePages_Surp:        0
    Hugepagesize:       4096 kB

(read Documentation/vm/hugetlbpage.txt for more information about it)

Standard Debian Kernel have HUGETLB enabled:

  • % grep HUGETLB /boot/config-$(uname -r)
    CONFIG_HUGETLBFS=y
    CONFIG_HUGETLB_PAGE=y

Various runtime settings (see documentation)

  • % grep -R ""  /sys/kernel/mm/hugepages/ /proc/sys/vm/*huge*
    /sys/kernel/mm/hugepages/hugepages-4096kB/nr_hugepages:256
    /sys/kernel/mm/hugepages/hugepages-4096kB/nr_overcommit_hugepages:0
    /sys/kernel/mm/hugepages/hugepages-4096kB/free_hugepages:256
    /sys/kernel/mm/hugepages/hugepages-4096kB/resv_hugepages:0
    /sys/kernel/mm/hugepages/hugepages-4096kB/surplus_hugepages:0
    /proc/sys/vm/hugepages_treat_as_movable:0
    /proc/sys/vm/hugetlb_shm_group:0
    /proc/sys/vm/nr_hugepages:256
    /proc/sys/vm/nr_overcommit_hugepages:0

Hugepage enabled applications

An application can allocate/use HugeTlbPage through two different means:

  1. mmap system call require a mounted hugetlbfs, with appropriate permissions.

  2. Shared memory segment (shmat/shmget system calls or mmap with MAP_HUGETLB), must be member of a group, configured in /proc/sys/vm/hugetlb_shm_group.

    Application hugetlbfs shared memory
    QEMU/KVM No Yes
    MySQM Yes No

KVM

MySQL

Linux HugeTLBfs: Improve MySQL Database Application Performance
http://www.cyberciti.biz/tips/linux-hugetlbfs-and-mysql-performance.html (note tested)

Documentation

See also


  1. Credits: Russell Coker (1)

  2. This is the most reliable method of allocating huge pages as memory has not yet become fragmented. Yes, you finally found an operation that actually recommend a reboot in Linux ;) (2)