E Very Large Memory on Linux x86

E.1 Shared Global Area Tuning

The basic memory structures associated with Oracle Database include System Global Area (SGA) and Program Global Area (PGA). The SGA is a group of shared memory structures, known as SGA components, that contain data and control information for one Oracle Database instance. The SGA is shared by all server and background processes. The PGA is a memory region that contains data and control information for a server process.

The following topics are covered in this section:

• Manual SGA Tuning

• Automatic SGA Tuning

E.2 Overview of HugePages

HugePages is a feature integrated into the Linux kernel with release 2.6. It is a method to have larger pages where it is useful for working with very large memory. It can be useful for both 32-bit and 64-bit configurations. HugePage sizes vary from 2MB to 256MB, depending on the kernel version and the hardware architecture. For Oracle Databases, using HugePages reduces the operating system maintenance of page states, and increases TLB (Translation Lookaside Buffer) hit ratio.

Without HugePages, the operating system keeps each 4 KB of memory as a page, and when it is allocated to the SGA, then the lifecycle of that page (dirty, free, mapped to a process, and so on) must be kept up to date by the operating system kernel.

With HugePages, the operating system page table (virtual memory to physical memory mapping) is smaller, since each page table entry is pointing to pages from 2 MB to 256 MB. Also, the kernel has fewer pages whose lifecyle must be monitored.

For example, if you use HugePages with 64-bit hardware, and you want to map 256 MB of memory, you may need one page table entry (PTE). If you do not use HugePages, and you want to map 256 MB of memory, then you must have 256 MB * 1024 KB/4 KB = 65536 PTEs.

E.3 Methods To Increase SGA Limits

The following are the methods to increase SGA limits on a 32-bit computer:

• Hugemem Kernel

• Hugemem Kernel with Very Large Memory

$ uname -r
2.6.9-5.0.3.ELhugemem

E.4 Configuring Very Large Memory for Oracle Database

Complete the following procedure to configure Very Large Memory on the computer:

1. Log in as a root user:

   sudo -sh
   Password:
   

2. Edit the /etc/rc.local file and add the following entries to it to configure the computer to mount ramfs over the /dev/shm directory, whenever you start the computer:

   umount /dev/shm
   mount -t ramfs ramfs /dev/shm
   chown oracle:oinstall /dev/shm
   

   In the preceding commands, oracle is the owner of Oracle software files and oinstall is the group for Oracle owner account.

3. Restart the server.

4. Log in as a root user.

5. Run the following command to check if the /dev/shm directory is mounted with the ramfs type:

   # mount | grep shm
   ramfs on /dev/shm type ramfs (rw)
   

6. Run the following command to check the permissions on the /dev/shm directory:

   # ls -ld /dev/shm
   drwxr-xr-x  3 oracle oinstall 0 Jan 13 12:12 /dev/shm
   

7. Edit the /etc/security/limits.conf file and add the following entries to it to increase the max locked memory limit:

   soft    memlock        3145728hard    memlock        3145728
   

8. Switch to the oracle user:

   # sudo - oracle
   Password:
   

9. Run the following command to check the max locked memory limit:

   $ ulimit -l
   3145728
   

10. Complete the following procedure to configure instance parameters for Very Large Memory:

    1. Replace the DB_CACHE_SIZE and DB_xK_CACHE_SIZE parameters with DB_BLOCK_BUFFERS parameter.

    2. Add the USE_INDIRECT_DATA_BUFFERS=TRUE parameter.

    3. Configure SGA size according to the SGA requirements.

    4. Remove SGA_TARGET, if set.

11. Start the database instance.

12. Run the following commands to check the memory allocation:

    $ ls -l /dev/shm
    $ ipcs -m
    

13. Run the following command to display the value of Hugepagesize variable:

    $ grep Hugepagesize /proc/meminfo
    

14. Complete the following procedure to create a script that computes recommended values for hugepages configuration for the current shared memory segments:

    1. Create a text file named hugepages_settings.sh.

    2. Add the following content in the file:

       #!/bin/bash
       #
       # hugepages_settings.sh
       #
       # Linux bash script to compute values for the
       # recommended HugePages/HugeTLB configuration
       #
       # Note: This script does calculation for all shared memory
       # segments available when the script is run, no matter it
       # is an Oracle RDBMS shared memory segment or not.
       # Check for the kernel version
       KERN='uname -r | awk -F. '{ printf("%d.%d\n",$1,$2); }''
       # Find out the HugePage size
       HPG_SZ=`grep Hugepagesize /proc/meminfo | awk {'print $2'}`
       # Start from 1 pages to be on the safe side and guarantee 1 free HugePage
       NUM_PG=1
       # Cumulative number of pages required to handle the running shared memory segments
       for SEG_BYTES in `ipcs -m | awk {'print $5'} | grep "[0-9][0-9]*"`
       do
          MIN_PG='echo "$SEG_BYTES/($HPG_SZ*1024)" | bc -q'
          if [ $MIN_PG -gt 0 ]; then
             NUM_PG=`echo "$NUM_PG+$MIN_PG+1" | bc -q`
          fi
       done
       # Finish with results
       case $KERN in
          '2.4') HUGETLB_POOL=`echo "$NUM_PG*$HPG_SZ/1024" | bc -q`;
                 echo "Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL" ;;
          '2.6') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
           *) echo "Unrecognized kernel version $KERN. Exiting." ;;
       esac
       # End
       

    3. Run the following command to change the permission of the file:

       $ chmod +x hugepages_settings.sh
       

15. Run the hugepages_settings.sh script to compute the values for hugepages configuration:

    $ ./hugepages_settings.sh
    

    Note:

    Before running this script, ensure that all the applications that must use hugepages run.

16. Set the following kernel parameter:

    # sysctl -w vm.nr_hugepages=value_displayed_in_step_15
    

17. To make the value of the parameter available for every time you restart the computer, edit the /etc/sysctl.conf file and add the following entry:

    vm.nr_hugepages=value_displayed_in_step_15
    

18. Run the following command to check the available hugepages:

    $ grep Huge /proc/meminfo
    

19. Restart the instance.

20. Run the following command to check the available hugepages (1 or 2 pages free):

    $ grep Huge /proc/meminfo
    

    Note:

    If the setting of the nr_hugepages parameter is not effective, you might must restart the server.

E.5 Restrictions Involved in Implementing Very Large Memory

Following are the limitations of running a computer in the Very Large Memory mode:

• Very Large Memory configurations do not support multiple database block sizes.

• The computer on which Oracle Database is installed must have more than 4 GB of memory.

• Initialization parameters SGA_TARGET or MEMORY_TARGET must be unset before you implement Very Large Memory.