Below are the steps taken to configure huge pages on each Exadata node.
1) Ensure that all DB instances are up an running on the node.
2) Ensure that the databases are not using Oracle 11g AMM. See Metalink note 749851.1 “HugePages and Oracle Database 11g Automatic Memory Management (AMM) on Linux”.
The note states:
If you want to use HugePages make sure that both MEMORY_TARGET / MEMORY_MAX_TARGET initialization parameters are unset (i.e. using “ALTER SYSTEM RESET”)
for the database instance.
This should just be a verification check, as we are not using AMM in DB aenvironments.
3) Create and execute the script provided in metalink note 401749.1 “Shell Script to Calculate Values Recommended Linux HugePages / HugeTLB Configuration”.
This can be done on every nodes i will be setting the same values across all nodes as i have only 2 db instances running with same SGA & PGA.
[root@radbvip1 ~]# ./hugepages_settings.sh
This script is provided by Doc ID 401749.1 from My Oracle Support
(http://support.oracle.com) where it is intended to compute values for
the recommended HugePages/HugeTLB configuration for the current shared
memory segments. Before proceeding with the execution please make sure
that:
* Oracle Database instance(s) are up and running
* Oracle Database 11g Automatic Memory Management (AMM) is not setup
(See Doc ID 749851.1)
* The shared memory segments can be listed by command:
# ipcs -m
Press Enter to proceed…
Recommended setting: vm.nr_hugepages = 16900
4) Set the kernel parameter. This can be done on all nodes as root via the following dcli command:
[root@radbvip1 ~]# dcli -l root -g dbs_group “sysctl -w vm.nr_hugepages=16900”
radbvip1: vm.nr_hugepages = 16900
radbvip2: vm.nr_hugepages = 16900
radbvip3: vm.nr_hugepages = 16900
radbvip4: vm.nr_hugepages = 16900
5) Ensure that the kernel setting is persistent on reboot. Edit the /etc/sysctl.conf configuration file and add the following in all nodes.
##################################################################################################
# Added by Oracle <date> to reserve huge pages for
# if memory is adjusted, this parameter will also need to be
# adjusted based upon the output of /root/hugepages_settings.sh
# MOS note 401749.1 for precise computations and see MOS 361323.1 for a description of Hugepages
# also MOS note 1274318.1 Oracle Sun Database Machine X2-2 Setup/Configuration Best Practices
vm.nr_hugepages = 16900
##################################################################################################
6) Verify Huge Page reservations:
# dcli -l root -g dbs_group “grep Huge /proc/meminfo”
7) Set the database init.ora parameter for using huge pages in both databases:
SYS@PROD> alter system set use_large_pages=ONLY scope=spfile sid=’*’;
SYS@PRE_PROD> alter system set use_large_pages=ONLY scope=spfile sid=’*’;
8) Restart databases
$ srvctl stop database -d prod
$ srvctl stop database -d pre_prod
$ srvctl start database -d prod
$ srvctl start database -d pre_prod
9) Verify HugePages used via review of alert log. You should see somethink like that below.
Starting ORACLE instance (normal)
****************** Huge Pages Information *****************
Huge Pages memory pool detected (total: 197662 free: 197662)
DFLT Huge Pages allocation successful (allocated: 10240)
…
DFLT Huge Pages allocation successful (allocated: 23552)
DFLT Huge Pages allocation successful (allocated: 1)
***********************************************************
Done !!
Oracle Buddies 