2 Administering Oracle Clusterware

Network Requirements for IPMI

IPMI requires a management network. The management network should be a secure, dedicated network, but it may also be a shared, multi-purpose network. In either case, the Ethernet port used by the BMC on each node must be connected to the network used for server management.

To enable fencing with IPMI, Oracle Clusterware must have access to the management network, so that it can communicate with the BMCs on cluster member nodes. You can provide that access in one of the following ways:

• Use the same network connection both for the private network for interconnects and the management network

• Use forwarding of network traffic for the management network to the BMC addresses from the private network or some other network accessible to Oracle Clusterware.

For flexibility in network management, you can configure the BMC for dynamic IP address assignment using DHCP. The design and implementation of fencing using IPMI supports this and assumes this to be the normal case. To support dynamic addressing of BMCs, the network used for IPMI-based management must have a DHCP server to assign the BMC's IP address.

IPMI Driver

The IPMI driver must be permanently installed on each node, so that it is available when the node is started, so that Oracle Clusterware can communicate with the BMCs.

Typically, the IMPI hardware on a server node obtains its IP address using DHCP when the node starts. While other fencing configuration information is stored in Oracle Registries, dynamic address information must be obtained from the BMC when Oracle Clusterware starts.

# /sbin/modprobe ipmi_msghandler
# /sbin/modprobe ipmi_si
# /sbin/modprobe ipmi_devintf

# /sbin/lsmod | grep ipmi
ipmi_devintf           12617  0
ipmi_si                33377  0
ipmi_msghandler        33701  2 ipmi_devintf,ipmi_si

# ls -l /dev/ipmi0
crw-------  1 root root 253, 0 Sep 23 06:29 /dev/ipmi0

# grep ipmi /proc/devices
253 ipmidev

# mknod /dev/ipmi0 c 253 0x0

About BMC Configuration

For IPMI-based fencing to function, the BMC hardware must be properly configured for remote control using a LAN. Requirements are the same for all platforms, but the configuration process is platform-dependent. The BMC configuration may be effected from the firmware interface (BIOS), using a platform-specific management utility or one of many publicly available utilities, which can be downloaded from the Internet. Two of the available utilities are ipmitool (http://ipmitool.sourceforge.net/), which is available for Linux, and ipmiutil (http://ipmiutil.sourceforge.net/), which is available for both Linux and Windows.

The BMC configuration must provide the following:

• Enable IPMI over LAN, which permits the BMC to be controlled over the network.

• Establish a static IP address, or enable dynamic IP-addressing using DHCP.

• Establish an administrator account with user name and password.

• Configure the BMC for VLAN tags, if the BMC is on a tagged VLAN

The configuration mechanism does not influence functionality if the above requirements are met.

BMC Configuration Example Using ipmitool on Linux

The following is an example of BMC configuration on Linux using ipmitool (version 1.8.6). Because the protection settings on the BMC device (/dev/ipmi0) restrict access to root, the following commands must be run with root privilege.

1. When the IPMI driver is loaded and the device special file is created, verify that ipmitool can communicate with the BMC using the driver using the command ipmitool bmc info. For example:

   # ipmitool bmc info
   Device ID                 : 32
   .
   .
   .
   

2. Enable IPMI over LAN:

   1. Determine the IPMI channel number for LAN (typically 1) by printing channel attributes, starting from channel "1" until the output shows typical LAN attributes, such as the IP address. For example:

      # ipmitool lan print 1
       
      . . . 
      IP Address Source       : 0x01
      IP Address              : 192.0.2.244
      . . .
      

   2. Turn on LAN access for the channel found. For example, where the LAN channel is 1:

      # ipmitool lan set 1 access on
      

3. Configure IP address settings for IPMI

   1. Use dynamic IP addressing (DHCP). This is the default assumed by the Oracle Universal Installer. Use of DHCP requires a DHCP-server on the subnet, but has the advantage of making the other addressing settings for you automatically. For example:

      # ipmitool lan set 1 ipsrc dhcp
      

   2. Use static IP address assignment. If the BMC shares a network connection, with the OS, then the IP address must be on the same subnet. You must set not only the IP address, but the proper values for netmask and default gateway for your network configuration.

      # ipmitool lan set 1 ipaddr 192.168.0.55
      # ipmitool lan set 1 netmask 255.255.255.0
      # ipmitool lan set 1 defgw ipaddr 192.168.0.1
      

      Note:

      The address that you specify in this example, 192.168.0.55, is associated only with the BMC and does not respond to normal pings.

4. Establish an administration account with username and password.

   1. Set BMC to require password authentication for ADMIN access over LAN. For example:

      # ipmitool lan set 1 auth ADMIN MD5,PASSWORD
      

   2. List the account slots on the BMC and identify an unused slot (a User ID with an empty User Name field - in this example):

      # ipmitool channel getaccess 1
      . . . 
      User ID              : 4
      User Name            :
      Fixed Name           : No
      Access Available     : call-in / callback
      Link Authentication  : disabled
      IPMI Messaging       : disabled
      Privilege Level      : NO ACCESS
      . . .
      

   3. Assign the administrator user name and password and enable messaging for the identified slot. (Note that for IPMI v1.5 the user name and password can be at most 16 characters). Finally, set the privilege level for that slot when accessed over LAN (channel 1) to ADMIN (level 4). For example, using the administrator IPMIadm and password myIPMIpwd:

      # ipmitool user set name 4 IPMIadm
      # ipmitool user set password 4 myIPMIpwd
      # ipmitool user enable 4
      # ipmitool channel setaccess 1 4 privilege=4
      

5. Verify the setup after you complete configuration using the command ipmitool lan print. For example (note configuration settings in italic)

   # ipmitool lan print 1
   Set in Progress         : Set Complete
   Auth Type Support       : NONE MD2 MD5 PASSWORD
   Auth Type Enable        : Callback : MD2 MD5
                           : User     : MD2 MD5
                           : Operator : MD2 MD5
                           : Admin    : MD5 PASSWORD
                           : OEM      : MD2 MD5
   IP Address Source       : Static Address (or DHCP Address)
   IP Address              : 192.168.0.55
   Subnet Mask             : 255.255.255.0
   MAC Address             : 00:14:22:23:fa:f9
   SNMP Community String   : public
   IP Header               : TTL=0x40 Flags=0x40 Precedence=… 
   Default Gateway IP      : 192.168.0.1
   Default Gateway MAC     : 00:00:00:00:00:00
   .
   .
   .
   # ipmitool channel getaccess 1 4
   Maximum User IDs     : 10
   Enabled User IDs     : 2
   User ID              : 4
   User Name            : IPMIadm
   Fixed Name           : No
   Access Available     : call-in / callback
   Link Authentication  : enabled
   IPMI Messaging       : enabled
   Privilege Level      : ADMINISTRATOR
   

6. Verify that the BMC is accessible and controllable from a remote node in your cluster. For example, on node1, with user IPMIadm, and password IPMIpwd, enter the following command:

   # ipmitool -H node1 -U IPMIadm -P IPMIpwd bmc info
   

   This command should return the BMC information from the node.

BMC Configuration Example Using ipmiutil on Windows 2003 R2

The following is an example of BMC configuration on Windows Server 2003 R2 using ipmiutil (version 2.2.3). These operations should be executed in a command window while logged in with Administrator privileges.

1. When the IPMI driver is loaded and the device special file is created, verify that ipmitool can communicate with the BMC using the driver using the command ipmitool lan. For example:

   C:\bin> ipmitool lan
   ipmiutil ver 2.23
   PEFilter parameters displayed . . .  
   pefconfig, GetLanEntry for channel 1 . . .
   Lan Param(0) Set in progress: 00
   .
   .
   .
   

   Caution:

   ipmiutil sets certain LAN parameters only in the context of enabling IPMI over LAN; for example, by supplying the -l option. This can have the undesired effect of resetting to default values of some previously established parameters if they are not supplied on the command line. For that reason, it is important to complete the following steps in order.

2. Establish remote LAN access with ADMIN privilege (-v 4) and the IPMI administrator user name and password (ipmiutil locates the LAN channel on its own). For example:

   C:\bin> ipmiutil lan -l -v 4 -u root -p password
   

3. Configure dynamic or static IP address settings for the BMC:

   1. Use dynamic IP-addressing (DHCP). This is the default assumed by the Oracle Universal Installer. Use of DHCP requires a DHCP-server on the subnet, but has the advantage of making the other addressing settings for you automatically.

      C:\bin> ipmiutil lan -l -D
      

   2. Use static IP address assignment. If the BMC shares a network connection wit the operating system, then the IP address must be on the same subnet. You must set not only the IP address, but also the proper values default gateway and net mask for your network configuration. For example:

      C:\bin> ipmiutil lan –l –I 192.0.2.244(IP addr)
      C:\bin> ipmiutil lan -l –G 192.0.2.1              (gateway IP)
      C:\bin> ipmiutil lan –l –S 255.255.248.0   (netmask)
      

      Note:

      The address that you specify in this example, 192.0.244, is associated only with the BMC and does not respond to normal pings.

      Also, enabling IPMI over LAN with the -l option resets the subnet mask to a value obtained from the operating system. Thus, when setting parameters one at a time as in the above example, the -S option should be specified last.

4. Verify the setup after you complete configuration using the command ipmitool lan, which displays the configuration. For example (note configuration settings in italic):

   C:\bin> ipmiutil lan 
   ipmiutil ver 2.23
   pefconfig ver 2.23
   -- BMC version 1.40, IPMI version 1.5
   pefconfig, GetPefEntry ...
   PEFilter(01): 04 h ? event ... (ignore PEF entries)
   ...
    
   pefconfig, GetLanEntry for channel 1 ...
   Lan Param(0) Set in progress: 00
   Lan Param(1) Auth type support: 17 : None MD2 MD5 Pswd
   Lan Param(2) Auth type enables: 16 16 16 16 00
   Lan Param(3) IP address: 192.0.2.244
   Lan Param(4) IP addr src: 01 : Static
   Lan Param(5) MAC addr: 00 11 43 d7 4f bd
   Lan Param(6) Subnet mask: 255 255 248 0
   Lan Param(7) IPv4 header: 40 40 10
   GetLanEntry: completion code=cc
   GetLanEntry(10), ret = -1
   GetLanEntry: completion code=cc
   GetLanEntry(11), ret = -1
   Lan Param(12) Def gateway IP: 192.0.2.1
   Lan Param(13) Def gateway MAC: 00 00 0c 07 ac dc
   ...
   Get User Access(1): 0a 01 01 0f : No access ()
   Get User Access(2): 0a 01 01 14 : IPMI, Admin  (root)
   Get User Access(3): 0a 01 01 0f : No access ()
   pefconfig, completed successfully
   

5. Use the ipmitool command to verify that the BMC is accessible and controllable from a remote node in your cluster. For example, where the IPMI administrator is root, and the node you are checking is node1:

   # ipmitool -H node1-U root -P password lan print 1
   

   This command should return the LAN configuration from the node.

IPMI Postinstallation Registration with Oracle Clusterware

When IPMI is installed during Oracle Clusterware installation, Failure Isolation is configured in two phases. Before you start installation, you install and enable the IPMI driver in the server operating system, and configure the IPMI hardware on each node (IP address mode, admin credentials, and so on). When you start Oracle Clusterware installation, the installer collects the IPMI administrator user ID and password, and stores them in an Oracle Wallet in node-local storage, in the Oracle Local Registry.

With postinstallation configuration, install and enable the IPMI driver, and configure the BMC as described in the preceding section, "Configuring Server Hardware for IPMI".

After you have completed the server configuration, complete the following procedure on each cluster node to register IPMI administrators and passwords on the nodes.

1. Start Oracle Clusterware, which allows it to obtain the current IP address from the BMC. This confirms the ability of the clusterware to communicate with the BMC, which is necessary at startup.

   If the clusterware was running before the BMC was configured, you can shut it down and restart it. Alternatively, you can use the BMC management utility to obtain the BMC's IP address and then use the cluster control utility crsctl to store the BMC's IP address in the Oracle Registry by issuing a command similar to the following:

   crsctl set css ipmiaddr 192.168.10.45
   

2. Use the cluster control utility crsctl to store the previously established user ID and password for the resident BMC in the Oracle Registry by issuing the crsctl set css ipmiadmin command, and supplying the administrator and password at the prompt. For example:

   crsctl set css ipmiadmin youradmin
   IPMI BMC password: yourpassword
   

   This command validates the supplied credentials and fails if another cluster node cannot access the local BMC using them.

   After you complete hardware and operating system configuration, and register the IPMI administrator on Oracle Clusterware, IPMI-based Failure Isolation should be fully functional.

Modifying Configurations with IPMI Postinstallation Configuration Utility

To modify an existing IPMI-based fencing configuration (for example to change BMC passwords, or to configure IPMI for fencing in an existing installation), use the cluster configuration utility crsctl with the BMC configuration tool appropriate to your platform. For example, to change the administrator password for the BMC, you must first modify the BMC configuration as described in "Configuring Server Hardware for IPMI", and then use crsctl to inform the clusterware that the password has changed.

The configuration data needed by Oracle Clusterware for IPMI is kept in an Oracle Wallet in the Oracle Registry. Because the configuration information is kept in a secure store, it must be written by the Oracle Clusterware installation owner account (the Grid user), so you must log in as that installation user.

Use the following procedure to modify an existing IPMI configuration:

1. Enter the command crsctl set css ipmiadmin admin_name. For example, with the user IPMIadm:

   $  crsctl set css ipmiadmin IPMIadm
   

   Provide the administrator password. Oracle Clusterware stores the administrator name and password for the local BMC in the Oracle Local Registry.

   After storing the new credentials, Oracle Clusterware can retrieve the new credentials and distribute them as required.

2. Enter the command crsctl set css ipmiaddr bmc_ip_address. For example:

   $ crsctl set css ipmiaddr 192.0.2.244
   

   This command stores the new BMC IP address of the local BMC in the Oracle Registry, After storing the BMC IP address, Oracle Clusterware can retrieve the new configuration and distribute it as required.

3. Enter the command crsctl get css ipmiaddr. For example:

   $ crsctl get css ipmiaddr
   

   This command retrieves the IP address for the local BMC from the Oracle Registry and displays it on the console.

How Server Pools Work

Server pools divide the cluster into groups of servers hosting the same or similar resources. They distribute a uniform workload (a set of Oracle Clusterware resources) over several servers in the cluster. For example, you can restrict Oracle databases to run only in a particular server pool. When you enable role-separated management, you can explicitly grant permission to operating system users to change attributes of certain server pools.

Top-level server pools:

• Logically divide the cluster

• Are always exclusive, meaning that one server can only reside in one particular server pool at a certain point in time

Server pools each have three attributes that they are assigned when they are created:

• MIN_SIZE: The minimum number of servers the server pool should contain. If the number of servers in a server pool is below the value of this attribute, then Oracle Clusterware automatically moves servers from elsewhere into the server pool until the number of servers reaches the attribute value.

• MAX_SIZE: The maximum number of servers the server pool should contain.

• IMPORTANCE: A number from 0 to 1000 (0 being least important) that ranks a server pool among all other server pools in a cluster.

When Oracle Clusterware is installed, two server pools are created automatically: Generic and Free. All servers in a new installation are assigned to the Free server pool, initially. Servers move from Free to newly defined server pools automatically. When you upgrade Oracle Clusterware, all nodes are assigned to the Generic server pool, to ensure compatibility with database releases before Oracle Database 11g release 2 (11.2).

The Free Server Pool

The Free server pool contains servers that are not assigned to any other server pools. The attributes of the Free server pool are restricted, as follows:

• SERVER_NAMES, MIN_SIZE, and MAX_SIZE cannot be edited by the user

• IMPORTANCE and ACL can be edited by the user

The Generic Server Pool

The Generic server pool stores pre-11g release 2 (11.2) databases and administrator-managed databases that have fixed configurations. Additionally, the Generic server pool contains servers that match either of the following:

• Servers that you specified in the HOSTING_MEMBERS attribute of all resources of the application resource type

  See Also:

  "HOSTING_MEMBERS" for more information about this attribute

• Servers with names you specified in the SERVER_NAMES attribute of the server pools that list the Generic server pool as a parent server pool

The Generic server pool's attributes are restricted, as follows:

• No one can modify configuration attributes of the Generic server pool (all attributes are read-only)

• When you specify a server name in the HOSTING_MEMBERS attribute, Oracle Clusterware only allows it if the server is:

  • Online and exists in the Generic server pool

  • Online and exists in the Free server pool, in which case Oracle Clusterware moves the server into the Generic server pool

  • Online and exists in any other server pool and the client is either a cluster administrator or is allowed to use the server pool's servers, in which case, the server is moved into the Generic server pool

  • Offline and the client is a cluster administrator

• When you register a child server pool with the Generic server pool, Oracle Clusterware only allows it if the server names pass the same requirements as previously specified for the resources.

  Servers are initially considered for assignment into the Generic server pool at cluster startup time or when a server is added to the cluster, and only after that to other server pools.

How Oracle Clusterware Assigns Servers to Server Pools

When a server joins the cluster, Oracle Clusterware assigns the server to a server pool in the following order:

1. Until all server pools are filled in order of importance to their minimum (MIN_SIZE).

2. Until all server pools are filled in order of importance to their maximum (MAX_SIZE).

3. By default, any servers not placed in a server pool go into the Free server pool.

   You can modify the IMPORTANCE attribute for the Free server pool.

Servers Moving from Server Pool to Server Pool

If the number of servers in a server pool falls below the value of the MIN_SIZE attribute for the server pool (such as when a server fails), based on values you set for the MIN_SIZE and IMPORTANCE attributes for all server pools, Oracle Clusterware can move servers from other server pools into the server pool whose number of servers has fallen below the value for MIN_SIZE. Oracle Clusterware selects servers from other server pools to move into the deficient server pool that meet the following criteria:

• For server pools that have a lower IMPORTANCE value than the deficient server pool, Oracle Clusterware can take servers from those server pools even if it means that the number of servers falls below the value for the MIN_SIZE attribute.

• For server pools with equal or greater IMPORTANCE, Oracle Clusterware only takes servers from those server pools if the number of servers in a server pool is greater than the value of its MIN_SIZE attribute.

Table 2-2 lists and describes all server pool attributes.

ACL

owner:user:rwx,pgrp:group:rwx,other::r—

user:username:rwx
group:group_name:rwx

ACTIVE_SERVERS

server_name1 server_name2 ...

EXCLUSIVE_POOLS

IMPORTANCE

MAX_SIZE

MIN_SIZE

NAME

PARENT_POOLS

sp1 sp2 ...

SERVER_NAMES

server1 server2 ...

You manage server pools that are managing Oracle RAC databases with the Server Control (SRVCTL) utility. Use the Oracle Clusterware Control (CRSCTL) utility to manage all other server pools. Only cluster administrators have permission to create top-level server pools.

Storing Voting Disks on Oracle ASM

Oracle ASM manages voting disks differently from other files that it stores. When you place voting disks on disks in an Oracle ASM disk group, Oracle Clusterware records exactly where they are located. If Oracle ASM fails, then Cluster Synchronization Services (CSS) can still access the voting disks.

If you choose to store your voting disks in Oracle ASM, then Oracle ASM stores all the voting disks for the cluster.

The number of voting files you can store in a particular Oracle ASM disk group depends upon the redundancy of the disk group.

• External redundancy: A disk group with external redundancy can store only one voting disk

• Normal redundancy: A disk group with normal redundancy can store up to three voting disks

• High redundancy: A disk group with high redundancy can store up to five voting disks

By default, Oracle ASM puts each voting disk in its own failure group within the disk group. A failure group is a subset of the disks in a disk group, which could fail at the same time because they share hardware. The failure of common hardware must be tolerated. For example, four drives that are in a single removable tray of a large JBOD (Just a Bunch of Disks) array are in the same failure group because the tray could be removed, making all four drives fail at the same time.

Conversely, drives in the same cabinet can be in multiple failure groups if the cabinet has redundant power and cooling so that it is not necessary to protect against failure of the entire cabinet. However, Oracle ASM mirroring is not intended to protect against a fire in the computer room that destroys the entire cabinet. If voting disks stored on Oracle ASM with Normal or High redundancy, and the storage hardware in one failure group suffers a failure, then if there is another disk available in a disk group in an unaffected failure group, Oracle ASM recovers the voting disk in the unaffected failure group.

A normal redundancy disk group must contain at least two failure groups but if you are storing your voting disks on Oracle ASM, then a normal redundancy disk group must contain at least three failure groups. A high redundancy disk group must contain at least three failure groups. However, Oracle recommends using several failure groups. A small number of failure groups, or failure groups of uneven capacity, can create allocation problems that prevent full use of all of the available storage.

You must specify enough failure groups in each disk group to support the redundancy type for that disk group.

Backing Up Voting Disks

In Oracle Clusterware 11g release 2 (11.2), you no longer have to back up the voting disk. The voting disk data is automatically backed up in OCR as part of any configuration change and is automatically restored to any voting disk added. If all voting disks are corrupted, however, you can restore them as described in "Restoring Voting Disks".

Restoring Voting Disks

If you have multiple voting disks, then you can remove the voting disks and add them back into your environment using the following commands, where GUID is the file universal identifier of the voting disk and path_to_voting_disk is the directory path to the voting disk:

$ crsctl delete css votedisk FUID
$ crsctl add css votedisk path_to_voting_disk

Run the crsctl query css votedisk command to obtain the GUIDs of the voting disks in the cluster.

If all of the voting disks are corrupted, then you can restore them, as follows:

1. Restore OCR as described in "Restoring Oracle Cluster Registry", if necessary.

   This step is necessary only if OCR is also corrupted or otherwise unavailable, such as if OCR is on Oracle ASM and the disk group is no longer available.

   See Also:

   Oracle Database Storage Administrator's Guide for more information about managing Oracle ASM disk groups

2. Run the following command as root from only one node to start the Oracle Clusterware stack in exclusive mode, which does not require voting files to be present or usable:

   # crsctl start crs -excl
   

3. Run the following command to retrieve the list of voting files currently defined:

   $ crsctl query css votedisk
   

   This list may be empty if all voting disks are corrupted, or may have entries that are marked as status 3 or OFF.

4. Depending on where you store your voting files, do one of the following:

   • If the voting disks are stored in Oracle ASM, then run the following command to migrate the voting disks to the Oracle ASM disk group you specify:

     crsctl replace votedisk +asm_disk_group
     

     The Oracle ASM disk group to which you migrate the voting files must exist in Oracle ASM. You can use this command whether the voting disks were stored in Oracle ASM or some other storage device.

   • If voting disks are OFFLINE and are not stored in Oracle ASM (as indicated by the absence of a name in the last column of the string of information returned by the crsctl query css votedisk command), then run the following command using the File Universal Identifier (FUID) obtained in the previous step:

     $ crsctl delete css votedisk FUID
     

     Add a voting disk, as follows:

     $ crsctl add css votedisk path_to_voting_disk
     

5. Stop the Oracle Clusterware stack as root:

   # crsctl stop crs
   

6. Restart the Oracle Clusterware stack in normal mode as root:

   # crsctl start crs
   

Adding, Deleting, or Migrating Voting Disks

You can add, remove, and migrate voting disks after you install Oracle Clusterware. Note that the commands you use to do this are different, depending on whether your voting disks are located on Oracle ASM, or are located on another storage option.

Use the following commands to modify voting disks, depending on your storage option:

• To display the voting disk FUID and file path of each current voting disk, run the following command:

  $ crsctl query css votedisk
  1. ONLINE   296641fd201f4f3fbf3452156d3b5881 (/ocfs2/staih09_vd3) []
  

  This command returns a disk sequence number, the status of the disk, the FUID, and the path of the disk or the name of the Oracle ASM disk group on which the disk is stored.

• To replace a voting disk with an Oracle ASM disk group, run the following command:

  $ crsctl replace votedisk +asm_disk_group
  

• To add one or more voting disks to a storage location other than Oracle ASM storage, run the following command, replacing the path_to_voting_disk variable with one or more space-delimited, complete paths to the voting disks you want to add:

  $ crsctl add css votedisk path_to_voting_disk [...]
  

• To replace voting disk A with voting disk B, you must add voting disk B, and then delete voting disk A. To add a new disk and remove the existing disk, run the following command, replacing the path_to_voting_diskB variable with the fully qualified path name of voting disk B:

  $ crsctl add css votedisk path_to_voting_diskB -purge
  

  The -purge option deletes existing voting disks.

  Use the crsctl replace votedisk command to replace a voting disk with an Oracle ASM disk group.

• To remove a voting disk, run the following command, specifying one or more space-delimited, voting disk FUIDs or comma-delimited directory paths to the voting disks you want to remove:

  $ crsctl delete css votedisk {FUID | path_to_voting_disk[...]}
  

• To migrate voting disks from a storage device other than Oracle ASM to Oracle ASM, or to migrate from Oracle ASM to an alternative storage device, specify the Oracle ASM disk group name or path to the non-Oracle ASM storage device in the following command:

  $ crsctl replace votedisk {+asm_disk_group | path_to_voting_disk}
  

  You can run this command on any node in the cluster.

$ crsctl start crs -excl

After modifying the voting disk, verify the voting disk location, as follows:

$ crsctl query css votedisk

Migrating Oracle Cluster Registry to Oracle Automatic Storage Management

To improve Oracle Clusterware storage manageability, OCR is configured, by default, to use Oracle ASM in Oracle Database 11g release 2 (11.2). With the Oracle Clusterware storage residing in an Oracle ASM disk group, you can manage both database and clusterware storage using Oracle Enterprise Manager.

However, if you upgrade from a previous version of Oracle Clusterware, you can migrate OCR to reside on Oracle ASM, and take advantage of the improvements in managing Oracle Clusterware storage.

To migrate OCR to Oracle ASM using OCRCONFIG:

1. Ensure that Oracle Clusterware upgrade to 11g release 2 (11.2) is complete. Run the following command to verify the current running version:

   $ crsctl query crs activeversion
   

2. Use the Oracle ASM Configuration Assistant (ASMCA) to configure and start Oracle ASM on all nodes in the cluster.

   See Also:

   Oracle Database Storage Administrator's Guide for more information about using ASMCA

3. Use ASMCA to create an Oracle ASM disk group that is at least the same size of the existing OCR and has at least normal redundancy.

   Notes:

   • You can store the OCR in an Oracle ASM disk group that has external redundancy.

     If a disk fails in the disk group, or if you bring down Oracle ASM, then you can lose the OCR because it depends on Oracle ASM for I/O.

     To avoid this issue, add another OCR to a different disk group. Alternatively, you can store OCR on a block device, or on a shared file system using OCRCONFIG to enable OCR redundancy.

     Oracle does not support storing the OCR on different storage types simultaneously, such as storing OCR on both Oracle ASM and a shared file system, except during a migration.

   • If Oracle ASM fails, then OCR is not accessible on the node on which Oracle ASM failed, but the cluster remains operational. The entire cluster only fails if the Oracle ASM instance on the OCR master node fails, if the majority of the OCR locations are in Oracle ASM, and if there is an OCR read or write access, then the crsd stops and the node becomes inoperative.

   • Ensure that Oracle ASM disk groups that you create are mounted on all of the nodes in the cluster.

   See Also:

   Oracle Grid Infrastructure Installation Guide for more detailed sizing information

4. To add OCR to an Oracle ASM disk group, ensure that the Oracle Clusterware stack is running and run the following command as root:

   # ocrconfig -add +new_disk_group
   

   You can run this command more than once if you add multiple OCR locations. You can have up to five OCR locations. However, each successive run must point to a different disk group.

5. To remove storage configurations no longer in use, run the following command as root:

   # ocrconfig -delete old_storage_location
   

   Run this command for every configured OCR.

The following example shows how to migrate two OCRs to Oracle ASM using OCRCONFIG.

# ocrconfig -add +new_disk_group
# ocrconfig -delete /dev/raw/raw2
# ocrconfig -delete /dev/raw/raw1

# ocrconfig -add /dev/sdd1
# ocrconfig -add /dev/sde1
# ocrconfig -add /dev/sdf1
# ocrconfig -delete +unused_disk_group

Adding, Replacing, Repairing, and Removing Oracle Cluster Registry Locations

The Oracle installation process for Oracle Clusterware gives you the option of automatically mirroring OCR. You can manually put the mirrored OCRs on a shared network file system (NFS), or on any cluster file system that is certified by Oracle. Alternatively, you can place the OCR on Oracle ASM and allow it to create mirrors automatically, depending on the redundancy option you select.

This section includes the following topics:

• Adding an Oracle Cluster Registry Location

• Replacing an Oracle Cluster Registry Location

• Repairing an Oracle Cluster Registry Configuration on a Local Node

• Removing an Oracle Cluster Registry Location

• Overriding the Oracle Cluster Registry Data Loss Protection Mechanism

You can manually mirror OCR, as described in the "Adding an Oracle Cluster Registry Location" section, if you:

• Upgraded to 11g release 2 (11.2) but did not choose to mirror OCR during the upgrade

• Created only one OCR location during the Oracle Clusterware installation

In addition to mirroring OCR locations, you can also:

• Replace an OCR location if there is a misconfiguration or other type of OCR error, as described in the "Replacing an Oracle Cluster Registry Location" section.

• Repair an OCR location if Oracle Database displays an OCR failure alert in Oracle Enterprise Manager or in the Oracle Clusterware alert log file, as described in the "Repairing an Oracle Cluster Registry Configuration on a Local Node" section.

• Remove an OCR location if, for example, your system experiences a performance degradation due to OCR processing or if you transfer your OCR to RAID storage devices and choose to no longer use multiple OCR locations, as described in the "Removing an Oracle Cluster Registry Location" section.

# ocrconfig -add +asm_disk_group | file_name

# ocrconfig -repair -add file_name | -delete file_name | -replace
current_file_name -replacement new_file_name

Backing Up Oracle Cluster Registry

This section describes how to back up OCR content and use it for recovery. The first method uses automatically generated OCR copies and the second method enables you to issue a backup command manually:

• Automatic backups: Oracle Clusterware automatically creates OCR backups every four hours. At any one time, Oracle Database always retains the last three backup copies of OCR. The CRSD process that creates the backups also creates and retains an OCR backup for each full day and at the end of each week. You cannot customize the backup frequencies or the number of files that Oracle Database retains.

• Manual backups: Run the ocrconfig -manualbackup command on a node where the Oracle Clusterware stack is up and running to force Oracle Clusterware to perform a backup of OCR at any time, rather than wait for the automatic backup. You must run the command as a user with administrative privileges. The -manualbackup option is especially useful when you want to obtain a binary backup on demand, such as before you make changes to the OCR. The OLR only supports manual backups.

When the clusterware stack is down on all nodes in the cluster, the backups that are listed by the command ocrconfig -showbackup may differ from node to node. After you install or upgrade Oracle Clusterware on a node, or add a node to the cluster, when the root.sh script finishes, it backs up the OLR.

Listing Backup Files

Run the following command to list the backup files:

# ocrconfig -showbackup

On Windows:

C:\>ocrconfig -showbackup

The ocrconfig -showbackup command to displays the backup location, timestamp, and the originating node name of the backup files that Oracle Clusterware creates. By default, the -showbackup option displays information for both automatic and manual backups but you can include the auto or manual flag to display only the automatic backup information or only the manual backup information, respectively.

Run the following command to inspect the contents and verify the integrity of the backup file:

# ocrdump -backupfile backup_file_name

On Windows:

C:\>ocrdump -backupfile backup_file_name

You can use any backup software to copy the automatically generated backup files at least once daily to a different device from where the primary OCR resides.

The default location for generating backups on Linux or UNIX systems is Grid_home/cdata/cluster_name, where cluster_name is the name of your cluster. The Windows default location for generating backups uses the same path structure. Because the default backup is on a local file system, Oracle recommends that you include the backup file created with the OCRCONFIG utility as part of your operating system backup using standard operating system or third-party tools.

Restoring Oracle Cluster Registry

If a resource fails, then before attempting to restore OCR, restart the resource. As a definitive verification that OCR failed, run ocrcheck and if the command returns a failure message, then both the primary OCR and the OCR mirror have failed. Attempt to correct the problem using the OCR restoration procedure for your platform.

• Restoring the Oracle Cluster Registry on Linux or UNIX Systems

• Restoring the Oracle Cluster Registry on Windows Systems

Restoring the Oracle Cluster Registry on Linux or UNIX Systems

If you are storing the OCR on an Oracle ASM disk group, and that disk group is corrupt, then you must restore the Oracle ASM disk group using Oracle ASM utilities, and then mount the disk group again before recovering the OCR. Recover the OCR by running the command ocrconfig -restore.

Use the following procedure to restore OCR on Linux or UNIX systems:

1. List the nodes in your cluster by running the following command on one node:

   $ olsnodes
   

2. Stop Oracle Clusterware by running the following command as root on all of the nodes:

   # crsctl stop crs
   

   If the preceding command returns any error due to OCR corruption, stop Oracle Clusterware by running the following command as root on all of the nodes:

   # crsctl stop crs -f
   

3. If you are restoring OCR to a cluster file system or network file system, then run the following command as root to restore OCR with an OCR backup that you can identify in "Listing Backup Files":

   # ocrconfig -restore file_name
   

   After you complete this step, skip to step 8.

4. Start the Oracle Clusterware stack on one node in exclusive mode by running the following command as root:

   # crsctl start crs -excl
   

   Ignore any errors that display.

   Check whether crsd is running. If it is, stop it by running the following command as root:

   # crsctl stop resource ora.crsd -init
   

   Caution:

   Do not use the -init flag with any other command.

5. Restore OCR with an OCR backup that you can identify in "Listing Backup Files" by running the following command as root:

   # ocrconfig -restore file_name
   

   Notes:

   • If the original OCR location does not exist, then you must create an empty (0 byte) OCR location before you run the ocrconfig -restore command.

   • Ensure that the OCR devices that you specify in the OCR configuration exist and that these OCR devices are valid.

   • If you configured OCR in an Oracle ASM disk group, then ensure that the Oracle ASM disk group exists and is mounted.

   See Also:

   • Oracle Grid Infrastructure Installation Guide for information about creating OCRs

   • Oracle Database Storage Administrator's Guide for more information about Oracle ASM disk group management

6. Verify the integrity of OCR:

   # ocrcheck
   

7. Stop Oracle Clusterware on the node where it is running in exclusive mode:

   # crsctl stop crs -f
   

8. Begin to start Oracle Clusterware by running the following command as root on all of the nodes:

   # crsctl start crs
   

9. Verify the OCR integrity of all of the cluster nodes that are configured as part of your cluster by running the following CVU command:

   $ cluvfy comp ocr -n all -verbose
   

Restoring the Oracle Cluster Registry on Windows Systems

If you are storing the OCR on an Oracle ASM disk group, and that disk group is corrupt, then you must restore the Oracle ASM disk group using Oracle ASM utilities, and then mount the disk group again before recovering the OCR. Recover the OCR by running the command ocrconfig -restore.

Use the following procedure to restore OCR on Windows systems:

1. List the nodes in your cluster by running the following command on one node:

   C:\>olsnodes
   

2. Stop Oracle Clusterware by running the following command as a Windows administrator on all of the nodes:

   C:\>crsctl stop crs
   

   If the preceding command returns any error due to OCR corruption, stop Oracle Clusterware by running the following command as a Windows administrator on all of the nodes:

   C:\>crsctl stop crs -f
   

3. Start the Oracle Clusterware stack on one node in exclusive mode by running the following command as a Windows administrator:

   C:\>crsctl start crs -excl
   

   Ignore any errors that display.

   Check whether crsd is running. If it is, stop it by running the following command as a Windows administrator:

   C:\>crsctl stop resource ora.crsd -init
   

   Caution:

   Do not use the -init flag in any other command.

4. Restore OCR with the OCR backup file that you identified in "Listing Backup Files" by running the following command as a Windows administrator:

   C:\>ocrconfig -restore file_name
   

   Make sure that the OCR devices that you specify in the OCR configuration exist and that these OCR devices are valid.

   Notes:

   • If the original OCR location does not exist, then you must create an empty (0 byte) OCR location before you run the ocrconfig -restore command.

   • Ensure that the OCR devices that you specify in the OCR configuration exist and that these OCR devices are valid.

   • Ensure that the Oracle ASM disk group you specify exists and is mounted.

   See Also:

   • Oracle Grid Infrastructure Installation Guide for information about creating OCRs

   • Oracle Database Storage Administrator's Guide for more information about Oracle ASM disk group management

5. Verify the integrity of OCR:

   C:\>ocrcheck
   

6. Stop Oracle Clusterware on the node where it is running in exclusive mode:

   C:\>crsctl stop crs -f
   

7. Begin to start Oracle Clusterware by running the following command as a Windows administrator on all of the nodes:

   C:\>crsctl start crs
   

8. Run the following Cluster Verification Utility (CVU) command to verify the OCR integrity of all of the nodes in your cluster database:

   C:\>cluvfy comp ocr -n all -verbose
   

   See Also:

   Appendix A, "Cluster Verification Utility Reference" for more information about enabling and using CVU

Diagnosing Oracle Cluster Registry Problems

You can use the OCRDUMP and OCRCHECK utilities to diagnose OCR problems as described under the following topics:

• Using the OCRDUMP Utility

• Using the OCRCHECK Utility

Administering Oracle Cluster Registry with Oracle Cluster Registry Export and Import Commands

In addition to using the automatically created OCR backup files, you should also export the OCR contents before and after making significant configuration changes, such as adding or deleting nodes from your environment, modifying Oracle Clusterware resources, and upgrading, downgrading or creating a database. Do this by using the ocrconfig -export command, which exports the OCR content to a file format.

Using the ocrconfig -export command also enables you to restore OCR using the -import option if your configuration changes cause errors. For example, if you have unresolvable configuration problems, or if you are unable to restart Oracle Clusterware after such changes, then restore your configuration using the procedure for your platform.

Oracle Local Registry

In Oracle Clusterware 11g release 2 (11.2), each node in a cluster has a local registry for node-specific resources, called an Oracle Local Registry (OLR), that is installed and configured when Oracle Clusterware installs OCR. Multiple processes on each node have simultaneous read and write access to the OLR particular to the node on which they reside, regardless of whether Oracle Clusterware is running or fully functional.

By default, OLR is located at Grid_home/cdata/host_name.olr on each node.

Manage OLR using the OCRCHECK, OCRDUMP, and OCRCONFIG utilities as root with the -local option.

• You can check the status of OLR on the local node using the OCRCHECK utility, as follows:

  # ocrcheck -local
  
  Status of Oracle Cluster Registry is as follows :
          Version                  :          3
          Total space (kbytes)     :     262132
          Used space (kbytes)      :       9200
          Available space (kbytes) :     252932
          ID                       :  604793089
          Device/File Name         : /private2/crs/cdata/localhost/dglnx6.olr
                                     Device/File integrity check succeeded
  
          Local OCR integrity check succeeded
  

• You can display the content of OLR on the local node to the text terminal that initiated the program using the OCRDUMP utility, as follows:

  # ocrdump -local -stdout
  

• You can perform administrative tasks on OLR on the local node using the OCRCONFIG utility.

  • To export OLR to a file:

    # ocrconfig –local –export file_name
    

    Notes:

    • Oracle recommends that you use the -manualbackup and -restore commands and not the -import and -export commands.

    • When exporting OLR, Oracle recommends including "olr", the host name, and the timestamp in the name string. For example:

      olr_myhost1_20090603_0130_export
      

  • To import a specified file to OLR:

    # ocrconfig –local –import file_name
    

  • To manually back up OLR:

    # ocrconfig –local –manualbackup
    

    Note:

    The OLR is backed up at the end of an installation or an upgrade. After that time, you can only manually back up the OLR. Automatic backups are not supported for the OLR. You should create a new backup when you migrate the OCR from Oracle ASM to other storage, or you migrate the OCR from other storage to Oracle ASM.

    The default backup location for the OLR is in the path Grid_home/cdata/host_name.

  • To view the contents of the OLR backup file:

    ocrdump -local -backupfile olr_backup_file_name
    

  • To change the ORL backup location:

    ocrconfig -local -backuploc new_olr_backup_path
    

  • To restore OLR:

    # crsctl stop crs
    # ocrconfig -local -restore file_name
    # ocrcheck -local
    # crsctl start crs
    $ cluvfy comp olr
    
    

Upgrading and Downgrading the Oracle Cluster Registry Configuration

When you install Oracle Clusterware, it automatically runs the ocrconfig -upgrade command. To downgrade, follow the downgrade instructions for each component and also downgrade OCR using the ocrconfig -downgrade command. If you are upgrading OCR, then you can use the OCRCHECK utility to verify the integrity of OCR.

About Private Networks, Network Interfaces, and Network Adapters

Oracle Clusterware requires that each node is connected through a private network (in addition to the public network). The private network connection is referred to as the cluster interconnect.

Oracle only supports clusters in which all of the nodes use the same network interface connected to the same subnet (defined as a global interface with the oifcfg command). You cannot use different network interfaces for each node (node-specific interfaces). Refer to Appendix D, "Oracle Interface Configuration Tool (OIFCFG) Command Reference" for more information about global and node-specific interfaces. Table 2-5 describes how the network interface card (NIC), or network adapter and the private IP address are stored.

Consequences of Changing Interface Names Using OIFCFG

The consequences of changing interface names depend on which name you are changing, and whether you are also changing the IP address. In cases where you are only changing the interface names, the consequences are minor. If you change the name for the Public Interface that is stored in the OCR, then you also must modify the node applications for each node. Therefore, you must stop the node applications for this change to take effect.

https://metalink.oracle.com

Changing or Deleting a Network Interface From a Cluster Configuration File

Use the following procedure to change or delete a network interface.

1. Ensure that Oracle Clusterware is running on all of the cluster nodes by running the following command:

   olsnodes -s
   

   The output from this command should show that Oracle Clusterware is running on all of the nodes in the cluster. For example:

   ./olsnodes -s
   myclustera Active
   myclusterc Active
   myclusterb Active
   

2. Ensure that the replacement interface is configured and operational in the operating system on all of the nodes. To do thus, use the form of the ifconfig command for your platform. For example, on Linux, use:

   /sbin/ifconfig..
   

3. Add the new interface to the cluster as follows, providing the name of the new interface and the subnet address, using the following commands:

   $ oifcfg setif -global
   /interface_name/subnet/:cluster_interconnect
   

   Note:

   You can use wildcards in the preceding command example.

4. Ensure that the previous step completes. Then you can remove the former subnet as follows by providing the name and subnet address of the former interface:

   $ oifcfg delif -global /interface_name/subnet/
   

5. Verify the current configuration using the following command:

   oifcfg getif
   

   For example:

   $ oifcfg getif
   eth2 10.220.52.0 global cluster_interconnect
   eth0 10.220.16.0 global public
   

6. On all of the nodes, stop Oracle Clusterware by running the following command as the root user:

   # crsctl stop crs
   

7. When the cluster stops, you can deconfigure the deleted network interface in the operating system using the following command:

   $ ifconfig down
   

   At this point, the IP address from network interfaces for the old subnet is deconfigured from Oracle Clusterware. This command does not affect the configuration of the IP address on the operating system.

8. Restart Oracle Clusterware by running the following command on each cluster member node as the root user:

   # crsctl start crs
   

   The changes take effect when Oracle Clusterware restarts.

   If you use the CLUSTER_INTERCONNECTS initialization parameter, then you must update it to reflect the changes.

Changing the Network Adapter for the Interconnect

To change the network interface for the private interconnect (for example, eth1), you must perform the change on all nodes (globally). This is because Oracle currently does not support the use of different network interface cards in the same subnet for the cluster interconnect.

To change the network interface, perform the following steps:

1. Make sure that the Oracle Clusterware stack is up and running on all cluster nodes.

2. Use operating system commands (ifconfig, or the command for your system) to ensure that the new or replacement interface is configured and up on all cluster member nodes.

3. On a single node in the cluster add the new global interface specification:

   $ oifcfg setif -global interface_name/subnet:cluster_interconnect
   

   Note:

   You can use wild cards with the interface name. For example, oifcfg setif -global "eth*/192.168.0.0:cluster_interconnect is valid syntax. However, be careful to avoid ambiguity with other addresses or masks used with other cluster interfaces. If you use wild cards, then you see warning similar to the following:

   eth* 192.168.0.0 global cluster_interconnect
   PRIF-29: Warning: wildcard in network parameters can cause mismatch
   among GPnP profile, OCR, and system
   

   Legacy network configuration does not support wildcards; thus wildcards are resolved using current node configuration at the time of the update.

4. On a node in the cluster, use the ifconfig command to ensure that the new IP address exists.

5. Add the new subnet, with the following command, providing the name of the interface and the subnet address. The changes take effect when Oracle Clusterware restarts:

   $ oifcfg setif -global interface_name/subnet:cluster_interconnect
   

   See Also:

   Appendix D, "Oracle Interface Configuration Tool (OIFCFG) Command Reference" for more information about using the OIFCFG command

6. Verify the configuration with the oifcfg getif command.

7. Stop Oracle Clusterware on all nodes by running the following command as root on each node:

   # crsctl stop crs
   

   Note:

   With cluster network configuration changes, the cluster must be fully stopped; do not use rolling stops and restarts.

8. Assign the current network address to the new network adapter using the ifconfig command.

   As root user, issue the ifconfig operating system command to assign the currently used private network address to the network adapter intended to be used for the interconnect. This usually requires some downtime for the current interface and the new interface. See your platform-specific operating system documentation for more information about issuing the ifconfig command.

9. Update the network configuration settings on the operating system.

   You must update the operating system configuration changes, because changes made using ifconfig are not persistent.

   See Also:

   Your operating system documentation for more information about how to make ifconfig commands persistent

10. Remove the former subnet, as follows, providing the name and subnet address of the former interface:

    oifcfg delif -global interface_name/subnet
    

    For example:

    $ oifcfg delif -global eth1/10.10.0.0
    

    Caution:

    This step should be performed only after a replacement interface is committed into the Grid Plug and Play configuration. Simple deletion of cluster interfaces without providing a valid replacement can result in invalid cluster configuration.

11. Restart Oracle Clusterware by issuing the following command as root user on all nodes:

    # crsctl start crs
    

    You must restart Oracle Clusterware after running the oifcfg delif command, because Oracle Clusterware, Oracle ASM, and Oracle RAC continue to use the former subnet until they are restarted.