10 Managing Oracle Software and Applying Patches

Configure the Software Library

The first step is configuring the software library. When configuring the Software Library you specify the directory paths that the software library will use to store binary data, such as patches.

To configure the Software Library prior to using the Patch Interface of Database Control:

1. On the Database Home page in DB Control, click Software and Support.

2. In the Deployment Procedure Manager section, click the Deployment and Provisioning Software Library link.

   The Provisioning page appears.

3. Click the link for the Administration page.

4. Scroll down to the Software Library Configuration section at the bottom of the page. Click Add.

   The Add Software Library Location page appears.

5. In the Software Library directory location field, enter the name of the directory where the patches will be uploaded to the node, for example, /home/oracle. Click OK.

6. When the utility completes, you have completed the Software Library configuration.

About My Oracle Support Credentials

To download patches from My Oracle Support (formerly OracleMetaLink) using Enterprise Manager, you can give Enterprise Manager Database Control (Database Control) your login credentials so that it can log in to My Oracle Support automatically and search for patch releases. You must set these credentials before you can run the Patch Advisor in Database Control.

After you have configured the My Oracle Support credentials, Enterprise Manager automatically creates and schedules two jobs for your system:

• Refresh From Metalink

• OPatch Update (which requires the Software Library to be configured)

Refer to Oracle Database 2 Day DBA for instructions on setting your My Oracle Support credentials.

Start the Provisioning Daemon

The Procedure Manager allows you to view, edit, run, and monitor deployment procedures. Deployment procedures are best practices provided by Oracle for various provisioning tasks.

In order for the Deployment Procedure Manager to function properly, the Provisioning Daemon job should be running. This job is used to monitor the status of running deployment procedures. You can start or stop a job and also view the status of the job using the pafctl utility. You can find this utility in the $ORACLE_HOME/bin directory. The syntax for this utility is as follows:

pafctl { start | stop | status } [sysman_password] [interval]

The keywords for this command indicate the following:

• start— Starts the Provisioning Daemon job. This takes the interval value from the command line. If no interval is specified in the command line, you will have to enter a value, which is the interval (in minutes) with which to run the Provisioning Daemon job.

• stop— Stop the Provisioning Daemon job.

• status—Checks if the Provisioning Daemon job is running.

• sysman_password—You must specify the SYSMAN password to connect to the repository. If not specified at the command line, you will be prompted to enter the value.

• interval—This is an integer that specifies the interval (in minutes) for which to run the Provisioning Daemon job. The default interval is 3 minutes. To specify a value that is not the default value, you must use the start action. You will then be prompted for both the SYSMAN user password and the interval.

To start the Provisioning Daemon job, enter the following command at the operating system prompt"

$ pafctl start
Enter the repository user password:
Enter interval [default 3]:

Patching an Oracle RAC Database Using the Rolling Upgrade Method

In rolling patching, one node is shut down, the patch is applied to the node, and the node is brought back up. This is performed node by node, separately, until all the nodes in the cluster are patched. This is the most efficient means of applying an interim patch to an Oracle RAC, ASM, or Oracle Clusterware installation. By patching nodes individually, there is zero downtime for the cluster database because at least one instance is available at all times on a different node.

While most patches can be applied in a rolling fashion, some patches cannot be applied in this fashion. The README file for the patch indicates whether or not you can apply the patch using the rolling patch method. If the patch cannot be applied using the rolling patch method, then you must use either "Minimum Downtime Patching" or the Patch Oracle RAC Database (All Nodes Upgrade) method to apply the patch.

To patch a non-shared Oracle home using the rolling patch method:

1. On the Database Home page, click Software and Support.

2. On the Software and Support page, select Deployment Procedures.

   The Deployment Procedure Manager page appears.

3. Select the option Patch Oracle RAC Database (Rolling Upgrade) then click the Schedule Deployment button.

   The Select Patches page appears.

4. Under the heading RAC Updates, click the Add Patches button.

   The Search and Select Patches page appears.

5. Select either the Search My Oracle Support option to check for patches online or the Search Software Library option to search for patches manually added to the Software Library, as described in "Uploading Patches to the Software Library".

   Enter parameters to limit the Search performed, if desired, then click Go.

6. Check the box next to the patches you want to install, then click Select.

   
   Description of the illustration patch_selecting.gif
   
   

   The Select Patches page appears again, with the selected patches displayed in the RAC Updates section.

7. After you have selected all the patches you want to install, click Next.

   The Credentials and Schedule page appears.

8. Choose to either use the Preferred Credentials or to override the Preferred Credentials. If you choose to override the Preferred Credentials, then enter the username and password of the operating system user that owns the Oracle Home being patched, for example, oracle. Check the option to save the username and password as Preferred Credentials.

   In the Schedule section of the page, choose the option One Time (Later) and then specify the time when you want the patch to be applied to your system, or choose the option One Time (Immediately) to have the patch applied right away.

   After you have specified the schedule for the patch, click Next. The Review page appears.

9. Review the information on this page regarding the updates that will be made to your system. At the bottom of the page, under the heading Warning, read the warning statement about not being able to access DB Control during the patching operation, then check the Select the check box to accept this warning and continue box.

   
   Description of the illustration patch_warning.gif
   
   

   After you have checked this box you can click Finish to submit the patching operation.

10. To view the progress of the patching operation, select Procedure Completion Status on the Deployment Procedure Manager page.

Preparing to Use OPatch

Before you apply the patch to your Oracle RAC database, your ASM installation, or to your Oracle Clusterware installation, there are a few steps to perform:

• Checking the ORACLE_HOME Environment Variable

• Performing a Backup Prior to Patching Software

• Staging the Patch on Each Node

• Updating the PATH Environment Variable

• Configuring SSH User Equivalency

Minimum Downtime Patching

In minimum downtime patching, one set of nodes is shut down and the patch is applied to those nodes. After the first set of nodes has been patched, the second set of nodes is shut down. The first set of nodes is then restarted and the patch is applied to the second set of nodes. After the patch has been applied to the second set of nodes, those nodes are restarted. This method leads to less downtime for Oracle RAC, compared to having all the nodes shut down at the same time.

When you use the minimum downtime patching method, the following actions occur:

• The local node is always patched first.

• The local node is used as a base to patch the other nodes.

• The user is prompted for the set of nodes to patch first from the remaining nodes.

• For each node in this first set, the user is asked to stop the instance and then the patch is propagated to that node before continuing to the next node. When the initial set of nodes has been patched, the user is asked to shut down the remaining nodes.

• After the local node is patched, the patch is propagated to the last set of nodes and the inventory is updated. The last instances are stopped on the remote nodes. You can then start up the patched nodes (the first set of nodes) before patching the remaining nodes.

To apply a patch to your cluster database using the minimum downtime method:

1. Change to the directory where the unzipped patch is staged on disk, for example:

   $ cd Oracle_home/EMStagedPatches/4519934/4519934
   

2. Stop all user applications that use the Oracle RAC home directory for the group of nodes being patched. For example, to stop Enterprise Manager Database Control on the local node, use the following command, where Oracle_home is the home directory for your Oracle RAC installation:

   $ Oracle_home/bin/emctl stop dbconsole
   

3. Shut down all Oracle RAC instances on the local node. To shut down an instance for an Oracle RAC database, enter a command similar to the following example, where Grid_home is the home directory for your Oracle grid infrastructure installation, orcl is the name of the database, and orcl1 is the name of the instance:

   $ CRS_home/bin/srvctl stop instance -d orcl -i "orcl1"
   

4. Make sure the ORACLE_HOME environment variable points to the software directory you want to patch, for example:

   $ echo $ORACLE_HOME
   /u01/app/oracle/11gR2/db_1
   

5. Use the following command from within the patch directory:

   $ opatch apply -minimize_downtime
   

   If you run the OPatch command from the directory where the patch is staged on disk, you do not need to specify the patch ID.

   OPatch asks if you are ready to patch the local node. After you confirm that the Oracle RAC instances on the local node have been shut down, OPatch applies the patch to the Oracle home directory on the local node. You are then asked to select the next nodes to be patched.

6. After you shut down the Oracle RAC instances on the other nodes in the cluster, you can restart the Oracle RAC instance on the local node. Then, instruct OPatch that you are ready to patch the remaining nodes.

7. After all the nodes have been patched, restart the Oracle RAC instances on the other nodes in the cluster. The following command shows how to start the orcl2 instance for the Oracle RAC database named orcl:

   $ CRS_home/bin/srvctl start instance -d orcl -i "orcl1"
    
   

8. Verify that all the nodeapps were restarted on the nodes in the cluster.

   $ crsctl check crs
   

   If any of the node applications did not restart, use the SRVCTL utility to restart them. For example, you can use commands similar the following to restart the listener on the racnode1 node, where Grid_home is the home directory of your Oracle grid infrastructure installation:

   $ CRS_home/bin/srvctl start listener -n racnode1
    
   

9. Run any post-patch scripts that are mentioned in the patch instructions, for example:

   $ sqlplus /nolog 
   SQL> connect sys/password@orcl1 AS SYSDBA
   SQL> @Oracle_home/cpu/CPUOct2007/catcpu.sql
   SQL> @Oracle_home/rdbms/admin/utlrp.sql
   SQL> exit
   

Updating the Node List for OPatch

If OPatch does not automatically detect Oracle RAC or its nodes, investigate the contents of the inventory and ensure they are complete.

To update the node list for OPatch:

If the list of nodes for your cluster is not complete, you can update it by using Oracle Universal Installer and the -updateNodeList flag, as demonstrated in the following example:

Oracle_home/oui/bin/runInstaller -updateNodeList 
ORACLE_ HOME=/u01/app/oracle/11gR2/db_1 
CLUSTER_NODES=racnode1,racnode2,racnode3 -noClusterEnabled

About OPatch Log and Trace Files

Logging and tracing is a common aid for debugging. OPatch maintains logs for all apply, rollback, and lsinventory operations. The log files are located in the Oracle_home/cfgtoollogs/opatch directory. Each log file will be tagged with the time stamp of the operation. Log files are named as opatch_mm-dd-yyyy_hh-mm-ss.log, where mm-dd-yyyy is the current date and hh-mm-ss is the current time. Each time you run OPatch, a new log file is created.

For example, if a log file is created on May 17, 2007 at 11:55 PM, then it will be named as follows:

opatch_05-17-2007_23-55-00.log

OPatch also maintains an index of the commands processed by OPatch and the log files associated with it in the opatch_history.txt file located in the Oracle_home/cfgtoollogs/opatch directory. A sample of the opatch_history.txt file is as follows:

Date & Time : Tue Apr 26 23:00:55 PDT 2007
Oracle Home : /u01/app/oracle/11gR2/db_1/
OPatch Ver. : 11.2.0.0.0
Current Dir : /scratch/oui/OPatch
Command     : lsinventory
Log File    : 
/u01/app/oracle/11gR2/db_1/cfgtoollogs/opatch/opatch-2007_Apr_26_23-00-55-PDT_Tue.log

Resolving the "Not a valid patch area" Error

You might get this error if the directory that the OPatch utility is using to do the patch does not match the template for what it is checking, or if the OPatch utility is run from an invalid directory.

The Patch_Shiphome directory should have the following structure:

• An etc directory that has the metadata files

• A files directory that has the patch files

• The etc/config/inventory.xml file and the actions.xml file under the same directory

To resolve the "Not a valid patch area" error:

• Perform one of the following actions:

  1. Remove the patch shiphome directory and re-create it with the proper structure (by extracting the files again).

  2. Start the OPatch utility from the directory where the patch to be installed has been unzipped and staged on disk.

  3. Use the following command when starting OPatch:

     opatch apply /Patch_Shiphome
     

     where Patch_Shiphome is the location where the patch has been staged on disk.

Resolving the "Unable to remove a partially installed interim patch" Error

If the patching process is interrupted, you might get the error "Unable to remove a partially installed interim patch" when you try to install the patch a second time.

To resolve the partially installed patch error:

1. Ensure that the environment variable ORACLE_HOME is set to the Oracle home directory you are attempting to patch.

2. Go to the Oracle_home/.patch_storage/patch-id_timestamp directory and run the restore.sh script (or restore.bat on Windows platforms) as follows:

   Oracle_home/.patch_storage/patch-id_timestamp/restore.sh
   

3. On Linux and UNIX systems, use the Oracle_home/.patch_storage/patch-id_timestamp/make.txt file (if available) to modify your operating system environment, as follows:

   /bin/sh make.txt
   

4. Attempt to apply the patch again.