Oracle 9i RAC Single Node Installation on Red Hat Linux 7.1Steve Bourgeois,
Polaris Database Systems
I've wanted to experiment with Oracle9i Real Application Clusters (RAC) for quite some time now. After working with Oracle Parallel Server (the predecessor to RAC) starting with version 6.2 in 1992 I haven't been fortunate enough to work with a client implementing 9i RAC (as of this writing). As someone who is always interested in learning about and tinkering with latest Oracle technology, I decided to setup a low cost environment where I could experiment with RAC. Seeking to utilize a home-built Intel-based PC as the server platform, I download copies of Red Hat Linux 7.1 and Oracle9i Database Release 1 (9.0.1) to build my environment. This document outlines the steps to install and configure Oracle 9i RAC on a single Linux node. It assumes the reader has a working knowledge of Oracle RDBMS, Oracle RAC, and Linux technology. All of the sample scripts and files referenced in this document can be downloaded here. After doing some research, I wasn't able to find a frugal solution to this problem. The simplest approach would be to use a supported solution like the Network Appliance Filer storage appliance, but I don't happen to have one of those hanging around in my spare computer parts closet. I looked into several open source software solutions like NBD, ENBD, DRBD, and PVFS. Unfortunately, each solution either wasn't designed to support multiple node concurrent access or didn't allow support for parallel concurrent writes. Interestingly, the author of DRBD indicates that adding this support wouldn't require much effort. The most promising software solution was Sistina's Global File System (GFS) for Linux. Apparently, GFS was an open source project until August, 2001, when Sistina decided to make it a commercial product. The GFS open source developers countered by starting the OpenGFS project, which does deserve checking out at some point. Finally, it is possible to use common SCSI controllers and disks to create a dual-hosted storage solution. It is possible to put multiple SCSI controllers on the same SCSI bus with a Y cable and external SCSI bus termination. The Linux High Availability HOWTO: Attaching Shared Storage has a nice section that details how to do this.
3. Red Hat Linux Configuration I chose Red Hat Linux 7.1 because it is easily downloaded via the internet (unlike SuSE Linux) and is a certified operating system for Oracle 8.1.7 and 9.0.1. If you need the Red Hat installation media, you can download iso images directly from RedHat or from a mirror site and burn your own copy. If you don't have access to a broadband internet connection or a cdrom burner, you can buy the cdrom set from Amazon.com or have a friend burn a copy for you. If you need help installing Red Hat Linux, the installation documentation can be found at RedHat. I chose a Workstation install from the Red Hat installer, which installs the X Windows software and development tools necessary for a successful Oracle installation. I installed the following Red Hat 7.1 modules and patches to the base Red Hat 7.1 install. If you patch your Red Hat installation, please be sure to install the kernel and glibc rpm files appropriate for your cpu architecture. For this installation, I used the i686 rpm files since my CPU is a Pentium IV.
Before patching the kernel, it is highly recommended to create a boot floppy as user root with the current kernel with the mkbootdisk command:
Once the rpm's are downloaded to a directory on the Linux server, they should be checked for corruption during download before being installed. If a rpm file does not return the message "md5 OK", then the rpm is corrupt and should be downloaded again :
The rpm's can be installed as user root with the following commands:
Once the rpm's are installed, the /etc/lilo.conf file should be updated to include an entry for the new kernel. The lilo.conf file from my system can be found here. Note: My motherboard's onboard ATA controller is not recognized properly by Linux, so I installed a separate Promise ATA-133 controller card. Therefore, the append="ide2=0x9400,0x9802 ide3=0x9c00,0xa002" line may not be necessary in your lilo.conf file. If you are using ATA devices and don't have an additional ATA controller installed, your boot and root partitions would be on hard drives hda, hdb, hdc, or hdd. If you are using SCSI devices, your boot and root partitions would be on hard drives sda, sdb, sdc, sdd, etc. Once the /etc/lilo.conf file is updated, the boot loader should be installed and the machine should be rebooted with the new 2.4.9-34 kernel:
I also installed some networking services rpm's as user root with the following commands:
Once these modules are installed the rexec, rlogin, rsh, telnet, and wu-ftpd files in the /etc/xinetd.d directory should be edited such that "disable" option is set to "no". Once the changes are complete, the xinetd daemon should be restarted.
4. Oracle 9i Pre-Installation Steps We must perform a few steps before launching the Oracle Universal Installer:
As the root user, I set the SHMMAX (the maximum allowable size of a shared memory segment) parameter to be 2 GB:
This change is not persistent across system reboots, so this command should be added to the end of the /etc/rc.d/rc.local script to ensure the change is made at system startup. The text I appended to the rc.local file in my environment can be found here. The oracle account is the Oracle software owner, and the OSDBA group is the privileged operating system group. They can be created as user root with the following commands:
As user root, set the password for the oracle account. You will be prompted for the new password and confirmation of that password.
The Oracle Cluster Manager software uses the watchdog timer to reboot the machine when the /dev/watchdog device isn't written to within a given timeout period. It should be configured as the root user with the following commands:
Loading the softdog module is not persistent across system reboots, so the insmod command should be added to the end of the /etc/rc.d/rc.local script to ensure the module is loaded at system startup. The text I appended to the rc.local file in my environment can be found here. You can verify that the softdog module is loaded by executing the lsmod command and looking for a "softdog" entry.
After the Oracle software is installed, the root.sh script that is generated attempts to create the RAC configuration file srvConfig.loc in this directory. Unfortunately, the OUI doesn't create this directory as part of the installation process. This appears to be an Oracle bug, but we can work around it by creating the directory as the root user with the following commands:
Since this will be a single node RAC environment, it really isn't necessary for the database to reside on raw devices. Since a clustered file system for Linux is not yet released (as of this writing), I decided to create an environment as realistic as possible by using raw devices. Although I didn't install every Oracle option, I did write the creation scripts to create a disk partition for each tablespace or device that might be asked for during the installation process. Since the database will reside on raw devices, each partition is 1 MB larger than the corresponding tablespace size to account for overhead. The file use, file size, disk partition, and raw device layout is shown below:
Since I needed to create 19 partitions and don't happen to be using LVM software, I simply created a single large extended partition in which I could create 19 logical partitions. The first step was to manually create block devices /dev/hdf17 - /dev/hdf23 with the mknod command since they aren't created as part of the default Linux installation. Please note that it is extremely important to use the correct major and minor numbers for the disk drive and partition numbers in your environment. In my environment, I am using disk hdf, so I can find the major and minor numbers for /dev/hdf17 by checking the values for /dev/hdf16 with the ls command:
Based on this result, I know that I need to create /dev/hdf17 with major 33 and minor 81, /dev/hdf18 with major 33 and minor 82, etc. The script that I used to create the block devices for my environment can be found here. The commands to create the devices for my environment would be:
The next step is to actually create the logical partitions. Although they can be created interactively with fdisk, I wrote a script to create the logical partitions on disk hdf. My script can be found here. The final step is to bind the raw Linux character devices to the logical partitions (block devices) we just created with the raw command and set file permissions and ownership. For example, to bind the first raw device /dev/raw/raw1 to block device /dev/hdf5:
The script I used to bind all of my raw devices can be found here. This change is not persistent across system reboots, so all of the bind commands should be added to the end of the /etc/rc.d/rc.local script to ensure the bindings are done at system startup. The text I appended to the rc.local file in my environment can be found here. You can verify the bindings by running the raw command in query mode:
You can optionally create symbolic links to the raw devices to give your Oracle database files meaningful names. The script that I used to create symbolic links in /u01/oradata/rac to the raw devices can be found here. The map file tells the DBCA which raw devices to use for each of the typical tablespaces created during database creation. It is simply a list of tablespace name variables and the corresponding raw device. The DBCA raw device map file for my environment can be found here. Oracle 9i uses the Blackdown JDK. Download the JDK distribution from Blackdown.org. As user root, extract the distribution from the archive:
This will create directory jdk118_v3, which should be moved to the /usr directory as user root:
When the Oracle Universal Installer prompts you for for the JDK location (in a later step), the directory /usr/jdk118_v3 should be used. Set the Oracle environment variables by adding the appropriate entries to the startup file for the oracle user. In my environment I added the following entries to the $HOME/.bashrc file since I was using bash as my shell:
These environment variables will be set automatically the next time you login as the oracle user. The .bashrc file for the oracle user can also be sourced manually with the following command:
5. Install the Oracle 9i RAC Software The Oracle Universal Installer (OUI) requires an X Windows interface to perform an interactive installation. The OUI can be run directly on the database server, or on any X Windows-capable computer.
The default cdrom mount point for Red Hat Linux is /mnt/cdrom. The cdrom can be mounted as user root with the following command:
Since the Oracle9i distribution contains multiple cdrom's, the OUI should be NOT be run from the command line where the current working directory is within the file system of the mounted Oracle9i cdrom. Otherwise, it will not be possible to eject the current Oracle9i cdrom when the OUI prompts for the next Oracle9i cdrom. The Oracle Universal Installer can be run as user oracle with the following command:
6. Configure and Start Oracle Cluster Manager Processes According to the Oracle9i Real Application Clusters Installation and Configuration Release 1 (9.0.1) manual, the Oracle Cluster Manager is:
Edit the $ORACLE_HOME/oracm/admin/nmcfg.ora file as user oracle to contain the following entries. The nmcfg.ora file from my environment can be found here.
I quickly found that the OCM cluster heartbeat daemon (watchdogd) is a bit aggressive with regards to shutting down my server. When the system was under heavy load (during database creation), I would see ping came too late messages in the $ORACLE_HOME/oracm/log/wdd.log file followed by the inevitable Shutting down the entire node... To give myself a little more breathing room, I increased the margin time for the watchdogd daemon to 30 seconds by adding the -m 30000 flag to the watchdogd entry in the $ORACLE_HOME/oracm/admin/ocmargs.ora file as user oracle. The ocmargs.ora file from my environment can be found here.
Please note that while this may be an appropriate workaround on a system you are experimenting with, the server can still be rebooted by watchdogd. Oracle Metalink note 166830.1, Setting up Real Application Cluster (RAC) environment on Linux - Single node, provides a nice overview of the steps to modify the Linux watchdog timer to skip the reboot completely. Run the ocmstart.sh script as user root with the following commands:
The Global Services Daemon allows the srvctl utility to perform system management tasks. To initialize the raw device (specified in the /var/opt/oracle/srvConfig.loc file), execute the following command as user oracle:
Execute the gsd program as user oracle with the following command:
These commands can be appended to the /etc/rc.d/rc.local script to automate the execution of these commands at system startup. The text I appended to the rc.local file in my environment can be found here. 7. Configure the Oracle 9i Net Listener
Run the Oracle Net Configuration Assistant as user oracle with the following command:
If the OCM processes are not running (see Configure and Start Oracle Cluster Manager Processes), you may see the following error returned after executing netca: "CM:CmThreadAttach: socket not connected (error=2) cannot allocate memory for -1 nodes" You can verify that the OCM processes are running by executing the ps command and looking for "oranm" and "oracm" entries:
We are now ready to configure the Oracle 9i Net Listener. 8. Oracle RAC Database Configuration and Creation We are now ready to configure and create a RAC database.
At this point, we are ready to generate the scripts to create our RAC database. For simplicity, I decided to use the Oracle Database Configuration Assistant (DBCA) to generate the scripts. The scripts I used to create my environment can be found here. The first step is to set the DBCA_RAW_CONFIG environment variable to point to the DBCA raw device map file created in Oracle 9i Pre-Installation Steps. To set DBCA_RAW_CONFIG and run the DBCA, execute the following commands as user oracle:
Because of the problems with the watchdogd daemon I described in Oracle 9i Pre-Installation Steps, I temporarily disabled RAC support to work around the many system reboots I experienced during the database creation process. To temporarily disable RAC support, execute the following commands as user oracle to disable RAC support and relink the oracle binary:
We are finally ready to execute the database creation scripts generated in Oracle RAC Database Configuration and Creation! I modified the DBCA-generated scripts to reduce the memory footprint of the SGA, the postDBCreation.sql script to create symbolic links to the init.ora files for each instance, and created the postDBCreation2.sql script to create the redo thread and the undo tablespace for the second RAC instance, configure the server parameter file (SPFILE) for RAC, and create an Oracle password file for the second RAC instance. The scripts I used to create my environment can be found here. Once the database creation is complete, RAC support can be re-enabled by executing the following commands as user oracle:
The instance names in my environment were rac1 and rac2. Execute the following commands to start up each RAC instance in shared mode:
In this document I have outlined the steps I took to install Oracle 9i RAC on an Intel PC with Red Hat Linux 7.1. It proved to be an interesting and educational experience due to the number of installation and configuration issues that I had to work around. Of course, we all have to deal with challenging environments when working with Oracle. If you have any questions or comments about this document or want to report any errata or omissions, please feel free to email me.
|
