Oracle Database Administration Blog » RAC

Installing Oracle 11g RAC on virtual servers using VMware

mudhalvan — Wed, 20 Jul 2011 02:14:22 +0000

Installing Oracle 11g RAC on virtual servers using VMware.

This post describes the installation of Oracle 11g RAC on virtual servers.Hardware
The following hardware was used for the installation:

Operating system	Processor	Memory	HDD’s
Windows XP SP2 (32 bit)	Intel P4 3Ghz	3Gb	1 x 160Gb 7200rpm & 1x 250Gb 7200rpm. Both on Ultra ATA Controllers

You’ll need at least 3Gb of RAM for this installation as the virtual machines will be configured with 1Gb of RAM each, leaving 1Gb for the host operating system. The virtual machines further requires a total of 40Gb of storage.

Software

General

For obvious reason this installation should never be used for Production or Development purposes. This type of environment is however extremely helpful if you want to learn and understand Oracle RAC and you do not have access to traditional hardware resources.

Virtual Machines

I used VMware Server (1.0.3) to facilitate my virtual machines. If you have not done so already download the software here. You’ll need to register to get an activation code, but it is a free product.

Configure your first virtual machine:

Install VMware Server and launch.
Select ‘local host’ and ‘OK’ to close the pop-up window.
Select ‘New Virtual Machine’ from the VMware Server Console to create the hardware specification for your first virtual machine.
This will launch the ‘New Virtual Machine Wizard’. ‘Next’ to continue.
Select ‘Custom’ to allow you to customize the virtual machine. ‘Next’ to continue.
I’ve settled for Oracle Enterprise Linux 4.5 (which is a rebranded version of Red Hat Enterprise Linux ), therefore select ‘Linux’ as the Guest Operating System and ‘Red Hat Enterprise Linux 4’ as the Version. Oracle Enterprise Linux 5 is not yet supported on VMware Server. ‘Next’ to continue.
Name your virtual machine and select a location to store the files.
I’ve named the virtual machines: EL4_11gRAC1 and EL4_11gRAC2. We will be creating 2 virtual machines with shared storage. You need to create separate directories for your virtual machines. Also create a separate directory for your shared storage. I’ve created them as follows:

Virtual Machine 1 E:\Virtual Machines\RAC\rac11g\rac1

Virtual Machine 2 E:\Virtual Machines\RAC\rac11g\rac2

Shared Storage E:\Virtual Machines\RAC\rac11g\share
Set Access Rights. Deselect ‘Make this virtual machine private’. ‘Next’ to continue.
Keep the default Startup/Shutdown Options (Run this virtual machines as the user that powers on the virtual machine). ‘Next’ to continue.
Keep the Processor Configuration default (One). ‘Next’ to continue.
Allocate 1Gb of memory for your virtual machine. ‘Next’ to continue.
Keep the default network type (bridged networking). ‘Next’ to continue.
Keep the default I/O Adapter Type (LSI Logic). ‘Next’ to continue.
Keep the default setting to create a new virtual disk. ‘Next’ to continue.
Keep the default disk type (SCSI). ‘Next’ to continue.
Specify a disk capacity of 20Gb for your local disk. We need enough space for the Linux installation, Oracle Clusterware software, Oracle Database software and a staging area. Make sure to deselect ‘Allocate all disk space now’ and select ‘Split disk into 2Gb files’.
‘Next’ to continue.
Specify a filename for your virtual disk (localdisk.vmdk) and create it in the directory for your first virtual machine (E:\Virtual Machines\RAC\rac11g\rac1). ‘Next’ to continue.
Edit your virtual machine settings to add the shared disks and an additional network card. Select ‘Edit virtual machine settings’ from the Commands section on the VMware Server Console. ‘Next’ to continue.
This will bring up a settings window. Select ‘Add’ to add new hardware. This will launch the ‘Add New Hardware’ wizard.
Select ‘Ethernet Adapter’ to add an additional network card. We will use this to configure the private interconnect. ‘Next’ to continue.
This time around select ‘Host-only’ as the network type. ‘Next’ to continue.
Launch the New Hardware Wizard again to add the following SCSI disks:

Purpose Size Virtual device node

Database files 3Gb SCSI 1:0

Database files 3Gb SCSI 1:1

Database files 3Gb SCSI 1:2

Oracle Cluster Registry 300Mb SCSI 1:3

Voting Disk 300Mb SCSI 1:4
Select ‘Hard Disk’ to add a new harddisk. ‘Next’ to continue.
Select ‘Create a new virtual disk’. ‘Next’ to continue.
Select ‘SCSI’ as the disk type. ‘Next’ to continue.
Allocate disk space as per the table in step 22. This time around keep the default options (‘Allocate all disk space now’ selected and ‘Split disk into 2Gb files’ deselected). ‘Next’ to continue.
Give the new disk a descriptive name (datadisk1.vmdk) and save it to the shared disk folder (E:\Virtual Machines\RAC\rac11g\share ). ‘Next’ to continue.
Once the new disk has been created, select it and click on the ‘Advanced’ button to edit. ‘Next’ to continue.
Change the Virtual device node as per the table in step 22 and change the Mode to ‘Independent’ and ‘Persistent’. ‘Next’ to continue.
Repeat steps 23 to 29 until all the disks in the table in step 22 have been configured.
Once all the disks have been added edit the virtual machine configuration file (E:\Virtual Machines\RAC\rac11g\rac1\Red Hat Enterprise Linux 4.vmx), but first close the Virtual Machine Settings screen to save the changes to the configuration file.Add the parameters in bold blue below to the configuration file in order to enable disk sharing:config.version = “8″
virtualHW.version = “4″
scsi0.present = “TRUE”
scsi0.virtualDev = “lsilogic”
memsize = “768″
scsi0:0.present = “TRUE”
scsi0:0.fileName = “rac1_localdisk.vmdk”
ide1:0.present = “TRUE”
ide1:0.fileName = “auto detect”
ide1:0.deviceType = “cdrom-raw”
floppy0.present = “FALSE”
Ethernet0.present = “TRUE”
displayName = “EL4_11gRAC1″
guestOS = “rhel4″
priority.grabbed = “normal”
priority.ungrabbed = “normal”
disk.locking = “FALSE”
diskLib.dataCacheMaxSize = “0″
diskLib.dataCacheMaxReadAheadSize = “0″
diskLib.dataCacheMinReadAheadSize = “0″
diskLib.dataCachePageSize = “0″
scsi1.sharedBus = “virtual”
scsi1.present = “TRUE”
scsi1:0.present = “TRUE”
scsi1:0.fileName = “E:\Virtual Machines\rac\rac11g\share\datadisk1.vmdk”
scsi1:0.mode = “independent-persistent”
scsi1:0.deviceType = “disk”
scsi1:1.present = “TRUE”
scsi1:1.fileName = “E:\Virtual Machines\rac\rac11g\share\datadisk2.vmdk”
scsi1:1.mode = “independent-persistent”
scsi1:1.deviceType = “disk”
scsi1:2.present = “TRUE”
scsi1:2.fileName = “E:\Virtual Machines\rac\rac11g\share\datadisk3.vmdk”
scsi1:2.mode = “independent-persistent”
scsi1:2.deviceType = “disk”
scsi1:3.present = “TRUE”
scsi1:3.fileName = “E:\Virtual Machines\rac\rac11g\share\ocrdisk.vmdk”
scsi1:3.mode = “independent-persistent”
scsi1:3.deviceType = “disk”
scsi1:4.present = “TRUE”
scsi1:4.fileName = “E:\Virtual Machines\rac\rac11g\share\votingdisk.vmdk”
scsi1:4.mode = “independent-persistent”
scsi1:4.deviceType = “disk”
scsi1.virtualDev = “lsilogic”
ide1:0.autodetect = “TRUE”
Ethernet1.present = “TRUE”

You have now successfully configured your first virtual machine. We’ll clone this at a later stage to provide for the second virtual machine.

Install Oracle Enterprise Linux

You are now ready to install Linux on the virtual machine. If you have not done so already, download the Oracle Enterprise Linux 4 (Update 5) files from hereand unzip. Once unzipped you should have the following cd image files:
Enterprise-R4-U5-i386-disc1.iso
Enterprise-R4-U5-i386-disc2.iso
Enterprise-R4-U5-i386-disc3.iso
Enterprise-R4-U5-i386-disc4.iso

Install Linux:

To start the installation select ‘Edit virtual machine settings’ from the main VMware Server Console. Select the CD-ROM drive. Under ‘Connection’ select ‘Use ISO image’ and browse to and select the ISO image for the Linux installation disk 1 (Enterprise-R4-U5-i386-disc1.iso).’OK’ to continue.
Select ‘Start this virtual machine’ to start it up.
The virtual machine will now boot from the CD-ROM and start the Linux installer. Press ‘Enter’ to select to install in graphical mode. Select ‘Skip’ on the following screen to skip media testing.
The graphical installer will now launch. ‘Next’ to continue.
Make your language selection. ‘Next’ to continue.
Select your keyboard configuration. ‘Next’ to continue.
Select ‘Manually partition with Disk Druid’. ‘Next’ to continue.
Select ‘Yes’ for all warning messages that indicate the partition table was unreadable.
Partition local disk
Partition the local disk (/dev/sda) as follows:

Mount point Size

/swap 2048Mb

/tmp 1024Mb

/ 4096Mb

/u01 13312Mb

To partition the local disk select the free space on /dev/sda and select ‘Edit’ to add partitions. Add all partitions as in the table above to /dev/sda. Do not configure the other disks at this stage. Select ‘Next’ to continue once done.
Keep the default boot loader configuration. ‘Next’ to continue.
Network configuration
Configure the network as follows:

Hostname Network device Network type IP address

rac1.startoracle.com eth0 Public 192.168.1.50

eth1 Private 10.10.10.50

Please note: Configure the public IP address for your virtual machines so that they are on the same network (192.168.1.1 in my case) as your host machine. You can also change the domain name (startoracle.com in my case) to your own. Remember to make these changes throughout the rest of the installation.

Tick both network devices to be ‘Active on boot’:

Select each device and select ‘Edit’ to edit advanced settings:
Deselect ‘Configure using DHCP’
Enter the IP address as per the table above.
Enter the Netmask: 255.255.255.0

Repeat for interface eth1:

Set the hostname: rac1.startoracle.com
Set the Gateway: 192.168.1.1
Set the Primary DNS: 192.168.1.1
‘Next’ to continue when done.
Disable the firewall and SELinux. ‘Next’ to continue.
Select additional language support. ‘Next’ to continue.
Select your time zone. ‘Next’ to continue.
Set your root password. ‘Next’ to continue.
Select ‘Customize software packages to be installed’. ‘Next’ to continue.
Package Group Selection
Keep all the default selected package groups and add the following groups:
– Legacy Software Development
– Graphical Internet (to add Firefox)
– Development Tools
– System Tools. Also select ‘Details’ for this group and add the 3 oracleasm packages and the sysstat package.
‘Next’ to continue.
Select ‘Next’ to start the installation.
During the installation you’ll be prompted to insert the installation cd’s. When this happens simply right-click on the CD-ROM icon in the bottom right corner and select the appropriate ISO image file. Select ‘Ok’ to continue the installation.
Reboot once installation is done.
After the reboot, finalize the setup. ‘Next’ to skip the Welcome screen.
Accept the license agreement. ‘Next’ to continue.
Check the date and time and set if incorrect. ‘Next’ to continue.
Check and set display. ‘Next’ to continue.
Setup a system user (not the oracle user). You can skip this. ‘Next’ to continue.
Additional CDs. No action required. ‘Next’ to continue.
Select ‘Next’ to complete the installation.
Ensure that you are using the SMP kernel
To ensure you are using this kernel open a terminal window as the root user and enter the following command: uname -rIf the following output is displayed then the correct kernel is used:

If any other kernel is used edit the /boot/grub/grub.conf file and set the default option to boot the SMP kernel. The first option is 0 and is usually set as the default boot option, the second option is 1 etc. Each kernel boot option starts with the ‘title’ line.
At this point power down the virtual machine, edit the virtual machine settings to reset the CD-ROM to ‘Use physical drive’ and ‘Auto detect’. Restart the virtual machine.
Install VMware tools in the virtual machine. This is necessary so that the time between the host and virtual machines stay synchronized. If the time between the RAC nodes are out of sync installation to the remote node will fail.To install VMware tools, log in as root in the virtual machine and select ‘Install VMware Tools…’ from the VM menu in the VMware Server Console.

Confirm installation. ‘Install’ to continue.

This will mount an image on the cd drive with the installation package.

Open the cd folder and double-click the VMwareTools package to install.

Open a terminal window and execute: /usr/bin/vmware-config-tools.pl to configure VMware Tools. Select the screen resolution during configuration.Configure VMware Tools to start every time you login. On the ‘Applications’ menu go to ‘Preferences’, ‘More Preferences’ and launch ‘Sessions’. Add: /usr/bin/vmware-toolbox to Startup Programs.

Log out and back in as root.Make sure to tick the ‘Time synchronization between the virtual machine and the host operating system’ box. This will add the following line to the Virtual Machine configuration file:tools.syncTime = “TRUE”

Minimize the ‘VMware Tools Properties’ window.
Edit /boot/grub/grub.conf and add the following options to the kernel /boot/ line of your default (SMP) kernel:
clock=pit nosmp noapic nolapic
Install the following packages from the indicated cd’s:
libaio-devel-0.3.105-2 (CD 3)
elfutils-libelf-devel-0.97.1-4 (CD 4)
unixODBC-devel-2.2.11-1 (CD 4)

Create disk partitions for shared disks
Prepare the following disks that will host the shared database files, the Oracle Cluster Registry and the Clusterware voting disk:

Purpose	Size	Virtual device node	Disk
Database files	3Gb	SCSI 1:0	/dev/sdb
Database files	3Gb	SCSI 1:1	/dev/sdc
Database files	3Gb	SCSI 1:2	/dev/sdd
Oracle Cluster Registry	300Mb	SCSI 1:3	/dev/sde
Voting Disk	300Mb	SCSI 1:4	/dev/sdf

Use the following command to list all disks & partitions: fdisk –l
This will list all disks (including /dev/sda which we’ve already configured).

Repeat the following steps for all disks in the table above:
Open a terminal window as the root user and enter the following command (replacing with the correct disk name): fdisk

[root@rac1 ~]# fdisk /dev/sdb
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-391, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-391, default 391):
Using default value 391
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.

The key sequence after entering the fdisk command is:
n [return] p [return] 1 [return] [return] [return] w [return]

Prepare raw disks
Bind the OCR and Voting Disk partitions to raw devices. To do this edit the file: /etc/sysconfig/rawdevices and add the following lines:
/dev/raw/raw1 /dev/sde1
/dev/raw/raw2 /dev/sdf1

Restart the raw devices service for the new settings to take effect:
[root@rac1 ~]# service rawdevices restart
Assigning devices:
/dev/raw/raw1 –> /dev/sde1
/dev/raw/raw1: bound to major 8, minor 65
/dev/raw/raw2 –> /dev/sdf1
/dev/raw/raw2: bound to major 8, minor 81When you restart an Oracle Enterprise Linux 4 system, ownership and permissions for raw devices will revert to the root owner. To prevent this from happening edit the raw devices section of the /etc/udev/permissions.d/50-udev.permissions file to look like the following:
Create users and groups
In Oracle 11g Oracle recommends a new system group (osasm) if you want to separate the ASM administration function from your DBA’s. Likewise they recommend creating a separate group (crs) for Clusterware administration. I will however create a single software owner (oracle) for all Oracle software.To create the necessary groups and users execute the following from the command line as the root user:
# groupadd -g 501 oinstall
# groupadd -g 502 dba
# useradd -g oinstall -G dba -s /bin/ksh oracle
# passwd oracle
Configuring kernel parameters and shell limits
Append the following to the /etc/sysctl.conf file as the root user:
kernel.sem = 250 32000 100 128
kernel.shmmax = 536870912
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default = 4194304
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 262144

You can issue the following command for these settings to take immediate effect: /sbin/sysctl –p

Add the following entries to the /etc/hosts file as the root user:

Public	192.168.1.50	rac1.startoracle.com	rac1
	192.168.1.60	rac2.startoracle.com	rac2
VIP	192.168.1.51	rac1-vip.startoracle.com	rac1-vip
	192.168.1.61	rac2-vip.startoracle.com	rac2-vip
Inter-connect	10.10.10.50	rac1-priv.startoracle.com	rac1-priv
	10.10.10.60	rac2-priv.startoracle.com	rac2-priv

The /etc/hosts file should now contain the following entries:

Restart the network services to pick up the new settings with the following command: # service network restart

As root create the following oracle directories and change the ownership to oracle:
# mkdir -p /u01/app/oracle/product/11.1.0/crs
# mkdir /u01/app/oracle/product/11.1.0/db_1
# mkdir /u01/Stage11g
# chown -R oracle:oinstall /u01/app
# chown oracle:oinstall /u01/Stage11g
Set shell limits for the Oracle user
As the root user add the following lines to the /etc/security/limits.conf file:
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536Add the following as the second last lines in the /etc/pam.d/login file if it does not exist:
session required pam_limits.so

Append the following to the /etc/profile file:
if [ $USER = “oracle” ]; then
if [ $SHELL = “/bin/ksh” ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
umask 022
fi
Install the ASMLib package
Download the ASM Lib package from the Oracle Technology Network. Download the package named ‘oracleasmlib-2.0.2-1.i386.rpm’ under the section ‘Intel IA32 (x86) Architecture’ and install it as the root user with the following command: # rpm -i oracleasmlib-2.0.2-1.i386.rpm
Clone the virtual machine
At this point shutdown the virtual machine and copy all the files from the virtual machine’s directory to the directory for the second virtual machine. Once this is complete add the second virtual machine as follows:
Select the ‘Home’ tab in the VMware Server Console.

Select ‘Open Existing Virtual Machine’ and browse to the cloned virtual machine directory and select the file ‘Red Hat Enterprise Linux 4.vmx’. The cloned virtual machine will now show up in the VMware Server Console.

Rename the new virtual machine:
To rename the virtual machine select ‘Edit virtual machine settings’ and select the ‘Options’ tab. Change the virtual machine name (EL4_11gRAC2) and click ‘OK’ to continue.

Start the new virtual machine (while leaving the old one powered off) and log in as root. When prompted if you want to create a new UUID for the virtual machine select ‘Create’ and ‘OK’ to continue.

Upon startup you will be asked to remove the 2 “old” network cards and add “new” network cards. Select ‘Remove Configuration’ to remove the network cards. You’ll need to do this twice.

Select ‘Configure’ to configure the first network card.

Enter the IP configuration for the second virtual machine’s first network card (Private) as per the table in step 36. ‘OK’ to continue.

Select ‘Configure’ again to configure the second network card and enter the IP configuration for the second virtual machine’s second network card (Public) as per the table in step 36. ‘OK’ to continue.

Login as the root user and update the network settings.
From the ‘Applications’ menu select ‘System Settings’ and ‘Network’ to edit network settings.
For each network device do the following:
Select the device and Select ‘Edit’.

Make sure the IP address is correct for each device – eth0 (the bridged network connection) should be on the public network and eth1 (the host-only network connection) should be on the private network. If the settings are incorrect update the IP address, subnet mask and default gateway.

Select the ‘Hardware Device’ tab and click ‘Probe’ to get the new MAC address for the device. Select ‘OK’ to continue.

Select the ‘DNS’ tab and update the Hostname to rac2.startoracle.com. From the file menu select ‘Save’ and ‘Quit’.

Restart the virtual machine for the changes to take effect and start the first virtual machine. Both virtual machines should now be up and running.Note: To test the network setup, ping the nodes from each other using the public and private ip addresses (the virtual ip address is not pingable at this stage).
Enabling SSH User Equivalency
The OUI (Oracle Universal Installer) uses the ssh and scp commands during installation to run remote commands and copy files to other nodes in the cluster, therefore you need to setup user equivalency for the Oracle user on all nodes.Perform the following in order to setup user equivalency:As the oracle user on node rac1:
$ mkdir ~/.ssh
$ chmod 700 ~/.ssh
$ /usr/bin/ssh-keygen -t rsa
Note: Whenever prompted for a passphrase leave it empty and press return.

As the oracle user on node rac2:
$ mkdir ~/.ssh
$ chmod 700 ~/.ssh
$ /usr/bin/ssh-keygen -t rsa

As the oracle user on node rac1:
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ ssh rac2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ scp ~/.ssh/authorized_keys rac2:/home/oracle/.ssh/

Perform the following as the oracle user on node rac1 and then node rac2:
ssh rac1 date
ssh rac2 date
ssh rac1.startoracle.com date
ssh rac2.startoracle.com date
ssh rac1-priv date
ssh rac2-priv date
ssh rac1-priv.startoracle.com date
ssh rac2-priv.startoracle.com date
Setup the ASM disksConfigure the Oracle ASM driver
As the root user on both nodes run the following:
# service oracleasm configure

Ouput:
[root@rac1 etc]# service oracleasm configure
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets (’[]’). Hitting without typing an
answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Fix permissions of Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: [ OK ]
Loading module “oracleasm”: [ OK ]
Mounting ASMlib driver filesystem: [ OK ]
Scanning system for ASM disks: [ OK ]

Stamp the ASM disks
As the root user on node rac1 (or rac2, but not both) run the following:
# service oracleasm createdisk DATA1 /dev/sdb1
# service oracleasm createdisk DATA2 /dev/sdc1
# service oracleasm createdisk RECO1 /dev/sdd1

Output:
[root@rac1 etc]# service oracleasm createdisk DATA1 /dev/sdb1
Marking disk “/dev/sdb1″ as an ASM disk: [ OK ]
[root@rac1 etc]# service oracleasm createdisk DATA2 /dev/sdc1
Marking disk “/dev/sdc1″ as an ASM disk: [ OK ]
[root@rac1 etc]# service oracleasm createdisk RECO1 /dev/sdd1
Marking disk “/dev/sdd1″ as an ASM disk: [ OK ]

Verify the newly configured ASM disks
As the root user on both nodes run the following:
# service oracleasm scandisks
# service oracleasm listdisks

Output:
[root@rac1 etc]# service oracleasm scandisks
Scanning system for ASM disks: [ OK ]
[root@rac1 etc]# service oracleasm listdisks
DATA1
DATA2
RECO1

You are now ready to install Oracle Clusterware.

Install Oracle Clusterware

Copy the Oracle Clusterware and Database Software zip files downloadedearlier to the staging area (/u01/Stage11g) on node rac1. Use pscp to copy the downloaded files from your Windows host machine to the virtual machine:

Alternatively download the zip files from your virtual machine. If your network is setup correctly you should be able to connect to the internet from within your virtual machine.
Unzip the Oracle 11g software in your staging area (/u01/Stage11g) as the oracle user:
$ unzip /u01/Stage11g/linux_11gR1_clusterware.zip
$ unzip /u01/Stage11g/linux_11gR1_database.zip
Install the cvuqdisks package
It can be found in the /u01/Stage11g/clusterware/rpm directory. Install on both nodes as the root user: # rpm -i cvuqdisk-1.0.1–1.rpm
Run the Cluster Verification Utility (CVU or Cluvfy) as the oracle user to verify all pre-installation tasks for Oracle Clusterware are completed.
Go to your Clusterware staging directory (/u01/Stage11g/clusterware) and run the following command as the oracle user on node rac1: $ ./runcluvfy.sh stage -pre crsinst -n rac1,rac2The Clufvy output will be similar to the following:
$ pwd
/u01/Stage11g/clusterware

$ ./runcluvfy.sh stage -pre crsinst -n rac1,rac2
Performing pre-checks for cluster services setup
Checking node reachability…
Node reachability check passed from node “rac1″.
Checking user equivalence…
User equivalence check passed for user “oracle”.
Checking administrative privileges…
User existence check passed for “oracle”.
Group existence check passed for “oinstall”.
Membership check for user “oracle” in group “oinstall” [as Primary] passed.
Administrative privileges check passed.
Checking node connectivity…
Node connectivity check passed for subnet “192.168.1.0″ with node(s) rac2,rac1.
Node connectivity check passed for subnet “10.10.10.0″ with node(s) rac2,rac1.
Interfaces found on subnet “10.10.10.0″ that are likely candidates for VIP:
rac2 eth1:10.10.10.60
rac1 eth1:10.10.10.50
Interfaces found on subnet “192.168.1.0″ that are likely candidates for a private interconnect:rac2 eth0:192.168.1.60
rac1 eth0:192.168.1.50
Node connectivity check passed.
Checking system requirements for ‘crs’…
Total memory check failed.
Check failed on nodes:
rac2,rac1
Free disk space check passed.
Swap space check passed.
System architecture check passed.
Kernel version check passed.
Package existence check passed for “make-3.80″.
Package existence check passed for “binutils-2.15.92.0.2″.
Package existence check passed for “gcc-3.4.5″.
Package existence check passed for “libaio-0.3.105″.
Package existence check passed for “libaio-devel-0.3.105″.
Package existence check passed for “libstdc++-3.4.5″.
Package existence check passed for “elfutils-libelf-devel-0.97″.
Package existence check passed for “sysstat-5.0.5″.
Package existence check passed for “libgcc-3.4.5″.
Package existence check passed for “libstdc++-devel-3.4.5″.
Package existence check passed for “unixODBC-2.2.11″.
Package existence check passed for “unixODBC-devel-2.2.11″.
Package existence check passed for “glibc-2.3.4-2.19″.
Group existence check passed for “dba”.
Group existence check passed for “oinstall”.
User existence check passed for “nobody”.
System requirement failed for ‘crs’
Pre-check for cluster services setup was unsuccessful on all the nodes.

The Clufvy pre-check will fail, because the ‘Total memory check’ failed, even though we have the recommended 1Gb RAM allocated to each instance. Make sure all other checks passed.
Launch the OUI for Oracle Clusterware
Prior to installing open a terminal window as the root user on node rac1 and type the following:
# xhost +Open a terminal window as the oracle user on node rac1 and go to the clusterware staging directory (/u01/Stage11g/clusterware) and start the installer with the following command:
$ ./runInstaller

This will launch the OUI (Oracle Universal Installer) for Oracle Clusterware 11g.
Welcome. Select ‘Next’ to continue.
Specify Inventory directory and credentials
Change the inventory directory from the default of /u01/app/oraInventroy to /u01/app/oracle/oraInventory. Keep the default (oinstall) Operating System group name. ‘Next’ to continue.
Specify Home Details
Change the path to /u01/app/oracle/product/11.1.0/crs. ‘Next’ to continue.
Product-Specific Prerequisite Checks
The OUI will now verify that the environment meets all the requirements. All pre-requisite steps should complete successfully. ‘Next’ to continue.
Specify Cluster Configuration
Specify all nodes in the cluster. Initially only node rac1 will be listed.
Select ‘Add’ and add the details for node rac2. Enter all details. ‘OK’ to continue.
After you’ve added node rac2 the Cluster configuration should look as below. ‘Next’ to continue.
Specify Network Interface Usage
Select the eth0 interface name and ‘Edit’ to correct the Interface Type.
Change the eth0 Interface Type to Public. ‘Ok’ to continue.
Verify that eth0 (192.168.1.0) is now your public network and eth1 (10.10.10.0) is your private network. ‘Next’ to continue.
Specify Oracle Cluster Registry (OCR) Location
Select External Redundancy and specify the first raw device (/dev/raw/raw1) setup earlier as the OCR location. ‘Next’ to continue.
Specify Voting Disk Location
Select External Redundancy and specify the second raw device (/dev/raw/raw2) setup earlier as the Voting Disk location. ‘Next’ to continue.
Summary
Review the summary and select ‘Install’ to start the installation.
Install
Monitor the progress of the installation.
Configuration Scripts
Once the installation is complete you will be prompted to run 2 configuration scripts as the root user. Open a terminal window and execute these scripts as the root user on each node. Select ‘Ok’ to continue after these scripts have completed successfully on both nodes. Do not run the scripts simultaneously on both nodes.

The output for the configuration scripts will look similar to below:
[root@rac1 ~]# /u01/app/oracle/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oracle/oraInventory to 770.
Changing groupname of /u01/app/oracle/oraInventory to oinstall.
The execution of the script is complete
[root@rac1 ~]# /u01/app/oracle/product/11.1.0/crs/root.sh
WARNING: directory ‘/u01/app/oracle/product/11.1.0′ is not owned by root
WARNING: directory ‘/u01/app/oracle/product’ is not owned by root
WARNING: directory ‘/u01/app/oracle’ is not owned by root
WARNING: directory ‘/u01/app’ is not owned by root
Checking to see if Oracle CRS stack is already configured
/etc/oracle does not exist. Creating it now.
Setting the permissions on OCR backup directory
Setting up Network socket directories
Oracle Cluster Registry configuration upgraded successfully
The directory ‘/u01/app/oracle/product/11.1.0′ is not owned by root. Changing owner to root
The directory ‘/u01/app/oracle/product’ is not owned by root. Changing owner to root
The directory ‘/u01/app/oracle’ is not owned by root. Changing owner to root
The directory ‘/u01/app’ is not owned by root. Changing owner to root
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node :
node 1: rac1 rac1-priv rac1
node 2: rac2 rac2-priv rac2
Creating OCR keys for user ‘root’, privgrp ‘root’..
Operation successful.
Now formatting voting device: /dev/raw/raw2
Format of 1 voting devices complete.
Startup will be queued to init within 30 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
Cluster Synchronization Services is active on these nodes.
rac1
Cluster Synchronization Services is inactive on these nodes.
rac2
Local node checking complete. Run root.sh on remaining nodes to start CRS daemons.
Configuration Assistants
The configuration assistants will now kick into gear. They all need to complete successfully in order for the installation to be successful.
End of Installation
Once the configuration assistants complete below screen will display. ‘Exit’ to complete the installation and exit the OUI.
Checking the status of Clusterware
To check the status of your installation issue the following commands as the root user:
# /u01/app/oracle/product/11.1.0/crs/bin/crsctl check crs
Cluster Synchronization Services appears healthy
Cluster Ready Services appears healthy
Event Manager appears healthyTo get more detailed information issue the following command as the root user: #/u01/app/oracle/product/11.1.0/crs/bin/crs_stat -t
Name Type R/RA F/FT Target State Host
———— ———– —- —- —— —— —-
ora.rac1.gsd application 0/5 0/0 ONLINE ONLINE rac1
ora.rac1.ons application 0/3 0/0 ONLINE ONLINE rac1
ora.rac1.vip application 0/0 0/0 ONLINE ONLINE rac1
ora.rac2.gsd application 0/5 0/0 ONLINE ONLINE rac2
ora.rac2.ons application 0/3 0/0 ONLINE ONLINE rac2
ora.rac2.vip application 0/0 0/0 ONLINE ONLINE rac2

Install Oracle Database and ASM Software

As root change ownership of the oracle directories to the oracle user on both nodes:
# chown oracle:oinstall /u01/app/oracle
# chown oracle:oinstall /u01/app/oracle/product
# chown oracle:oinstall /u01/app/oracle/product/11.1.0
To start the OUI change to the database directory in your staging area (/u01/Stage11g/database) and launch the OUI as the oracle user with the following command: $ ./runInstaller
Welcome
‘Next’ to skip the Welcome screen.
Select Installation Type
Keep the default installation type (Enterprise Edition). ‘Next’ to continue.
Install Location
Keep the default Oracle Base location (/u01/app/oracle) and default Oracle Home location (/u01/app/oracle/product/11.1.0/db_1). ‘Next’ to continue.
Specify Hardware Cluster Installation Mode
Keep the default ‘Cluster Installation’ selection and select both nodes. ‘Next’ to continue.
Product-Specific Prerequisite Checks
The OUI will now verify that the environment meets all the requirements. All pre-requisite steps should complete successfully. Select ‘Next’ to continue.
Select Configuration Option
Select ‘Install Software Only’. We will use the DBCA (Database Configuration Assistant) at a later stage to configure ASM and create a database. ‘Next’ to continue.
Privileged Operating System Groups
Keep the default options (dba, oinstall and oinstall). ‘Next’ to continue.
Summary
Review the summary and select ‘Install’ to start the installation.
Install
Monitor the progress of the installation.
Configuration Script
Once the installation is complete you will be prompted to run a script as the root user. Open a terminal window and execute the script as the root user on each node. Select ‘Ok’ to continue after the script has been run successfully on both nodes.
End of Installation
Once the installation is complete select ‘Exit’ to complete the installation and exit the OUI.

Create Oracle ASM instances

Launch the DBCA (Database Configuration Assistant) to configure ASM and create a RAC database.Open a terminal window as the oracle user. From the /u01/app/oracle/product/11.1.0/db_1/bin directory launch the dbca with the following command: $ ./dbca
Welcome
Keep the default selection (Oracle RAC database). ‘Next’ to continue.
Operations
Select ‘Configure ASM’. ‘Next’ to continue.
Node Selection
Select all the nodes and ‘Next’ to continue.
Create ASM Instance
Select a SYS pasword for the ASM instance. Keep the default type of parameter file (IFILE) to create. ‘Next’ to continue.
Select ‘Ok’ to confirm creation of the ASM instances.
Select ‘Yes’ to let the DBCA create default listeners.
ASM Disk Groups
Select ‘Create New’ to create new ASM disk groups.
Enter ‘DATA’ as the first disk group name. Keep the default redundancy settings (Normal) and select the 2 data disks (DATA1 and DATA2). ‘Ok’ to continue.
The DATA disk group should now be mounted. This will be the location for all our database files. Select ‘Create new’ again to add another disk group.
Enter ‘RECO’ as the disk group name. This time select ‘External’ for redundancy and select the remaining disk (RECO1). ‘OK’ to continue.
All the disk groups are now created. ‘Finish’ to complete.
Select ‘Yes’ to perform the database creation.

Create Oracle RAC Database

Operations
Select ‘Create a Database’. ‘Next’ to continue.
Node Selection
Select all the nodes and ‘Next’ to continue.
Database Templates
Keep the default setting (General Purpose or Transaction Processing). ‘Next’ to continue.
Database Identification
Enter ‘rac’ (without the quotes) as the global database name. ‘Next’ to continue.
Management Options
Keep the default settings (‘Configure Enterprise Manager’ and ‘Configure Database Control for local management’ selected, but ‘Enable Alert Notifications’ and ‘Enable Daily Disk Backup to Recovery Area’ deselected). ‘Next’ to continue.
Database Credentials
Select ‘Use the same Administrative Password for All Accounts’ and enter a password. ‘Next’ to continue.
Storage Options
Select ‘ASM’ for storage. ‘Next’ to continue.
ASM Disk Groups
Select both disk groups (DATA and RECO). ‘Next’ to continue.
Database File Locations
Keep the default (Use Oracle-Managed Files). Make sure ‘+DATA’ is entered as the Database Area. ‘Next’ to continue.
Select ‘Specify Flash Recovery Area’ and enter ‘+RECO’ as the Flash Recovery Area. Select ‘Enable archiving’ and click on the ‘Edit Archive Mode Parameters’ button.
Edit Archive Mode Parameters
Make sure the Flash Recovery Area (RECO) is shown as the archiving destination. If not change it to ‘+RECO/’ (without the quotes). ‘OK’ and then ‘Next’ to continue.
Database Content
Select ‘Sample Schemas’ if you want to install them. I did not. ‘Next’ to continue.
Initialization Parameters
Reduce the ‘Memory Size’ to 250Mb and select ‘Use Automatic Memory Management’ Leave all other settings as is. ‘Next’ to continue.
Security Settings
Keep the default, enhanced 11g security settings. ‘Next’ to continue.
Automatic Maintenance Tasks
Make sure ‘Enable automatic maintenance tasks’ is seleced. ‘Next’ to continue.
Database Storage
Review the storage options for all files. ‘Next’ to continue.
Creation Options
Select ‘Generate database creation scripts’ if you want to review these at a later stage. ‘Finish’ to review the installation.
Summary
Select ‘Ok’ to close the review page and ‘Finish’ to start the installation.
The DBCA will first generate the database creation scripts if you selected this. A message will be displayed once this is done. Acknowledging the message will start the database creation.
Once database creation is done a summary screen will be displayed. ‘Exit’ to exit the OUI.
As the oracle user edit the /etc/oratab file on both nodes.Replace the database name with the instance name for the rac database i.e. replace the rac keyword with rac1 or rac2 depending on the node. Furthermore add details for your clusterware home to this file. This will enable you to set the Clusterware home using the oraenv script. Once edited the /etc/oratab file should contain the following:
On node RAC1:
+ASM1:/u01/app/oracle/products/11.1.0/db_1:N
rac1:/u01/app/oracle/products/11.1.0/db_1:N
crs:/u01/app/oracle/products/11.1.0/crs:N
On node RAC2
+ASM2:/u01/app/oracle/products/11.1.0/db_1:N
rac2:/u01/app/oracle/products/11.1.0/db_1:N
crs:/u01/app/oracle/products/11.1.0/crs:N
Setup the environment for the Oracle user
As the oracle user create the file /home/oracle/.profile on all nodes and add the following lines:
umask 022
set -o vi
export PS1=”[${USER}@`hostname -s`]$ “
export ORACLE_BASE=/u01/app/oracle
export ORACLE_SID=`hostname -s`
export ORAENV_ASK=NO
. oraenv
unset ORAENV_ASKCongratulations! You have successfully installed and created an Oracle 11g RAC database.

Oracle 11g RAC on virtual servers. Done.

How the sequence and Cache size of RAC instances work?

mudhalvan — Tue, 03 May 2011 03:58:42 +0000

Sequences in RAC are handled separately by each instance.Each instance reads the sequence from the database and start allocating number until is it finish the cache range.Then it reads the sequence again and start allocating new numbers.

Assuming cache size of 3:

RAC1 – first call – 1 (save next value = 4 in the database).

RAC2 – first call – 4 (save next value = 7 in the database).

RAC1 – second call – 2

RAC1 – third call – 3

RAC1 – forth call – 7

RAC2 – second call – 5

There also contention between the nodes when they need to update the next value for the sequence.
On RAC, where inserts are performed on both nodes it depends if you care about the order or you just want to make sure that each row has a different value.

1) Order is important – Define the sequence with ordered and big cache. This will cause global lock between the nodes and will degrade performence.

2) Order is not important – Just define with big cache.
Big cache minimize the contention for updating the sequence.

RAC – Load Balance

mudhalvan — Wed, 23 Mar 2011 12:01:31 +0000

Understanding and Troubleshooting Instance Load Balancing [ID 263599.1]

10g & 11g :Configuration of TAF(Transparent Application Failover) and Load Balancing [ID 453293.1]

How to find out the Master Node of a RAC

mudhalvan — Tue, 22 Feb 2011 12:18:13 +0000

How to find out the Master Node of a RAC

Option 1:

# ocrconfig -showbackup

The node that store OCR backups is the master node.

Option 2:

$ cssd >grep -i “master node“ ocssd.log | tail -1

[CSSD]CLSS-3001: local node number 1, master node number 1

Above grep shows the master node in the cluster is node number 1.

Option 3:

$ grep master rac3_diag_4217.trc

I‘m the master node

Option 4:

Query V$GES_RESOURCE to identified master node.

How do I identify the OCR file location

mudhalvan — Sun, 05 Dec 2010 09:58:55 +0000

Do simple search for ocr.loc

/var/opt/oracle/ocr.loc
or
/etc/ocr.loc
or
# ocrcheck

Major RAC Wait Events

aarthimudhalvan — Fri, 09 Apr 2010 04:17:35 +0000

In a RAC environment the buffer cache is global across all instances in the cluster and hence the processing
differs.The most common wait events related to this are gc cr request and gc buffer busy

GC CR request: the time it takes to retrieve the data from the remote cache

Reason: RAC Traffic Using Slow Connection or Inefficient queries (poorly tuned queries will increase the amount of data blocks
requested by an Oracle session. The more blocks requested typically means the more often a block will need to be read from a remote instance via the interconnect.)

GC BUFFER BUSY: It is the time the remote instance locally spends accessing the requested data block.

SRVCTL

aarthimudhalvan — Fri, 09 Apr 2010 04:15:56 +0000

srvctl start instance -d db_name -i “inst_name_list” [-o start_options]
srvctl stop instance -d name -i “inst_name_list” [-o stop_options]
srvctl stop instance -d orcl -i “orcl3,orcl4″ -o immediate
srvctl start database -d name [-o start_options]
srvctl stop database -d name [-o stop_options]
srvctl start database -d orcl -o mount

Oracle clusterware tools

aarthimudhalvan — Fri, 09 Apr 2010 04:14:36 +0000

OIFCFG – allocating and deallocating network interfaces
OCRCONFIG – Command-line tool for managing Oracle Cluster Registry
OCRDUMP – Identify the interconnect being used
CVU – Cluster verification utility to get status of CRS resources

OCR

aarthimudhalvan — Wed, 07 Apr 2010 05:19:46 +0000

Oracle clusterware manages CRS resources based on the configuration information of CRS resources stored in OCR(Oracle Cluster Registry).

CRS Resource

aarthimudhalvan — Wed, 07 Apr 2010 05:18:31 +0000

Oracle clusterware is used to manage high-availability operations in a cluster.Anything that Oracle Clusterware manages is known as a CRS resource.Some examples of CRS resources are database,an instance,a service,a listener,a VIP address,an application process etc.

Virtual Machine 1	E:\Virtual Machines\RAC\rac11g\rac1
Virtual Machine 2	E:\Virtual Machines\RAC\rac11g\rac2
Shared Storage	E:\Virtual Machines\RAC\rac11g\share

Mount point	Size
/swap	2048Mb
/tmp	1024Mb
/	4096Mb
/u01	13312Mb

Hostname	Network device	Network type	IP address
rac1.startoracle.com	eth0	Public	192.168.1.50
	eth1	Private	10.10.10.50