Multinode RAC 12c cluster on VirtualBox using linked clones

Recently I’ve had to install a four-node RAC cluster on my laptop in order to do some tests. I have found an “easy” (well, easy, it depends), fast and space-efficient way to do it so I would like to track it down.

The quick step list

Install the OS on the first node
Add the shared disks
Install the clusterware in RAC mode on on the first node only
Remove temporarily the shared disks
Clone the server as linked clone as many times as you want
Reconfigure the new nodes with the new ip and naming
Add back the shared disks on the first node and on all other nodes
Clone the GI + database homes in order to add them to the cluster

Using this method the Oracle binaries (the most space consuming portion of the RAC installation) are installed and allocated on the first node only.

The long step list

Actually you can follow many instruction steps from the RAC Attack 12c book.

Review the HW requirements but let at least 3Gb RAM for each guest + 2Gb more for your host (you may try with less RAM but everything will slow down).
Download all the SW components , additionally you may download the latest PSU (12.1.0.1.2) from MOS.
Prepare the host and install linux on the first node. When configuring the OS, make sure you enter all the required IP addresses for the additional nodes. RAC Attack has two nodes collabn1, collabn2. Add as many nodes as you want to configure. As example, the DNS config may have four nodes

collabn1        A       192.168.78.51
collabn2        A       192.168.78.52
collabn3        A       192.168.78.53
collabn4        A       192.168.78.54
collabn1-vip    A       192.168.78.61
collabn2-vip    A       192.168.78.62
collabn3-vip    A       192.168.78.63
collabn4-vip    A       192.168.78.64
collabn1-priv   A       172.16.100.51
collabn2-priv   A       172.16.100.52
collabn3-priv   A       172.16.100.53
collabn4-priv   A       172.16.100.54
collabn-cluster-scan     A       192.168.78.251
collabn-cluster-scan     A       192.168.78.252
collabn-cluster-scan     A       192.168.78.253

collabn1 A 192.168.78.51

collabn2 A 192.168.78.52

collabn3 A 192.168.78.53

collabn4 A 192.168.78.54

collabn1-vip A 192.168.78.61

collabn2-vip A 192.168.78.62

collabn3-vip A 192.168.78.63

collabn4-vip A 192.168.78.64

collabn1-priv A 172.16.100.51

collabn2-priv A 172.16.100.52

collabn3-priv A 172.16.100.53

collabn4-priv A 172.16.100.54

collabn-cluster-scan A 192.168.78.251

collabn-cluster-scan A 192.168.78.252

collabn-cluster-scan A 192.168.78.253

Create the shared disks and configure the persistent storage naming using udev.

At this point, the procedure starts differing from the RAC Attack book.

Skip the creation of the second host and go directly to the VNC Server setup.
Install the Grid Infrastructure and the Database software using only the first node.
You may want to install the latest PSU (12.1.0.1.2), use my previous post as guideline
Once the GI + DB are installed correctly, stop and disable the crs on the first node:

# <GIHOME>/bin/crsctl stop crs
# <GIHOME>/bin/crsctl disable crs
# shutdown -h now

# <GIHOME>/bin/crsctl stop crs

# <GIHOME>/bin/crsctl disable crs

# shutdown -h now

Go to the VirtualBox VM settings and delete all the shared disks
Clone the first server as linked clone (right-click, clone, choose the name, flag “Linked Clone” as many times as the number of additional servers you want.

By using this method the new servers will use the same virtual disk file of the first server and a second file will be used to track the differences. This will save a lot of space on the disk.
Add back the shared disks to all the servers.
Start the other nodes and configure them following the RAC Attack instructions again.
Once all the nodes are configured, the GI installation has to be cleaned out on all the cloned servers using these guidelines:

cd $GI_HOME
rm -rf log/$(hostname)
rm -rf gpnp/$(hostname)
find gpnp -type f -exec rm -f {} \;
rm -rf cfgtoollogs/*
rm -rf crs/init/*
rm -rf cdata/*
rm -rf crf/*
rm -rf network/admin/*.ora
rm -rf crs/install/crsconfig_params
find . -name '*.ouibak' -exec rm {} \;
find . -name '*.ouibak.1' -exec rm {} \;
rm -rf root.sh*
rm -rf rdbms/audit/*
rm -rf rdbms/log/*
rm -rf inventory/backup/*
chown -R oracle:oinstall /u01/app

rm -f /etc/init.d/ohasd 
rm -rf /etc/oracle 
rm -rf /u01/app/oraInventory/*

cd $GI_HOME

rm -rf log/$(hostname)

rm -rf gpnp/$(hostname)

find gpnp -type f -exec rm -f {} \;

rm -rf cfgtoollogs/*

rm -rf crs/init/*

rm -rf cdata/*

rm -rf crf/*

rm -rf network/admin/*.ora

rm -rf crs/install/crsconfig_params

find . -name '*.ouibak' -exec rm {} \;

find . -name '*.ouibak.1' -exec rm {} \;

rm -rf root.sh*

rm -rf rdbms/audit/*

rm -rf rdbms/log/*

rm -rf inventory/backup/*

chown -R oracle:oinstall /u01/app

rm -f /etc/init.d/ohasd

rm -rf /etc/oracle

rm -rf /u01/app/oraInventory/*

Then, on each cloned server, run the perl clone.pl as follows to clone the GI home, but change the LOCAL_NODE accordingly (note: the GI Home name must be identical to the one specified in the original installation!):

[oracle@collabn2 bin]$ perl clone.pl -silent ORACLE_BASE=/u01/app/oracle ORACLE_HOME=/u01/app/12.1.0/grid \
 ORACLE_HOME_NAME=OraGI12Home1 INVENTORY_LOCATION=/u01/app/oraInventory \
 LOCAL_NODE=collabn2 "CLUSTER_NODES={collabn1,collabn2,collabn3,collabn4}"  CRS=TRUE 

 ./runInstaller -clone -waitForCompletion "ORACLE_BASE=/u01/app/oracle" "ORACLE_HOME=/u01/app/12.1.0/grid" "ORACLE_HOME_NAME=OraGI12Home1" "INVENTORY_LOCATION=/u01/app/oraInventory" "LOCAL_NODE=collabn2" "CLUSTER_NODES={collabn1,collabn2}" "CRS=TRUE" -silent -paramFile /u01/app/12.1.0/grid/clone/clone_oraparam.ini 
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 500 MB. Actual 5537 MB Passed 
Checking swap space: must be greater than 500 MB. Actual 3012 MB Passed 
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2014-02-18_03-40-00PM. Please wait ...
You can find the log of this install session at: /u01/app/oraInventory/logs/cloneActions2014-02-18_03-40-00PM.log
 .................................................. 5% Done.
 .................................................. 10% Done.
 .................................................. 15% Done.
 .................................................. 20% Done.
 .................................................. 25% Done.
 .................................................. 30% Done.
 .................................................. 35% Done.
 .................................................. 40% Done.
 .................................................. 45% Done.
 .................................................. 50% Done.
 .................................................. 55% Done.
 .................................................. 60% Done.
 .................................................. 65% Done.
 .................................................. 70% Done.
 .................................................. 75% Done.
 .................................................. 80% Done.
 .................................................. 85% Done.
 .................................................. 90% Done.
 .................................................. 95% Done.
 Copy files in progress.

 Copy files successful.

 Link binaries in progress.

 Link binaries successful.

 Setup files in progress.

 Setup files successful.

 Setup Inventory in progress.

 Setup Inventory successful.

 Finish Setup successful.
 The cloning of OraGI12Home1 was successful. 
 Please check '/u01/app/oraInventory/logs/cloneActions2014-02-18_03-40-00PM.log' for more details.

 As a root user, execute the following script(s): 
 	1. /u01/app/12.1.0/grid/root.sh 

Execute /u01/app/12.1.0/grid/root.sh on the following nodes: 
[collabn2,collabn2,collabn3,collabn4]
.................................................. 100% Done.
[oracle@collabn2 bin]$ su - Password: 
[root@collabn2 ~]# /u01/app/oraInventory/orainstRoot.sh 
Changing permissions of /u01/app/oraInventory. 
Adding read,write permissions for group. 
Removing read,write,execute permissions for world. 
Changing groupname of /u01/app/oraInventory to oinstall. 
The execution of the script is complete. 
[root@collabn2 ~]#

[oracle@collabn2 bin]$ perl clone.pl -silent ORACLE_BASE=/u01/app/oracle ORACLE_HOME=/u01/app/12.1.0/grid \

ORACLE_HOME_NAME=OraGI12Home1 INVENTORY_LOCATION=/u01/app/oraInventory \

LOCAL_NODE=collabn2 "CLUSTER_NODES={collabn1,collabn2,collabn3,collabn4}" CRS=TRUE

./runInstaller -clone -waitForCompletion "ORACLE_BASE=/u01/app/oracle" "ORACLE_HOME=/u01/app/12.1.0/grid" "ORACLE_HOME_NAME=OraGI12Home1" "INVENTORY_LOCATION=/u01/app/oraInventory" "LOCAL_NODE=collabn2" "CLUSTER_NODES={collabn1,collabn2}" "CRS=TRUE" -silent -paramFile /u01/app/12.1.0/grid/clone/clone_oraparam.ini

Starting Oracle Universal Installer...

Checking Temp space: must be greater than 500 MB. Actual 5537 MB Passed

Checking swap space: must be greater than 500 MB. Actual 3012 MB Passed

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2014-02-18_03-40-00PM. Please wait ...

You can find the log of this install session at: /u01/app/oraInventory/logs/cloneActions2014-02-18_03-40-00PM.log

.................................................. 5% Done.

.................................................. 10% Done.

.................................................. 15% Done.

.................................................. 20% Done.

.................................................. 25% Done.

.................................................. 30% Done.

.................................................. 35% Done.

.................................................. 40% Done.

.................................................. 45% Done.

.................................................. 50% Done.

.................................................. 55% Done.

.................................................. 60% Done.

.................................................. 65% Done.

.................................................. 70% Done.

.................................................. 75% Done.

.................................................. 80% Done.

.................................................. 85% Done.

.................................................. 90% Done.

.................................................. 95% Done.

Copy files in progress.

Copy files successful.

Link binaries in progress.

Link binaries successful.

Setup files in progress.

Setup files successful.

Setup Inventory in progress.

Setup Inventory successful.

Finish Setup successful.

The cloning of OraGI12Home1 was successful.

Please check '/u01/app/oraInventory/logs/cloneActions2014-02-18_03-40-00PM.log' for more details.

As a root user, execute the following script(s):

1. /u01/app/12.1.0/grid/root.sh

Execute /u01/app/12.1.0/grid/root.sh on the following nodes:

[collabn2,collabn2,collabn3,collabn4]

.................................................. 100% Done.

[oracle@collabn2 bin]$ su - Password:

[root@collabn2 ~]# /u01/app/oraInventory/orainstRoot.sh

Changing permissions of /u01/app/oraInventory.

Adding read,write permissions for group.

Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.

The execution of the script is complete.

[root@collabn2 ~]#

Then, on the first node (that you have started and you have reactivated the clusterware stack on it with crsctl enable crs / crsctl start crs ;-)), run this command to add the new nodes in the definition of the cluster:

[oracle@collabn1 addnode]$ ./addnode.sh -silent -noCopy ORACLE_HOME=/u01/app/12.1.0/grid "CLUSTER_NEW_NODES={collabn2,collabn3,collabn4}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={collabn2-vip,collabn3-vip,collabn4-vip}"
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 120 MB.   Actual 6125 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 3017 MB    Passed
[WARNING] [INS-13014] Target environment does not meet some optional requirements.
   CAUSE: Some of the optional prerequisites are not met. See logs for details. /u01/app/oraInventory/logs/addNodeActions2014-02-18_03-43-22PM.log
   ACTION: Identify the list of failed prerequisite checks from the log: /u01/app/oraInventory/logs/addNodeActions2014-02-18_03-43-22PM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.

Prepare Configuration in progress.

Prepare Configuration successful.
..................................................   40% Done.

As a root user, execute the following script(s):
        1. /u01/app/12.1.0/grid/root.sh

Execute /u01/app/12.1.0/grid/root.sh on the following nodes:
[collabn2,collabn3,collabn4]

The scripts can be executed in parallel on all the nodes. If there are any policy managed databases managed by cluster, proceed with the addnode procedure without executing the root.sh script. Ensure that root.sh script is executed after all the policy managed databases managed by clusterware are extended to the new nodes.
..................................................   60% Done.

Update Inventory in progress.
..................................................   100% Done.

Update Inventory successful.
Successfully Setup Software.

[oracle@collabn1 addnode]$ ./addnode.sh -silent -noCopy ORACLE_HOME=/u01/app/12.1.0/grid "CLUSTER_NEW_NODES={collabn2,collabn3,collabn4}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={collabn2-vip,collabn3-vip,collabn4-vip}"

Starting Oracle Universal Installer...

Checking Temp space: must be greater than 120 MB. Actual 6125 MB Passed

Checking swap space: must be greater than 150 MB. Actual 3017 MB Passed

[WARNING] [INS-13014] Target environment does not meet some optional requirements.

CAUSE: Some of the optional prerequisites are not met. See logs for details. /u01/app/oraInventory/logs/addNodeActions2014-02-18_03-43-22PM.log

ACTION: Identify the list of failed prerequisite checks from the log: /u01/app/oraInventory/logs/addNodeActions2014-02-18_03-43-22PM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.

Prepare Configuration in progress.

Prepare Configuration successful.

.................................................. 40% Done.

As a root user, execute the following script(s):

1. /u01/app/12.1.0/grid/root.sh

Execute /u01/app/12.1.0/grid/root.sh on the following nodes:

[collabn2,collabn3,collabn4]

The scripts can be executed in parallel on all the nodes. If there are any policy managed databases managed by cluster, proceed with the addnode procedure without executing the root.sh script. Ensure that root.sh script is executed after all the policy managed databases managed by clusterware are extended to the new nodes.

.................................................. 60% Done.

Update Inventory in progress.

.................................................. 100% Done.

Update Inventory successful.

Successfully Setup Software.

from the first server copy these files on all the other nodes:

scp -rp /u01/app/12.1.0/grid/crs/install/crsconfig_addparams collabn2:/u01/app/12.1.0/grid/crs/install/crsconfig_addparams
scp -rp /u01/app/12.1.0/grid/crs/install/crsconfig_params collabn2:/u01/app/12.1.0/grid/crs/install/crsconfig_params
scp -rp /u01/app/12.1.0/grid/gpnp collabn2:/u01/app/12.1.0/grid/gpnp

scp -rp /u01/app/12.1.0/grid/crs/install/crsconfig_addparams collabn2:/u01/app/12.1.0/grid/crs/install/crsconfig_addparams

scp -rp /u01/app/12.1.0/grid/crs/install/crsconfig_params collabn2:/u01/app/12.1.0/grid/crs/install/crsconfig_params

scp -rp /u01/app/12.1.0/grid/gpnp collabn2:/u01/app/12.1.0/grid/gpnp

Then clone also the DB Home (again, run it on each new server and specify the same DB home name that you have used in the original installation):

[oracle@collabn2 bin]$ perl clone.pl -O 'CLUSTER_NODES={collabn1,collabn2,collabn3,collabn4}' -O LOCAL_NODE=collabn2 ORACLE_BASE=/u01/app/oracle ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1 ORACLE_HOME_NAME=OraDB12Home1 -O -noConfig
./runInstaller -clone -waitForCompletion   "CLUSTER_NODES={collabn1,collabn2,collabn3,collabn4}"  "LOCAL_NODE=collabn2" "ORACLE_BASE=/u01/app/oracle" "ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1" "ORACLE_HOME_NAME=OraDB12Home1"  -noConfig  -silent -paramFile /u01/app/oracle/product/12.1.0/dbhome_1/clone/clone_oraparam.ini
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 500 MB.   Actual 3896 MB    Passed
Checking swap space: must be greater than 500 MB.   Actual 3005 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2014-02-18_05-22-22PM. Please wait ...You can find the log of this install session at:
 /u01/app/oraInventory/logs/cloneActions2014-02-18_05-22-22PM.log
..................................................   5% Done.
..................................................   10% Done.
..................................................   15% Done.
..................................................   20% Done.
..................................................   25% Done.
..................................................   30% Done.
..................................................   35% Done.
..................................................   40% Done.
..................................................   45% Done.
..................................................   50% Done.
..................................................   55% Done.
..................................................   60% Done.
..................................................   65% Done.
..................................................   70% Done.
..................................................   75% Done.
..................................................   80% Done.
..................................................   85% Done.
..................................................   90% Done.
..................................................   95% Done.

Copy files in progress.

Copy files successful.

Link binaries in progress.

Link binaries successful.

Setup files in progress.

Setup files successful.

Setup Inventory in progress.

Setup Inventory successful.

Finish Setup in progress.

Finish Setup successful.
The cloning of OraDB12Home1 was successful.
Please check '/u01/app/oraInventory/logs/cloneActions2014-02-18_05-22-22PM.log' for more details.

As a root user, execute the following script(s):
        1. /u01/app/oracle/product/12.1.0/dbhome_1/root.sh

Execute /u01/app/oracle/product/12.1.0/dbhome_1/root.sh on the following nodes:
[collabn2]

..................................................   100% Done.

[oracle@collabn2 bin]$ perl clone.pl -O 'CLUSTER_NODES={collabn1,collabn2,collabn3,collabn4}' -O LOCAL_NODE=collabn2 ORACLE_BASE=/u01/app/oracle ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1 ORACLE_HOME_NAME=OraDB12Home1 -O -noConfig

./runInstaller -clone -waitForCompletion "CLUSTER_NODES={collabn1,collabn2,collabn3,collabn4}" "LOCAL_NODE=collabn2" "ORACLE_BASE=/u01/app/oracle" "ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1" "ORACLE_HOME_NAME=OraDB12Home1" -noConfig -silent -paramFile /u01/app/oracle/product/12.1.0/dbhome_1/clone/clone_oraparam.ini

Starting Oracle Universal Installer...

Checking Temp space: must be greater than 500 MB. Actual 3896 MB Passed

Checking swap space: must be greater than 500 MB. Actual 3005 MB Passed

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2014-02-18_05-22-22PM. Please wait ...You can find the log of this install session at:

/u01/app/oraInventory/logs/cloneActions2014-02-18_05-22-22PM.log