ASM using 11gR2 Grid

Lately at work I've been working on migrating our databases to RAC. I'm doing this in incremental steps. The eventual goal is to get to 11gR2 running on a 2 node RAC. Currently we are running 10gR2 on RHEL 5.4.

The initial step I've taken is to migrate to ASM for storage. I've decided to use 11gR2 Grid for the ASM component. The ASM install is very straightforward, providing you plan appropriately.

• First thing you should do is get the Grid software and stage it on your linux box.
• Now you will have to install some asmlib rpm's. I got mine from:

http://www.oracle.com/technetwork/jp/topics/linux/downloads/rhel5-083144-ja.html

• Be sure to get the correct version for the linux flavor you have. You can do uname -rm to see.
• Install the rpms (these are the ones I needed, your will probably be different):

• rpm -i oracleasm-support-2.1.3-1.el5.x86_64.rpm oracleasmlib-2.0.4-1.el5.x86_64.rpm
  rpm -i oracleasm-2.6.18-164.el5-2.0.5-1.el5.x86_64.rpm
  

• Configure ASMLib

  • /etc/init.d/oracleasm configure
    Configuring the Oracle ASM library driver.This will configure the on-boot properties of the Oracle ASM library
    driver. The following questions will determine whether the driver is
    loaded on boot and what permissions it will have. The current values
    will be shown in brackets ('[]'). Hitting without typing an
    answer will keep that current value. Ctrl-C will abort.
    Default user to own the driver interface [oracle]:
    Default group to own the driver interface [dba]:
    Start Oracle ASM library driver on boot (y/n) [y]:
    Scan for Oracle ASM disks on boot (y/n) [y]:
    Writing Oracle ASM library driver configuration: done
    Initializing the Oracle ASMLib driver: [ OK ]
    Scanning the system for Oracle ASMLib disks: [ OK ]
    

• Alter the configuration for multipath (if you have multipath).
  • Multipath provides redundant paths to a volume that is connected to the server. In a typical multipath setup, you might have 4 devices defined that all resolve to the same volume. One of these devices is a human readable name setup by udev. This cannot be used. The other 3 are going to be something like dm-1, sda1, sdb1. the dm-1 is the multipath device. This we want to use. The other 2 are devices for each individual path. If ASM were to grab hold of one of these, well, you wouldn't have multipath, would you…
  • We will modify the asmlib config file located at /etc/sysconfig/oracleasm
  • There are two parameters that are of interest:
    • ORACLEASM_SCANORDER - specify which devices are looked at first
    • ORACLEASM_SCANEXCLUDE - specify devices to ignore
  • We will set ORACLEASM_SCANORDER
    • ORACLEASM_SCANORDER=“dm”
  • NOTE: devices in these lists are space separated, and match any device that starts with the characters specified. So dm will match dm-1, dm-2, etc.
• Create the disks as ASM. If using multipath, you can use any of the device names to do this

  • oracleasm createdisk VOL1 /dev/sda1
    oracleasm createdisk VOL2 /dev/sdc1
    oracleasm createdisk VOL3 /dev/sde1
    

• rescan for ASM disks, and then list the disks

  • oracleasm scandisks
    oracleasm listdisks
    

• Now, setup Grid
  • run the runInstaller located in the grid staging area that you downloaded above.
  • Walking thru the steps is pretty straightforward. Some spots where you must stop and think are creating the ASM diskgroup. Here you have some choices. How many diskgroups will you create, which volumes will you assign, and what redundancy level will you use.
    • A note on redundancy levels:
      • High Redundancy = triple mirroring
      • Normal redundancy = double mirroring
      • External redundancy = no mirroring
• When the setup is complete, you will have an ASM instance up and running!

As luck would have it, I made a mistake when creating my disk group. I chose normal redundancy, when I needed to choose external. It turns out that my volumes were already mirrored on the SAN, and so it was very unnecessary to mirror at the software level. No problem I thought, I'll just drop and recreate the diskgroup, since nothing is using it yet.

When I tried to drop the diskgroup using asmca, I got the above error message. Same thing when trying alter diskgroup data dismount or alter diskgroup data drop.

Documentation states to query v$asm_client to find out who is using the diskgroup, and then stop that client (database). My query returned no rows. To be expected because I hadn't set any database to use this ASM instance.

Poking around using asmcmd, I found the culprit. The asm instance stored it's spfile inside the asm diskgroup. The instance itself was using the diskgroup. So, to fix this:

sqlplus / as sysasm

SQL>create pfile='init+ASM.ora' from spfile;
SQL>shutdown immediate;
SQL>startup pfile='init+ASM.ora';

Note the sqplus / as sysasm. You have to connect as sysasm to do any operations on the asm instance. sysdba does not have rights!

Now you can drop the group and recreate it with the correct parameters (I used asmca for this), then back to sqlplus:

SQL>create spfile='+DATA' from pfile;
SQL>shutdown immediate;
SQL>startup;

Fixed!