PRVF-4557 During installation of Oracle Binaries ( 11.2.0.4)

 

Introduction: During an Oracle software install for a new RAC environment once again  I was surprised by Oracle . I had installed the Grid infrastructure (GI) 11.2.0.3 on RedHat Linux with success and I was about to install the Oracle Binaries which should be a piece of cake with the installed  GI up and running properly.  According to the old quote “Never stop learning ( or being surprised  )”   during the preparations the runInstaller  managed to give me a surprise  and I was unable to continue with installation. So obviously this  surprise needed  fixing first.

Details: When running the runInstaller , after choosing the option to install the Oracle binaries on a Real Application Cluster environment  and after  a couple of options  I received a pop up telling me :
PRVF-4557 : Node application “ora.svrb1hr.vip” is offline on node
I was unable to go ahead and had to investigate. As always Google and the Oracle Community where  my brothers in oracle arms for this so i Came across this scenario :
First I checked my hosts file to make sure  information to be present on both nodes. That showed  following details which looked ok:

oracle@svrb1hr:/opt/oracle [CRM]# grep  vip /etc/hosts 
10.97.242.32 svrb1hr-vip.myenv.dc-world.de svrb1hr-vip
10.97.242.33 svrb2hr-vip.myenv.dc-world.de svrb2hr-vip

Next step was  checking my cluster resources. As  mentioned before GI install had finished  properly the day before so that really made me wonder:

oracle@svrb1hr:/opt/oracle [CRM]# crsctl status resource -t 
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CLUSTERDATA.dg
               ONLINE  ONLINE       svrb1hr                                     
               ONLINE  ONLINE       svrb2hr                                     
ora.LISTENER.lsnr
               ONLINE  OFFLINE      svrb1hr                                     
               ONLINE  ONLINE       svrb2hr                                     
ora.asm
               ONLINE  ONLINE       svrb1hr                 Started             
               ONLINE  ONLINE       svrb2hr                 Started             
ora.gsd
               OFFLINE OFFLINE      svrb1hr                                     
               OFFLINE OFFLINE      svrb2hr                                     
ora.net1.network
               ONLINE  ONLINE       svrb1hr                                     
               ONLINE  ONLINE       svrb2hr                                     
ora.ons
               ONLINE  ONLINE       svrb1hr                                     
               ONLINE  ONLINE       svrb2hr                                     
ora.registry.acfs
               ONLINE  ONLINE       svrb1hr                                     
               ONLINE  ONLINE       svrb2hr                                     
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       svrb1hr                                     
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       svrb2hr                                     
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       svrb1hr                                     
ora.cvu
      1        ONLINE  ONLINE       svrb2hr                                     
ora.svrb1hr.vip
      1        ONLINE  INTERMEDIATE svrb2hr                 FAILED OVER         
ora.svrb2hr.vip
      1        ONLINE  ONLINE       svrb2hr                                     
ora.oc4j
      1        ONLINE  ONLINE       svrb1hr                                     
ora.scan1.vip
      1        ONLINE  ONLINE       svrb1hr                                     
ora.scan2.vip
      1        ONLINE  ONLINE       svrb2hr                                     
ora.scan3.vip
      1        ONLINE  ONLINE       svrb1hr                                     

Well that was unexpected, because  expectation was that the srvrb1hr.vip should be running on the first node. Still don’t understand what happened to cause this. Hmm and frankly if you have suggestions what happened please let me know . But  I did know  it was needed to bring back the vip  address to the first server.
First attempt  was to issue the command needed  ( crs_relocate ) on the node where  I was already working ( node 1) .
oracle@svrb1hr:/opt/oracle [CRM]# which crs_relocate
/opt/crs/product/11203/crs/bin/crs_relocate
oracle@svrb1hr:/opt/oracle [CRM]# crs_relocate svrb1hr.vip
CRS-0210: Could not find resource ‘svrb1hr.vip’.
###  activities needed 2 b done from second node
Grumbling with this  but well at least it was explained what to do next ….
So I opened a session against the second node , made sure my Oracle Home was pointing to the GI.
oracle@svrb1hr:/opt/oracle [CRM]# ssh svrb2hr
On the second box  the command crs_relocate was entered:

                     
oracle@svrb2hr:/opt/oracle [CRS]# crs_relocate ora.svrb1hr.vip
Attempting to stop `ora.svrb1hr.vip` on member `svrb2hr`
Stop of `ora.svrb1hr.vip` on member `svrb2hr` succeeded.
Attempting to start `ora.svrb1hr.vip` on member `svrb1hr`
Start of `ora.svrb1hr.vip` on member `svrb1hr` succeeded.
Attempting to start `ora.LISTENER.lsnr` on member `svrb1hr`
Start of `ora.LISTENER.lsnr` on member `svrb1hr` succeeded.

Well that  looked promessing  so let’s check one more time then:

oracle@svrb2hr:/opt/oracle [CRS]#  crsctl status resource -t 
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CLUSTERDATA.dg
               ONLINE  ONLINE       svrb1hr                                     
               ONLINE  ONLINE       svrb2hr                                     
ora.LISTENER.lsnr
               ONLINE  ONLINE       svrb1hr                                     
               ONLINE  ONLINE       svrb2hr                                     
ora.asm
               ONLINE  ONLINE       svrb1hr                 Started             
               ONLINE  ONLINE       svrb2hr                 Started             
ora.gsd
               OFFLINE OFFLINE      svrb1hr                                     
               OFFLINE OFFLINE      svrb2hr                                     
ora.net1.network
               ONLINE  ONLINE       svrb1hr                                     
               ONLINE  ONLINE       svrb2hr                                     
ora.ons
               ONLINE  ONLINE       svrb1hr                                     
               ONLINE  ONLINE       svrb2hr                                     
ora.registry.acfs
               ONLINE  ONLINE       svrb1hr                                     
               ONLINE  ONLINE       svrb2hr                                     
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       svrb1hr                                     
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       svrb2hr                                     
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       svrb1hr                                     
ora.cvu
      1        ONLINE  ONLINE       svrb2hr                                     
ora.svrb1hr.vip
      1        ONLINE  ONLINE       svrb1hr                                     
ora.svrb2hr.vip
      1        ONLINE  ONLINE       svrb2hr                                     
ora.oc4j
      1        ONLINE  ONLINE       svrb1hr                                     
ora.scan1.vip
      1        ONLINE  ONLINE       svrb1hr                                     
ora.scan2.vip
      1        ONLINE  ONLINE       svrb2hr                                     
ora.scan3.vip
      1        ONLINE  ONLINE       svrb1hr

Much better!!  after this I restarted the runInstaller again from the first node and indeed installation of  the Oracle Binaries was flawless.

Please refer to : https://mathijsbruggink.wordpress.com/2014/10/03/prvf-4557-during-installation-of-oracle-binaries-11-2-0-3/

ASM new features in Oracle Database 11g

ASM was first time introduced in Oracle Database 10g. Since then lot of improvement has been done. Here I am presenting changes and new feature when you use ASM in Oracle Database 11h environment.
--------------------------------------------------------------------------------------
The default parameter settings work perfectly for ASM. The only parameters needed for 11g ASM:
ASM_DISKSTRING
ASM_DISKGROUPS
INSTANCE_TYPE
--------------------------------------------------------------------------------------
ASM path are not part of regular OS level I/O. Because of this RDBMS does a raw I/O which will be faster as compared to file system I/O. Since there is no intervention OS.
-------------------------------------------------------------------------------------
ASM Fast Mirror Resync - When first time ASM features was released in 10g, Whenever it is unable to complete the write or read operation in a disk, disk was dropped from ASM group. This was relatively costlier operation. It could take hours to do that.
but now in 11g, When a disk goes offline following a transient failure, ASM tracks the ASM data extents that are modified during the outage. After the transient failure is repaired, ASM quickly resynchronizes only those ASM data extents that have been affected during the outage. This feature assumes that the content of the affected ASM disks has not been damaged or modified.
conclusion is this feature is beneficial by reducing the time required to resynchronize a transient failure of a disk.
ALTER DISKGROUP dgroupA SET ATTRIBUTE 'DISK_REPAIR_TIME'='4H';
ALTER DISKGROUP dgroupA
OFFLINE DISKS IN FAILGROUP site2 DROP AFTER 6H;
ALTER DISKGROUP dgroupA
ONLINE DISKS IN FAILGROUP site2 POWER 2 WAIT;
ALTER DISKGROUP dgroupA DROP DISKS IN FAILGROUP site2
FORCE;
V$ASM_ATTRIBUTE,V$ASM_DISK, V$ASM_DISK_IOSTAT,V$ASM_OPERATION views can be checked for further investigation.
ASM Preferred Mirror Read
============================================
Mirroring is done to ensure the protection from data loss. ASM failure groups in Oracle Database 10g always reads the primary copy of a mirrored ASM data extent. If a node is allowed to read the data from local or from closest node inspite that data is secondary, then it can improve the performance by reducing network traffic. Oracle Database 11g can do this by configuring preferred mirror read using the new ASM_PREFERRED_READ_FAILURE_GROUPS initialization parameter to specify a list of preferred mirror read names.
ASM_PREFERRED_READ_FAILURE_GROUPS=DATA.SITEA On first instance
ASM_PREFERRED_READ_FAILURE_GROUPS=DATA.SITEB On second instance
This can be monitored from v$asm_disk(preferred_read) and v$asm_disk_iostat tables.
ASM Scalability and Performance Enhancements
=================================================
ASM uses a predetermined number of ASM data extents of each size. As soon as a file crosses a certain threshold, the next ASM data extent size is used. An ASM file can begin with 1 AU; as the file’s size increases, the ASM data extent size also increases to 8 or 64 AUs based on predefined file size thresholds.
ASM also automatically defragments during allocation if the desired size is unavailable, thereby potentially affecting allocation times but offering much faster file opens, given the reduction in the memory required to store file ASM data extents.
SYSASM privilege
=======================================================
In Oracle Database 11g, if you log in to an ASM instance as SYSDBA, warnings are written in the corresponding alert.log file.
SYSASM privilege is replacing SYSDBA privilege as compared to proir version. SYSDBA privilege in ASM instance will be restricted.
ASM Disk Group Compatibility
================================================
An ASM instance can support different RDBMS clients running at different compatibility settings. The database compatible version setting of each instance must be greater than or equal to the RDBMS compatibility of all disk groups used by that database. Database instances are typically run from a different Oracle home than the ASM instance. This implies that the database instance may be running a different software version than the ASM instance. When a database instance first connects to an ASM instance, it negotiates the highest version that they both can support.
Main Objective behind this new feature is to ensure heterogenous environment should also supproted by ASM.
Few New Maintenance command in ASM
==========================================
ALTER DISKGROUP DATA CHECK; Checks all the metadata directories by default
ALTER DISKGROUP data MOUNT RESTRICT; When a disk group is mounted in RESTRICTED mode, clients cannot access the files in a disk group.
ALTER DISKGROUP data MOUNT FORCE; Mounts the disk group even if some disks belonging to the disk group are not accessible
DROP DISKGROUP data FORCE INCLUDING CONTENTS; Enables users to drop a disk group that cannot be mounted; fails if the disk group is mounted anywhere
ASMCMD is now more powerful in 11g ASM
========================================
The lsdsk command lists ASM disk information. This command can run in two modes:
• Connected mode: ASMCMD uses the V$ and GV$ views to retrieve disk information.
• Nonconnected mode: ASMCMD scans disk headers to retrieve disk information, using an ASM disk string to restrict the discovery set. The connected mode is always attempted first.
MD_BACKUP command is used to do backup of metadata from ASM disk group.
MD_RESTORE command is used to restore the lost metadata to ASM disk group.
ASMCMD> md_backup –b jfv_backup_file -g data
Disk group to be backed up: DATA#
Current alias directory path: jfv
ASMCMD>
ASMCMD> md_restore -b jfv_backup_file -t full -g data
Disk group to be restored: DATA#
ASMCMDAMBR-09358, Option -t newdg specified without any override options.
Current Diskgroup being restored: DATA
Diskgroup DATA created!
User Alias directory +DATA/jfv
created!
ASMCMD now support cp command by which we can copy the ASM files from local to remote node or vice versa and from file system to ASM disk or vice versa.
cp +DATA/ORCL/DATAFILE/TBSJFV.256.629730771 +DATA/ORCL/tbsjfv.bak
cp /home/oracle/tbsjfv.dbf +data/jfv
cp +DATA/orcl/datafile/tbsjfv.256.629989893 \sys@edcdr12p1.+ASM2:+D2/jfv/tbsjfv.dbf

ASM Preffered Mirror

Introduction

The concept of the Preferred Read is not a new idea, but is now implemented in Oracle’s ASM volume 
management in Oracle 11g.The concept is to read from the storage that can present the needed data at a lower
latency.  Initially, this was designed for WAN or site-specific storage in order to avoid higher-latency site
connections. By restricting data reads to the local storage, the application would be able to service requests at
nominal read speeds while writes were the only communication needed to traverse the long haul site link. This is
a feature that is available to most Operating Systems with their included volume manager and as a feature to
Symantec/Veritas through the title Preferred Plex. This paper will discuss the merits of using PRG technology with Oracle ASM

eg:


SELECT name, failgroup FROM v$asm_disk;

NAME                           FAILGROUP
------------------------------ ------------------------------
DATA_0000                      DATA_0000
DATA_0001                      DATA_0001
DATA_0002                      DATA_0002

3 rows selected.

SQL> ALTER SYSTEM SET ASM_PREFERRED_READ_FAILURE_GROUPS = 'data.data_0000', 'data.data_0001', 'data.data_0002';

Allocating ASM Extent Sets 

With ASM redundancy, the first file extent allocated is chosen as primary extent, and
the mirrored extent is called the secondary extent. In the case of high redundancy,
there will two secondary extents. This logical grouping of primary and secondary
extents is called an extent set. When a block is read from disk, it is always read from
the primary extent, unless the primary extent cannot be read. In Oracle Database
11g, the preferred read feature allows the database to read the secondary extent first
instead of reading the primary extent. This is especially important for RAC Extended 

Cluster implementations.

As stated earlier, ASM in Oracle Database 10g always reads the primary copy of a mirrored extent set. Thus, a read for a specific block may require a read of the primary extent at the remote site across the interconnect. Accessing a remote disk through a metropolitan area or wide area storage network is substantially slower than accessing a local disk. This can tax the interconnect as well as result in high I/ O and network latency. To assuage this, Oracle Database 11g provides a feature called preferred reads. This feature enables ASM administrators to specify a failure group for local reads–– that is, provide preferred reads. In a normal or high-redundancy diskgroup, when a secondary extent is on a preferred disk and the primary extent is remote, the secondary extent is read rather than the primary one on that node. This feature is especially beneficial for extended cluster configurations.

To configure preferred read failure groups the disk group compatibility attributes must be set to 11.1 or higher. Once the compatibility options are correct, the ASM_PREFERRED_READ_FAILURE_GROUPS parameter is set to the preferred failure groups for each node.

SQL> select compatibility, database_compatibility
  2  from  v$asm_diskgroup
  3  where name = 'DG1'
  4  /

(

ASM_PREFERRED_READ_FAILURE_GROUPS=DATA.SITEA On first instance
ASM_PREFERRED_READ_FAILURE_GROUPS=DATA.SITEB On second instance

)