Hey everyone,
Hope you’re all doing good.
A few days ago, I received a call from a client regarding an issue during a RAC instance startup running on 19.17 version.
This environment consists in a 2-node RAC, running with user segregation, so, for Grid Infrastructure the owner is grid and for database the owner is oracle.
When I tried to start the instance, I was getting this issue:
[oracle@DROBDDBRL02 ~]$ srvctl start instance -db OBD_RAC -instance OBD2
PRCR-1013 : Failed to start resource ora.obd_rac.db
PRCR-1064 : Failed to start resource ora.obd_rac.db on node drobddbrl02
CRS-5017: The resource action "ora.obd_rac.db start" encountered the following error:
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+OBD_DATA/OBD_RAC/spfileobd.ora'
ORA-17503: ksfdopn:10 Failed to open file +OBD_DATA/OBD_RAC/spfileobd.ora
ORA-01034: ORACLE not available
ORA-27121: unable to determine size of shared memory segment
Linux-x86_64 Error: 13: Permission denied
Additional information: 8702
Additional information: 57
. For details refer to "(:CLSN00107:)" in "/oracle/app/grid/diag/crs/drobddbrl02/crs/trace/crsd_oraagent_oracle.trc".
CRS-2674: Start of 'ora.obd_rac.db' on 'drobddbrl02' failed
So, it looks like that instance startup failed because SPFILE cannot be read.
When I tried to start the instance manually using SQL*Plus, the following error was displayed:
Total System Global Area 4294963272 bytes
Fixed Size 8904776 bytes
Variable Size 1308622848 bytes
Database Buffers 2952790016 bytes
Redo Buffers 24645632 bytes
ORA-00205: error in identifying control file, check alert log for more info
Humm, the instance is not able to read controlfile, so, the MOUNT failed.
We have two symptoms:
  • SPFILE is not read;
  • Control File is not read.

We can conclude that the cause is related to ASM, because controlfile and spfile are on ASM and cannot be read.

Let’s check how are the ASM DiskGroups:

SQL> select name,state from v$asm_diskgroup;

USERNAME        OSUSER
--------------- ----------
OBD_DATA        MOUNTED
OBD_FRA         MOUNTED
OBD_REDO1       MOUNTED
OBD_REDO2       MOUNTED
OK, ASM DiskGroups are mounted.
Then I did a check on oracle binary under GRID_HOME:
In Node 1, where the instance is working fine:
[root@DROBDDBRL01 ~]# ls -l $ORACLE_HOME/bin/oracle
-rwsr-s--x 1 grid oinstall 427363064 Mar 8 04:33 /oracle/app/193/grid/bin/oracle
In Node 2, where the instance is not able to start:
[root@DROBDDBRL02 ~]# ls -l $ORACLE_HOME/bin/oracle
-rwxrwxrwx 1 grid oinstall 427363064 Mar 8 04:33 /oracle/app/193/grid/bin/oracle

That’s it! Someone changed the permissions of binary to 777.

Let’s fix the permissions to the right one:
[root@DROBDDBRL02 ~]# chmod 6751 $ORACLE_HOME/bin/oracle
[root@DROBDDBRL02 ~]# ls -l $ORACLE_HOME/bin/oracle
-rwsr-s--x 1 grid oinstall 427363064 Mar 8 04:33 /oracle/app/193/grid/bin/oracle
After this we need to restart the cluster:
[root@DROBDDBRL02 ~]# crsctl stop crs -f
[root@DROBDDBRL02 ~]# crsctl start crs
Problem solved.
[oracle@DROBDDBRL01 ~]$ srvctl status database -db OBD_RAC
Instance OBD1 is running on node drobddbrl01
Instance OBD2 is running on node drobddbrl02
There was no GI patch, but the support note below explains the issue and the solution:
Database Not Started with ORA-00205 After Applying GI PSU (Doc ID 1487382.1)
Hope this helps.

Peace,

Vinicius