Description of problem:
orainstance.sh not detecting failures as best it could, thus allowing a failed DB instance to stay down/failed while the cluster continues to see it in a started state and get_db_status() continually tries to restart the DB. While the steps used here are an edge case, any false positive on a failure on a cluster service created for fail-over is an issue.

Version-Release number of selected component (if applicable):
rgmanager-2.0.52-37.el5_9.1

How reproducible:
Always.

Steps to Reproduce:
1. Setup DB instance in cluster
2. Start DB instance successfully
3. Break DB by moving pfile.
4. Kill Oracle processes
5. Allow cluster to detect failed DB
  
Actual results:
DB tries to restart, does not see failure and loop begins.

Expected results:
DB start failure detected and proper service recovery policy followed.

Additional info:
Logs from failure.

Apr 26 17:09:01 ilsvm0072 orainstance.sh: Restarting Oracle Database...
Apr 26 17:09:01 ilsvm0072 cat:
Apr 26 17:09:01 ilsvm0072 cat: SQL*Plus: Release 11.2.0.3.0 Production on Fri Apr 26 17:09:01 2013
Apr 26 17:09:01 ilsvm0072 cat:
Apr 26 17:09:01 ilsvm0072 cat: Copyright (c) 1982, 2011, Oracle.  All rights reserved.
Apr 26 17:09:01 ilsvm0072 cat:
Apr 26 17:09:01 ilsvm0072 cat: Connected to an idle instance.
Apr 26 17:09:01 ilsvm0072 cat:
Apr 26 17:09:01 ilsvm0072 cat: SQL> ORA-01078: failure in processing system parameters
Apr 26 17:09:01 ilsvm0072 cat: LRM-00109: could not open parameter file '/db/oratest/oracle/product/11203_64/dbs/initorarhel5.ora'
Apr 26 17:09:01 ilsvm0072 cat: SQL> Disconnected
Apr 26 17:10:01 ilsvm0072 orainstance.sh: Restarting Oracle Database...
Apr 26 17:10:01 ilsvm0072 cat:
Apr 26 17:10:01 ilsvm0072 cat: SQL*Plus: Release 11.2.0.3.0 Production on Fri Apr 26 17:10:01 2013
Apr 26 17:10:01 ilsvm0072 cat:
Apr 26 17:10:01 ilsvm0072 cat: Copyright (c) 1982, 2011, Oracle.  All rights reserved.
Apr 26 17:10:01 ilsvm0072 cat:
Apr 26 17:10:01 ilsvm0072 cat: Connected to an idle instance.
Apr 26 17:10:01 ilsvm0072 cat:
Apr 26 17:10:01 ilsvm0072 cat: SQL> ORA-01078: failure in processing system parameters
Apr 26 17:10:01 ilsvm0072 cat: LRM-00109: could not open parameter file '/db/oratest/oracle/product/11203_64/dbs/initorarhel5.ora'
Apr 26 17:10:01 ilsvm0072 cat: SQL> Disconnected
Apr 26 17:11:01 ilsvm0072 orainstance.sh: Restarting Oracle Database...

and so on...

The issue seems to be this:

# egrep -n "grep.*\^ORA" /usr/share/cluster/orainstance.sh
108:    grep -q "^ORA-" $logfile
158:    grep -q "^ORA-" $logfile

The $logfile entry is "^SQL> ORA-"