Bug 240523 - Big problem with raw device binding with EMC powerpath devices, on dual core server Dell PowerEdge 2900, for Oracle RAC database server
Big problem with raw device binding with EMC powerpath devices, on dual core ...
Status: CLOSED CANTFIX
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: initscripts (Show other bugs)
4.3
i686 Linux
medium Severity urgent
: ---
: ---
Assigned To: Bill Nottingham
Brock Organ
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2007-05-18 01:18 EDT by Hai Wu
Modified: 2014-03-16 23:06 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2008-12-08 16:15:32 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Hai Wu 2007-05-18 01:18:51 EDT
Description of problem:
Some random raw devices could not be bound to emcpower devices in boot-up
process per setting in '/etc/sysconfig/rawdevices' file, it always complains
that emcpower block device file can not be found. Manually run
'/etc/init.d/rawdevices start' will always work, after server boots up.

Version-Release number of selected component (if applicable):
EMC PowerPath 4.5.1 for Linux
RHEL ES4U3

How reproducible:


Steps to Reproduce:
1. Install EMC PowerPath 4.5.1 for Linux on Dell PowerEdge 2900 (DUAL CORE
server), using qlogic card. Have 4-path for each LUN (The same issue is not
happening to 2-path LUN).
2. put the following (similar) in /etc/sysconfigure/rawdevices file:
/dev/raw/raw1 /dev/emcpowerk1
/dev/raw/raw2 /dev/emcpowerj1
/dev/raw/raw3 /dev/emcpowera1
/dev/raw/raw4 /dev/emcpowern1
/dev/raw/raw5 /dev/emcpowero1
/dev/raw/raw6 /dev/emcpowerp1
/dev/raw/raw7 /dev/emcpowerq1
/dev/raw/raw8 /dev/emcpowerr1
/dev/raw/raw9 /dev/emcpowert1
/dev/raw/raw10 /dev/emcpowerl1
/dev/raw/raw11 /dev/emcpowerm1
/dev/raw/raw12 /dev/emcpowers1
/dev/raw/raw13 /dev/emcpoweru1
/dev/raw/raw14 /dev/emcpowerb1
/dev/raw/raw15 /dev/emcpowere1
/dev/raw/raw16 /dev/emcpowerf1
/dev/raw/raw17 /dev/emcpowerg1
/dev/raw/raw18 /dev/emcpowerh1
/dev/raw/raw19 /dev/emcpowerc1
/dev/raw/raw20 /dev/emcpowerd1
/dev/raw/raw21 /dev/emcpoweri1

3.
Reboot the system.
  
Actual results:
Most of the time, it would complain about one EMC device file could not be
found, for example, /dev/emcpowerd1 could not be found. Sometimes, it complains
about 2 EMC device files could not be found. In some rare cases, it would work fine.


Expected results:
It should always work.

Additional info:
I have to change file /etc/init.d/rawdevices to add one line:
change from:
       raw $RAW $BLOCK
change to:
       sleep 1
       raw $RAW $BLOCK

Does it sound like an EMC issue or RHEL supporting issue with dual core?

Thanks,
Hai
Comment 1 Bill Nottingham 2007-05-18 02:01:59 EDT
Seems like an issue in how long it takes EMC to become available after it has
started.
Comment 2 Hai Wu 2007-05-18 02:15:28 EDT
EMC powerpath inserts 3/4 lines into file /etc/rc.sysinit, to call
/etc/init.d/PowerPath start, I tried to put 5 seconds delay after that, it
didn't help. I went into the script /etc/init.d/PowerPath, and add some delay,
that didn't help.

The only place that it helps is this /etc/sysconfig/rawdevices file. I tried to
just sleep 0.2 seconds, and that also helps, I tested that for a few times, and
it worked fine.

It is a timing issue, but I don't know who needs to fix this. If we don't use
rawdevices, EMC powerpath always works fine. It is the interoperation between
these two that created this problem. It could also due to udev did not create
emcpower device in time for raw command to take that as its input.
Comment 3 Hai Wu 2007-05-18 02:25:11 EDT
Just tried to put 60 seconds delay after /etc/init.d/PowerPath start in the boot
up, and that didn't help, it complained that 2 emcpower device files were not found.
Comment 4 Bill Nottingham 2007-05-18 02:42:17 EDT
Is udev actually what's creating the emc devices ,or is that done by the
PowerPath script?
Comment 5 Hai Wu 2007-05-18 03:53:30 EDT
I am not sure, but in that script, it has some comments:
    #
    #  Wait for udev to finish creating emcpower devices
    #
    pdfound=1
    for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15; do

From this script, I can' tell how emc asks udev to create emcpower devices.
Comment 6 Bill Nottingham 2007-05-18 04:11:16 EDT
Hm, perhaps that script should run 'udevsettle' or similar to wait for events to
finish.
Comment 7 Bill Nottingham 2007-05-18 04:11:41 EDT
Actually, udevsettle probably postdates RHEL 4 - never mind.
Comment 8 Andy Speagle 2007-06-01 18:05:09 EDT
I was able to make it work by adding a 15 second sleep before 
the "pp_start_lvm" call in the "configure_pp" function in /etc/init.d/PowerPath:

--- snip
rc_check

sleep 15

pp_start_lvm
}
--- snip

This seemed to allow enough time for things to settle down on my system.  Of 
course, while this helps with my rawdevice startup bindings; I'm still having 
problems with LVM not finding the emcpower devices.... *grumble* ... it's 
definitely a timing issue that EMC needs to work on.  

I don't like having to hack /etc/rc.sysinit and other files to make a system 
wait for emcpower devices to show up.

I'm going to make some modifications to the ugly /etc/init.d/PowerPath script 
to make this work consistently.  Perhaps I can convince EMC to fix their 
package.
Comment 9 Bill Nottingham 2008-12-08 16:15:32 EST
Given that this seems to be solely an issue with EMC powerpath itself, I don't see anything we can fix in the initscripts themselves. Therefore, closing. (Apologies for the delay.)

Note You need to log in before you can comment on or make changes to this bug.