Bug 112300 - services do not failover after a member is disconnected from shared storage and reboots
services do not failover after a member is disconnected from shared storage a...
Status: CLOSED ERRATA
Product: Red Hat Cluster Suite
Classification: Red Hat
Component: clumanager (Show other bugs)
3
All Linux
medium Severity high
: ---
: ---
Assigned To: Lon Hohberger
David Lawrence
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2003-12-17 08:48 EST by Lon Hohberger
Modified: 2009-04-16 16:14 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2004-01-23 12:59:11 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Patch to fix failover (7.04 KB, patch)
2003-12-17 08:52 EST, Lon Hohberger
no flags Details | Diff

  None (edit)
Description Lon Hohberger 2003-12-17 08:48:27 EST
Description of problem: Disconnecting the shared storage from a member
when it is running services causes the member to reboot (expected).
However, services do not fail-over to the other member.

Version-Release number: 1.2.6-1

How reproducible: 30%

Steps to Reproduce:
1. Start cluster on two members.
2. Move all services to member A
3. Disconnect shared storage from member A.
  
Actual results: Member A still "runs" all services, even after it is
marked 'Inactive'.

Expected results: Member B should take over services.

Additional info: Tested on shared SCSI RAID array; fibre channel may
have different behavior given that the timeouts are different.  For
instance, SCSI cable disconnects are more adequately handled in the
drivers, whereas some FC drivers continue to retry for several minutes
before giving up.
Comment 1 Lon Hohberger 2003-12-17 08:52:52 EST
Created attachment 96584 [details]
Patch to fix failover

This was due to a strange condition which may exist outside of the
shared-storage disconnect, but is certainly exascerbated by it.
Comment 3 Lon Hohberger 2003-12-17 09:07:29 EST
This fix breaks rolling upgrade in real-world testing. Fix it.
Comment 4 Lon Hohberger 2003-12-17 12:21:41 EST
Ok, rolling upgrade still works - but only if it is done in a special
way.  The lowest-ordered member must be upgraded *last*.  

This is because the new code checks for a cluster quorum incarnation
number, while the old code does not.  Since the lowest-ordered member
is the lock keeper, the check must be disabled until all other members
have restarted their services.

This workaround should be sufficient; as I believe this fix is
necessary for long-term support.
Comment 5 Lon Hohberger 2003-12-18 16:09:25 EST
(Official) Fix for this should appear in Update 1
Comment 8 Lon Hohberger 2004-01-15 16:21:38 EST
There was a second issue causing services not to fail over when the
high node was unplugged because the lock daemon was waiting for a
message from the quorum daemon, who was waiting for the STONITH
operations to complete.  The lock client would give up prematurely -
where it should have retried the locking operation during a failover.
 Fixed in 1.2.9-1
Comment 9 Suzanne Hillman 2004-01-15 16:41:33 EST
Verified, closed as errata.
Comment 10 Lon Hohberger 2004-01-15 17:47:28 EST
The latest CVS build fixes this:

http://people.redhat.com/lhh/packages.html
Comment 12 Gary Lerhaupt 2004-01-23 12:06:39 EST
We are still experiencing this issue with 1.2.9 on top of RHEL3 U1.
Comment 13 Lon Hohberger 2004-01-23 12:59:11 EST
Appropriate bug for your specific issue is:

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=113226

Comment 14 Lon Hohberger 2007-12-21 10:10:21 EST
Fixing product name.  Clumanager on RHEL3 was part of RHCS3, not RHEL3

Note You need to log in before you can comment on or make changes to this bug.