Bug 460909

Summary: Two new nodes with dirty flag don't see each other
Product: Red Hat Enterprise Linux 5 Reporter: Christine Caulfield <ccaulfie>
Component: cmanAssignee: Christine Caulfield <ccaulfie>
Status: CLOSED ERRATA QA Contact: Cluster QE <mspqa-list>
Severity: medium Docs Contact:
Priority: urgent    
Version: 5.2CC: bstevens, cluster-maint, edamato, nstraz, riek, tao
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-01-20 21:50:07 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 471273    
Attachments:
Description Flags
Small program to set the dirty flag in cman
none
Proposed patch none

Description Christine Caulfield 2008-09-02 15:34:57 UTC 
Description of problem:

If two nodes start up in a partitioned network - ie they can't see each other, often because the switch separating them hasn't sorted out its multicast routes yet - and have the dirty flags set, when the network connection is restored the two nodes don't see each other. No node is fenced or even marked "disallowed".


Version-Release number of selected component (if applicable):
5.0+

How reproducible:
Every time

Steps to Reproduce:
1. Separate the two nodes eg:
   on 131:  iptables -A INPUT -s 10.15.84.132 -p udp -j DROP
   on 132:  iptables -A INPUT -s 10.15.84.131 -p udp -j DROP
2. Join them to the cluster.
3. Set the dirty flag, I think fenced does this.
4. Join the cluster nodes with iptables -D INPUT 1

Actual results:

cman_tool nodes on both systems shows only the local node, and not the other. syslog shows that openais/Clm can see both nodes.


Expected results:

The other node shows up as "disallowed" or there is a fence race to kill one node.

Additional info:

This bug is similar to bz#443358

See also bz#460190
Comment 1 Christine Caulfield 2008-09-02 15:36:17 UTC 
Created attachment 315566 [details]
Small program to set the dirty flag in cman
Comment 2 Christine Caulfield 2008-09-02 15:37:24 UTC 
Created attachment 315567 [details]
Proposed patch

This patch should fix the problem. I'd like to see it tested rather a lot before releasing it though.
Comment 3 RHEL Program Management 2008-09-09 15:46:52 UTC 
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.
Comment 4 Christine Caulfield 2008-09-10 08:07:46 UTC 
On RHEL5:

commit 74721309f73dc6dc38abd07dc7c08e0ecb8ec602
Author: Christine Caulfield <ccaulfie>
Date:   Wed Sep 10 09:06:25 2008 +0100

cman: honour the dirty flag on a node we haven't seen before

and STABLE2
Comment 10 errata-xmlrpc 2009-01-20 21:50:07 UTC 
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-0189.html