Bug 207004 - RHEL5 cman allows node to join cluster even though it's being fenced.
RHEL5 cman allows node to join cluster even though it's being fenced.
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: cman (Show other bugs)
5.0
All Linux
medium Severity medium
: ---
: ---
Assigned To: Christine Caulfield
Cluster QE
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2006-09-18 14:36 EDT by Josef Bacik
Modified: 2009-04-16 18:29 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2006-09-19 11:08:05 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Josef Bacik 2006-09-18 14:36:07 EDT
Description of problem:
If you setup a 2 node cluster, node1 and node2, and start cman on node1 and wait
for it to try to fence node2 because it hasn't joined within the post_join_delay
time period, if you're fence script is like mine and is messed up, fenced will
fail to fence the node and will continue trying, which is expected.  However, if
you start cman on node2, node1 will allow it to join, even though it is
currently trying to fence that node.  Everything will work perfectly, and fenced
will continue on in the background trying to fence that node.

Version-Release number of selected component (if applicable):
[root@rh3cluster1 ~]# rpm -q cman
cman-2.0.15-3.fc6

How reproducible:
Every time

Steps to Reproduce:
1.start cman on node1, wait until it starts to try and fence node 2
2.make sure your fence script will fail to fence the node
3.start cman on node2
  
Actual results:
Node2 will join properly and it will work

Expected results:
node2 shouldn't be allowed to join until the fencing script succeeds.

Additional info:
This is how RHEL4 works and I assume is the correct behavior.  I'm going to try
on 2.0.16 when it finishes getting built, and if its no longer a problem I will
close this ticket.
Comment 1 Christine Caulfield 2006-09-19 03:18:13 EDT
Joining the cluster is fine. What should /not/ happen is that an unfenced node
re-joins the fence domain.

That would be a bug in the fencing daemon I suspect.
Comment 2 Josef Bacik 2006-09-19 08:27:51 EDT
I think the fence domain was in JOIN_START_WAIT or something like that, the
thing that worried me was that rgmanager started fine.  I will upgrade
everything this morning and try again and see if it still happens.
Comment 3 Christine Caulfield 2006-09-19 08:40:15 EDT
JOIN_START_WAIT sounds fine then. 
Maybe rgmanager needs to check the fenced flag in cman ?


I'm not sure why it shoud be different though, because RHEL4 cman allowed
unfenced nodes to join the cman layer of the cluster too. fence & rgmanager are
layered above the cluster manager.
Comment 4 Josef Bacik 2006-09-19 10:57:15 EDT
Sep 19 10:55:03 rh5cluster1 fenced[5340]: rh5cluster2.gsslab.rdu.redhat.com not
a cluster member after 3 sec post_join_delay
Sep 19 10:55:03 rh5cluster1 fenced[5340]: fencing node
"rh5cluster2.gsslab.rdu.redhat.com"
Sep 19 10:55:08 rh5cluster1 fenced[5340]: agent "fence_apc" reports: failed:
unrecognised menu response
Sep 19 10:55:08 rh5cluster1 fenced[5340]: fence
"rh5cluster2.gsslab.rdu.redhat.com" failed
Sep 19 10:55:13 rh5cluster1 fenced[5340]: fencing node
"rh5cluster2.gsslab.rdu.redhat.com"
Sep 19 10:55:15 rh5cluster1 fenced[5340]: agent "fence_apc" reports: failed:
unrecognised menu response
Sep 19 10:55:15 rh5cluster1 fenced[5340]: fence
"rh5cluster2.gsslab.rdu.redhat.com" failed


[root@rh5cluster2 ~]# cman_tool services
type             level name     id       state       
fence            0     default  00010001 none        
[1 2]

[root@rh5cluster2 ~]# cman_tool nodes
Node  Sts   Inc   Joined               Name
   1   M     88   2006-09-19 10:55:18  rh5cluster1.gsslab.rdu.redhat.com
   2   M     84   2006-09-19 10:55:18  rh5cluster2.gsslab.rdu.redhat.com

and now rh5cluster1 is no longer trying to fence rh5cluster2.
Comment 5 Josef Bacik 2006-09-19 11:07:10 EDT
I guess i like this better, that way if the fencing fails you don't have this
hang because its trying to fence a node that took too long to bootup. 
rh5cluster1 has stopped trying to fence the node after the node joined the
cluster, which works for me.
Comment 6 Josef Bacik 2006-09-19 11:20:05 EDT
Well crap looks like fenced kicked back on when I tried to start clvm, now we're
back in a messed up state

[root@rh5cluster1 ~]# cman_tool services
type             level name     id       state       
fence            0     default  00010001 JOIN_START_WAIT
[1 2]
dlm              1     clvmd    00020001 JOIN_START_WAIT
[1 2]

tho it looks like this is the reason fenced kicked back in

Sep 19 11:08:52 rh5cluster1 openais[5314]: [TOTEM] Sending initial ORF token
Sep 19 11:08:52 rh5cluster1 openais[5314]: [CLM  ] CLM CONFIGURATION CHANGE
Sep 19 11:08:52 rh5cluster1 openais[5314]: [CLM  ] New Configuration:
Sep 19 11:08:52 rh5cluster1 openais[5314]: [CLM  ]      r(0) ip(10.10.1.13)
Sep 19 11:08:52 rh5cluster1 fenced[5340]: rh5cluster2.gsslab.rdu.redhat.com not
a cluster member after 0 sec post_fail_delay
Sep 19 11:08:52 rh5cluster1 openais[5314]: [CLM  ] Members Left:
Sep 19 11:08:52 rh5cluster1 openais[5314]: [CLM  ]      no interface found for
nodeid
Sep 19 11:08:52 rh5cluster1 fenced[5340]: fencing node
"rh5cluster2.gsslab.rdu.redhat.com"
Sep 19 11:08:52 rh5cluster1 openais[5314]: [CLM  ] Members Joined:
Sep 19 11:08:52 rh5cluster1 openais[5314]: [SYNC ] This node is within the
primary component and will provide service.
Sep 19 11:08:52 rh5cluster1 openais[5314]: [CLM  ] CLM CONFIGURATION CHANGE
Sep 19 11:08:52 rh5cluster1 openais[5314]: [CLM  ] New Configuration:
Sep 19 11:08:52 rh5cluster1 openais[5314]: [CLM  ]      r(0) ip(10.10.1.13)
Sep 19 11:08:52 rh5cluster1 openais[5314]: [CLM  ] Members Left:
Sep 19 11:08:52 rh5cluster1 openais[5314]: [CLM  ] Members Joined:

I don't know why rh5cluster2 just randomly dropped out, there were no network
problems, though that time corresponds with the time that i did

service clvmd start

on rh5cluster2.  The servcie clvmd start hung for a while, but succeeded, with a
bunch of errors in /var/log/messages

Sep 19 11:18:13 rh5cluster2 kernel: dlm: clvmd: recover 7
Sep 19 11:18:13 rh5cluster2 kernel: dlm: clvmd: add member 1
Sep 19 11:18:13 rh5cluster2 kernel: dlm: clvmd: config mismatch: 32,0 nodeid 1: 0,0
Sep 19 11:18:13 rh5cluster2 kernel: dlm: clvmd: ping_members aborted -22 last
nodeid 1
Sep 19 11:18:13 rh5cluster2 kernel: dlm: clvmd: total members 2 error -22
Sep 19 11:18:13 rh5cluster2 kernel: dlm: clvmd: recover_members failed -22
Sep 19 11:18:13 rh5cluster2 kernel: dlm: clvmd: recover 7 error -22
Sep 19 11:18:15 rh5cluster2 kernel: dlm: lockspace 20001 from 1 not found
Sep 19 11:18:47 rh5cluster2 last message repeated 55 times

and everything still in JOIN_START_WAIT
Comment 7 Christine Caulfield 2006-09-19 11:39:48 EDT
But isn't that what you wanted? everything waiting for the node to be fenced ?
Comment 8 Josef Bacik 2006-09-19 11:44:03 EDT
Yes, but thats not what's happening.

Node1 starts
Node1 tries to fence node2 because its not up
Node2 joins
Node1 stops trying to fence node2
Node2 starts fenced, it succeeds
Node1 starts clvm, it succeeds
Node2 starts clvm, it hangs
Node1 thinks that node2 has gone out to lunch and tries to fence again
Node2 eventually clvm succeeds
All   services in JOIN_START_WAIT

node1 shouldn't stop trying to fenced node2 until its successful imho, but thats
what is happening.
Comment 9 Josef Bacik 2006-09-19 11:51:27 EDT
Ok i'm going to close this, every other bug I hit isn't related to this as this
is working properly, having the node be allowed to join and stop the fencing
action is proper.
Comment 10 Nate Straz 2007-12-13 12:22:20 EST
Moving all RHCS ver 5 bugs to RHEL 5 so we can remove RHCS v5 which never existed.

Note You need to log in before you can comment on or make changes to this bug.