Bug 361371

Summary: RHEL5U1beta: reboot service cman on one node can cause inconsistent cluster status.
Product: Red Hat Enterprise Linux 5 Reporter: Zhenyong(Jerry) Jiang <jzhenyon>
Component: cmanAssignee: Christine Caulfield <ccaulfie>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Cluster QE <mspqa-list>
Severity: high Docs Contact:
Priority: high    
Version: 5.0CC: cluster-maint, rrosario
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2008-07-07 14:54:00 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
message from gfs1
none
message from gfs2 none

Description Zhenyong(Jerry) Jiang 2007-11-01 08:41:22 UTC 
Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:

Always

Steps to Reproduce:

1. setup a tow-nodes cluster on RHEL5U1beta,（or RHEL5GA） 

2. make sure the cluster status is ok on both nodes.

3. restart service cman on node-a ( /etc/init.d/cman restart )

4. check the cluster status on both nodes, the command of clustat will show
different status.
  
Actual results:

node-a（in my example, it called gfs1）

  Member Name                        ID   Status
  ------ ----                        ---- ------
  gfs1                                  1 Online
  gfs2                                  2 Online


node-b（in my example, it called gfs2）

  Member Name                        ID   Status
  ------ ----                        ---- ------
  gfs1                                  1 Offline
  gfs2                                  2 Online

Expected results:

I think the status should be same.

Additional info:
Comment 1 Christine Caulfield 2007-11-01 08:57:12 UTC 
Can you post the output of 'cman_tool status' and 'cman_tool nodes' from bothe
systems please ?
Comment 2 Zhenyong(Jerry) Jiang 2007-11-05 09:12:02 UTC 
I restart cman on gfs1.

[root@gfs1 ~]# clustat 
msg_open: No such file or directory
Member Status: Quorate

  Member Name                        ID   Status
  ------ ----                        ---- ------
  gfs1                                  1 Online, Local
  gfs2                                  2 Online

[root@gfs1 ~]# 
[root@gfs1 ~]# 
[root@gfs1 ~]# 
[root@gfs1 ~]#  > /var/log/messages
[root@gfs1 ~]# /etc/init.d/cman restart
Stopping cluster: 
   Stopping fencing... done
   Stopping cman... done
   Stopping ccsd... done
   Unmounting configfs... done
                                                           [  OK  ]
Starting cluster: 
   Loading modules... done
   Mounting configfs... done
   Starting ccsd... done
   Starting cman... done
   Starting daemons... done
   Starting fencing... done
                                                           [  OK  ]
[root@gfs1 ~]# clustat
msg_open: No such file or directory
Member Status: Quorate

  Member Name                        ID   Status
  ------ ----                        ---- ------
  gfs1                                  1 Online, Local
  gfs2                                  2 Online

[root@gfs1 ~]# 



And this is shown on gfs2.

[root@gfs2 ~]# clustat 
msg_open: No such file or directory
Member Status: Quorate

  Member Name                        ID   Status
  ------ ----                        ---- ------
  gfs1                                  1 Online
  gfs2                                  2 Online, Local

[root@gfs2 ~]# << At this time, cman was restarted on gfs1 >>
[root@gfs2 ~]# clustat 
msg_open: No such file or directory
Member Status: Quorate

  Member Name                        ID   Status
  ------ ----                        ---- ------
  gfs1                                  1 Online
  gfs2                                  2 Online, Local
Comment 3 Zhenyong(Jerry) Jiang 2007-11-05 09:15:52 UTC 
Created attachment 247811 [details]
message from gfs1

message from gfs1
Comment 4 Zhenyong(Jerry) Jiang 2007-11-05 09:16:31 UTC 
Created attachment 247821 [details]
message from gfs2

message from gfs2
Comment 5 Christine Caulfield 2007-11-05 09:27:06 UTC 
I really need the cman_tool outputs. This was I can determine whether the
problem is in cman itself, or clustat/rgmanager.

The logs you have provided hint that openais is not at fault, but it's hard to
tell without the cman_tool information.