Bug 683105

Summary: Error joining the fence group.
Product: Red Hat Enterprise Linux 5 Reporter: Hans van Leeuwen <hansvl>
Component: cmanAssignee: Lon Hohberger <lhh>
Status: CLOSED WORKSFORME QA Contact: Cluster QE <mspqa-list>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 5.5CC: cluster-maint, edamato
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-02-10 18:05:56 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
cluster.conf and sysconfig for all 3 nodes none

Description Hans van Leeuwen 2011-03-08 15:14:28 UTC 
Created attachment 482922 [details]
cluster.conf and sysconfig for all 3 nodes

Description of problem:

When starting a 3-node cluster on xen, all nodes take a long time (about 5 minutes) to complete booting. This seems to be caused by the fenced being
unable to join the fence group. After the long wait, cluster and fencing seem to work just fine.


Version-Release number of selected component (if applicable):

RHEL 5.5 Release (no updates)


How reproducible:

Always our setup


Steps to Reproduce:

1. Setup a RHEL 5.5 server with xen support
2. Create 3 vm's (voswla, voswlb, voswlc)
3. Setup fence_xvm on nodes and dom0
4. Copy the attached cluster.conf to /etc/cluster on all nodes
5. Enable cman, rgmanager using chkconfig
6. Start all nodes
  
Actual results:

All nodes hang for 5 minutes+ on "Starting fencing..."

Expected results:

Nodes start up right away.


Additional info:

After disabeling redirecting to /dev/null in /etc/init.d/cman line 188, the startup output looks like this:

Starting portmap: [  OK  ]
Starting NFS statd: [  OK  ]
Starting RPC idmapd: [  OK  ]
Starting cluster: 
   Loading modules... DLM (built Mar 16 2010 22:01:45) installed
GFS2 (built Mar 16 2010 22:02:11) installed
done
   Mounting configfs... done
   Starting ccsd... done
   Starting cman... done
   Starting daemons... done
   Starting fencing... Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Waiting for fenced to join the fence group.
Error joining the fence group.
done
[  OK  ]
Starting system message bus: [  OK  ]
Mounting other filesystems:  [  OK  ]
Starting PC/SC smart card daemon (pcscd): [  OK  ]
Starting scsi_reserve:[FAILED]
Starting HAL daemon: [  OK  ]
Starting hidd: [  OK  ]
Starting autofs:  Loading autofs4: [  OK  ]
Starting automount: [  OK  ]
[  OK  ]
Starting sshd: [  OK  ]
Starting sendmail: [  OK  ]
Starting sm-client: [  OK  ]
Starting console mouse services: [  OK  ]
Starting crond: [  OK  ]
Starting anacron: [  OK  ]
[  OK  ] atd: [  OK  ]
Starting yum-updatesd: [  OK  ]
Starting Avahi daemon... [  OK  ]
Starting up CIM server: [  OK  ]
Starting luci: [  OK  ]

Point your web browser to https://voswlb.vmb3test:8084 to access luci

Starting Cluster Module - cluster monitor: Setting verbosity level to LogBasic
[  OK  ]
Starting Cluster Service Manager: [  OK  ]
dlm: Using TCP for communications
Starting oddjobd: [  OK  ]
Starting ricci: [  OK  ]
Starting smartd: [  OK  ]
Comment 1 Lon Hohberger 2012-02-10 18:05:56 UTC 
This fell through the cracks.

This sounds like iptables was blocking traffic.

This worked for me in my RHEL 6 and RHEL 5 clusters.