Bug 137021

Summary: ccs doesn't find most recent cluster.conf
Product: [Retired] Red Hat Cluster Suite Reporter: David Teigland <teigland>
Component: ccsAssignee: Jonathan Earl Brassow <jbrassow>
Status: CLOSED CURRENTRELEASE QA Contact: GFS Bugs <gfs-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 4CC: cluster-maint
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-01-27 18:03:56 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description David Teigland 2004-10-25 08:21:46 UTC
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7)
Gecko/20040626 Firefox/0.9.1

Description of problem:
All my nodes had just been rebooted and I wanted to make a change to
cluster.conf.  I made the change on one node, bumped the
config_version, and then started ccsd on all the nodes.

In theory, when cman_tool connects to ccsd, ccsd will go out, find the
most recent cluster.conf, and put it in /etc/cluster.  The config 
values read by cman_tool will then be the latest.  I ran "cman_tool 
join -c <clustername>" on all nodes but three of the eight didn't 
get the new cluster.conf, the other four did.


Version-Release number of selected component (if applicable):


How reproducible:
Didn't try

Steps to Reproduce:
1. reset all nodes
2. edit cluster.conf and increment config_version on one node
3. start ccsd on all nodes
4. run cman_tool join on all nodes
    

Additional info:

Comment 1 Kiersten (Kerri) Anderson 2004-11-04 15:08:05 UTC
Updates with the proper version and component name.

Comment 2 Kiersten (Kerri) Anderson 2004-11-04 15:16:56 UTC
Updates with the proper version and component name.

Comment 3 Kiersten (Kerri) Anderson 2004-11-04 15:21:02 UTC
Updates with the proper version and component name. Again, just love out tools.

Comment 4 Kiersten (Kerri) Anderson 2004-11-16 19:11:52 UTC
Updating version to the right level in the defects.  Sorry for the storm.

Comment 5 Jonathan Earl Brassow 2005-01-05 23:03:04 UTC
This boiled down to a timing issue.  ccsd processes request serially
(broadcast, or otherwise).

When cman_tool was started, it issued a connect.  While not yet
quorate, ccsd must broadcast to see if there are any more recent
versions of the config file.  This happens as part of the connect. 
select is used to set a timeout on just how long it waits for replies.

2 problems were encountered.  1) the timeout was not being properly
reset after the select returned.  2) everyone uses the same timeout

What results is that if a connect is issued simultaneously on every
node, they first try to process the connect - then any broadcast
requests.  Because they all have the same timeout, they never recieve
broadcast responses from their peers (because they are also stuck
processing connects).

The current solution is to add a random component to the timeout and
make sure to set the timeout properly after the select returns.