Red Hat Bugzilla – Bug 137021
ccs doesn't find most recent cluster.conf
Last modified: 2010-01-27 13:03:56 EST
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7)
Description of problem:
All my nodes had just been rebooted and I wanted to make a change to
cluster.conf. I made the change on one node, bumped the
config_version, and then started ccsd on all the nodes.
In theory, when cman_tool connects to ccsd, ccsd will go out, find the
most recent cluster.conf, and put it in /etc/cluster. The config
values read by cman_tool will then be the latest. I ran "cman_tool
join -c <clustername>" on all nodes but three of the eight didn't
get the new cluster.conf, the other four did.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. reset all nodes
2. edit cluster.conf and increment config_version on one node
3. start ccsd on all nodes
4. run cman_tool join on all nodes
Updates with the proper version and component name.
Updates with the proper version and component name. Again, just love out tools.
Updating version to the right level in the defects. Sorry for the storm.
This boiled down to a timing issue. ccsd processes request serially
(broadcast, or otherwise).
When cman_tool was started, it issued a connect. While not yet
quorate, ccsd must broadcast to see if there are any more recent
versions of the config file. This happens as part of the connect.
select is used to set a timeout on just how long it waits for replies.
2 problems were encountered. 1) the timeout was not being properly
reset after the select returned. 2) everyone uses the same timeout
What results is that if a connect is issued simultaneously on every
node, they first try to process the connect - then any broadcast
requests. Because they all have the same timeout, they never recieve
broadcast responses from their peers (because they are also stuck
The current solution is to add a random component to the timeout and
make sure to set the timeout properly after the select returns.