Bug 240508

Description of problem:
When a third node is added to a formerly 2node-mode cluster, cman fails
to clear the 2node flag.  The cluster is functional (failover of a service 
to/from the third node still works fine) but the only way to clear the 
flag on nodes 1 & 2 is to reboot them.


Version-Release number of selected component (if applicable):
cman-2.0.60-1.el5


How reproducible:
Every time.


Steps to Reproduce:
1. Start with functional 2-node failover cluster running (web) service.
2. Use s-c-c to add a 3rd node, then Send to Cluster.  Manual copy the
   cluster.conf to 3rd node.
3. node-3# service cman start; service rgmanager start

  
Actual results:

After node-3 joins cluster, output of "cman_tool status" on node-1 
and node-3, and output of "cman_tool nodes" on node-1, respectively:

[root@node-1 ~]# cman_tool status
Version: 6.0.1
Config Version: 2
Cluster Name: cluster7
Cluster Id: 26783
Cluster Member: Yes
Cluster Generation: 12
Membership state: Cluster-Member
Nodes: 3
Expected votes: 1
Total votes: 3
Quorum: 2
Active subsystems: 7
Flags: 2node
Ports Bound: 0 177
Node name: node-1.cluster-7.example.com
Node ID: 1
Multicast addresses: 239.192.104.8
Node addresses: 172.16.36.71

[root@node-3 ~]# cman_tool status
Version: 6.0.1
Config Version: 2
Cluster Name: cluster7
Cluster Id: 26783
Cluster Member: Yes
Cluster Generation: 12
Membership state: Cluster-Member
Nodes: 3
Expected votes: 3
Total votes: 3
Quorum: 2
Active subsystems: 7
Flags:
Ports Bound: 0 177
Node name: node-3.cluster-7.example.com
Node ID: 3
Multicast addresses: 239.192.104.8
Node addresses: 172.16.36.73

[root@node-1 ~]# cman_tool nodes
Node  Sts   Inc   Joined               Name
   1   M      4   2007-05-17 19:15:37  node-1.cluster-7.example.com
   2   M      8   2007-05-17 19:15:37  node-2.cluster-7.example.com
   3   M     12   2007-05-17 19:35:11  node-3.cluster-7.example.com


Expected results:
When the third node joined the cluster, nodes 1 and 2 should have 
had their 2node flag cleared in recognition of the new member.


Additional info:
I manually changed expected votes to 3 on nodes 1 & 2, and the service
was able to failover to node-3 (and back) with no problem.  
If node1 is rebooted, quorum is maintained and the services continue 
fine.  After node-1 comes back into the cluster, its 2node flag was 
cleared and it showed the proper nodes/votes/quorum.