Bug 617161

If we tell people that use '-r0' or [proposed] configupdate, then I agree - it should not allow decrease.

If people use explicitly -r(X-2) (ex: restore from backup config by force), maybe it's not a big deal.

CMAN doesn't care that the config decreases; it only cares that the config is the same across the nodes.  A quorate cluster w/ version 10 will reject joins from a cluster member claiming version 12.

OTOH, I do not know if other daemons rely on non-decreasing config_version ...  So, you're probably right.

There are several problems involved allowing config downgrades, but that´s not the only issue.

First of all, the behavior is triggered by a bug in cman config reload code.

Second, we can´t assume it´s ok to downgrade. Other daemons and config interfaces might get a fit.

We always assumed config is incremental and we need to stick to that.

http://git.fedorahosted.org/git/?p=cluster.git;a=commitdiff;h=ea01b37babe400a1f4545bd65e89d0ee6b35880e

Note for QE: the test cases for those fixes are complex. I´ll need to prepare a proper document comparing before and after the fix.

Both Lon and I did test the fixes so I have built for Snap9. Stay tuned.

Created attachment 434712 [details]
test cases

test cases to reproduce the issues with the old packages and how it works with the new ones.

Initial sanity check failed:

5 node cluster:

----- On all nodes, cman_tool version shows:

#  cman_tool version
6.2.0 config 1


-----Upgrade config:

[root@marathon-01 ~]# head /etc/cluster/cluster.conf
        <cluster name="marathon" config_version="9">
                <clusternodes>

[root@marathon-01 ~]# cman_tool version -r

----- On all nodes cman_tool shows:

# cman_tool version
6.2.0 config 9
# head /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster config_version="9" name="marathon">


----- Attempt to downgrade 

[root@marathon-01 ~]# head /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster config_version="5" name="marathon">

[root@marathon-01 ~]# cman_tool version -r
cman_tool: Error loading configuration in corosync/cman

[root@marathon-01 ~]# tail /var/log/messages
Jul 29 15:12:13 marathon-01 corosync[12263]:   [CMAN  ] Can't get updated config version: New configuration version has to be newer than current running configuration#012.


----- Check cman_tool version on all nodes:
[root@marathon-01 ~]# cman_tool version
6.2.0 config 9
[root@marathon-02 ~]# cman_tool version
6.2.0 config 9
[root@marathon-03 ~]# cman_tool version
6.2.0 config 9
[root@marathon-04 ~]# cman_tool version
6.2.0 config 9
[root@marathon-05 ~]# cman_tool version
6.2.0 config 9

But, look at the cluster.conf, it got the config_version="5" file!

[root@marathon-01 ~]# grep config_version /etc/cluster/cluster.conf
<cluster config_version="5" name="marathon">
[root@marathon-02 ~]# grep config_version /etc/cluster/cluster.conf
<cluster config_version="5" name="marathon">
[root@marathon-03 ~]# grep config_version /etc/cluster/cluster.conf
<cluster config_version="5" name="marathon">
[root@marathon-04 ~]# grep config_version /etc/cluster/cluster.conf
<cluster config_version="5" name="marathon">
[root@marathon-05 ~]# grep config_version /etc/cluster/cluster.conf
<cluster config_version="5" name="marathon">

(In reply to comment #10)
> Initial sanity check failed:
> 
> 5 node cluster:
> 
> ----- On all nodes, cman_tool version shows:
> 
> #  cman_tool version
> 6.2.0 config 1
> 
> 
> -----Upgrade config:
> 
> [root@marathon-01 ~]# head /etc/cluster/cluster.conf
>         <cluster name="marathon" config_version="9">
>                 <clusternodes>
> 
> [root@marathon-01 ~]# cman_tool version -r
> 
> ----- On all nodes cman_tool shows:
> 
> # cman_tool version
> 6.2.0 config 9
> # head /etc/cluster/cluster.conf
> <?xml version="1.0"?>
> <cluster config_version="9" name="marathon">
> 
> 
> ----- Attempt to downgrade 
> 
> [root@marathon-01 ~]# head /etc/cluster/cluster.conf
> <?xml version="1.0"?>
> <cluster config_version="5" name="marathon">
> 
> [root@marathon-01 ~]# cman_tool version -r
> cman_tool: Error loading configuration in corosync/cman
> 
> [root@marathon-01 ~]# tail /var/log/messages
> Jul 29 15:12:13 marathon-01 corosync[12263]:   [CMAN  ] Can't get updated
> config version: New configuration version has to be newer than current running
> configuration#012.
> 
> 
> ----- Check cman_tool version on all nodes:
> [root@marathon-01 ~]# cman_tool version
> 6.2.0 config 9
> [root@marathon-02 ~]# cman_tool version
> 6.2.0 config 9
> [root@marathon-03 ~]# cman_tool version
> 6.2.0 config 9
> [root@marathon-04 ~]# cman_tool version
> 6.2.0 config 9
> [root@marathon-05 ~]# cman_tool version
> 6.2.0 config 9
> 
> But, look at the cluster.conf, it got the config_version="5" file!
> 
> [root@marathon-01 ~]# grep config_version /etc/cluster/cluster.conf
> <cluster config_version="5" name="marathon">
> [root@marathon-02 ~]# grep config_version /etc/cluster/cluster.conf
> <cluster config_version="5" name="marathon">
> [root@marathon-03 ~]# grep config_version /etc/cluster/cluster.conf
> <cluster config_version="5" name="marathon">
> [root@marathon-04 ~]# grep config_version /etc/cluster/cluster.conf
> <cluster config_version="5" name="marathon">
> [root@marathon-05 ~]# grep config_version /etc/cluster/cluster.conf
> <cluster config_version="5" name="marathon">    

This is a different bug.

You have not been able to downgrade the configuration in the running cluster and that´s what this bug was all about.

The issue you are seeing here is that ccs_sync will still propagate the configuration regardless. We need to clone this bug in ricci (for ccs_sync) to implement a version check prior executing propagation.

This bug is indeed fixed since the runtime version of the cluster configuration has not downgraded.

Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.