Bug 617161 - config reload allows configuration to be downgraded
config reload allows configuration to be downgraded
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: cluster (Show other bugs)
6.0
All Linux
low Severity medium
: rc
: ---
Assigned To: Fabio Massimo Di Nitto
Cluster QE
:
Depends On:
Blocks: 619680 625560
  Show dependency treegraph
 
Reported: 2010-07-22 07:15 EDT by Fabio Massimo Di Nitto
Modified: 2010-11-10 15:00 EST (History)
7 users (show)

See Also:
Fixed In Version: cluster-3.0.12-18.el6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 619680 (view as bug list)
Environment:
Last Closed: 2010-11-10 15:00:01 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
test cases (3.82 KB, text/plain)
2010-07-27 10:49 EDT, Fabio Massimo Di Nitto
no flags Details

  None (edit)
Description Fabio Massimo Di Nitto 2010-07-22 07:15:12 EDT
(this assume that the fix for #616222 to be in place or corosync will spin the CPU 100%)

start the cluster with cluster.conf at version 10

cman_tool version
6.2.0 config 10

edit cluster.conf to version 8

cman_tool version -r(any number < 8)

cman_tool version 
6.2.0 config 8

this should never be the case....
Comment 1 Lon Hohberger 2010-07-22 09:50:49 EDT
If we tell people that use '-r0' or [proposed] configupdate, then I agree - it should not allow decrease.

If people use explicitly -r(X-2) (ex: restore from backup config by force), maybe it's not a big deal.

CMAN doesn't care that the config decreases; it only cares that the config is the same across the nodes.  A quorate cluster w/ version 10 will reject joins from a cluster member claiming version 12.
Comment 2 Lon Hohberger 2010-07-22 09:51:38 EDT
OTOH, I do not know if other daemons rely on non-decreasing config_version ...  So, you're probably right.
Comment 3 Fabio Massimo Di Nitto 2010-07-22 13:24:48 EDT
There are several problems involved allowing config downgrades, but that´s not the only issue.

First of all, the behavior is triggered by a bug in cman config reload code.

Second, we can´t assume it´s ok to downgrade. Other daemons and config interfaces might get a fit.

We always assumed config is incremental and we need to stick to that.
Comment 5 Fabio Massimo Di Nitto 2010-07-27 04:50:34 EDT
Note for QE: the test cases for those fixes are complex. I´ll need to prepare a proper document comparing before and after the fix.

Both Lon and I did test the fixes so I have built for Snap9. Stay tuned.
Comment 6 Fabio Massimo Di Nitto 2010-07-27 10:49:02 EDT
Created attachment 434712 [details]
test cases

test cases to reproduce the issues with the old packages and how it works with the new ones.
Comment 10 Dean Jansa 2010-07-29 16:16:50 EDT
Initial sanity check failed:

5 node cluster:

----- On all nodes, cman_tool version shows:

#  cman_tool version
6.2.0 config 1


-----Upgrade config:

[root@marathon-01 ~]# head /etc/cluster/cluster.conf
        <cluster name="marathon" config_version="9">
                <clusternodes>

[root@marathon-01 ~]# cman_tool version -r

----- On all nodes cman_tool shows:

# cman_tool version
6.2.0 config 9
# head /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster config_version="9" name="marathon">


----- Attempt to downgrade 

[root@marathon-01 ~]# head /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster config_version="5" name="marathon">

[root@marathon-01 ~]# cman_tool version -r
cman_tool: Error loading configuration in corosync/cman

[root@marathon-01 ~]# tail /var/log/messages
Jul 29 15:12:13 marathon-01 corosync[12263]:   [CMAN  ] Can't get updated config version: New configuration version has to be newer than current running configuration#012.


----- Check cman_tool version on all nodes:
[root@marathon-01 ~]# cman_tool version
6.2.0 config 9
[root@marathon-02 ~]# cman_tool version
6.2.0 config 9
[root@marathon-03 ~]# cman_tool version
6.2.0 config 9
[root@marathon-04 ~]# cman_tool version
6.2.0 config 9
[root@marathon-05 ~]# cman_tool version
6.2.0 config 9

But, look at the cluster.conf, it got the config_version="5" file!

[root@marathon-01 ~]# grep config_version /etc/cluster/cluster.conf
<cluster config_version="5" name="marathon">
[root@marathon-02 ~]# grep config_version /etc/cluster/cluster.conf
<cluster config_version="5" name="marathon">
[root@marathon-03 ~]# grep config_version /etc/cluster/cluster.conf
<cluster config_version="5" name="marathon">
[root@marathon-04 ~]# grep config_version /etc/cluster/cluster.conf
<cluster config_version="5" name="marathon">
[root@marathon-05 ~]# grep config_version /etc/cluster/cluster.conf
<cluster config_version="5" name="marathon">
Comment 11 Fabio Massimo Di Nitto 2010-07-30 02:24:38 EDT
(In reply to comment #10)
> Initial sanity check failed:
> 
> 5 node cluster:
> 
> ----- On all nodes, cman_tool version shows:
> 
> #  cman_tool version
> 6.2.0 config 1
> 
> 
> -----Upgrade config:
> 
> [root@marathon-01 ~]# head /etc/cluster/cluster.conf
>         <cluster name="marathon" config_version="9">
>                 <clusternodes>
> 
> [root@marathon-01 ~]# cman_tool version -r
> 
> ----- On all nodes cman_tool shows:
> 
> # cman_tool version
> 6.2.0 config 9
> # head /etc/cluster/cluster.conf
> <?xml version="1.0"?>
> <cluster config_version="9" name="marathon">
> 
> 
> ----- Attempt to downgrade 
> 
> [root@marathon-01 ~]# head /etc/cluster/cluster.conf
> <?xml version="1.0"?>
> <cluster config_version="5" name="marathon">
> 
> [root@marathon-01 ~]# cman_tool version -r
> cman_tool: Error loading configuration in corosync/cman
> 
> [root@marathon-01 ~]# tail /var/log/messages
> Jul 29 15:12:13 marathon-01 corosync[12263]:   [CMAN  ] Can't get updated
> config version: New configuration version has to be newer than current running
> configuration#012.
> 
> 
> ----- Check cman_tool version on all nodes:
> [root@marathon-01 ~]# cman_tool version
> 6.2.0 config 9
> [root@marathon-02 ~]# cman_tool version
> 6.2.0 config 9
> [root@marathon-03 ~]# cman_tool version
> 6.2.0 config 9
> [root@marathon-04 ~]# cman_tool version
> 6.2.0 config 9
> [root@marathon-05 ~]# cman_tool version
> 6.2.0 config 9
> 
> But, look at the cluster.conf, it got the config_version="5" file!
> 
> [root@marathon-01 ~]# grep config_version /etc/cluster/cluster.conf
> <cluster config_version="5" name="marathon">
> [root@marathon-02 ~]# grep config_version /etc/cluster/cluster.conf
> <cluster config_version="5" name="marathon">
> [root@marathon-03 ~]# grep config_version /etc/cluster/cluster.conf
> <cluster config_version="5" name="marathon">
> [root@marathon-04 ~]# grep config_version /etc/cluster/cluster.conf
> <cluster config_version="5" name="marathon">
> [root@marathon-05 ~]# grep config_version /etc/cluster/cluster.conf
> <cluster config_version="5" name="marathon">    

This is a different bug.

You have not been able to downgrade the configuration in the running cluster and that´s what this bug was all about.

The issue you are seeing here is that ccs_sync will still propagate the configuration regardless. We need to clone this bug in ricci (for ccs_sync) to implement a version check prior executing propagation.

This bug is indeed fixed since the runtime version of the cluster configuration has not downgraded.
Comment 14 releng-rhel@redhat.com 2010-11-10 15:00:01 EST
Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.

Note You need to log in before you can comment on or make changes to this bug.