Bug 1389443 - CIB upgrade does not check what version is possible to upgrade to clusterwise
Summary: CIB upgrade does not check what version is possible to upgrade to clusterwise
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: pcs
Version: 7.3
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: rc
: ---
Assignee: Tomas Jelinek
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On:
Blocks: 1397408
TreeView+ depends on / blocked
 
Reported: 2016-10-27 15:15 UTC by Tomas Jelinek
Modified: 2017-08-01 18:24 UTC (History)
6 users (show)

Fixed In Version: pcs-0.9.156-1.el7
Doc Type: Bug Fix
Doc Text:
Cause: The user configures a new pacemaker feature (alerts in this case). Consequence: Only the nodes which support the feature get the configuration. On the other nodes the CIB is unchanged and does not contain the new configuration. This makes the CIB inconsistent across the cluster. Fix: Run the pacemaker tool for upgrading CIB version in such a way that it actually checks what version is supported clusterwise. Result: If some nodes do not support the new feature, print an error message and do not change current cluster configuration on any node.
Clone Of:
: 1397408 (view as bug list)
Environment:
Last Closed: 2017-08-01 18:24:40 UTC
Target Upstream Version:


Attachments (Terms of Use)
proposed fix (24.65 KB, patch)
2016-11-09 13:42 UTC, Tomas Jelinek
no flags Details | Diff
proposed fix (2.37 KB, patch)
2016-12-09 15:06 UTC, Tomas Jelinek
no flags Details | Diff


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:1958 normal SHIPPED_LIVE pcs bug fix and enhancement update 2017-08-01 18:09:47 UTC

Description Tomas Jelinek 2016-10-27 15:15:02 UTC
This is based on the discussion in bz1305130.

How the CIB upgrade process works in pacemaker:
1. Pacemaker keeps track of what pacemaker version is running on each node.
2. Pacemaker elects the DC in such a way that DC is always the node with the oldest pacemaker version in a cluster.
3. When the "cibadmin --upgrade" command is run, the request is sent to the DC.
4. DC bumps the CIB schema version to the newest version supported by that very DC and tells other nodes about the change.
5. As a result, the CIB never gets upgraded to a schema version which is not supported by all nodes.

If the CIB upgrade is requested on a file, then there is no communication in a cluster. CIB schema version simply gets bumped to the newest version supported by that particular node.


When moving to the new pcs architecture, the CIB upgrade process was moved to the pcs library. In order to get rid of the side effect, we switched the live CIB upgrade to file-based CIB upgrade. That way we bypass all the checking done in pacemaker (see steps 1-5 above).

What we need to do is switch back to the live upgrade (unless -f was specified on the command line) to ensure the correct upgrade procedure is used and deal with the resulting side effect.

Comment 1 Tomas Jelinek 2016-11-09 13:42:53 UTC
Created attachment 1218962 [details]
proposed fix

Test:

1) Setup:
- Have a cluster with no pacemaker alerts support (RHEL7.2).
- Upgrade pcs and pacemaker on one node to RHEL7.3 version with alerts support.
[root@rh72-node1:~]# rpm -q pcs
pcs-0.9.152-10.el7.x86_64
[root@rh72-node1:~]# rpm -q pacemaker
pacemaker-1.1.15-11.el7.x86_64
[root@rh72-node2:~]# rpm -q pacemaker
pacemaker-1.1.13-10.el7.x86_64
[root@rh72-node3:~]# rpm -q pacemaker
pacemaker-1.1.13-10.el7.x86_64

[root@rh72-node1:~]# pcs cluster cib | tr ' ' '\n' | grep validate-with
validate-with="pacemaker-2.3"
[root@rh72-node2:~]# pcs cluster cib | tr ' ' '\n' | grep validate-with
validate-with="pacemaker-2.3"
[root@rh72-node3:~]# pcs cluster cib | tr ' ' '\n' | grep validate-with
validate-with="pacemaker-2.3"


2) Before fix:
[root@rh72-node1:~]# pcs alert create path=/some/path
CIB has been upgraded to the latest schema version.
[root@rh72-node1:~]# pcs cluster cib | tr ' ' '\n' | grep validate-with
validate-with="pacemaker-2.5"
[root@rh72-node2:~]# pcs cluster cib | tr ' ' '\n' | grep validate-with
validate-with="pacemaker-2.3"
[root@rh72-node3:~]# pcs cluster cib | tr ' ' '\n' | grep validate-with
validate-with="pacemaker-2.3"


3) After fix:
[root@rh72-node1:~]# pcs alert create path=/some/path
Error: Upgrading of CIB to the latest schema failed: Call cib_upgrade failed (-62): Timer expired
[root@rh72-node1:~]# pcs cluster cib | tr ' ' '\n' | grep validate-with
validate-with="pacemaker-2.3"
[root@rh72-node2:~]# pcs cluster cib | tr ' ' '\n' | grep validate-with
validate-with="pacemaker-2.3"
[root@rh72-node3:~]# pcs cluster cib | tr ' ' '\n' | grep validate-with
validate-with="pacemaker-2.3"

Note pacemaker 1.1.13 exits with an error (timer expired) if current CIB schema version matches the latest available version. This is fixed in newer builds of pacemaker. It does not have any effect on pcs bug / behavior other than pcs printing different error message instead of saying the CIB is already at the newest schema available.


Only "pcs alert" and "pcs acl" commands are affected. The bug is in the new pcs library which other commands do not use yet. Acls require schema version 2.0 which is quite old so the bug may not manifest there.

The bug was introduced in pcs-0.9.152-3.el7.

Comment 2 Tomas Jelinek 2016-12-09 15:06:24 UTC
Created attachment 1230027 [details]
proposed fix

Since we now update the CIB in the live cluster prior to making the actual requested changes, we need to report the CIB has been upgraded even if we do not make any changes due to an error.

Before fix:
[root@rh73-node1:~]# pcs cluster cib | tr ' ' '\n' | grep validate-with
validate-with="pacemaker-2.3"
[root@rh73-node1:~]# pcs alert create path=/some/path id=b@d
Error: invalid alert-id 'b@d', '@' is not a valid character for a alert-id
[root@rh73-node1:~]# pcs cluster cib | tr ' ' '\n' | grep validate-with
validate-with="pacemaker-2.5"

After fix:
[root@rh73-node1:~]# pcs cluster cib | tr ' ' '\n' | grep validate-with
validate-with="pacemaker-2.3"
[root@rh73-node1:~]# pcs alert create path=/some/path id=b@d
CIB has been upgraded to the latest schema version.
Error: invalid alert-id 'b@d', '@' is not a valid character for a alert-id
[root@rh73-node1:~]# pcs cluster cib | tr ' ' '\n' | grep validate-with
validate-with="pacemaker-2.5"

Comment 4 Ivan Devat 2017-02-20 07:40:37 UTC
Setup:

> Have cluster with enough old pacemaker.

[vm-rhel72-1 ~] $ rpm -q pacemaker
pacemaker-1.1.13-10.el7.x86_64
[vm-rhel72-1 ~] $ pcs cluster cib | tr ' ' '\n' | grep validate-with
validate-with="pacemaker-2.3"

[vm-rhel72-3 ~] $ rpm -q pacemaker
pacemaker-1.1.13-10.el7.x86_64
[vm-rhel72-3 ~] $ pcs cluster cib | tr ' ' '\n' | grep validate-with
validate-with="pacemaker-2.3"

> Upgrade pacemaker on one node.

[vm-rhel72-1 ~] $ rpm -q pacemaker
pacemaker-1.1.16-2.el7.x86_64


After Fix:
[vm-rhel72-1 ~] $ rpm -q pcs
pcs-0.9.156-1.el7.x86_64

[vm-rhel72-1 ~] $ pcs alert create path=/some/path
Error: Upgrading of CIB to the latest schema failed: Call cib_upgrade failed (-62): Timer expired

[vm-rhel72-1 ~] $ pcs cluster cib | tr ' ' '\n' | grep validate-with
validate-with="pacemaker-2.3"
[vm-rhel72-3 ~] $ pcs cluster cib | tr ' ' '\n' | grep validate-with
validate-with="pacemaker-2.3"

Comment 7 errata-xmlrpc 2017-08-01 18:24:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1958


Note You need to log in before you can comment on or make changes to this bug.