Bug 1404233
| Summary: | Make pcs avoid a full CIB replacement when possible | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Michele Baldessari <michele> | ||||||
| Component: | pcs | Assignee: | Tomas Jelinek <tojeline> | ||||||
| Status: | CLOSED ERRATA | QA Contact: | cluster-qe <cluster-qe> | ||||||
| Severity: | urgent | Docs Contact: | |||||||
| Priority: | urgent | ||||||||
| Version: | 7.3 | CC: | abeekhof, cfeist, cluster-maint, fdinitto, idevat, jpokorny, kgaillot, michele, omular, rsteiger, snagar, tojeline | ||||||
| Target Milestone: | rc | Keywords: | ZStream | ||||||
| Target Release: | --- | ||||||||
| Hardware: | All | ||||||||
| OS: | Linux | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | pcs-0.9.156-1.el7 | Doc Type: | If docs needed, set a value | ||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | |||||||||
| : | 1412309 1413046 (view as bug list) | Environment: | |||||||
| Last Closed: | 2017-08-01 18:24:40 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Bug Depends On: | 1412309 | ||||||||
| Bug Blocks: | 1413046 | ||||||||
| Attachments: |
|
||||||||
I talked with Andrew and I think we want to do two things here (at least to start). Add a command to push only a diff between 2 cib versions, so the workflow would look like this: 1. pcs cib file1 2. cp file1 file1.orig 3. pcs -f file1 <do stuff here> 4. pcs cib-push file1 --diff-only=file1.orig So we would only be sending the difference between the original and the new files instead of updating the entire CIB. The other thing we can do is when we're not working with files we should only push diffs instead of the full CIB. That way it should reduce race issues when two programs are updating different parts of the CIB. There's a pacemaker tool that will give you the diff between two CIBs that is suitable (crm_diff). Just to make things clear. The real world usage does not involve using -f when running pcs. It was only used in the reproducer to show the issue. We aim to fix the issue where more that one change is done to the LIVE CIB in parallel. Created attachment 1239461 [details]
proposed fix
Test: pcs cluster cib > original.xml cp original.xml updated.xml pcs -f updated.xml resource create dummy dummy pcs cluster cib-push updated.xml diff-against=original.xml Verify that the resource was created in the cluster. If run in parallel all changes done are applied and no change gets lost due to it being overwritten by another. Per discussion with Andrew Beekhof, there is a bug in pacemaker's crm_diff that will need to be fixed for this to work properly without needing to loop, so I will clone this for pacemaker. Created attachment 1250141 [details]
additional fix
After Fix: [vm-rhel72-1 ~] $ rpm -q pcs pcs-0.9.156-1.el7.x86_64 [vm-rhel72-1 ~] $ pcs resource NO resources configured [vm-rhel72-1 ~] $ pcs cluster cib > original.xml [vm-rhel72-1 ~] $ cp original.xml updated.xml [vm-rhel72-1 ~] $ cp original.xml updated2.xml [vm-rhel72-1 ~] $ pcs -f updated.xml resource create A ocf:heartbeat:Dummy [vm-rhel72-1 ~] $ pcs -f updated2.xml resource create B ocf:heartbeat:Dummy [vm-rhel72-1 ~] $ pcs -f updated2.xml resource create C ocf:heartbeat:Dummy [vm-rhel72-1 ~] $ pcs cluster cib-push updated.xml diff-against=original.xml CIB updated [vm-rhel72-1 ~] $ pcs cluster cib-push updated2.xml diff-against=original.xml CIB updated [vm-rhel72-1 ~] $ pcs resource B (ocf::heartbeat:Dummy): Started vm-rhel72-2 C (ocf::heartbeat:Dummy): Started vm-rhel72-3 A (ocf::heartbeat:Dummy): Started vm-rhel72-2 See also reflection of this new provision in a "pull-modify-push chaining workflow" proposal, [bug 1359057 comment 4]. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:1958 |
Description of problem: (Bug opened after an email discussion with Ken, Tomas and Andrew) Currently pcs does a full CIB replacement when adding a resource or a constraint. So if I start with a cluster that has one resource: [root@centos tests]# pcs status ... Online: [ centos ] Full list of resources: Clone Set: delay1-clone [delay1] Started: [ centos ] ... I then make two copies of the CIB and add two separate resources to each CIB [root@centos tests]# pcs cluster cib 1.xml [root@centos tests]# pcs cluster cib 2.xml [root@centos tests]# /usr/sbin/pcs -f 1.xml resource create delayfoo ocf:heartbeat:Delay op start timeout=200s stop timeout=200s monitor timeout=200s --clone interleave=true [root@centos tests]# /usr/sbin/pcs -f 2.xml resource create delaybar ocf:heartbeat:Delay op start timeout=200s stop timeout=200s monitor timeout=200s --clone interleave=true [root@centos tests]# pcs cluster cib-push --config 1.xml CIB updated [root@centos tests]# pcs status ... Online: [ centos ] Full list of resources: Clone Set: delay1-clone [delay1] Started: [ centos ] Clone Set: delayfoo-clone [delayfoo] Stopped: [ centos ] ... Now when I push the second CIB with a separate resource I see that it actually replaces the CIB: [root@centos tests]# pcs cluster cib-push --config 2.xml CIB updated [root@centos tests]# pcs status ... Online: [ centos ] Full list of resources: Clone Set: delay1-clone [delay1] Started: [ centos ] Clone Set: delaybar-clone [delaybar] Stopped: [ centos ] ... The expected result would be one of the following: A) An error when trying to push 2.xml B) Both delayfoo and delaybar created Version-Release number of selected component (if applicable): pcs-0.9.152-10.el7.centos.x86_64 pacemaker-1.1.15-11.el7_3.2.x86_64 corosync-2.4.0-4.el7.x86_64 Additional Info: The reason for needing either an error or a "differential CIB update" is so that we can implement composable HA race-free in TripleO.