Bug 1361795

Summary: rhel-osp-director: 7.2->7.3Async->8.0 fails: Error: unable to push cib Call cib_replace failed (-205): Update was older than existing configuration
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: rhosp-directorAssignee: Angus Thomas <athomas>
Status: CLOSED DUPLICATE QA Contact: Omri Hochman <ohochman>
Severity: high Docs Contact:
Priority: high    
Version: 8.0 (Liberty)CC: aschultz, dbecker, jstransk, mandreou, mburns, morazi, rhel-osp-director-maint, sathlang
Target Milestone: async   
Target Release: 8.0 (Liberty)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-02-09 17:34:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alexander Chuzhoy 2016-07-30 15:15:08 UTC
rhel-osp-director:  7.2->7.3Async->8.0 fails: Error: unable to push cib  Call cib_replace failed (-205): Update was older than existing configuration

Environment:
openstack-tripleo-heat-templates-kilo-0.8.14-16.el7ost.noarch
openstack-tripleo-heat-templates-0.8.14-16.el7ost.noarch
instack-undercloud-2.2.7-7.el7ost.noarch
openstack-puppet-modules-7.1.3-1.el7ost.noarch

Steps to reproduce:
1. deploy 7.2 with:
openstack overcloud deploy --templates --control-scale 3 --compute-scale 1 --neutron-network-type vxlan --neutron-tunnel-types vxlan --ntp-server clock.redhat.com --timeout 90 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e network-environment.yaml --ceph-storage-scale 1

2. Populate the overcloud
3. Update the setup to 7.3async
4. Attempt to upgrade the setup to 8.0
Result:
The upgrade fails during the major-upgrade-pacemaker-init.yaml step:

2016-07-30 05:14:07 [ControllerConfig]: CREATE_IN_PROGRESS  state changed
2016-07-30 05:14:09 [ControllerConfig]: CREATE_COMPLETE  state changed
2016-07-30 05:14:10 [NodeTLSData]: UPDATE_COMPLETE  state changed
2016-07-30 05:14:11 [ControllerConfig]: UPDATE_IN_PROGRESS  state changed
2016-07-30 05:14:11 [ControllerConfig]: CREATE_IN_PROGRESS  state changed
2016-07-30 05:14:12 [ControllerConfig]: CREATE_COMPLETE  state changed
ERROR: Authentication failed: Authentication required


[stack@instack ~]$ heat resource-list -n5 overcloud|grep -v COMPLE
+-------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+---------------------+----------------------------------------------------------------------------------+
| resource_name                             | physical_resource_id                          | resource_type                                     | resource_status | updated_time        | stack_name                                                                       |
+-------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+---------------------+----------------------------------------------------------------------------------+
| CephStorage                               | 855b0fea-0ca4-48b3-ba42-28c6c137369a          | OS::Heat::ResourceGroup                           | UPDATE_FAILED   | 2016-07-30T05:11:24 | overcloud                                                                        |
| 0                                         | 117b0cbe-8086-4bb0-8502-4105c37b5986          | OS::TripleO::CephStorage                          | UPDATE_FAILED   | 2016-07-30T05:11:26 | overcloud-CephStorage-kalfd7gcktuj                                               |
| Compute                                   | 99827da2-4942-4638-ba3e-8fb87ffe0d92          | OS::Heat::ResourceGroup                           | UPDATE_FAILED   | 2016-07-30T05:11:46 | overcloud                                                                        |
| 0                                         | 81ae7c82-376e-4815-b558-2f28b9b38ba7          | OS::TripleO::Compute                              | UPDATE_FAILED   | 2016-07-30T05:11:49 | overcloud-Compute-s57fmy7wi4pq                                                   |
| Controller                                | e5393d15-bee5-4793-ac53-3cf05cff9818          | OS::Heat::ResourceGroup                           | UPDATE_FAILED   | 2016-07-30T05:11:51 | overcloud                                                                        |
| 1                                         | 690c706a-730e-4320-a2b1-b15f54f64ff8          | OS::TripleO::Controller                           | UPDATE_FAILED   | 2016-07-30T05:11:59 | overcloud-Controller-oewxp33qsmih                                                |
| UpdateDeployment                          | 0ab759e2-8bed-48ce-9548-1d9693a8c8df          | OS::Heat::SoftwareDeployment                      | UPDATE_FAILED   | 2016-07-30T05:12:00 | overcloud-CephStorage-kalfd7gcktuj-0-z3cpkaozmlf5                                |
| 2                                         | 082e9ec1-1b37-4ed6-94ad-ee20f253f5d0          | OS::TripleO::Controller                           | UPDATE_FAILED   | 2016-07-30T05:12:03 | overcloud-Controller-oewxp33qsmih                                                |
| 0                                         | 6b5be9dd-22c3-4bb2-9685-610691cb5e8a          | OS::TripleO::Controller                           | UPDATE_FAILED   | 2016-07-30T05:12:09 | overcloud-Controller-oewxp33qsmih                                                |
| UpdateDeployment                          | 60eb88da-685c-4b57-9c98-8dc8d409c493          | OS::Heat::SoftwareDeployment                      | UPDATE_FAILED   | 2016-07-30T05:12:26 | overcloud-Compute-s57fmy7wi4pq-0-cze7ikb45td6                                    |
| UpdateDeployment                          | 79b712d2-b329-4872-aa5d-3d26f9ba103a          | OS::Heat::SoftwareDeployment                      | UPDATE_FAILED   | 2016-07-30T05:12:54 | overcloud-Controller-oewxp33qsmih-1-tvyjay4p73wa                                 |
| UpdateDeployment                          | b30ea3b6-5ef1-418e-ad9c-51aac8b57372          | OS::Heat::SoftwareDeployment                      | UPDATE_FAILED   | 2016-07-30T05:13:13 | overcloud-Controller-oewxp33qsmih-0-okt4h7ois7wo                                 |
| UpdateDeployment                          | a578aa2e-6089-4b1d-ad72-df8677681d33          | OS::Heat::SoftwareDeployment                      | UPDATE_FAILED   | 2016-07-30T05:13:20 | overcloud-Controller-oewxp33qsmih-2-7ho5f2fms7wi                                 |
+-------------------------------------------+---------------


[stack@instack ~]$ echo -e `heat deployment-show 79b712d2-b329-4872-aa5d-3d26f9ba103a`|grep -i -A2 -B2 error
+ echo 'Applying new Pacemaker config'
+ pcs cluster cib-push /tmp/tmp.CBVk9DMtTj
Error: unable to push cib
Call cib_replace failed (-205): Update was older than existing configuration


No stopped pacemaker resources.

Comment 3 Marios Andreou 2016-08-01 08:33:45 UTC
Hi sasha o/ I left a comment at the other bug you mentioned that you were chasing when you hit the bug you report here, at https://bugzilla.redhat.com/show_bug.cgi?id=1361746#c3

I think it may also apply here as the root cause, in particular the "ERROR: Authentication failed: Authentication required"

Comment 4 Jiri Stransky 2016-08-01 09:41:13 UTC
(In reply to Alexander Chuzhoy from comment #0)
> [stack@instack ~]$ echo -e `heat deployment-show
> 79b712d2-b329-4872-aa5d-3d26f9ba103a`|grep -i -A2 -B2 error
> + echo 'Applying new Pacemaker config'
> + pcs cluster cib-push /tmp/tmp.CBVk9DMtTj
> Error: unable to push cib
> Call cib_replace failed (-205): Update was older than existing configuration

This comes from yum_update.sh, which should be no-opped at the time of upgrade init:

[stack@instack ~]$ grep PackageUpdate /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker-init.yaml
  OS::TripleO::Tasks::PackageUpdate:  ../extraconfig/tasks/yum_update_noop.yaml

Perhaps the error with Keystone trusts that Marios linked above could be the root cause of the cib_replace error too (maybe yum_update.sh wasn't correctly no-opped because of that error?).

Comment 5 Sofer Athlan-Guyot 2017-02-09 17:34:04 UTC
Hi Sasha,

closing it as very old.  Don't hesitate to re-open it if the _member_ doc tricks wasn't enough to fix this one.

*** This bug has been marked as a duplicate of bug 1361746 ***