rhel-osp-director: 7.2->7.3Async->8.0 fails: Error: unable to push cib Call cib_replace failed (-205): Update was older than existing configuration Environment: openstack-tripleo-heat-templates-kilo-0.8.14-16.el7ost.noarch openstack-tripleo-heat-templates-0.8.14-16.el7ost.noarch instack-undercloud-2.2.7-7.el7ost.noarch openstack-puppet-modules-7.1.3-1.el7ost.noarch Steps to reproduce: 1. deploy 7.2 with: openstack overcloud deploy --templates --control-scale 3 --compute-scale 1 --neutron-network-type vxlan --neutron-tunnel-types vxlan --ntp-server clock.redhat.com --timeout 90 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e network-environment.yaml --ceph-storage-scale 1 2. Populate the overcloud 3. Update the setup to 7.3async 4. Attempt to upgrade the setup to 8.0 Result: The upgrade fails during the major-upgrade-pacemaker-init.yaml step: 2016-07-30 05:14:07 [ControllerConfig]: CREATE_IN_PROGRESS state changed 2016-07-30 05:14:09 [ControllerConfig]: CREATE_COMPLETE state changed 2016-07-30 05:14:10 [NodeTLSData]: UPDATE_COMPLETE state changed 2016-07-30 05:14:11 [ControllerConfig]: UPDATE_IN_PROGRESS state changed 2016-07-30 05:14:11 [ControllerConfig]: CREATE_IN_PROGRESS state changed 2016-07-30 05:14:12 [ControllerConfig]: CREATE_COMPLETE state changed ERROR: Authentication failed: Authentication required [stack@instack ~]$ heat resource-list -n5 overcloud|grep -v COMPLE +-------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+---------------------+----------------------------------------------------------------------------------+ | resource_name | physical_resource_id | resource_type | resource_status | updated_time | stack_name | +-------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+---------------------+----------------------------------------------------------------------------------+ | CephStorage | 855b0fea-0ca4-48b3-ba42-28c6c137369a | OS::Heat::ResourceGroup | UPDATE_FAILED | 2016-07-30T05:11:24 | overcloud | | 0 | 117b0cbe-8086-4bb0-8502-4105c37b5986 | OS::TripleO::CephStorage | UPDATE_FAILED | 2016-07-30T05:11:26 | overcloud-CephStorage-kalfd7gcktuj | | Compute | 99827da2-4942-4638-ba3e-8fb87ffe0d92 | OS::Heat::ResourceGroup | UPDATE_FAILED | 2016-07-30T05:11:46 | overcloud | | 0 | 81ae7c82-376e-4815-b558-2f28b9b38ba7 | OS::TripleO::Compute | UPDATE_FAILED | 2016-07-30T05:11:49 | overcloud-Compute-s57fmy7wi4pq | | Controller | e5393d15-bee5-4793-ac53-3cf05cff9818 | OS::Heat::ResourceGroup | UPDATE_FAILED | 2016-07-30T05:11:51 | overcloud | | 1 | 690c706a-730e-4320-a2b1-b15f54f64ff8 | OS::TripleO::Controller | UPDATE_FAILED | 2016-07-30T05:11:59 | overcloud-Controller-oewxp33qsmih | | UpdateDeployment | 0ab759e2-8bed-48ce-9548-1d9693a8c8df | OS::Heat::SoftwareDeployment | UPDATE_FAILED | 2016-07-30T05:12:00 | overcloud-CephStorage-kalfd7gcktuj-0-z3cpkaozmlf5 | | 2 | 082e9ec1-1b37-4ed6-94ad-ee20f253f5d0 | OS::TripleO::Controller | UPDATE_FAILED | 2016-07-30T05:12:03 | overcloud-Controller-oewxp33qsmih | | 0 | 6b5be9dd-22c3-4bb2-9685-610691cb5e8a | OS::TripleO::Controller | UPDATE_FAILED | 2016-07-30T05:12:09 | overcloud-Controller-oewxp33qsmih | | UpdateDeployment | 60eb88da-685c-4b57-9c98-8dc8d409c493 | OS::Heat::SoftwareDeployment | UPDATE_FAILED | 2016-07-30T05:12:26 | overcloud-Compute-s57fmy7wi4pq-0-cze7ikb45td6 | | UpdateDeployment | 79b712d2-b329-4872-aa5d-3d26f9ba103a | OS::Heat::SoftwareDeployment | UPDATE_FAILED | 2016-07-30T05:12:54 | overcloud-Controller-oewxp33qsmih-1-tvyjay4p73wa | | UpdateDeployment | b30ea3b6-5ef1-418e-ad9c-51aac8b57372 | OS::Heat::SoftwareDeployment | UPDATE_FAILED | 2016-07-30T05:13:13 | overcloud-Controller-oewxp33qsmih-0-okt4h7ois7wo | | UpdateDeployment | a578aa2e-6089-4b1d-ad72-df8677681d33 | OS::Heat::SoftwareDeployment | UPDATE_FAILED | 2016-07-30T05:13:20 | overcloud-Controller-oewxp33qsmih-2-7ho5f2fms7wi | +-------------------------------------------+--------------- [stack@instack ~]$ echo -e `heat deployment-show 79b712d2-b329-4872-aa5d-3d26f9ba103a`|grep -i -A2 -B2 error + echo 'Applying new Pacemaker config' + pcs cluster cib-push /tmp/tmp.CBVk9DMtTj Error: unable to push cib Call cib_replace failed (-205): Update was older than existing configuration No stopped pacemaker resources.
Hi sasha o/ I left a comment at the other bug you mentioned that you were chasing when you hit the bug you report here, at https://bugzilla.redhat.com/show_bug.cgi?id=1361746#c3 I think it may also apply here as the root cause, in particular the "ERROR: Authentication failed: Authentication required"
(In reply to Alexander Chuzhoy from comment #0) > [stack@instack ~]$ echo -e `heat deployment-show > 79b712d2-b329-4872-aa5d-3d26f9ba103a`|grep -i -A2 -B2 error > + echo 'Applying new Pacemaker config' > + pcs cluster cib-push /tmp/tmp.CBVk9DMtTj > Error: unable to push cib > Call cib_replace failed (-205): Update was older than existing configuration This comes from yum_update.sh, which should be no-opped at the time of upgrade init: [stack@instack ~]$ grep PackageUpdate /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-pacemaker-init.yaml OS::TripleO::Tasks::PackageUpdate: ../extraconfig/tasks/yum_update_noop.yaml Perhaps the error with Keystone trusts that Marios linked above could be the root cause of the cib_replace error too (maybe yum_update.sh wasn't correctly no-opped because of that error?).
Hi Sasha, closing it as very old. Don't hesitate to re-open it if the _member_ doc tricks wasn't enough to fix this one. *** This bug has been marked as a duplicate of bug 1361746 ***