Hide Forgot
+++ This bug was initially created as a clone of Bug #1381317 +++ Description of problem: Trying to update the Openstack 8 to latest. Executed the openstack overcloud update and it returned "COMPLETE" in seconds, but a heat resource-list overcloud show items in progress. [rlp@paisley-dir ~]$ openstack overcloud update stack overcloud -i --templates ~/pilot/templates/overcloud -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e ~/pilot/templates/overcloud/environments/network-isolation.yaml -e ~/pilot/templates/overcloud/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml -e ~/pilot/templates/dell-environment.yaml -e ~/pilot/templates/network-environment.yaml starting package update on stack overcloud COMPLETE update finished with status COMPLETE' [rlp@paisley-dir ~]$ heat resource-list overcloud +-------------------------------------------+-----------------------------------------------+---------------------------------------------------+--------------------+---------------------+ | resource_name | physical_resource_id | resource_type | resource_status | updated_time | +-------------------------------------------+-----------------------------------------------+---------------------------------------------------+--------------------+---------------------+ | AllNodesExtraConfig | 7a75e1a3-5035-4158-b838-5e7899d5546d | OS::TripleO::AllNodesExtraConfig | CREATE_COMPLETE | 2016-10-03T12:47:40 | | AllNodesValidationConfig | 938b4fb4-d0ce-448c-815d-00d1bff2b5dc | OS::TripleO::AllNodes::Validation | CREATE_COMPLETE | 2016-10-03T12:47:40 | | BlockStorageAllNodesDeployment | 1fc5450d-5a23-4b86-85fc-fad5d1cecb4a | OS::Heat::StructuredDeployments | CREATE_COMPLETE | 2016-10-03T12:47:40 | | BlockStorageAllNodesValidationDeployment | a4c6ce81-611e-458f-913e-0239a9b1ed79 | OS::Heat::StructuredDeployments | CREATE_COMPLETE | 2016-10-03T12:47:40 | | BlockStorageNodesPostDeployment | 9f12010b-70a1-4661-8708-156eae0894b2 | OS::TripleO::BlockStoragePostDeployment | CREATE_COMPLETE | 2016-10-03T12:47:40 | | CephClusterConfig | 651858f8-ae97-4472-bcc4-4fc6e39c8ae3 | OS::TripleO::CephClusterConfig::SoftwareConfig | CREATE_COMPLETE | 2016-10-03T12:47:40 | | CephStorageAllNodesDeployment | bdd57215-6db1-4f66-b973-d6d1d6f83015 | OS::Heat::StructuredDeployments | CREATE_COMPLETE | 2016-10-03T12:47:40 | | CephStorageAllNodesValidationDeployment | a1c249b2-b597-4276-a8ba-bdaeb5814bf7 | OS::Heat::StructuredDeployments | CREATE_COMPLETE | 2016-10-03T12:47:40 | | CephStorageCephDeployment | 3b5abf14-a25d-40c4-a870-d1305b363d17 | OS::Heat::StructuredDeployments | CREATE_COMPLETE | 2016-10-03T12:47:40 | | CephStorageNodesPostDeployment | b9f0e4fb-e490-4e5d-9df1-47fd29131e69 | OS::TripleO::CephStoragePostDeployment | CREATE_COMPLETE | 2016-10-03T12:47:40 | | ComputeAllNodesDeployment | 6a88157c-6957-44d9-9ca1-185333f3779f | OS::Heat::StructuredDeployments | CREATE_COMPLETE | 2016-10-03T12:47:40 | | ComputeAllNodesValidationDeployment | 5660e3d7-d1c5-44f6-bce4-2c9ecb926e6b | OS::Heat::StructuredDeployments | CREATE_COMPLETE | 2016-10-03T12:47:40 | | ComputeCephDeployment | 2b9d3b1d-70ea-4f8f-aa99-e1ca6cd0d337 | OS::Heat::StructuredDeployments | CREATE_COMPLETE | 2016-10-03T12:47:40 | | ComputeNodesPostDeployment | d57db68c-448c-4093-9187-9775b203e53e | OS::TripleO::ComputePostDeployment | CREATE_COMPLETE | 2016-10-03T12:47:40 | | ControlVirtualIP | 9ad92792-f1cb-4b6a-93fa-f1039e259d5b | OS::Neutron::Port | CREATE_COMPLETE | 2016-10-03T12:47:40 | | ControllerAllNodesDeployment | d9c0fb92-1158-4c41-a0e2-aa77cf2af0b8 | OS::Heat::StructuredDeployments | CREATE_COMPLETE | 2016-10-03T12:47:40 | | ControllerAllNodesValidationDeployment | 191d389a-9379-4c4d-bf44-b059fc58abfb | OS::Heat::StructuredDeployments | CREATE_COMPLETE | 2016-10-03T12:47:40 | | ControllerBootstrapNodeConfig | 390fc2f4-7863-4fd4-b82b-508c72f73a16 | OS::TripleO::BootstrapNode::SoftwareConfig | CREATE_COMPLETE | 2016-10-03T12:47:40 | | ControllerBootstrapNodeDeployment | aea62014-3504-4d54-a24e-81b19f2b4066 | OS::Heat::StructuredDeployments | CREATE_COMPLETE | 2016-10-03T12:47:40 | | ControllerCephDeployment | 43984b9f-3937-47cd-b573-1b49369cfc6c | OS::Heat::StructuredDeployments | CREATE_COMPLETE | 2016-10-03T12:47:40 | | ControllerClusterConfig | 122585b3-0766-449c-a6df-6bf70a9f5eb5 | OS::Heat::StructuredConfig | CREATE_COMPLETE | 2016-10-03T12:47:40 | | ControllerClusterDeployment | e1402010-e7ed-40dd-9550-d4fee3730674 | OS::Heat::StructuredDeployments | CREATE_COMPLETE | 2016-10-03T12:47:40 | | ControllerIpListMap | 48bf22cc-fc67-4d66-87ac-6730fed0f113 | OS::TripleO::Network::Ports::NetIpListMap | CREATE_COMPLETE | 2016-10-03T12:47:40 | | ControllerNodesPostDeployment | 15614bab-8286-476e-8149-6ee53daf1b9f | OS::TripleO::ControllerPostDeployment | CREATE_COMPLETE | 2016-10-03T12:47:40 | | ControllerSwiftDeployment | e2a87b75-8c0e-406c-8fd3-df9a8f1c6710 | OS::Heat::StructuredDeployments | CREATE_COMPLETE | 2016-10-03T12:47:40 | | HeatAuthEncryptionKey | overcloud-HeatAuthEncryptionKey-uprxipn5izjo | OS::Heat::RandomString | CREATE_COMPLETE | 2016-10-03T12:47:40 | | HorizonSecret | overcloud-HorizonSecret-r5qqdgdoi3ay | OS::Heat::RandomString | CREATE_COMPLETE | 2016-10-03T12:47:40 | | MysqlClusterUniquePart | overcloud-MysqlClusterUniquePart-cw5l3dijhrpg | OS::Heat::RandomString | CREATE_COMPLETE | 2016-10-03T12:47:40 | | MysqlRootPassword | overcloud-MysqlRootPassword-bp7yltgtop47 | OS::Heat::RandomString | CREATE_COMPLETE | 2016-10-03T12:47:40 | | ObjectStorageAllNodesDeployment | 1f2661e2-244f-4e3a-95ed-724c63a43d60 | OS::Heat::StructuredDeployments | CREATE_COMPLETE | 2016-10-03T12:47:40 | | ObjectStorageAllNodesValidationDeployment | 4e950ffa-76db-497f-81df-94c455569500 | OS::Heat::StructuredDeployments | CREATE_COMPLETE | 2016-10-03T12:47:40 | | ObjectStorageNodesPostDeployment | 87242aff-3496-4f4d-b7b1-6d9a8ca353df | OS::TripleO::ObjectStoragePostDeployment | CREATE_COMPLETE | 2016-10-03T12:47:40 | | ObjectStorageSwiftDeployment | 4a5178ad-7797-4e6e-93a0-3baf74909811 | OS::Heat::StructuredDeployments | CREATE_COMPLETE | 2016-10-03T12:47:40 | | PcsdPassword | overcloud-PcsdPassword-4ck57yz2plz7 | OS::Heat::RandomString | CREATE_COMPLETE | 2016-10-03T12:47:40 | | RabbitCookie | overcloud-RabbitCookie-7uemvh2hz3mx | OS::Heat::RandomString | CREATE_COMPLETE | 2016-10-03T12:47:40 | | SwiftDevicesAndProxyConfig | baea166f-ebc9-42d5-8691-69764ba575ad | OS::TripleO::SwiftDevicesAndProxy::SoftwareConfig | CREATE_COMPLETE | 2016-10-03T12:47:40 | | UpdateWorkflow | 62a09843-f1a4-4420-9686-9fcb1e03356f | OS::TripleO::Tasks::UpdateWorkflow | CREATE_COMPLETE | 2016-10-03T12:47:40 | | VipDeployment | f9fa7685-da01-45e1-bf4a-40de276f0646 | OS::Heat::StructuredDeployments | CREATE_COMPLETE | 2016-10-03T12:47:40 | | allNodesConfig | 8c1b20d3-4366-463e-a54c-28788bfa9076 | OS::TripleO::AllNodes::SoftwareConfig | CREATE_COMPLETE | 2016-10-03T12:47:40 | | Networks | 32edcb3f-ed6d-4273-bc2f-179266f88716 | OS::TripleO::Network | UPDATE_COMPLETE | 2016-10-03T16:23:59 | | VipConfig | b290430b-ef87-4c9a-b7e9-3efe5bf3ef6e | OS::TripleO::VipConfig | UPDATE_COMPLETE | 2016-10-03T16:23:59 | | CephStorage | 00aed9c8-cea6-47cf-96d8-4ee94488db83 | OS::Heat::ResourceGroup | UPDATE_IN_PROGRESS | 2016-10-03T16:24:09 | | PublicVirtualIP | 72a1ee1d-b6bd-417c-a4dc-edc80b524e44 | OS::TripleO::Network::Ports::ExternalVipPort | UPDATE_COMPLETE | 2016-10-03T16:24:11 | | RedisVirtualIP | d688bc34-8369-4f80-bd90-44d5f1a660f7 | OS::TripleO::Network::Ports::RedisVipPort | UPDATE_COMPLETE | 2016-10-03T16:24:12 | | StorageVirtualIP | b5c96020-fd08-4197-835c-94cc5706be37 | OS::TripleO::Network::Ports::StorageVipPort | UPDATE_COMPLETE | 2016-10-03T16:24:12 | | InternalApiVirtualIP | 5baad0fe-d728-476e-8012-f64947d55538 | OS::TripleO::Network::Ports::InternalApiVipPort | UPDATE_COMPLETE | 2016-10-03T16:24:13 | | ObjectStorage | 34653dcc-aaae-4cce-9360-2e4b47d0e029 | OS::Heat::ResourceGroup | UPDATE_COMPLETE | 2016-10-03T16:24:14 | | StorageMgmtVirtualIP | 9fc71c64-79a3-4e3c-8033-5ea7fda0cac7 | OS::TripleO::Network::Ports::StorageMgmtVipPort | UPDATE_COMPLETE | 2016-10-03T16:24:15 | | VipMap | 341eb1fd-c3cd-41a5-b921-40ce68c782c6 | OS::TripleO::Network::Ports::NetVipMap | UPDATE_COMPLETE | 2016-10-03T16:24:19 | | EndpointMap | f502318a-e9ff-4347-9545-8acc29098ea9 | OS::TripleO::EndpointMap | UPDATE_COMPLETE | 2016-10-03T16:24:21 | | BlockStorage | 8574e5ac-6ddd-4d3d-835b-24a63a2644c6 | OS::Heat::ResourceGroup | UPDATE_COMPLETE | 2016-10-03T16:24:23 | | Compute | 8b1aa3d4-bc7c-4d0d-8a22-2d3cabf70d3c | OS::Heat::ResourceGroup | UPDATE_IN_PROGRESS | 2016-10-03T16:24:23 | | Controller | c4711e8b-13d7-4b18-9c00-c148060966dd | OS::Heat::ResourceGroup | UPDATE_IN_PROGRESS | 2016-10-03T16:24:26 | +-------------------------------------------+-----------------------------------------------+---------------------------------------------------+--------------------+---------------------+ [rlp@paisley-dir ~]$ [rlp@paisley-dir ~]$ openstack overcloud update stack overcloud -i --templates ~/pilot/templates/overcloud -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e ~/pilot/templates/overcloud/environments/network-isolation.yaml -e ~/pilot/templates/overcloud/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml -e ~/pilot/templates/dell-environment.yaml -e ~/pilot/templates/network-environment.yaml WAITING on_breakpoint: [u'overcloud-cephstorage-2', u'overcloud-compute-2', u'overcloud-controller-1', u'overcloud-cephstorage-1', u'overcloud-cephstorage-0', u'overcloud-compute-0', u'overcloud-compute-1', u'overcloud-controller-0', u'overcloud-controller-2'] removing breakpoint on overcloud-controller-2 Breakpoint reached, continue? Regexp or Enter=proceed, no=cancel update, C-c=quit interactive mode: IN_PROGRESS IN_PROGRESS --- Additional comment from Randy Perryman on 2016-10-03 14:46:55 EDT --- So running the update command again. It completed on controller2 and 1 and compute 0 and 1. completed: [u'overcloud-controller-2', u'overcloud-controller-0'] on_breakpoint: [u'overcloud-cephstorage-2', u'overcloud-compute-2', u'overcloud-controller-1', u'overcloud-cephstorage-1', u'overcloud-cephstorage-0', u'overcloud-compute-0', u'overcloud-compute-1'] removing breakpoint on overcloud-compute-1 Breakpoint reached, continue? Regexp or Enter=proceed, no=cancel update, C-c=quit interactive mode: IN_PROGRESS IN_PROGRESS IN_PROGRESS IN_PROGRESS WAITING completed: [u'overcloud-controller-2', u'overcloud-compute-1', u'overcloud-controller-0'] on_breakpoint: [u'overcloud-cephstorage-2', u'overcloud-compute-2', u'overcloud-controller-1', u'overcloud-cephstorage-1', u'overcloud-cephstorage-0', u'overcloud-compute-0'] removing breakpoint on overcloud-compute-0 Breakpoint reached, continue? Regexp or Enter=proceed, no=cancel update, C-c=quit interactive mode: IN_PROGRESS IN_PROGRESS IN_PROGRESS WAITING completed: [u'overcloud-controller-2', u'overcloud-compute-0', u'overcloud-compute-1', u'overcloud-controller-0'] on_breakpoint: [u'overcloud-cephstorage-0', u'overcloud-compute-2', u'overcloud-cephstorage-2', u'overcloud-controller-1', u'overcloud-cephstorage-1'] I logged onto the four and did a yum update, controllers nothing, computes 206 packages to update. Including all the OpenStack packages. --- Additional comment from Brad P. Crochet on 2016-10-13 15:00:08 EDT --- There are multiple issues here. The reported problem, about the update reporting COMPLETE even though it was actually in progress, was fixed here: https://review.openstack.org/#/c/328260/ However, that was done for Mitaka, not Liberty. As for the packages not updating on the non-controllers, see this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1384640 --- Additional comment from Randy Perryman on 2016-11-04 14:11:16 EDT --- Is there a backport for this? --- Additional comment from Randy Perryman on 2016-11-04 14:43:35 EDT --- Thanks for the code review link. Two Scenarios come to mind here: 1. First run you manage to have everything in COMPLETE, before the next step starts, because the previous run (Install) Was successful. or 2. Just read over this, so if you have had "Failed" install and are rerunning the install. There will be points which could be at a FAILED/COMPLETE where the next step is to TRANSITION a "FAILED" to IN-PROGRESS, thus making the Re Run of the Update "FAIL" again. while status not in ['COMPLETE', 'FAILED']: status, resources = self.get_status() print(status) if status == 'WAITING': for state in resources: if resources[state]: print("{0}: {1}".format(state, self._server_names( resources[state].keys()))) user_input = raw_input( "Breakpoint reached, continue? Regexp or " How do we avoid this? --- Additional comment from Sean Merrow on 2016-11-17 11:43:18 EST --- Brad, are you able to respond to Randy's question? --- Additional comment from Mike Burns on 2016-11-17 15:12:15 EST --- This issue needs fixing in both OSP 8 and OSP 9. Will clone to OSP 9
Issue of showing complete/in-progress was merged upstream last June, but not in OSP. Will get it into OSP 9 z-stream: https://review.openstack.org/#/c/328260/
When can we expect in zstream for OSP 9?
According to our records, this should be resolved by openstack-tripleo-common-2.0.0-8.el7ost. This build is available now.
verified with osp10 openstack-tripleo-common-0.3.1-1.el7ost.noarch, 04:27:18 "Breakpoint reached, continue? Regexp or Enter=proceed (will clear 4a647ff9-f077-4938-a3f3-48ecbc6b6c28), no=cancel update, C-c=quit interactive mode: IN_PROGRESS", 04:27:18 "IN_PROGRESS", 04:27:18 "IN_PROGRESS", 04:27:18 "IN_PROGRESS", 04:27:18 "IN_PROGRESS", 04:27:18 "IN_PROGRESS", 04:27:18 "IN_PROGRESS", 04:27:18 "IN_PROGRESS", 04:27:18 "IN_PROGRESS", 04:27:18 "IN_PROGRESS", 04:27:18 "IN_PROGRESS", 04:27:18 "IN_PROGRESS", 04:27:18 "IN_PROGRESS", 04:27:18 "IN_PROGRESS", 04:27:18 "IN_PROGRESS", 04:27:18 "IN_PROGRESS", 04:27:18 "IN_PROGRESS", 04:27:18 "IN_PROGRESS", 04:27:18 "IN_PROGRESS", 04:27:18 "IN_PROGRESS", 04:27:18 "IN_PROGRESS", 04:27:18 "IN_PROGRESS", 04:27:18 "IN_PROGRESS", 04:27:18 "IN_PROGRESS", 04:27:18 "IN_PROGRESS",