Bug 1396271

Summary: Doing an update returns Complete, but heat resource-list shows IN-PROGRESS
Product: Red Hat OpenStack Reporter: Mike Burns <mburns>
Component: rhosp-directorAssignee: Angus Thomas <athomas>
Status: CLOSED CURRENTRELEASE QA Contact: Omri Hochman <ohochman>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 9.0 (Mitaka)CC: arkady_kanevsky, brad, david_paterson, dbecker, John_walsh, kasmith, kurt_hey, mbultel, mburns, morazi, ohochman, randy_perryman, rhel-osp-director-maint, sasha, smerrow, wayne_allen
Target Milestone: ---Keywords: TestOnly, ZStream
Target Release: 9.0 (Mitaka)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-common-2.0.0-8.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1381317 Environment:
Last Closed: 2017-01-03 13:15:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On: 1381317    
Bug Blocks: 1305654    

Description Mike Burns 2016-11-17 20:13:04 UTC 
+++ This bug was initially created as a clone of Bug #1381317 +++

Description of problem: Trying to update the Openstack 8 to latest.  Executed the openstack overcloud update and it returned "COMPLETE" in seconds, but a heat resource-list overcloud show items in progress.


[rlp@paisley-dir ~]$ openstack overcloud update stack overcloud -i --templates ~/pilot/templates/overcloud -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e ~/pilot/templates/overcloud/environments/network-isolation.yaml -e ~/pilot/templates/overcloud/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml -e ~/pilot/templates/dell-environment.yaml -e ~/pilot/templates/network-environment.yaml
starting package update on stack overcloud
COMPLETE
update finished with status COMPLETE'
[rlp@paisley-dir ~]$ heat resource-list overcloud                                                                                      +-------------------------------------------+-----------------------------------------------+---------------------------------------------------+--------------------+---------------------+                                                                                  | resource_name                             | physical_resource_id                          | resource_type                                     | resource_status    | updated_time        |
+-------------------------------------------+-----------------------------------------------+---------------------------------------------------+--------------------+---------------------+
| AllNodesExtraConfig                       | 7a75e1a3-5035-4158-b838-5e7899d5546d          | OS::TripleO::AllNodesExtraConfig                  | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| AllNodesValidationConfig                  | 938b4fb4-d0ce-448c-815d-00d1bff2b5dc          | OS::TripleO::AllNodes::Validation                 | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| BlockStorageAllNodesDeployment            | 1fc5450d-5a23-4b86-85fc-fad5d1cecb4a          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| BlockStorageAllNodesValidationDeployment  | a4c6ce81-611e-458f-913e-0239a9b1ed79          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| BlockStorageNodesPostDeployment           | 9f12010b-70a1-4661-8708-156eae0894b2          | OS::TripleO::BlockStoragePostDeployment           | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| CephClusterConfig                         | 651858f8-ae97-4472-bcc4-4fc6e39c8ae3          | OS::TripleO::CephClusterConfig::SoftwareConfig    | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| CephStorageAllNodesDeployment             | bdd57215-6db1-4f66-b973-d6d1d6f83015          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| CephStorageAllNodesValidationDeployment   | a1c249b2-b597-4276-a8ba-bdaeb5814bf7          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| CephStorageCephDeployment                 | 3b5abf14-a25d-40c4-a870-d1305b363d17          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| CephStorageNodesPostDeployment            | b9f0e4fb-e490-4e5d-9df1-47fd29131e69          | OS::TripleO::CephStoragePostDeployment            | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| ComputeAllNodesDeployment                 | 6a88157c-6957-44d9-9ca1-185333f3779f          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| ComputeAllNodesValidationDeployment       | 5660e3d7-d1c5-44f6-bce4-2c9ecb926e6b          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| ComputeCephDeployment                     | 2b9d3b1d-70ea-4f8f-aa99-e1ca6cd0d337          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| ComputeNodesPostDeployment                | d57db68c-448c-4093-9187-9775b203e53e          | OS::TripleO::ComputePostDeployment                | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| ControlVirtualIP                          | 9ad92792-f1cb-4b6a-93fa-f1039e259d5b          | OS::Neutron::Port                                 | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| ControllerAllNodesDeployment              | d9c0fb92-1158-4c41-a0e2-aa77cf2af0b8          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| ControllerAllNodesValidationDeployment    | 191d389a-9379-4c4d-bf44-b059fc58abfb          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| ControllerBootstrapNodeConfig             | 390fc2f4-7863-4fd4-b82b-508c72f73a16          | OS::TripleO::BootstrapNode::SoftwareConfig        | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| ControllerBootstrapNodeDeployment         | aea62014-3504-4d54-a24e-81b19f2b4066          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| ControllerCephDeployment                  | 43984b9f-3937-47cd-b573-1b49369cfc6c          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| ControllerClusterConfig                   | 122585b3-0766-449c-a6df-6bf70a9f5eb5          | OS::Heat::StructuredConfig                        | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| ControllerClusterDeployment               | e1402010-e7ed-40dd-9550-d4fee3730674          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| ControllerIpListMap                       | 48bf22cc-fc67-4d66-87ac-6730fed0f113          | OS::TripleO::Network::Ports::NetIpListMap         | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| ControllerNodesPostDeployment             | 15614bab-8286-476e-8149-6ee53daf1b9f          | OS::TripleO::ControllerPostDeployment             | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| ControllerSwiftDeployment                 | e2a87b75-8c0e-406c-8fd3-df9a8f1c6710          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| HeatAuthEncryptionKey                     | overcloud-HeatAuthEncryptionKey-uprxipn5izjo  | OS::Heat::RandomString                            | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| HorizonSecret                             | overcloud-HorizonSecret-r5qqdgdoi3ay          | OS::Heat::RandomString                            | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| MysqlClusterUniquePart                    | overcloud-MysqlClusterUniquePart-cw5l3dijhrpg | OS::Heat::RandomString                            | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| MysqlRootPassword                         | overcloud-MysqlRootPassword-bp7yltgtop47      | OS::Heat::RandomString                            | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| ObjectStorageAllNodesDeployment           | 1f2661e2-244f-4e3a-95ed-724c63a43d60          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| ObjectStorageAllNodesValidationDeployment | 4e950ffa-76db-497f-81df-94c455569500          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| ObjectStorageNodesPostDeployment          | 87242aff-3496-4f4d-b7b1-6d9a8ca353df          | OS::TripleO::ObjectStoragePostDeployment          | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| ObjectStorageSwiftDeployment              | 4a5178ad-7797-4e6e-93a0-3baf74909811          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| PcsdPassword                              | overcloud-PcsdPassword-4ck57yz2plz7           | OS::Heat::RandomString                            | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| RabbitCookie                              | overcloud-RabbitCookie-7uemvh2hz3mx           | OS::Heat::RandomString                            | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| SwiftDevicesAndProxyConfig                | baea166f-ebc9-42d5-8691-69764ba575ad          | OS::TripleO::SwiftDevicesAndProxy::SoftwareConfig | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| UpdateWorkflow                            | 62a09843-f1a4-4420-9686-9fcb1e03356f          | OS::TripleO::Tasks::UpdateWorkflow                | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| VipDeployment                             | f9fa7685-da01-45e1-bf4a-40de276f0646          | OS::Heat::StructuredDeployments                   | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| allNodesConfig                            | 8c1b20d3-4366-463e-a54c-28788bfa9076          | OS::TripleO::AllNodes::SoftwareConfig             | CREATE_COMPLETE    | 2016-10-03T12:47:40 |
| Networks                                  | 32edcb3f-ed6d-4273-bc2f-179266f88716          | OS::TripleO::Network                              | UPDATE_COMPLETE    | 2016-10-03T16:23:59 |
| VipConfig                                 | b290430b-ef87-4c9a-b7e9-3efe5bf3ef6e          | OS::TripleO::VipConfig                            | UPDATE_COMPLETE    | 2016-10-03T16:23:59 |
| CephStorage                               | 00aed9c8-cea6-47cf-96d8-4ee94488db83          | OS::Heat::ResourceGroup                           | UPDATE_IN_PROGRESS | 2016-10-03T16:24:09 |
| PublicVirtualIP                           | 72a1ee1d-b6bd-417c-a4dc-edc80b524e44          | OS::TripleO::Network::Ports::ExternalVipPort      | UPDATE_COMPLETE    | 2016-10-03T16:24:11 |
| RedisVirtualIP                            | d688bc34-8369-4f80-bd90-44d5f1a660f7          | OS::TripleO::Network::Ports::RedisVipPort         | UPDATE_COMPLETE    | 2016-10-03T16:24:12 |
| StorageVirtualIP                          | b5c96020-fd08-4197-835c-94cc5706be37          | OS::TripleO::Network::Ports::StorageVipPort       | UPDATE_COMPLETE    | 2016-10-03T16:24:12 |
| InternalApiVirtualIP                      | 5baad0fe-d728-476e-8012-f64947d55538          | OS::TripleO::Network::Ports::InternalApiVipPort   | UPDATE_COMPLETE    | 2016-10-03T16:24:13 |
| ObjectStorage                             | 34653dcc-aaae-4cce-9360-2e4b47d0e029          | OS::Heat::ResourceGroup                           | UPDATE_COMPLETE    | 2016-10-03T16:24:14 |
| StorageMgmtVirtualIP                      | 9fc71c64-79a3-4e3c-8033-5ea7fda0cac7          | OS::TripleO::Network::Ports::StorageMgmtVipPort   | UPDATE_COMPLETE    | 2016-10-03T16:24:15 |
| VipMap                                    | 341eb1fd-c3cd-41a5-b921-40ce68c782c6          | OS::TripleO::Network::Ports::NetVipMap            | UPDATE_COMPLETE    | 2016-10-03T16:24:19 |
| EndpointMap                               | f502318a-e9ff-4347-9545-8acc29098ea9          | OS::TripleO::EndpointMap                          | UPDATE_COMPLETE    | 2016-10-03T16:24:21 |
| BlockStorage                              | 8574e5ac-6ddd-4d3d-835b-24a63a2644c6          | OS::Heat::ResourceGroup                           | UPDATE_COMPLETE    | 2016-10-03T16:24:23 |
| Compute                                   | 8b1aa3d4-bc7c-4d0d-8a22-2d3cabf70d3c          | OS::Heat::ResourceGroup                           | UPDATE_IN_PROGRESS | 2016-10-03T16:24:23 |
| Controller                                | c4711e8b-13d7-4b18-9c00-c148060966dd          | OS::Heat::ResourceGroup                           | UPDATE_IN_PROGRESS | 2016-10-03T16:24:26 |
+-------------------------------------------+-----------------------------------------------+---------------------------------------------------+--------------------+---------------------+
[rlp@paisley-dir ~]$
[rlp@paisley-dir ~]$ openstack overcloud update stack overcloud -i --templates ~/pilot/templates/overcloud -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e ~/pilot/templates/overcloud/environments/network-isolation.yaml -e ~/pilot/templates/overcloud/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml -e ~/pilot/templates/dell-environment.yaml -e ~/pilot/templates/network-environment.yaml
WAITING
on_breakpoint: [u'overcloud-cephstorage-2', u'overcloud-compute-2', u'overcloud-controller-1', u'overcloud-cephstorage-1', u'overcloud-cephstorage-0', u'overcloud-compute-0', u'overcloud-compute-1', u'overcloud-controller-0', u'overcloud-controller-2']
removing breakpoint on overcloud-controller-2
Breakpoint reached, continue? Regexp or Enter=proceed, no=cancel update, C-c=quit interactive mode: IN_PROGRESS
IN_PROGRESS

--- Additional comment from Randy Perryman on 2016-10-03 14:46:55 EDT ---

So running the update command again.  It completed on controller2 and 1 and compute 0 and 1. 
completed: [u'overcloud-controller-2', u'overcloud-controller-0']
on_breakpoint: [u'overcloud-cephstorage-2', u'overcloud-compute-2', u'overcloud-controller-1', u'overcloud-cephstorage-1', u'overcloud-cephstorage-0', u'overcloud-compute-0', u'overcloud-compute-1']
removing breakpoint on overcloud-compute-1
Breakpoint reached, continue? Regexp or Enter=proceed, no=cancel update, C-c=quit interactive mode: IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
WAITING
completed: [u'overcloud-controller-2', u'overcloud-compute-1', u'overcloud-controller-0']
on_breakpoint: [u'overcloud-cephstorage-2', u'overcloud-compute-2', u'overcloud-controller-1', u'overcloud-cephstorage-1', u'overcloud-cephstorage-0', u'overcloud-compute-0']
removing breakpoint on overcloud-compute-0
Breakpoint reached, continue? Regexp or Enter=proceed, no=cancel update, C-c=quit interactive mode: IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
WAITING
completed: [u'overcloud-controller-2', u'overcloud-compute-0', u'overcloud-compute-1', u'overcloud-controller-0']
on_breakpoint: [u'overcloud-cephstorage-0', u'overcloud-compute-2', u'overcloud-cephstorage-2', u'overcloud-controller-1', u'overcloud-cephstorage-1']



I logged onto the four and did a yum update,  controllers nothing, computes 206 packages to update. Including all the OpenStack packages.

--- Additional comment from Brad P. Crochet on 2016-10-13 15:00:08 EDT ---

There are multiple issues here. The reported problem, about the update reporting COMPLETE even though it was actually in progress, was fixed here: https://review.openstack.org/#/c/328260/

However, that was done for Mitaka, not Liberty.

As for the packages not updating on the non-controllers, see this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1384640

--- Additional comment from Randy Perryman on 2016-11-04 14:11:16 EDT ---

Is there a backport for this?

--- Additional comment from Randy Perryman on 2016-11-04 14:43:35 EDT ---

Thanks for the code review link. 
Two Scenarios come to mind here:
1. First run you manage to have everything in COMPLETE, before the next step starts, because the previous run (Install) Was successful.
 or 

2. Just read over this, so if you have had "Failed" install and are rerunning the install. There will be points which could be at a FAILED/COMPLETE where the next step is to TRANSITION a "FAILED" to IN-PROGRESS, thus making the Re Run of the  Update "FAIL" again.

 while status not in ['COMPLETE', 'FAILED']:
            status, resources = self.get_status()
            print(status)
            if status == 'WAITING':
                for state in resources:
                    if resources[state]:
                        print("{0}: {1}".format(state, self._server_names(
                                                resources[state].keys())))
                user_input = raw_input(
                    "Breakpoint reached, continue? Regexp or "

How do we avoid this?

--- Additional comment from Sean Merrow on 2016-11-17 11:43:18 EST ---

Brad, are you able to respond to Randy's question?

--- Additional comment from Mike Burns on 2016-11-17 15:12:15 EST ---

This issue needs fixing in both OSP 8 and OSP 9.  Will clone to OSP 9
Comment 1 Sean Merrow 2016-11-18 14:02:23 UTC 
Issue of showing complete/in-progress was merged upstream last June, but not in OSP. Will get it into OSP 9 z-stream:

https://review.openstack.org/#/c/328260/
Comment 2 David Paterson 2016-11-22 19:41:57 UTC 
When can we expect in zstream for OSP 9?
Comment 3 Lon Hohberger 2016-12-01 19:53:55 UTC 
According to our records, this should be resolved by openstack-tripleo-common-2.0.0-8.el7ost.  This build is available now.
Comment 4 Omri Hochman 2016-12-19 16:18:22 UTC 
verified with osp10  openstack-tripleo-common-0.3.1-1.el7ost.noarch, 


04:27:18             "Breakpoint reached, continue? Regexp or Enter=proceed (will clear 4a647ff9-f077-4938-a3f3-48ecbc6b6c28), no=cancel update, C-c=quit interactive mode: IN_PROGRESS",
04:27:18             "IN_PROGRESS",
04:27:18             "IN_PROGRESS",
04:27:18             "IN_PROGRESS",
04:27:18             "IN_PROGRESS",
04:27:18             "IN_PROGRESS",
04:27:18             "IN_PROGRESS",
04:27:18             "IN_PROGRESS",
04:27:18             "IN_PROGRESS",
04:27:18             "IN_PROGRESS",
04:27:18             "IN_PROGRESS",
04:27:18             "IN_PROGRESS",
04:27:18             "IN_PROGRESS",
04:27:18             "IN_PROGRESS",
04:27:18             "IN_PROGRESS",
04:27:18             "IN_PROGRESS",
04:27:18             "IN_PROGRESS",
04:27:18             "IN_PROGRESS",
04:27:18             "IN_PROGRESS",
04:27:18             "IN_PROGRESS",
04:27:18             "IN_PROGRESS",
04:27:18             "IN_PROGRESS",
04:27:18             "IN_PROGRESS",
04:27:18             "IN_PROGRESS",
04:27:18             "IN_PROGRESS",