Bug 1382174

Summary: rhel-osp-director: Overcloud update fails after a long time, yum update didn't run on any node.
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: openstack-tripleo-commonAssignee: Brad P. Crochet <brad>
Status: CLOSED ERRATA QA Contact: Alexander Chuzhoy <sasha>
Severity: unspecified Docs Contact:
Priority: high    
Version: 10.0 (Newton)CC: brad, dbecker, dnavale, jcoufal, jschluet, jslagle, lbezdick, mburns, morazi, rhallise, rhel-osp-director-maint, slinaber, smerrow
Target Milestone: rcKeywords: Triaged
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-common-5.3.0-3.el7ost Doc Type: Bug Fix
Doc Text:
Previously, the 'DeployIdentifier' was not being updated for package update, resulting in Puppet not being run on the non-controller nodes. With this update, the 'DeployIdentifier' value is incremented. As a result, Puppet runs and updates packages on the non-controller nodes.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2016-12-14 16:10:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1337797    

Description Alexander Chuzhoy 2016-10-05 23:48:21 UTC 
rhel-osp-director:   Overcloud pdate fails after a long time, yum update didn't run on any node.

Environment:
instack-undercloud-5.0.0-0.20160929132429.56e65af.el7ost.noarch
openstack-puppet-modules-9.0.0-0.20160915155755.8c758d6.el7ost.noarch
openstack-tripleo-heat-templates-5.0.0-0.20160929150845.4cdc4fc.el7ost.noarch


Steps to reproduce:
1. Deploy 10 with:
openstack overcloud deploy --templates --control-scale 3 --compute-scale 1 --neutron-network-type vxlan --neutron-tunnel-types vxlan --ntp-server clock.redhat.com --timeout 90 -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e network-environment.yaml --ceph-storage-scale 1

2. update the undercloud.
3. Apply patch https://bugzilla.redhat.com/show_bug.cgi?id=1381738#c3
4. Attempt to  update the overcloud with:
openstack overcloud update stack overcloud -i --templates -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e network-environment.yaml


Result:
...
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
FAILED
update finished with status FAILED
Stack update failed.

[stack@instack ~]$ heat resource-list -n5 overcloud|grep -v COMPLE
WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
+-------------------------------------------+----------------------------------------------+-------------------------------------------------------------------------------------------------------+-----------------+----------------------+------------------------------------------------------------------------------------------------------------------------+
| resource_name                             | physical_resource_id                         | resource_type                                                                                         | resource_status | updated_time         | stack_name                                                                                                             |
+-------------------------------------------+----------------------------------------------+-------------------------------------------------------------------------------------------------------+-----------------+----------------------+------------------------------------------------------------------------------------------------------------------------+
| Compute                                   | d4c46f3d-9fbc-46ff-aa84-077acc84f289         | OS::Heat::ResourceGroup                                                                               | UPDATE_FAILED   | 2016-10-05T19:28:43Z | overcloud                                                                                                              |
| Controller                                | 47fc209c-de76-4717-bcd5-47c00ce15055         | OS::Heat::ResourceGroup                                                                               | UPDATE_FAILED   | 2016-10-05T19:28:46Z | overcloud                                                                                                              |
| 0                                         | 32666b1e-ea29-451f-b1e8-61a5db000bf9         | OS::TripleO::Compute                                                                                  | UPDATE_FAILED   | 2016-10-05T19:28:47Z | overcloud-Compute-u4ksjjq2t7en                                                                                         |
| 2                                         | 2e852706-84b9-4bcf-8e10-dc14f66073a3         | OS::TripleO::Controller                                                                               | UPDATE_FAILED   | 2016-10-05T19:28:54Z | overcloud-Controller-vwrccmz54vcz                                                                                      |
| 0                                         | ed57df7c-4afe-47d3-8b55-e2c2538bbe57         | OS::TripleO::Controller                                                                               | UPDATE_FAILED   | 2016-10-05T19:29:46Z | overcloud-Controller-vwrccmz54vcz                                                                                      |
| 1                                         | a657c768-66ad-4ccd-97d0-00a563a49ee4         | OS::TripleO::Controller                                                                               | UPDATE_FAILED   | 2016-10-05T19:29:49Z | overcloud-Controller-vwrccmz54vcz                                                                                      |
+-------------------------------------------+----------------------------------------------+-------------------------------------------------------------------------------------------------------+-----------------+----------------------+------------------------------------------------------------------------------------------------------------------------+
Comment 3 Alexander Chuzhoy 2016-10-06 23:33:41 UTC 
Succeeded after including a yaml with:
parameter_defaults:
    UpdateIdentifier: "yes"
EOF
Comment 4 James Slagle 2016-10-20 12:24:01 UTC 
Brad, is the upstream patch I linked in here the fix for this issue?

https://review.openstack.org/#/c/379516/

That patch is in WIP, is it still the fix or is there another fix in progress?
Comment 5 Brad P. Crochet 2016-10-20 12:25:48 UTC 
No. The current fix is here: https://review.openstack.org/#/c/387531/

The patch you are referring to is slated for Ocata.
Comment 6 Brad P. Crochet 2016-10-20 12:53:34 UTC 
*** Bug 1384640 has been marked as a duplicate of this bug. ***
Comment 8 Alexander Chuzhoy 2016-11-08 23:15:42 UTC 
Verified:

Environment:
openstack-tripleo-common-5.3.0-3.el7ost.noarch

The reported issue doesn't reproduce.
Comment 9 Sean Merrow 2016-11-17 16:41:08 UTC 
Mike, are we planning to backport this to OSP 9?
Comment 10 Mike Burns 2016-11-17 17:44:13 UTC 
(In reply to Sean Merrow from comment #9)
> Mike, are we planning to backport this to OSP 9?

Is it identified as an issue in OSP 9?  Everything mentioned in this bug is OSP 10 oriented.
Comment 12 errata-xmlrpc 2016-12-14 16:10:17 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html