Bug 1382174 - rhel-osp-director: Overcloud update fails after a long time, yum update didn't run on any node.
Summary: rhel-osp-director: Overcloud update fails after a long time, yum update did...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-common
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: rc
: 10.0 (Newton)
Assignee: Brad P. Crochet
QA Contact: Alexander Chuzhoy
URL:
Whiteboard:
: 1384640 (view as bug list)
Depends On:
Blocks: 1337797
TreeView+ depends on / blocked
 
Reported: 2016-10-05 23:48 UTC by Alexander Chuzhoy
Modified: 2016-12-29 16:56 UTC (History)
13 users (show)

Fixed In Version: openstack-tripleo-common-5.3.0-3.el7ost
Doc Type: Bug Fix
Doc Text:
Previously, the 'DeployIdentifier' was not being updated for package update, resulting in Puppet not being run on the non-controller nodes. With this update, the 'DeployIdentifier' value is incremented. As a result, Puppet runs and updates packages on the non-controller nodes.
Clone Of:
Environment:
Last Closed: 2016-12-14 16:10:17 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:2948 normal SHIPPED_LIVE Red Hat OpenStack Platform 10 enhancement update 2016-12-14 19:55:27 UTC
OpenStack gerrit 387531 None None None 2016-10-20 12:26:43 UTC
Launchpad 1614928 None None None 2016-10-06 19:58:44 UTC

Description Alexander Chuzhoy 2016-10-05 23:48:21 UTC
rhel-osp-director:   Overcloud pdate fails after a long time, yum update didn't run on any node.

Environment:
instack-undercloud-5.0.0-0.20160929132429.56e65af.el7ost.noarch
openstack-puppet-modules-9.0.0-0.20160915155755.8c758d6.el7ost.noarch
openstack-tripleo-heat-templates-5.0.0-0.20160929150845.4cdc4fc.el7ost.noarch


Steps to reproduce:
1. Deploy 10 with:
openstack overcloud deploy --templates --control-scale 3 --compute-scale 1 --neutron-network-type vxlan --neutron-tunnel-types vxlan --ntp-server clock.redhat.com --timeout 90 -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e network-environment.yaml --ceph-storage-scale 1

2. update the undercloud.
3. Apply patch https://bugzilla.redhat.com/show_bug.cgi?id=1381738#c3
4. Attempt to  update the overcloud with:
openstack overcloud update stack overcloud -i --templates -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e network-environment.yaml


Result:
...
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
FAILED
update finished with status FAILED
Stack update failed.

[stack@instack ~]$ heat resource-list -n5 overcloud|grep -v COMPLE
WARNING (shell) "heat resource-list" is deprecated, please use "openstack stack resource list" instead
+-------------------------------------------+----------------------------------------------+-------------------------------------------------------------------------------------------------------+-----------------+----------------------+------------------------------------------------------------------------------------------------------------------------+
| resource_name                             | physical_resource_id                         | resource_type                                                                                         | resource_status | updated_time         | stack_name                                                                                                             |
+-------------------------------------------+----------------------------------------------+-------------------------------------------------------------------------------------------------------+-----------------+----------------------+------------------------------------------------------------------------------------------------------------------------+
| Compute                                   | d4c46f3d-9fbc-46ff-aa84-077acc84f289         | OS::Heat::ResourceGroup                                                                               | UPDATE_FAILED   | 2016-10-05T19:28:43Z | overcloud                                                                                                              |
| Controller                                | 47fc209c-de76-4717-bcd5-47c00ce15055         | OS::Heat::ResourceGroup                                                                               | UPDATE_FAILED   | 2016-10-05T19:28:46Z | overcloud                                                                                                              |
| 0                                         | 32666b1e-ea29-451f-b1e8-61a5db000bf9         | OS::TripleO::Compute                                                                                  | UPDATE_FAILED   | 2016-10-05T19:28:47Z | overcloud-Compute-u4ksjjq2t7en                                                                                         |
| 2                                         | 2e852706-84b9-4bcf-8e10-dc14f66073a3         | OS::TripleO::Controller                                                                               | UPDATE_FAILED   | 2016-10-05T19:28:54Z | overcloud-Controller-vwrccmz54vcz                                                                                      |
| 0                                         | ed57df7c-4afe-47d3-8b55-e2c2538bbe57         | OS::TripleO::Controller                                                                               | UPDATE_FAILED   | 2016-10-05T19:29:46Z | overcloud-Controller-vwrccmz54vcz                                                                                      |
| 1                                         | a657c768-66ad-4ccd-97d0-00a563a49ee4         | OS::TripleO::Controller                                                                               | UPDATE_FAILED   | 2016-10-05T19:29:49Z | overcloud-Controller-vwrccmz54vcz                                                                                      |
+-------------------------------------------+----------------------------------------------+-------------------------------------------------------------------------------------------------------+-----------------+----------------------+------------------------------------------------------------------------------------------------------------------------+

Comment 3 Alexander Chuzhoy 2016-10-06 23:33:41 UTC
Succeeded after including a yaml with:
parameter_defaults:
    UpdateIdentifier: "yes"
EOF

Comment 4 James Slagle 2016-10-20 12:24:01 UTC
Brad, is the upstream patch I linked in here the fix for this issue?

https://review.openstack.org/#/c/379516/

That patch is in WIP, is it still the fix or is there another fix in progress?

Comment 5 Brad P. Crochet 2016-10-20 12:25:48 UTC
No. The current fix is here: https://review.openstack.org/#/c/387531/

The patch you are referring to is slated for Ocata.

Comment 6 Brad P. Crochet 2016-10-20 12:53:34 UTC
*** Bug 1384640 has been marked as a duplicate of this bug. ***

Comment 8 Alexander Chuzhoy 2016-11-08 23:15:42 UTC
Verified:

Environment:
openstack-tripleo-common-5.3.0-3.el7ost.noarch

The reported issue doesn't reproduce.

Comment 9 Sean Merrow 2016-11-17 16:41:08 UTC
Mike, are we planning to backport this to OSP 9?

Comment 10 Mike Burns 2016-11-17 17:44:13 UTC
(In reply to Sean Merrow from comment #9)
> Mike, are we planning to backport this to OSP 9?

Is it identified as an issue in OSP 9?  Everything mentioned in this bug is OSP 10 oriented.

Comment 12 errata-xmlrpc 2016-12-14 16:10:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html


Note You need to log in before you can comment on or make changes to this bug.