Bug 1302880 - rhel-osp-director: Update overcloud 7.2->7.3 Timed out waiting for a reply to message ID 5c0c681a9efc40b0bde965ae625bf2aa
Summary: rhel-osp-director: Update overcloud 7.2->7.3 Timed out waiting for a reply to...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 7.0 (Kilo)
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: y3
: 7.0 (Kilo)
Assignee: Zane Bitter
QA Contact: Amit Ugol
URL:
Whiteboard:
Depends On:
Blocks: 1299613 1309816 1309823
TreeView+ depends on / blocked
 
Reported: 2016-01-28 21:26 UTC by Alexander Chuzhoy
Modified: 2016-02-18 18:51 UTC (History)
9 users (show)

Fixed In Version: openstack-tripleo-heat-templates-0.8.6-114.el7ost
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1309823 (view as bug list)
Environment:
Last Closed: 2016-02-18 16:52:01 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
OpenStack gerrit 275437 None None None 2016-02-02 23:56:37 UTC
Red Hat Product Errata RHBA-2016:0264 normal SHIPPED_LIVE Red Hat Enterprise Linux OSP 7 director Bug Fix Advisory 2016-02-18 21:41:29 UTC

Description Alexander Chuzhoy 2016-01-28 21:26:01 UTC
rhel-osp-director: Update overcloud 7.2->7.3 Timed out waiting for a reply to message ID 5c0c681a9efc40b0bde965ae625bf2aa


Environment:
openstack-tripleo-heat-templates-0.8.6-112.el7ost.noarch
instack-undercloud-2.1.2-37.el7ost.noarch


Steps to reproduce:
1. Deploy overcloud 7.2 HA+1 compute + 1 swift + 1 cinder node.
2. Attempt to update the overcloud to 7.3

Result:

yes "" |openstack overcloud update stack overcloud -i --templates -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/updates/update-from-vip.yaml  -e network-environment.yaml
starting package update on stack overcloud
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
WAITING
not_started: [u'overcloud-compute-0', u'overcloud-blockstorage-0', u'overcloud-controller-0', u'overcloud-controller-1', u'overcloud-controller-2']
on_breakpoint: [u'overcloud-objectstorage-0']
WARNING: tripleo_common.stack_update removing breakpoint on overcloud-objectstorage-0
Breakpoint reached, continue? Regexp or Enter=proceed, no=cancel update, C-c=quit interactive mode: IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
ERROR: openstack ERROR: Timed out waiting for a reply to message ID 5c0c681a9efc40b0bde965ae625bf2aa


I edited /etc/heat/heat.conf on the undercloud:
rpc_response_timeout = 300
num_engine_workers = 4

and restarted openstack-heat-engine.service


The yum update didn't complete on nodes.

Comment 2 Steve Baker 2016-01-28 21:42:52 UTC
Could you please look into which resource failed, and if still available attach the heat-engine logs?

Comment 5 Zane Bitter 2016-02-02 23:56:37 UTC
I'm pretty sure this is just due to the fact that the EndpointMap stack creates 30(!) nested stacks, all at exactly the same time, just as an extremely heavyweight way of defining a custom function. I submitted a change upstream to generate it statically instead, which should speed it up by a factor of 31.

Comment 9 errata-xmlrpc 2016-02-18 16:52:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0264.html


Note You need to log in before you can comment on or make changes to this bug.