Bug 1308702

Summary: rpc_response_timeout not set in undercloud heat.conf when upgrading to 7.3 from 7.2
Product: Red Hat OpenStack Reporter: Dan Yasny <dyasny>
Component: rhosp-directorAssignee: Angus Thomas <athomas>
Status: CLOSED NOTABUG QA Contact: yeylon <yeylon>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 7.0 (Kilo)CC: dbecker, mburns, morazi, rhel-osp-director-maint, srevivo, zbitter
Target Milestone: y3   
Target Release: 7.0 (Kilo)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
Cause: The RPC response timeout for communication between the heat-api and heat-engine processes is supposed to be increased from the default value in a director undercloud. However, due to https://bugzilla.redhat.com/show_bug.cgi?id=1305947 this configuration change is missing on underclouds originally installed with director 7.2. Consequence: Heat stack operations on the undercloud may fail with a MessagingTimeout error. Workaround (if any): In the file /etc/heat/heat.conf on the undercloud, change the line: #rpc_response_timeout=60 to rpc_response_timeout=600 (i.e. remove the initial # to uncomment it, and change the value from 60 to 600.) Then restart the heat services: systemctl restart openstack-heat-api.service systemctl restart openstack-heat-api-cfn.service systemctl restart openstack-heat-engine.service Result: Heat will be running using the correct timeouts.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2016-02-15 21:15:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Dan Yasny 2016-02-15 19:29:18 UTC 
Description of problem:


Upgrade from 7.x to 7.3 fails because rpc_response_timeout isn't set to 600 as per BZ#1305947. While BZ#1305947 is fixed for a fresh install, it still fails for the same reason on upgrade.

[stack@instack ~]$ sudo grep rpc_response_timeout /etc/heat/heat.conf 
#rpc_response_timeout = 60

stack_status_reason:   
resources.Compute: MessagingTimeout: resources[1]: Timed out waiting for a reply to message ID db32425adde042db9f31ab0100dd4880 


Version-Release number of selected component (if applicable):

[stack@instack ~]$ exit
[root@instack heat]# rpm -qa | grep heat
openstack-tripleo-heat-templates-0.8.6-120.el7ost.noarch
python-heatclient-0.6.0-1.el7ost.noarch
openstack-heat-engine-2015.1.2-9.el7ost.noarch
heat-cfntools-1.2.8-2.el7.noarch
openstack-heat-api-2015.1.2-9.el7ost.noarch
openstack-heat-templates-0-0.8.20150605git.el7ost.noarch
openstack-heat-api-cloudwatch-2015.1.2-9.el7ost.noarch
openstack-heat-api-cfn-2015.1.2-9.el7ost.noarch
openstack-heat-common-2015.1.2-9.el7ost.noarch
[root@instack heat]# rpm -qa | grep instack
instack-0.0.7-2.el7ost.noarch
instack-undercloud-2.1.2-39.el7ost.noarch
[root@instack heat]# rpm -q openstack-puppet-modules
openstack-puppet-modules-2015.1.8-51.el7ost.noarch


How reproducible:
always

Steps to Reproduce:
1. deploy 7.2
2. update
3.

Actual results:
resources.Compute: MessagingTimeout: resources[1]: Timed out waiting for a reply to message ID db32425adde042db9f31ab0100dd4880 


Expected results:
upgrade should pass

Additional info:

 Deployment command: openstack overcloud deploy --templates --control-scale 3 --compute-scale 2 --ceph-storage-scale 1 --swift-storage-scale 1 --block-storage-scale 1 --neutron-network-type vxlan --neutron-tunnel-types vxlan  --ntp-server 10.5.26.10 --timeout 90 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo:-heat-templates/environments/network-isolation.yaml -e network-environment.yaml

Update command is: openstack overcloud update stack overcloud -i --templates -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/updates/update-from-vip.yaml  -e network-environment.yaml

[stack@instack ~]$ heat resource-list -n 5 overcloud | grep -v COMPLETE
+------------------+-----------------------------------------+-----------------------------+-----------------+----------------------+------------------+
| resource_name    | physical_resource_id                    | resource_type               | resource_status | updated_time         | parent_resource  |
+------------------+-----------------------------------------+-----------------------------+-----------------+----------------------+------------------+
| Controller       | 83e2dd2f-7e10-4d8b-9e4f-c73fa60c705d    | OS::Heat::ResourceGroup     | UPDATE_FAILED   | 2016-02-13T03:57:42Z |                  |
| Compute          | 96430ccd-aa55-47f6-8296-9b68fab76261    | OS::Heat::ResourceGroup     | UPDATE_FAILED   | 2016-02-13T03:57:51Z |                  |
| 1                | 39f20df7-ed7b-4b3d-922b-b69a86fa22b5    | OS::TripleO::Controller     | UPDATE_FAILED   | 2016-02-13T03:57:53Z | Controller       |
| BlockStorage     | 791042dd-a757-4570-bf30-551ccce66760    | OS::Heat::ResourceGroup     | UPDATE_FAILED   | 2016-02-13T03:58:02Z |                  |
| 0                | 6031a436-58b2-462c-9c80-d1a43177395d    | OS::TripleO::Compute        | UPDATE_FAILED   | 2016-02-13T03:58:03Z | Compute          |
| 0                | b1b28097-adbc-4cfa-98eb-686d18daedf2    | OS::TripleO::BlockStorage   | UPDATE_FAILED   | 2016-02-13T03:58:11Z | BlockStorage     |
| 1                | c423e398-4051-4d5f-8948-698bc231b0a3    | OS::TripleO::Compute        | UPDATE_FAILED   | 2016-02-13T03:58:30Z | Compute          |
| 2                | e919f699-4239-4e56-a9e4-b4d13d664bb3    | OS::TripleO::Controller     | UPDATE_FAILED   | 2016-02-13T03:58:43Z | Controller       |
| 0                | ca59a220-8fb9-4055-8976-a903d551977c    | OS::TripleO::Controller     | UPDATE_FAILED   | 2016-02-13T03:59:44Z | Controller       |
+------------------+-----------------------------------------+-----------------------------+-----------------+----------------------+------------------+
Comment 1 Zane Bitter 2016-02-15 19:33:36 UTC 
Our options here are basically:

1) Tell users to re-run puppet after yum-updating the undercloud; or
2) Tell users to manually set this configuration option and restart heat-api after upgrading from 7.2.
Comment 3 Angus Thomas 2016-02-15 21:02:00 UTC 
Closed with doctext for 7.3