1231825 – Stack creation failed with: AttributeError: 'module' object has no attribute 'MessagingTimeout'

Bug 1231825 - Stack creation failed with: AttributeError: 'module' object has no attribute 'MessagingTimeout'

Summary: Stack creation failed with: AttributeError: 'module' object has no attribute ...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-heat
Sub Component:
Version:	7.0 (Kilo)
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	unspecified
Target Milestone:	async
Target Release:	7.0 (Kilo)
Assignee:	Ryan Brown
QA Contact:	Amit Ugol
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2015-06-15 12:55 UTC by Udi Kalifon
Modified:	2016-04-26 20:37 UTC (History)
CC List:	13 users (show)
Fixed In Version:	openstack-heat-2015.1.0-5.el7ost
Doc Type:	Bug Fix
Doc Text:	Previously, remote procedure call messages would take longer than the configured default, triggering a mishandled exception case. This would obscure the correct traceback for the timeout due to a traceback in the exception handling code, making the error more difficult to diagnose. With this update, the exception handling code has now been revised so that a remote procedure call response timeout is raised, and timeouts are less frequent due to a higher threshold.
Clone Of:
Environment:
Last Closed:	2015-08-17 16:54:46 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
heat commands that show the resource fail reason (4.94 KB, text/plain) 2015-06-15 12:55 UTC, Udi Kalifon	no flags	Details
heat engine log showing traceback (4.38 MB, text/plain) 2015-06-17 11:59 UTC, James Slagle	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Launchpad	1466239	0	None	None	None	Never
Red Hat Product Errata	RHBA-2015:1637	0	normal	SHIPPED_LIVE	openstack-heat bug fix advisory	2015-08-17 20:54:38 UTC

Description Udi Kalifon 2015-06-15 12:55:36 UTC

Created attachment 1039026 [details]
heat commands that show the resource fail reason

Description of problem:
Using the latest poodle, I tried to deploy with the following commands:

export COMPUTESCALE=3
export CEPHSTORAGESCALE=3
instack-deploy-overcloud --tuskar

Creation failed in one of the compute nodes. See attached text file with the output of the heat commands that show the error message.


Version-Release number of selected component (if applicable):
openstack-tripleo-0.0.6-0.1.git812abe0.el7ost.noarch
openstack-heat-engine-2015.1.1-dev12.el7.centos.noarch


How reproducible:
randomly


Steps to Reproduce:
1. Install a virt environment with enough virtual nodes (I had 8)
2. Deploy and overcloud with 3 Ceph nodes and 3 Compute nodes


Actual results:
CREATE_FAILED

Comment 3 chris alfonso 2015-06-16 18:06:08 UTC

Udi, You have a mix of osp and centos packages installed. You might need to fix your environment to have the right set of dependencies. Please retest this once you have done so.

Comment 5 James Slagle 2015-06-17 11:53:52 UTC

I've repro'd this only osp packages.

Here's the traceback from the heat engine log:

2015-06-17 07:27:46.451 8273 DEBUG heat.engine.scheduler [-] Task stack_task from Stack "overcloud" [46a3cf85-14a4-4e11-8768-3e7f99f2be1c] running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:220
2015-06-17 07:27:46.452 8273 DEBUG heat.engine.scheduler [-] Task resource_action running step /usr/lib/python2.7/site-packages/heat/engine/scheduler.py:220
2015-06-17 07:27:46.502 8273 INFO heat.engine.resource [-] CREATE: TemplateResource "2" Stack "overcloud-Controller-k7hlqldpslh5" [e87d1a38-8c6e-43fe-8a97-86c2a563c453]
2015-06-17 07:27:46.502 8273 TRACE heat.engine.resource Traceback (most recent call last):
2015-06-17 07:27:46.502 8273 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 500, in _action_recorder
2015-06-17 07:27:46.502 8273 TRACE heat.engine.resource     yield
2015-06-17 07:27:46.502 8273 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 570, in _do_action
2015-06-17 07:27:46.502 8273 TRACE heat.engine.resource     yield self.action_handler_task(action, args=handler_args)
2015-06-17 07:27:46.502 8273 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 296, in wrapper
2015-06-17 07:27:46.502 8273 TRACE heat.engine.resource     step = next(subtask)
2015-06-17 07:27:46.502 8273 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 541, in action_handler_task
2015-06-17 07:27:46.502 8273 TRACE heat.engine.resource     handler_data = handler(*args)
2015-06-17 07:27:46.502 8273 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/template_resource.py", line 257, in handle_create
2015-06-17 07:27:46.502 8273 TRACE heat.engine.resource     self.child_params())
2015-06-17 07:27:46.502 8273 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 265, in create_with_template
2015-06-17 07:27:46.502 8273 TRACE heat.engine.resource     self.raise_local_exception(ex)
2015-06-17 07:27:46.502 8273 TRACE heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 284, in raise_local_exception
2015-06-17 07:27:46.502 8273 TRACE heat.engine.resource     local_ex = copy.copy(getattr(exception, ex_type))
2015-06-17 07:27:46.502 8273 TRACE heat.engine.resource AttributeError: 'module' object has no attribute 'MessagingTimeout'
2015-06-17 07:27:46.502 8273 TRACE heat.engine.resource

Comment 6 James Slagle 2015-06-17 11:59:53 UTC

Created attachment 1039899 [details]
heat engine log showing traceback

Comment 7 James Slagle 2015-06-17 12:00:32 UTC

at the time i was using:
openstack overcloud deploy --control-scale 3 --plan-uuid 6d0fc8d6-194f-41ca-a8df-048c398331df

my package versions:
[root@instack ~]# rpm -qa | grep openstack
openstack-tripleo-common-0.0.1.dev6-0.git49b57eb.el7ost.noarch
openstack-neutron-common-2015.1.0-8.el7ost.noarch
openstack-heat-engine-2015.1.0-3.el7ost.noarch
openstack-ceilometer-common-2015.1.0-2.el7ost.noarch
openstack-heat-api-cfn-2015.1.0-3.el7ost.noarch
openstack-swift-2.3.0-1.el7ost.noarch
openstack-nova-novncproxy-2015.1.0-11.el7ost.noarch
openstack-swift-plugin-swift3-1.7-3.el7ost.noarch
openstack-heat-templates-0-0.6.20150605git.el7ost.noarch
openstack-ceilometer-collector-2015.1.0-2.el7ost.noarch
openstack-ironic-common-2015.1.0-4.el7ost.noarch
openstack-ceilometer-alarm-2015.1.0-2.el7ost.noarch
openstack-swift-account-2.3.0-1.el7ost.noarch
redhat-access-plugin-openstack-7.0.0-0.el7ost.noarch
openstack-glance-2015.1.0-6.el7ost.noarch
openstack-heat-api-2015.1.0-3.el7ost.noarch
openstack-tripleo-image-elements-0.9.6-1.el7ost.noarch
python-openstackclient-1.0.3-2.el7ost.noarch
openstack-ironic-discoverd-1.1.0-3.el7ost.noarch
openstack-puppet-modules-2015.1.6-1.el7ost.noarch
openstack-tripleo-puppet-elements-0.0.1-2.el7ost.noarch
openstack-dashboard-2015.1.0-10.el7ost.noarch
openstack-ceilometer-central-2015.1.0-2.el7ost.noarch
openstack-nova-compute-2015.1.0-11.el7ost.noarch
openstack-nova-conductor-2015.1.0-11.el7ost.noarch
openstack-neutron-openvswitch-2015.1.0-8.el7ost.noarch
openstack-swift-container-2.3.0-1.el7ost.noarch
openstack-nova-api-2015.1.0-11.el7ost.noarch
openstack-tripleo-0.0.6-0.1.git812abe0.el7ost.noarch
openstack-nova-common-2015.1.0-11.el7ost.noarch
openstack-heat-common-2015.1.0-3.el7ost.noarch
openstack-tuskar-0.4.18-2.el7ost.noarch
openstack-dashboard-theme-2015.1.0-10.el7ost.noarch
openstack-tuskar-ui-extras-0.0.4-1.el7ost.noarch
openstack-nova-scheduler-2015.1.0-11.el7ost.noarch
openstack-swift-object-2.3.0-1.el7ost.noarch
openstack-nova-cert-2015.1.0-11.el7ost.noarch
openstack-nova-console-2015.1.0-11.el7ost.noarch
openstack-neutron-2015.1.0-8.el7ost.noarch
openstack-ironic-conductor-2015.1.0-4.el7ost.noarch
openstack-tempest-kilo-20150507.2.el7ost.noarch
openstack-neutron-ml2-2015.1.0-8.el7ost.noarch
openstack-keystone-2015.1.0-1.el7ost.noarch
python-django-openstack-auth-1.2.0-2.el7ost.noarch
openstack-tuskar-ui-0.3.0-2.el7ost.noarch
openstack-utils-2014.2-1.el7ost.noarch
openstack-heat-api-cloudwatch-2015.1.0-3.el7ost.noarch
openstack-ceilometer-notification-2015.1.0-2.el7ost.noarch
openstack-selinux-0.6.32-1.el7ost.noarch
openstack-ceilometer-api-2015.1.0-2.el7ost.noarch
openstack-ironic-api-2015.1.0-4.el7ost.noarch
openstack-swift-proxy-2.3.0-1.el7ost.noarch
openstack-tripleo-heat-templates-0.8.6-9.el7ost.noarch

Comment 8 James Slagle 2015-06-17 12:01:03 UTC

incidentally, even though Heat tracebacked, Nova kept going and I got my 4 deployed servers:

[stack@instack ~]$ nova list
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                   | Status | Task State | Power State | Networks            |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+
| 5bcafeaf-475e-495b-a3c8-28838ef41033 | overcloud-compute-0    | ACTIVE | -          | Running     | ctlplane=192.0.2.10 |
| ad1ac9c2-f727-46a6-9e99-cf6fee3bdd42 | overcloud-controller-0 | ACTIVE | -          | Running     | ctlplane=192.0.2.11 |
| 49aca586-0c5d-4a09-9ff6-2b06e04cc255 | overcloud-controller-1 | ACTIVE | -          | Running     | ctlplane=192.0.2.9  |
| 38021d1d-16a3-4c8f-b8f5-8d6df4e71166 | overcloud-controller-2 | ACTIVE | -          | Running     | ctlplane=192.0.2.12 |
+--------------------------------------+------------------------+--------+------------+-------------+---------------------+

Comment 9 James Slagle 2015-06-17 12:07:25 UTC

there's no MessagingTimeout class in heat.common.exception which explains the traceback. I guess the real question is why we got the timeout to begin with

Comment 10 Zane Bitter 2015-06-17 16:26:55 UTC

Heat makes internal RPC calls to perform operations on nested stacks... looks like one of those timed out, and the error handling is faulty.

Comment 11 Steve Baker 2015-06-17 22:08:38 UTC

Raised upstream

Comment 12 Dougal Matthews 2015-07-01 14:51:07 UTC

I ran into this today. I deployed a Tuskar plan with one compute and one control node. Then I done an update adding two ceph storage.

So, I done:

    openstack overcloud deploy --plan overcloud
    openstack management plan set -S Ceph-Storage-1=2 $PLAN_ID
    openstack overcloud deploy --plan overcloud

Error:

    [stack@instack ~]$ heat resource-show f46fdf1b-96d8-4982-a42b-df5767b2bf99 0
    +------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
    | Property               | Value                                                                                                                                                        |
    +------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
    | attributes             | {                                                                                                                                                            |
    |                        |   "hostname": "overcloud-compute-0",                                                                                                                         |
    |                        |   "config_identifier": "os-apply-config deployment 3cbbc555-071e-4ad0-aa9e-e005917db07d completed",                                                          |
    |                        |   "nova_server_resource": "a7d63517-ce89-4fb1-952a-4b5b2aee5742",                                                                                            |
    |                        |   "tenant_ip_address": "192.0.2.10",                                                                                                                         |
    |                        |   "hosts_entry": "192.0.2.10 overcloud-compute-0.localdomain overcloud-compute-0",                                                                           |
    |                        |   "storage_ip_address": "192.0.2.10",                                                                                                                        |
    |                        |   "internal_api_ip_address": "192.0.2.10",                                                                                                                   |
    |                        |   "ip_address": "192.0.2.10"                                                                                                                                 |
    |                        | }                                                                                                                                                            |
    | description            |                                                                                                                                                              |
    | links                  | http://192.0.2.1:8004/v1/f54eef71829942ecb7f14933de5e9a9f/stacks/overcloud-Compute-b5dvske3wgkf/f46fdf1b-96d8-4982-a42b-df5767b2bf99/resources/0 (self)      |
    |                        | http://192.0.2.1:8004/v1/f54eef71829942ecb7f14933de5e9a9f/stacks/overcloud-Compute-b5dvske3wgkf/f46fdf1b-96d8-4982-a42b-df5767b2bf99 (stack)                 |
    |                        | http://192.0.2.1:8004/v1/f54eef71829942ecb7f14933de5e9a9f/stacks/overcloud-Compute-b5dvske3wgkf-0-hk4ofuyi4poz/10583659-eefc-4e9f-bc8d-c6f96ad7a22d (nested) |
    | logical_resource_id    | 0                                                                                                                                                            |
    | parent_resource        | Compute                                                                                                                                                      |
    | physical_resource_id   | 10583659-eefc-4e9f-bc8d-c6f96ad7a22d                                                                                                                         |
    | required_by            |                                                                                                                                                              |
    | resource_name          | 0                                                                                                                                                            |
    | resource_status        | UPDATE_FAILED                                                                                                                                                |
    | resource_status_reason | AttributeError: 'module' object has no attribute 'MessagingTimeout'                                                                                          |
    | resource_type          | Tuskar::Compute-1                                                                                                                                            |
    | updated_time           | 2015-07-01T12:53:24Z                                                                                                                                         |
    +------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+

Packages:

[stack@instack ~]$ rpm -qa | grep openstack
openstack-neutron-common-2015.1.0-10.el7ost.noarch
openstack-heat-engine-2015.1.0-4.el7ost.noarch
openstack-ceilometer-common-2015.1.0-6.el7ost.noarch
openstack-heat-api-cfn-2015.1.0-4.el7ost.noarch
openstack-swift-2.3.0-1.el7ost.noarch
openstack-nova-novncproxy-2015.1.0-13.el7ost.noarch
openstack-swift-object-2.3.0-1.el7ost.noarch
openstack-ceilometer-collector-2015.1.0-6.el7ost.noarch
openstack-tripleo-image-elements-0.9.6-5.el7ost.noarch
python-openstackclient-1.0.3-2.el7ost.noarch
openstack-ironic-common-2015.1.0-8.el7ost.noarch
openstack-ceilometer-alarm-2015.1.0-6.el7ost.noarch
openstack-keystone-2015.1.0-1.el7ost.noarch
redhat-access-plugin-openstack-7.0.0-0.el7ost.noarch
openstack-tripleo-0.0.7-0.1.1664e566.el7ost.noarch
openstack-glance-2015.1.0-6.el7ost.noarch
openstack-heat-api-2015.1.0-4.el7ost.noarch
openstack-ironic-discoverd-1.1.0-4.el7ost.noarch
openstack-puppet-modules-2015.1.7-5.el7ost.noarch
openstack-tripleo-heat-templates-0.8.6-22.el7ost.noarch
openstack-dashboard-2015.1.0-10.el7ost.noarch
openstack-ceilometer-central-2015.1.0-6.el7ost.noarch
openstack-nova-compute-2015.1.0-13.el7ost.noarch
openstack-nova-conductor-2015.1.0-13.el7ost.noarch
openstack-swift-account-2.3.0-1.el7ost.noarch
openstack-swift-proxy-2.3.0-1.el7ost.noarch
openstack-nova-common-2015.1.0-13.el7ost.noarch
openstack-tripleo-common-0.0.1.dev6-0.git49b57eb.el7ost.noarch
openstack-heat-common-2015.1.0-4.el7ost.noarch
openstack-tuskar-0.4.18-3.el7ost.noarch
openstack-tripleo-puppet-elements-0.0.1-3.el7ost.noarch
openstack-dashboard-theme-2015.1.0-10.el7ost.noarch
openstack-tuskar-ui-extras-0.0.4-1.el7ost.noarch
openstack-heat-templates-0-0.6.20150605git.el7ost.noarch
openstack-neutron-openvswitch-2015.1.0-10.el7ost.noarch
openstack-swift-container-2.3.0-1.el7ost.noarch
openstack-nova-api-2015.1.0-13.el7ost.noarch
openstack-nova-console-2015.1.0-13.el7ost.noarch
openstack-neutron-2015.1.0-10.el7ost.noarch
openstack-ironic-conductor-2015.1.0-8.el7ost.noarch
openstack-tempest-kilo-20150507.2.el7ost.noarch
openstack-neutron-ml2-2015.1.0-10.el7ost.noarch
openstack-nova-scheduler-2015.1.0-13.el7ost.noarch
openstack-nova-cert-2015.1.0-13.el7ost.noarch
openstack-tuskar-ui-0.3.0-7.el7ost.noarch
openstack-utils-2014.2-1.el7ost.noarch
openstack-heat-api-cloudwatch-2015.1.0-4.el7ost.noarch
openstack-ceilometer-notification-2015.1.0-6.el7ost.noarch
openstack-selinux-0.6.35-1.el7ost.noarch
openstack-ceilometer-api-2015.1.0-6.el7ost.noarch
openstack-ironic-api-2015.1.0-8.el7ost.noarch
openstack-swift-plugin-swift3-1.7-3.el7ost.noarch
python-django-openstack-auth-1.2.0-3.el7ost.noarch

Comment 13 Dougal Matthews 2015-07-01 15:23:20 UTC

I just got this for a second time in a row with the above steps. Mentioning this as I heard it isn't always reproducible.

Comment 20 Amit Ugol 2015-08-11 05:31:48 UTC

I haven't managed to reproduce it, and failed deployments elsewhere have not failed due to this timeout as far as I have checked.

Comment 22 errata-xmlrpc 2015-08-17 16:54:46 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:1637

Note You need to log in before you can comment on or make changes to this bug.