Bug 1364240

Summary:	RHOSP Deployment dies at 30%
Product:	Red Hat Quickstart Cloud Installer	Reporter:	Thom Carlin <tcarlin>
Component:	Installation - RHELOSP	Assignee:	John Matthews <jmatthew>
Status:	CLOSED CANTFIX	QA Contact:	Dave Johnson <dajohnso>
Severity:	medium	Docs Contact:	Julie <juwu>
Priority:	unspecified
Version:	1.0	CC:	bthurber, dcadzow, jmontleo
Target Milestone:	ga	Keywords:	Triaged
Target Release:	1.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-08-12 13:41:25 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Thom Carlin 2016-08-04 19:04:33 UTC

Description of problem:

RHOSP deployment errors at 30%

Version-Release number of selected component (if applicable):

QCI-1.0-RHEL-7-20160801.t.2
QCIOOO-8.0-RHEL-7-20160801.t.0

How reproducible:

Uncertain

Steps to Reproduce:
1. Install/configure QCI/TripleO
2. Deploy RHOSP in error
3. Remove errant deployment
4. Deploy RHOSP again

Actual results:

Error at 30%
Failed deployment

Expected results:

Successful deployment

Additional info:

Actions::Fusor::Deployment::OpenStack::Deploy
Deploy Red Hat RHEL OpenStack Platform Platform overcloud

ERROR: deployment failed with status: CREATE_FAILED and reason: Resource CREATE failed: resources.Controller: Property error: resources[0].properties: Property MysqlClustercheckPassword not assigned

Comment 1 Thom Carlin 2016-08-04 19:17:37 UTC

heat-api[10150]: 2016-08-05 02:56:13.969 10150 DEBUG heat.common.serializers [req-cc93ba28-960f-4d93-a857-b142cf1b7427 58ebb51dfef449fd9806661fb09eb503 53a34f2f0360432b9b6c58e1c0a15c33] JSON response : {"stacks": [{"description": "Deploy an OpenStack environment, consisting of several node types (roles), Controller, Compute, BlockStorage, SwiftStorage and CephStorage.  The Storage roles enable independent scaling of the storage components, but the minimal deployment is one Controller and one Compute node.\n", "parent": null, "stack_status_reason": "Resource CREATE failed: resources.Controller: Property error: resources[0].properties: Property MysqlClustercheckPassword not assigned", "stack_name": "overcloud", "stack_user_project_id": "960fbf8186e840adaf70d545094024cc", "tags": null, "creation_time": "2016-08-05T06:53:08", "links": [{"href": "http://<<undercloud_ip_address>>:8004/v1/53a34f2f0360432b9b6c58e1c0a15c33/stacks/overcloud/bc9082b7-454e-4711-b9a0-1a5e5621845d", "rel": "self"}], "updated_time": null, "stack_owner": "admin", "stack_status": "CREATE_FAILED", "id": "bc9082b7-454e-4711-b9a0-1a5e5621845d"}]} to_json /usr/lib/python2.7/site-packages/heat/common/serializers.py:42

heat-engine: 2016-08-05 02:53:34.204 10258 DEBUG dogpile.core.dogpile [req-a4a24a7c-2cbe-47c8-ab83-3475ba332b31 58ebb51dfef449fd9806661fb09eb503 53a34f2f0360432b9b6c58e1c0a15c33] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154
heat-engine: 2016-08-05 02:53:34.205 10258 DEBUG heat.engine.stack [req-a4a24a7c-2cbe-47c8-ab83-3475ba332b31 58ebb51dfef449fd9806661fb09eb503 53a34f2f0360432b9b6c58e1c0a15c33] Property error: resources[0].properties: Property MysqlClustercheckPassword not assigned validate /usr/lib/python2.7/site-packages/heat/engine/stack.py:678
heat-engine: 2016-08-05 02:53:34.206 10258 DEBUG oslo_messaging.rpc.dispatcher [req-a4a24a7c-2cbe-47c8-ab83-3475ba332b31 58ebb51dfef449fd9806661fb09eb503 53a34f2f0360432b9b6c58e1c0a15c33] Expected exception during message handling (Property error: resources[0].properties: Property MysqlClustercheckPassword not assigned) _dispatch_and_reply /usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py:145
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource [-] update_stack
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource Traceback (most recent call last):
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 434, in update_with_template
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource     args)
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource   File "/usr/lib/python2.7/site-packages/heat/rpc/client.py", line 269, in update_stack
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource     args=args))
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource   File "/usr/lib/python2.7/site-packages/heat/rpc/client.py", line 61, in call
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource     return client.call(ctxt, method, **kwargs)
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 403, in call
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource     return self.prepare().call(ctxt, method, **kwargs)
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource     retry=self.retry)
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource     timeout=timeout, retry=retry)
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource     retry=retry)
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 422, in _send
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource     raise result
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource StackValidationFailed_Remote: Property error: resources[0].properties: Property MysqlClustercheckPassword not assigned
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource
heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource
heat-engine: 2016-08-05 02:53:34.287 10248 INFO heat.engine.resource [-] CREATE: ResourceGroup "Controller" [79e92a81-4740-4a4b-9398-fa6541cb3f3e] Stack "overcloud" [bc9082b7-454e-4711-b9a0-1a5e5621845d]
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource Traceback (most recent call last):
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 638, in _action_recorder
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource     yield
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 708, in _do_action
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource     yield self.action_handler_task(action, args=handler_args)
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 292, in wrapper
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource     step = next(subtask)
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 679, in action_handler_task
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource     handler_data = handler(*args)
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/resource_group.py", line 370, in handle_create
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource     checkers[0].start()
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 194, in start
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource     self.step()
:
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 217, in step
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource     next(self._runner)
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/resource_group.py", line 385, in _run_to_completion
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource     timeout)
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 437, in update_with_template
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource     self.raise_local_exception(ex)
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource   File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 328, in raise_local_exception
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource     raise exception.ResourceFailure(message, self, action=self.action)
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource ResourceFailure: resources.Controller: Property error: resources[0].properties: Property MysqlClustercheckPassword not assigned
heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource

Comment 2 Thom Carlin 2016-08-04 19:21:15 UTC

2016-08-05 02:56:13.923 10150 DEBUG oslo_messaging._drivers.amqpdriver [req-cc93ba28-960f-4d93-a857-b142cf1b7427 58ebb51dfef449fd9806661fb09eb503 53a34f2f0360432b9b6c58e1c0a15c33] MSG_ID is bfa8225b5b
9046ccac20d94b1f04d580 _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:392
2016-08-05 02:56:13.969 10150 DEBUG heat.common.serializers [req-cc93ba28-960f-4d93-a857-b142cf1b7427 58ebb51dfef449fd9806661fb09eb503 53a34f2f0360432b9b6c58e1c0a15c33] JSON response : {"stacks": [{"d
escription": "Deploy an OpenStack environment, consisting of several node types (roles), Controller, Compute, BlockStorage, SwiftStorage and CephStorage.  The Storage roles enable independent scaling 
of the storage components, but the minimal deployment is one Controller and one Compute node.\n", "parent": null, "stack_status_reason": "Resource CREATE failed: resources.Controller: Property error: 
resources[0].properties: Property MysqlClustercheckPassword not assigned", "stack_name": "overcloud", "stack_user_project_id": "960fbf8186e840adaf70d545094024cc", "tags": null, "creation_time": "2016-
08-05T06:53:08", "links": [{"href": "http://<<undercloud_ip_address>>:8004/v1/53a34f2f0360432b9b6c58e1c0a15c33/stacks/overcloud/bc9082b7-454e-4711-b9a0-1a5e5621845d", "rel": "self"}], "updated_time": null, "stack
_owner": "admin", "stack_status": "CREATE_FAILED", "id": "bc9082b7-454e-4711-b9a0-1a5e5621845d"}]} to_json /usr/lib/python2.7/site-packages/heat/common/serializers.py:42
2016-08-05 02:56:13.970 10150 DEBUG eventlet.wsgi.server [req-cc93ba28-960f-4d93-a857-b142cf1b7427 58ebb51dfef449fd9806661fb09eb503 53a34f2f0360432b9b6c58e1c0a15c33] <<gateway>> - - [05/Aug/2016 0
2:56:13] "GET /v1/53a34f2f0360432b9b6c58e1c0a15c33/stacks HTTP/1.1" 200 1103 0.125543 write /usr/lib/python2.7/site-packages/heat/common/wsgi.py:267

Comment 3 Thom Carlin 2016-08-04 19:29:41 UTC

Resume results in "ERROR: deployment failed with status: UPDATE_IN_PROGRESS and reason: Stack UPDATE started"

Comment 4 Thom Carlin 2016-08-04 20:44:28 UTC

CREATED_FAILED, UPDATE_FAILED on compute node:
resources.Compute: resources[0]: ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"

Comment 5 Thom Carlin 2016-08-05 19:08:37 UTC

Possibly bz 1358779 (Testing https://review.openstack.org/#/c/337304/ against /usr/share/openstack-tripleo-heat-templates/puppet/controller.yaml /home/stack/openstack-tripleo-heat-templates/puppet/controller.yaml

Comment 7 Jason Montleon 2016-08-12 13:34:00 UTC

Known issue with Openstack it appears:
https://access.redhat.com/solutions/2472361

Comment 8 Jason Montleon 2016-08-12 13:41:25 UTC

There's nothing much we can do for this. Either don't do a yum update or keep these packages at the versions listed until a newer version resolves the issue.

Comment 9 Thom Carlin 2016-08-12 13:46:02 UTC

https://access.redhat.com/solutions/2446961 and https://bugzilla.redhat.com/show_bug.cgi?id=1347063