Description of problem: RHOSP deployment errors at 30% Version-Release number of selected component (if applicable): QCI-1.0-RHEL-7-20160801.t.2 QCIOOO-8.0-RHEL-7-20160801.t.0 How reproducible: Uncertain Steps to Reproduce: 1. Install/configure QCI/TripleO 2. Deploy RHOSP in error 3. Remove errant deployment 4. Deploy RHOSP again Actual results: Error at 30% Failed deployment Expected results: Successful deployment Additional info: Actions::Fusor::Deployment::OpenStack::Deploy Deploy Red Hat RHEL OpenStack Platform Platform overcloud ERROR: deployment failed with status: CREATE_FAILED and reason: Resource CREATE failed: resources.Controller: Property error: resources[0].properties: Property MysqlClustercheckPassword not assigned
heat-api[10150]: 2016-08-05 02:56:13.969 10150 DEBUG heat.common.serializers [req-cc93ba28-960f-4d93-a857-b142cf1b7427 58ebb51dfef449fd9806661fb09eb503 53a34f2f0360432b9b6c58e1c0a15c33] JSON response : {"stacks": [{"description": "Deploy an OpenStack environment, consisting of several node types (roles), Controller, Compute, BlockStorage, SwiftStorage and CephStorage. The Storage roles enable independent scaling of the storage components, but the minimal deployment is one Controller and one Compute node.\n", "parent": null, "stack_status_reason": "Resource CREATE failed: resources.Controller: Property error: resources[0].properties: Property MysqlClustercheckPassword not assigned", "stack_name": "overcloud", "stack_user_project_id": "960fbf8186e840adaf70d545094024cc", "tags": null, "creation_time": "2016-08-05T06:53:08", "links": [{"href": "http://<<undercloud_ip_address>>:8004/v1/53a34f2f0360432b9b6c58e1c0a15c33/stacks/overcloud/bc9082b7-454e-4711-b9a0-1a5e5621845d", "rel": "self"}], "updated_time": null, "stack_owner": "admin", "stack_status": "CREATE_FAILED", "id": "bc9082b7-454e-4711-b9a0-1a5e5621845d"}]} to_json /usr/lib/python2.7/site-packages/heat/common/serializers.py:42 heat-engine: 2016-08-05 02:53:34.204 10258 DEBUG dogpile.core.dogpile [req-a4a24a7c-2cbe-47c8-ab83-3475ba332b31 58ebb51dfef449fd9806661fb09eb503 53a34f2f0360432b9b6c58e1c0a15c33] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/core/dogpile.py:154 heat-engine: 2016-08-05 02:53:34.205 10258 DEBUG heat.engine.stack [req-a4a24a7c-2cbe-47c8-ab83-3475ba332b31 58ebb51dfef449fd9806661fb09eb503 53a34f2f0360432b9b6c58e1c0a15c33] Property error: resources[0].properties: Property MysqlClustercheckPassword not assigned validate /usr/lib/python2.7/site-packages/heat/engine/stack.py:678 heat-engine: 2016-08-05 02:53:34.206 10258 DEBUG oslo_messaging.rpc.dispatcher [req-a4a24a7c-2cbe-47c8-ab83-3475ba332b31 58ebb51dfef449fd9806661fb09eb503 53a34f2f0360432b9b6c58e1c0a15c33] Expected exception during message handling (Property error: resources[0].properties: Property MysqlClustercheckPassword not assigned) _dispatch_and_reply /usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py:145 heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource [-] update_stack heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource Traceback (most recent call last): heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 434, in update_with_template heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource args) heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/heat/rpc/client.py", line 269, in update_stack heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource args=args)) heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/heat/rpc/client.py", line 61, in call heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource return client.call(ctxt, method, **kwargs) heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 403, in call heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource return self.prepare().call(ctxt, method, **kwargs) heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource retry=self.retry) heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource timeout=timeout, retry=retry) heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource retry=retry) heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 422, in _send heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource raise result heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource StackValidationFailed_Remote: Property error: resources[0].properties: Property MysqlClustercheckPassword not assigned heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource heat-engine: 2016-08-05 02:53:34.229 10248 ERROR heat.engine.resources.stack_resource heat-engine: 2016-08-05 02:53:34.287 10248 INFO heat.engine.resource [-] CREATE: ResourceGroup "Controller" [79e92a81-4740-4a4b-9398-fa6541cb3f3e] Stack "overcloud" [bc9082b7-454e-4711-b9a0-1a5e5621845d] heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource Traceback (most recent call last): heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 638, in _action_recorder heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource yield heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 708, in _do_action heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource yield self.action_handler_task(action, args=handler_args) heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 292, in wrapper heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource step = next(subtask) heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 679, in action_handler_task heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource handler_data = handler(*args) heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/resource_group.py", line 370, in handle_create heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource checkers[0].start() heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 194, in start heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource self.step() : heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 217, in step heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource next(self._runner) heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/resource_group.py", line 385, in _run_to_completion heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource timeout) heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 437, in update_with_template heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource self.raise_local_exception(ex) heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py", line 328, in raise_local_exception heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource raise exception.ResourceFailure(message, self, action=self.action) heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource ResourceFailure: resources.Controller: Property error: resources[0].properties: Property MysqlClustercheckPassword not assigned heat-engine: 2016-08-05 02:53:34.287 10248 ERROR heat.engine.resource
2016-08-05 02:56:13.923 10150 DEBUG oslo_messaging._drivers.amqpdriver [req-cc93ba28-960f-4d93-a857-b142cf1b7427 58ebb51dfef449fd9806661fb09eb503 53a34f2f0360432b9b6c58e1c0a15c33] MSG_ID is bfa8225b5b 9046ccac20d94b1f04d580 _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:392 2016-08-05 02:56:13.969 10150 DEBUG heat.common.serializers [req-cc93ba28-960f-4d93-a857-b142cf1b7427 58ebb51dfef449fd9806661fb09eb503 53a34f2f0360432b9b6c58e1c0a15c33] JSON response : {"stacks": [{"d escription": "Deploy an OpenStack environment, consisting of several node types (roles), Controller, Compute, BlockStorage, SwiftStorage and CephStorage. The Storage roles enable independent scaling of the storage components, but the minimal deployment is one Controller and one Compute node.\n", "parent": null, "stack_status_reason": "Resource CREATE failed: resources.Controller: Property error: resources[0].properties: Property MysqlClustercheckPassword not assigned", "stack_name": "overcloud", "stack_user_project_id": "960fbf8186e840adaf70d545094024cc", "tags": null, "creation_time": "2016- 08-05T06:53:08", "links": [{"href": "http://<<undercloud_ip_address>>:8004/v1/53a34f2f0360432b9b6c58e1c0a15c33/stacks/overcloud/bc9082b7-454e-4711-b9a0-1a5e5621845d", "rel": "self"}], "updated_time": null, "stack _owner": "admin", "stack_status": "CREATE_FAILED", "id": "bc9082b7-454e-4711-b9a0-1a5e5621845d"}]} to_json /usr/lib/python2.7/site-packages/heat/common/serializers.py:42 2016-08-05 02:56:13.970 10150 DEBUG eventlet.wsgi.server [req-cc93ba28-960f-4d93-a857-b142cf1b7427 58ebb51dfef449fd9806661fb09eb503 53a34f2f0360432b9b6c58e1c0a15c33] <<gateway>> - - [05/Aug/2016 0 2:56:13] "GET /v1/53a34f2f0360432b9b6c58e1c0a15c33/stacks HTTP/1.1" 200 1103 0.125543 write /usr/lib/python2.7/site-packages/heat/common/wsgi.py:267
Resume results in "ERROR: deployment failed with status: UPDATE_IN_PROGRESS and reason: Stack UPDATE started"
CREATED_FAILED, UPDATE_FAILED on compute node: resources.Compute: resources[0]: ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
Possibly bz 1358779 (Testing https://review.openstack.org/#/c/337304/ against /usr/share/openstack-tripleo-heat-templates/puppet/controller.yaml /home/stack/openstack-tripleo-heat-templates/puppet/controller.yaml
Known issue with Openstack it appears: https://access.redhat.com/solutions/2472361
There's nothing much we can do for this. Either don't do a yum update or keep these packages at the versions listed until a newer version resolves the issue.
https://access.redhat.com/solutions/2446961 and https://bugzilla.redhat.com/show_bug.cgi?id=1347063