Bug 1302845

Created attachment 1119232 [details]
Undercloud description of error

OSP + RHEV + CFME deployment stops in the middle w/ ControllerNodePostDeployment failure. Current workaround is to undeploy from Undercloud UI and resume the OSP deployment task in Satellite UI. 

Logging from heat-api.log regarding the error:

2016-01-27 19:31:59.174 18689 INFO eventlet.wsgi.server [req-40210273-fab2-4e9a-aa7e-29b601069be8 admin admin] 192.0.2.1 - - [27/Jan/2016 19:31:59] "GET /v1/369b087f32454cfc890a36a604b6bdd2/stacks HTTP/1.1" 200 916 0.143380
2016-01-28 08:34:48.045 18689 INFO eventlet.wsgi.server [-] (18689) accepted ('192.0.2.254', 60252)
2016-01-28 08:34:48.047 18689 DEBUG heat.api.middleware.version_negotiation [-] Processing request: GET /v1/369b087f32454cfc890a36a604b6bdd2/stacks Accept: application/json process_request /usr/lib/python2.7/site-packages/heat/api/middleware/version_negotiation.py:50
2016-01-28 08:34:48.047 18689 DEBUG heat.api.middleware.version_negotiation [-] Matched versioned URI. Version: 1.0 process_request /usr/lib/python2.7/site-packages/heat/api/middleware/version_negotiation.py:65
2016-01-28 08:34:48.048 18689 DEBUG keystoneclient.auth.identity.v2 [-] Making authentication request to http://192.0.2.254:35357/v2.0/tokens get_auth_ref /usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v2.py:76
2016-01-28 08:34:48.143 18689 DEBUG keystoneclient.session [-] REQ: curl -g -i -X GET http://192.0.2.254:35357/v3/auth/tokens -H "X-Subject-Token: {SHA1}17dae1b61b7814e6a1e53340cbe803e24253c85f" -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}d47835b90aa754e0b0ab2ef7ec6141685d64de11" _http_log_request /usr/lib/python2.7/site-packages/keystoneclient/session.py:195
2016-01-28 08:34:48.185 18689 DEBUG keystoneclient.session [-] RESP: [200] content-length: 6435 x-subject-token: {SHA1}17dae1b61b7814e6a1e53340cbe803e24253c85f vary: X-Auth-Token connection: keep-alive date: Thu, 28 Jan 2016 13:34:48 GMT content-type: application/json x-openstack-request-id: req-f0561ed2-8b03-4809-90b9-6e370042b065
RESP BODY: {"token": {"methods": ["password", "token"], "roles": [{"id": "0472321ad9e244ef837e75c9af853888", "name": "admin"}, {"id": "9fe2ff9ee4384b1894a90878d3e92bab", "name": "_member_"}], "expires_at": "2016-01-28T17:34:45.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "369b087f32454cfc890a36a604b6bdd2", "name": "admin"}, "catalog": "<removed>", "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "1660416dcad649e5b908789bde0c084c", "name": "admin"}, "audit_ids": ["e4UWZ5e6THiVHuRSdqT88w", "W0lKQintSF6IxIo7U7goag"], "issued_at": "2016-01-28T13:34:45.174348"}}
 _http_log_response /usr/lib/python2.7/site-packages/keystoneclient/session.py:224
2016-01-28 08:34:48.188 18689 DEBUG heat.openstack.common.policy [req-37124495-f1ba-468b-b81a-56462b00caa3 - admin] Rules successfully reloaded _load_policy_file /usr/lib/python2.7/site-packages/heat/openstack/common/policy.py:295
2016-01-28 08:34:48.188 18689 INFO heat.openstack.common.policy [req-37124495-f1ba-468b-b81a-56462b00caa3 - admin] Can not find policy directory: policy.d
2016-01-28 08:34:48.189 18689 DEBUG heat.common.wsgi [req-37124495-f1ba-468b-b81a-56462b00caa3 - admin] Calling <heat.api.openstack.v1.stacks.StackController object at 0x30f36d0> : index __call__ /usr/lib/python2.7/site-packages/heat/common/wsgi.py:647
2016-01-28 08:34:48.189 18689 INFO heat.openstack.common.policy [req-37124495-f1ba-468b-b81a-56462b00caa3 - admin] Can not find policy directory: policy.d
2016-01-28 08:34:48.190 18689 DEBUG oslo_messaging._drivers.amqpdriver [req-37124495-f1ba-468b-b81a-56462b00caa3 - admin] MSG_ID is a330fde8de244e19b5658bdfb0ac56ab _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:311
2016-01-28 08:34:48.191 18689 DEBUG oslo_messaging._drivers.amqp [req-37124495-f1ba-468b-b81a-56462b00caa3 - admin] UNIQUE_ID is a2ad677fd5cf46748e245c7e74d09da7. _add_unique_id /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqp.py:258
2016-01-28 08:34:48.251 18689 DEBUG heat.common.serializers [req-37124495-f1ba-468b-b81a-56462b00caa3 - admin] JSON response : {"stacks": [{"parent": null, "description": "No description", "links": [{"href": "http://192.0.2.254:8004/v1/369b087f32454cfc890a36a604b6bdd2/stacks/overcloud/da4ea309-d27b-46cf-9126-aed91b1fac99", "rel": "self"}], "stack_status_reason": "Resource CREATE failed: Error: resources.ControllerNodesPostDeployment.resources.ControllerServicesBaseDeployment_Step2.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6", "stack_name": "overcloud", "stack_user_project_id": "618ccf0bc8974e73b91e451e3a915602", "creation_time": "2016-01-28T00:12:43Z", "updated_time": null, "stack_owner": "admin", "stack_status": "CREATE_FAILED", "id": "da4ea309-d27b-46cf-9126-aed91b1fac99"}]} to_json /usr/lib/python2.7/site-packages/heat/common/serializers.py:42
2016-01-28 08:34:48.252 18689 INFO eventlet.wsgi.server [req-37124495-f1ba-468b-b81a-56462b00caa3 - admin] 192.0.2.254 - - [28/Jan/2016 08:34:48] "GET /v1/369b087f32454cfc890a36a604b6bdd2/stacks?sort_dir=desc&sort_key=created_at&limit=1000 HTTP/1.1" 200 940 0.206061
2016-01-28 08:34:48.255 18689 INFO eventlet.wsgi.server [-] (18689) accepted ('192.0.2.254', 60255)