Description of problem: Using the latest puddle code, deploying the overcloud fails on both Dell and HP hardware. instack-deploy-overcloud --tuskar leaves one node in ACTIVE state and one in ERROR. - On Dell: [stack@host15 ~]$ nova list +--------------------------------------+-------------------------------------------------------+--------+------------+-------------+----------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------------------------------------------------------+--------+------------+-------------+----------------------+ | 41c880ca-e263-4677-bbe3-8aeb0bc8787c | ov-qcojsahar5-0-mkn6tsc735qu-NovaCompute-axich2jzb5z4 | ACTIVE | - | Running | ctlplane=10.8.147.9 | | d980eb9d-13c3-487d-a1c2-af7c39bdfe1a | ov-t7f35hco2lq-0-ga6yyh6wggfb-Controller-2yrm3pmelpep | ERROR | - | NOSTATE | ctlplane=10.8.147.10 | +--------------------------------------+-------------------------------------------------------+--------+------------+-------------+----------------------+ [stack@host15 ~]$ heat stack-list +--------------------------------------+------------+---------------+----------------------+ | id | stack_name | stack_status | creation_time | +--------------------------------------+------------+---------------+----------------------+ | 06e139ce-315c-41dd-a861-26b53e12a02b | overcloud | CREATE_FAILED | 2015-06-01T02:53:29Z | +--------------------------------------+------------+---------------+----------------------+ - On HP: [stack@virtblade11 ~]$ nova list +--------------------------------------+-------------------------------------------------------+--------+------------+-------------+---------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------------------------------------------------------+--------+------------+-------------+---------------------+ | 3e97833a-3721-4f77-b2a5-cba70cd4279d | ov-5jdzx4elxx-0-dqy757egupbf-NovaCompute-7sfiap2m6m34 | ERROR | - | NOSTATE | ctlplane=192.0.2.10 | | eca52463-48d6-44c8-b873-60e6dd498243 | ov-yv43oxz5fx5-0-3474l6vv27ws-Controller-brndcdjrfyzs | ACTIVE | - | Running | ctlplane=192.0.2.11 | +--------------------------------------+-------------------------------------------------------+--------+------------+-------------+---------------------+ [stack@virtblade11 ~]$ heat stack-list +--------------------------------------+------------+---------------+----------------------+ | id | stack_name | stack_status | creation_time | +--------------------------------------+------------+---------------+----------------------+ | 559effe7-6bb6-47c8-8e91-9a8daaa4a6e8 | overcloud | CREATE_FAILED | 2015-06-01T01:16:43Z | +--------------------------------------+------------+---------------+----------------------+ - heat stack-show overcloud reports: | stack_status | CREATE_FAILED | | stack_status_reason | Resource CREATE failed: ResourceUnknownStatus: Resource | | | failed - Unknown status FAILED due to "Resource CREATE | | | failed: ResourceUnknownStatus: Resource failed - | | | Unknown status FAILED due to "Resource CREATE failed: | | | BadRequest: Expecting to find username or userId in | | | passwordCredentials - the server could not comply with | the request since it is either malformed or otherwise | | incorrect. The client is assumed to be in error. (HTTP | | | 400)"" journalctl shows a lot of : Jun 01 03:38:36 host15.beaker.tripleo.lab.eng.rdu2.redhat.com ironic-conductor[33968]: 2015-06-01 03:38:36.964 33968 ERROR oslo_messaging.rpc.dispatcher [-] Exception during message handling: Node 6e25f27f-bf16-47c7-a3cd-4340f3ad07cf is associated with instance 41c880ca-e263-4677-bbe3-8aeb0bc8787c. Jun 01 03:38:36 host15.beaker.tripleo.lab.eng.rdu2.redhat.com ironic-conductor[33968]: 2015-06-01 03:38:36.965 33968 ERROR oslo_messaging._drivers.common [-] Returning exception Node 6e25f27f-bf16-47c7-a3cd-4340f3ad07cf is associated with instance 41c880ca-e263-4677-bbe3-8aeb0bc8787c. to caller Jun 01 03:38:36 host15.beaker.tripleo.lab.eng.rdu2.redhat.com ironic-conductor[33968]: 2015-06-01 03:38:36.965 33968 ERROR oslo_messaging._drivers.common [-] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n executor_callback))\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 142, in inner\n return func(*args, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/ironic/conductor/manager.py", line 388, in update_node\n node_obj.save()\n', ' File "/usr/lib/python2.7/site-packages/ironic/objects/base.py", line 143, in wrapper\n return fn(self, ctxt, *args, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/ironic/objects/node.py", line 265, in save\n self.dbapi.update_node(self.uuid, updates)\n', ' File "/usr/lib/python2.7/site-packages/ironic/db/sqlalchemy/api.py", line 338, in update_node\n return self._do_update_node(node_id, values)\n', ' File "/usr/lib/python2.7/site-packages/ironic/db/sqlalchemy/api.py", line 364, in _do_update_node\n instance=ref.instance_uuid)\n', 'NodeAssociated: Node 6e25f27f-bf16-47c7-a3cd-4340f3ad07cf is associated with instance 41c880ca-e263-4677-bbe3-8aeb0bc8787c.\n'] Jun 01 03:38:39 host15.beaker.tripleo.lab.eng.rdu2.redhat.com ironic-conductor[33968]: 2015-06-01 03:38:39.060 33968 ERROR Version-Release number of selected component (if applicable): latest puddle How reproducible: Always - all hardware we have access to Steps to Reproduce: 1. Install undercloud and setup for overcloud deploy on baremetal following docs https://repos.fedorapeople.org/repos/openstack-m/docs/master/basic_deployment/basic_deployment.html using bits from the latest puddle 2. Run instack-deploy-overcloud --tuskar 3. Wait until heat stack-list shows CREATE_FAILED Actual results: Deploy fails Expected results: CREATE_COMPLETE - deploy should succeed Additional info:
We are getting the same error when trying to deploy overcloud via Tuskar-UI [root@localhost heat]# cat heat-engine.log | grep error 2015-05-21 09:35:03.940 2051 TRACE heat.engine.resource BadRequest: Expecting to find username or userId in passwordCredentials - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) 2015-05-21 09:35:04.987 2051 INFO heat.engine.stack [-] Stack CREATE FAILED (overcloud-Compute-uoogcnoy5gru-0-4wbvn3zxvcf5): Resource CREATE failed: BadRequest: Expecting to find username or userId in passwordCredentials - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) [root@localhost heat]# cat heat-api.log | grep error 2015-05-22 07:32:18.435 1229 DEBUG heat.common.serializers [-] JSON response : {"explanation": "The resource could not be found.", "code": 404, "error": {"message": "The Resource (Controller-1-servers) could not be found in Stack overcloud.", "traceback": "Traceback (most recent call last):\n\n File \"/usr/lib/python2.7/site-packages/heat/common/context.py\", line 302, in wrapped\n return func(self, ctx, *args, **kwargs)\n\n File \"/usr/lib/python2.7/site-packages/heat/engine/service.py\", line 1095, in describe_stack_resource\n stack_name=stack.name)\n\nResourceNotFound: The Resource (Controller-1-servers) could not be found in Stack overcloud.\n", "type": "ResourceNotFound"}, "title": "Not Found"} to_json /usr/lib/python2.7/site-packages/heat/common/serializers.py:42 2015-05-22 07:32:18.656 1229 DEBUG heat.common.serializers [-] JSON response : {"explanation": "The resource could not be found.", "code": 404, "error": {"message": "The Resource (Cinder-Storage-1-servers) could not be found in Stack overcloud.", "traceback": "Traceback (most recent call last):\n\n File \"/usr/lib/python2.7/site-packages/heat/common/context.py\", line 302, in wrapped\n return func(self, ctx, *args, **kwargs)\n\n File \"/usr/lib/python2.7/site-packages/heat/engine/service.py\", line 1095, in describe_stack_resource\n stack_name=stack.name)\n\nResourceNotFound: The Resource (Cinder-Storage-1-servers) could not be found in Stack overcloud.\n", "type": "ResourceNotFound"}, "title": "Not Found"} to_json /usr/lib/python2.7/site-packages/heat/common/serializers.py:42 2015-05-22 07:32:18.869 1229 DEBUG heat.common.serializers [-] JSON response : {"explanation": "The resource could not be found.", "code": 404, "error": {"message": "The Resource (Compute-1-servers) could not be found in Stack overcloud.", "traceback": "Traceback (most recent call last):\n\n File \"/usr/lib/python2.7/site-packages/heat/common/context.py\", line 302, in wrapped\n return func(self, ctx, *args, **kwargs)\n\n File \"/usr/lib/python2.7/site-packages/heat/engine/service.py\", line 1095, in describe_stack_resource\n stack_name=stack.name)\n\nResourceNotFound: The Resource (Compute-1-servers) could not be found in Stack overcloud.\n", "type": "ResourceNotFound"}, "title": "Not Found"} to_json /usr/lib/python2.7/site-packages/heat/common/serializers.py:42 2015-05-22 07:32:19.031 1229 DEBUG heat.common.serializers [-] JSON response : {"explanation": "The resource could not be found.", "code": 404, "error": {"message": "The Resource (Swift-Storage-1-servers) could not be found in Stack overcloud.", "traceback": "Traceback (most recent call last):\n\n File \"/usr/lib/python2.7/site-packages/heat/common/context.py\", line 302, in wrapped\n return func(self, ctx, *args, **kwargs)\n\n File \"/usr/lib/python2.7/site-packages/heat/engine/service.py\", line 1095, in describe_stack_resource\n stack_name=stack.name)\n\nResourceNotFound: The Resource (Swift-Storage-1-servers) could not be found in Stack overcloud.\n", "type": "ResourceNotFound"}, "title": "Not Found"} to_json /usr/lib/python2.7/site-packages/heat/common/serializers.py:42 2015-05-22 07:32:19.195 1229 DEBUG heat.common.serializers [-] JSON response : {"explanation": "The resource could not be found.", "code": 404, "error": {"message": "The Resource (Ceph-Storage-1-servers) could not be found in Stack overcloud.", "traceback": "Traceback (most recent call last):\n\n File \"/usr/lib/python2.7/site-packages/heat/common/context.py\", line 302, in wrapped\n return func(self, ctx, *args, **kwargs)\n\n File \"/usr/lib/python2.7/site-packages/heat/engine/service.py\", line 1095, in describe_stack_resource\n stack_name=stack.name)\n\nResourceNotFound: The Resource (Ceph-Storage-1-servers) could not be found in Stack overcloud.\n", "type": "ResourceNotFound"}, "title": "Not Found"} to_json /usr/lib/python2.7/site-packages/heat/common/serializers.py:42
This should now be resolved.
Verified in puddle 2015-06-24
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2015:1549