Bug 1226957 - CREATE_FAILED for overcloud deploy on baremetal, 1 node active, 1 in error state: "BadRequest: Expecting to find username or userId in passwordCredentials"
Summary: CREATE_FAILED for overcloud deploy on baremetal, 1 node active, 1 in error st...
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director
Version: 7.0 (Kilo)
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ga
: Director
Assignee: Jay Dobies
QA Contact: Udi
URL:
Whiteboard:
Keywords: Automation, TestOnly, Triaged
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-06-01 14:11 UTC by Ronelle Landy
Modified: 2015-08-05 13:52 UTC (History)
7 users (show)

(edit)
Clone Of:
(edit)
Last Closed: 2015-08-05 13:52:16 UTC


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2015:1549 normal SHIPPED_LIVE Red Hat Enterprise Linux OpenStack Platform director Release 2015-08-05 17:49:10 UTC

Description Ronelle Landy 2015-06-01 14:11:31 UTC
Description of problem:

Using the latest puddle code, deploying the overcloud fails on both Dell and HP hardware. instack-deploy-overcloud --tuskar leaves one node in ACTIVE state and one in ERROR.

 - On Dell:

[stack@host15 ~]$ nova list
+--------------------------------------+-------------------------------------------------------+--------+------------+-------------+----------------------+
| ID                                   | Name                                                  | Status | Task State | Power State | Networks             |
+--------------------------------------+-------------------------------------------------------+--------+------------+-------------+----------------------+
| 41c880ca-e263-4677-bbe3-8aeb0bc8787c | ov-qcojsahar5-0-mkn6tsc735qu-NovaCompute-axich2jzb5z4 | ACTIVE | -          | Running     | ctlplane=10.8.147.9  |
| d980eb9d-13c3-487d-a1c2-af7c39bdfe1a | ov-t7f35hco2lq-0-ga6yyh6wggfb-Controller-2yrm3pmelpep | ERROR  | -          | NOSTATE     | ctlplane=10.8.147.10 |
+--------------------------------------+-------------------------------------------------------+--------+------------+-------------+----------------------+
[stack@host15 ~]$ heat stack-list
+--------------------------------------+------------+---------------+----------------------+
| id                                   | stack_name | stack_status  | creation_time        |
+--------------------------------------+------------+---------------+----------------------+
| 06e139ce-315c-41dd-a861-26b53e12a02b | overcloud  | CREATE_FAILED | 2015-06-01T02:53:29Z |
+--------------------------------------+------------+---------------+----------------------+

 - On HP:

[stack@virtblade11 ~]$ nova list
+--------------------------------------+-------------------------------------------------------+--------+------------+-------------+---------------------+
| ID                                   | Name                                                  | Status | Task State | Power State | Networks            |
+--------------------------------------+-------------------------------------------------------+--------+------------+-------------+---------------------+
| 3e97833a-3721-4f77-b2a5-cba70cd4279d | ov-5jdzx4elxx-0-dqy757egupbf-NovaCompute-7sfiap2m6m34 | ERROR  | -          | NOSTATE     | ctlplane=192.0.2.10 |
| eca52463-48d6-44c8-b873-60e6dd498243 | ov-yv43oxz5fx5-0-3474l6vv27ws-Controller-brndcdjrfyzs | ACTIVE | -          | Running     | ctlplane=192.0.2.11 |
+--------------------------------------+-------------------------------------------------------+--------+------------+-------------+---------------------+
[stack@virtblade11 ~]$ heat stack-list
+--------------------------------------+------------+---------------+----------------------+
| id                                   | stack_name | stack_status  | creation_time        |
+--------------------------------------+------------+---------------+----------------------+
| 559effe7-6bb6-47c8-8e91-9a8daaa4a6e8 | overcloud  | CREATE_FAILED | 2015-06-01T01:16:43Z |
+--------------------------------------+------------+---------------+----------------------+


 - heat stack-show overcloud reports:

| stack_status          | CREATE_FAILED                                                                                                                                                                                                                                                                                                                                                                                      |
| stack_status_reason   | Resource CREATE failed: ResourceUnknownStatus: Resource                                                                                                                                                                                                                                                                                                                                                    |
|                       | failed - Unknown status FAILED due to "Resource CREATE                                                                                                                                                                                                                                                                                                                                                     |
|                       | failed: ResourceUnknownStatus: Resource failed -                                                                                                                                                                                                                                                                                                                                                            |
|                       | Unknown status FAILED due to "Resource CREATE failed:                                                                                                                                                                                                                                                                                                                                                       |
|                       | BadRequest: Expecting to find username or userId in                                                                                                                                                                                                                                                                                                                                                        |
|                       | passwordCredentials - the server could not comply with                                                                                                                                                                                                                                     | the request since it is either malformed or otherwise                                                                                                                                                                                                           |                       | incorrect. The client is assumed to be in error. (HTTP                                                                                                                                                                                                                                                                                                                                                      |
|                       | 400)""                                              


journalctl shows a lot of :

Jun 01 03:38:36 host15.beaker.tripleo.lab.eng.rdu2.redhat.com ironic-conductor[33968]: 2015-06-01 03:38:36.964 33968 ERROR oslo_messaging.rpc.dispatcher [-] Exception during message handling: Node 6e25f27f-bf16-47c7-a3cd-4340f3ad07cf is associated with instance 41c880ca-e263-4677-bbe3-8aeb0bc8787c.
Jun 01 03:38:36 host15.beaker.tripleo.lab.eng.rdu2.redhat.com ironic-conductor[33968]: 2015-06-01 03:38:36.965 33968 ERROR oslo_messaging._drivers.common [-] Returning exception Node 6e25f27f-bf16-47c7-a3cd-4340f3ad07cf is associated with instance 41c880ca-e263-4677-bbe3-8aeb0bc8787c. to caller
Jun 01 03:38:36 host15.beaker.tripleo.lab.eng.rdu2.redhat.com ironic-conductor[33968]: 2015-06-01 03:38:36.965 33968 ERROR oslo_messaging._drivers.common [-] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n    result = func(ctxt, **new_args)\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 142, in inner\n    return func(*args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/ironic/conductor/manager.py", line 388, in update_node\n    node_obj.save()\n', '  File "/usr/lib/python2.7/site-packages/ironic/objects/base.py", line 143, in wrapper\n    return fn(self, ctxt, *args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/ironic/objects/node.py", line 265, in save\n    self.dbapi.update_node(self.uuid, updates)\n', '  File "/usr/lib/python2.7/site-packages/ironic/db/sqlalchemy/api.py", line 338, in update_node\n    return self._do_update_node(node_id, values)\n', '  File "/usr/lib/python2.7/site-packages/ironic/db/sqlalchemy/api.py", line 364, in _do_update_node\n    instance=ref.instance_uuid)\n', 'NodeAssociated: Node 6e25f27f-bf16-47c7-a3cd-4340f3ad07cf is associated with instance 41c880ca-e263-4677-bbe3-8aeb0bc8787c.\n']
Jun 01 03:38:39 host15.beaker.tripleo.lab.eng.rdu2.redhat.com ironic-conductor[33968]: 2015-06-01 03:38:39.060 33968 ERROR 


Version-Release number of selected component (if applicable):

latest puddle

How reproducible:

Always - all hardware we have access to

Steps to Reproduce:
1. Install undercloud and setup for overcloud deploy on baremetal following docs https://repos.fedorapeople.org/repos/openstack-m/docs/master/basic_deployment/basic_deployment.html using bits from the latest puddle
2. Run instack-deploy-overcloud --tuskar
3. Wait until heat stack-list shows CREATE_FAILED

Actual results:

Deploy fails

Expected results:

CREATE_COMPLETE - deploy should succeed

Additional info:

Comment 4 Jiri Tomasek 2015-06-04 09:33:45 UTC
We are getting the same error when trying to deploy overcloud via Tuskar-UI

[root@localhost heat]# cat heat-engine.log | grep error
2015-05-21 09:35:03.940 2051 TRACE heat.engine.resource BadRequest: Expecting to find username or userId in passwordCredentials - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400)
2015-05-21 09:35:04.987 2051 INFO heat.engine.stack [-] Stack CREATE FAILED (overcloud-Compute-uoogcnoy5gru-0-4wbvn3zxvcf5): Resource CREATE failed: BadRequest: Expecting to find username or userId in passwordCredentials - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400)
[root@localhost heat]# cat heat-api.log | grep error
2015-05-22 07:32:18.435 1229 DEBUG heat.common.serializers [-] JSON response : {"explanation": "The resource could not be found.", "code": 404, "error": {"message": "The Resource (Controller-1-servers) could not be found in Stack overcloud.", "traceback": "Traceback (most recent call last):\n\n  File \"/usr/lib/python2.7/site-packages/heat/common/context.py\", line 302, in wrapped\n    return func(self, ctx, *args, **kwargs)\n\n  File \"/usr/lib/python2.7/site-packages/heat/engine/service.py\", line 1095, in describe_stack_resource\n    stack_name=stack.name)\n\nResourceNotFound: The Resource (Controller-1-servers) could not be found in Stack overcloud.\n", "type": "ResourceNotFound"}, "title": "Not Found"} to_json /usr/lib/python2.7/site-packages/heat/common/serializers.py:42
2015-05-22 07:32:18.656 1229 DEBUG heat.common.serializers [-] JSON response : {"explanation": "The resource could not be found.", "code": 404, "error": {"message": "The Resource (Cinder-Storage-1-servers) could not be found in Stack overcloud.", "traceback": "Traceback (most recent call last):\n\n  File \"/usr/lib/python2.7/site-packages/heat/common/context.py\", line 302, in wrapped\n    return func(self, ctx, *args, **kwargs)\n\n  File \"/usr/lib/python2.7/site-packages/heat/engine/service.py\", line 1095, in describe_stack_resource\n    stack_name=stack.name)\n\nResourceNotFound: The Resource (Cinder-Storage-1-servers) could not be found in Stack overcloud.\n", "type": "ResourceNotFound"}, "title": "Not Found"} to_json /usr/lib/python2.7/site-packages/heat/common/serializers.py:42
2015-05-22 07:32:18.869 1229 DEBUG heat.common.serializers [-] JSON response : {"explanation": "The resource could not be found.", "code": 404, "error": {"message": "The Resource (Compute-1-servers) could not be found in Stack overcloud.", "traceback": "Traceback (most recent call last):\n\n  File \"/usr/lib/python2.7/site-packages/heat/common/context.py\", line 302, in wrapped\n    return func(self, ctx, *args, **kwargs)\n\n  File \"/usr/lib/python2.7/site-packages/heat/engine/service.py\", line 1095, in describe_stack_resource\n    stack_name=stack.name)\n\nResourceNotFound: The Resource (Compute-1-servers) could not be found in Stack overcloud.\n", "type": "ResourceNotFound"}, "title": "Not Found"} to_json /usr/lib/python2.7/site-packages/heat/common/serializers.py:42
2015-05-22 07:32:19.031 1229 DEBUG heat.common.serializers [-] JSON response : {"explanation": "The resource could not be found.", "code": 404, "error": {"message": "The Resource (Swift-Storage-1-servers) could not be found in Stack overcloud.", "traceback": "Traceback (most recent call last):\n\n  File \"/usr/lib/python2.7/site-packages/heat/common/context.py\", line 302, in wrapped\n    return func(self, ctx, *args, **kwargs)\n\n  File \"/usr/lib/python2.7/site-packages/heat/engine/service.py\", line 1095, in describe_stack_resource\n    stack_name=stack.name)\n\nResourceNotFound: The Resource (Swift-Storage-1-servers) could not be found in Stack overcloud.\n", "type": "ResourceNotFound"}, "title": "Not Found"} to_json /usr/lib/python2.7/site-packages/heat/common/serializers.py:42
2015-05-22 07:32:19.195 1229 DEBUG heat.common.serializers [-] JSON response : {"explanation": "The resource could not be found.", "code": 404, "error": {"message": "The Resource (Ceph-Storage-1-servers) could not be found in Stack overcloud.", "traceback": "Traceback (most recent call last):\n\n  File \"/usr/lib/python2.7/site-packages/heat/common/context.py\", line 302, in wrapped\n    return func(self, ctx, *args, **kwargs)\n\n  File \"/usr/lib/python2.7/site-packages/heat/engine/service.py\", line 1095, in describe_stack_resource\n    stack_name=stack.name)\n\nResourceNotFound: The Resource (Ceph-Storage-1-servers) could not be found in Stack overcloud.\n", "type": "ResourceNotFound"}, "title": "Not Found"} to_json /usr/lib/python2.7/site-packages/heat/common/serializers.py:42

Comment 5 chris alfonso 2015-06-15 19:58:24 UTC
This should now be resolved.

Comment 7 Udi 2015-06-29 10:35:59 UTC
Verified in puddle 2015-06-24

Comment 9 errata-xmlrpc 2015-08-05 13:52:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2015:1549


Note You need to log in before you can comment on or make changes to this bug.