Bug 1296889 - OSPd deployment fails with "Expecting to find username or userId in passwordCredentials"
Summary: OSPd deployment fails with "Expecting to find username or userId in passwordC...
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director
Version: 7.0 (Kilo)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 10.0 (Newton)
Assignee: Hugh Brock
QA Contact: Shai Revivo
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-01-08 11:05 UTC by Filip Hubík
Modified: 2016-10-10 04:21 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-10-10 04:21:50 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Filip Hubík 2016-01-08 11:05:15 UTC
Description of problem:
CI is unable to deploy OSPd7(8 too with same error) puddle (2016-01-06.1) on baremetals with any topology or network variant on RHEL 7.2 Server. We are deploying on 5 BM nodes, first node assigned for undercloud.

Overcloud deployment failed. OC controllers are not accessible through SSH, they are in ERROR state. Heat-stack list reports overcloud CREATE_FAILED, there are errors in heat resource-list & event-list.

1) Introspection result:
rdomanager_oscplugin.utils.wait_for_node_discovery Discovery didn't finish for nodes e083f04c-1929-4432-9d95-408bc9520f87,f3a5ca66-5352-4a0c-9894-db70063d4af3,00bc9ed0-87d2-406e-b426-ecb90acce257,86a513c4-65b8-4be8-a6b1-04fff7bf64d1", "stdout": "Setting available nodes to manageable...\nStarting introspection of node: e083f04c-1929-4432-9d95-408bc9520f87\nStarting introspection of node: f3a5ca66-5352-4a0c-9894-db70063d4af3\nStarting introspection of node: 00bc9ed0-87d2-406e-b426-ecb90acce257\nStarting introspection of node: 86a513c4-65b8-4be8-a6b1-04fff7bf64d1\nWaiting for discovery to finish...\nSetting manageable nodes to available...\nNode e083f04c-1929-4432-9d95-408bc9520f87 has been set to available.\nNode f3a5ca66-5352-4a0c-9894-db70063d4af3 has been set to available.\nNode 00bc9ed0-87d2-406e-b426-ecb90acce257 has been set to available.\nNode 86a513c4-65b8-4be8-a6b1-04fff7bf64d1 has been set to available.\nDiscovery completed.", "warnings": []}

04:43:50 stdout: Setting available nodes to manageable...
04:43:50 Starting introspection of node: e083f04c-1929-4432-9d95-408bc9520f87
04:43:50 Starting introspection of node: f3a5ca66-5352-4a0c-9894-db70063d4af3
04:43:50 Starting introspection of node: 00bc9ed0-87d2-406e-b426-ecb90acce257
04:43:50 Starting introspection of node: 86a513c4-65b8-4be8-a6b1-04fff7bf64d1
04:43:50 Waiting for discovery to finish...
04:43:50 Setting manageable nodes to available...
04:43:50 Node e083f04c-1929-4432-9d95-408bc9520f87 has been set to available.
04:43:50 Node f3a5ca66-5352-4a0c-9894-db70063d4af3 has been set to available.
04:43:50 Node 00bc9ed0-87d2-406e-b426-ecb90acce257 has been set to available.
04:43:50 Node 86a513c4-65b8-4be8-a6b1-04fff7bf64d1 has been set to available.
04:43:50 Discovery completed.

2) Deploy command:

openstack overcloud deploy --debug --log-file overcloud_deployment_14.log --templates --libvirt-type=kvm --neutron-network-type gre --neutron-tunnel-types gre --ntp-server 10.5.26.10 --control-scale 1 --compute-scale 1 --ceph-storage-scale 1 --block-storage-scale 0 --swift-storage-scale 0 --control-flavor baremetal --compute-flavor baremetal --ceph-storage-flavor baremetal --block-storage-flavor baremetal --swift-storage-flavor baremetal --neutron-public-interface=nic2 --timeout=480 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml

3) nova list; ironic node-list; openstack server list; after deploy command

08:46:46 | ID                                   | Name                    | Status | Task State | Power State | Networks               |
08:46:46 +--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
08:46:46 | 8376ad16-4f01-464d-841e-037d066d5c9b | overcloud-cephstorage-0 | BUILD  | spawning   | NOSTATE     | ctlplane=10.200.200.14 |
08:46:46 | 0f227da2-e1d1-4ec9-9278-b29c88579965 | overcloud-compute-0     | BUILD  | spawning   | NOSTATE     | ctlplane=10.200.200.12 |
08:46:46 | 4f90b498-0e7b-4d35-9204-d92aae0794f7 | overcloud-controller-0  | BUILD  | spawning   | NOSTATE     | ctlplane=10.200.200.13 |
08:46:46 +--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
08:46:46 +--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+
08:46:46 | UUID                                 | Name | Instance UUID                        | Power State | Provision State | Maintenance |
08:46:46 +--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+
08:46:46 | e083f04c-1929-4432-9d95-408bc9520f87 | None | 0f227da2-e1d1-4ec9-9278-b29c88579965 | power on    | wait call-back  | False       |
08:46:46 | f3a5ca66-5352-4a0c-9894-db70063d4af3 | None | 8376ad16-4f01-464d-841e-037d066d5c9b | power on    | wait call-back  | False       |
08:46:46 | 00bc9ed0-87d2-406e-b426-ecb90acce257 | None | 4f90b498-0e7b-4d35-9204-d92aae0794f7 | power on    | wait call-back  | False       |
08:46:46 | 86a513c4-65b8-4be8-a6b1-04fff7bf64d1 | None | None                                 | power off   | available       | False       |
08:46:46 +--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+
08:46:46 +--------------------------------------+-------------------------+--------+------------------------+
08:46:46 | ID                                   | Name                    | Status | Networks               |
08:46:46 +--------------------------------------+-------------------------+--------+------------------------+
08:46:46 | 8376ad16-4f01-464d-841e-037d066d5c9b | overcloud-cephstorage-0 | BUILD  | ctlplane=10.200.200.14 |
08:46:46 | 4f90b498-0e7b-4d35-9204-d92aae0794f7 | overcloud-controller-0  | BUILD  | ctlplane=10.200.200.13 |
08:46:46 | 0f227da2-e1d1-4ec9-9278-b29c88579965 | overcloud-compute-0     | BUILD  | ctlplane=10.200.200.12 |
08:46:46 +--------------------------------------+-------------------------+--------+------------------------+

4) Heat resource-list overcloud
08:46:54 | CephStorage                               | 2a22f3cf-5503-4a99-baee-63af747c7e72          | OS::Heat::ResourceGroup                           | CREATE_FAILED   | 2016-01-08T04:46:27Z |
08:46:54 | Controller                                | 939f672a-6ae4-4575-a71b-1a4c4db5e25d          | OS::Heat::ResourceGroup                           | CREATE_FAILED   | 2016-01-08T04:46:27Z |

5) heat event-list overcloud
...
|\n| CephStorage            | e0b7755e-9cf7-4d33-bcc6-2f723d170a88 | BadRequest: resources.CephStorage.resources[0].resources.CephStorage: Expecting to find username or userId in passwordCredentials - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to  | CREATE_FAILED      | 2016-01-08T08:46:22Z |\n| Compute                | 8db490cc-d77f-42fc-9320-79c67a4a0eb9 | BadRequest: resources.Compute.resources[0].resources.NovaCompute: Expecting to find username or userId in passwordCredentials - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be i | CREATE_FAILED      | 2016-01-08T08:46:23Z |\n| Controller             | 48ef3a0b-9dee-4403-a453-1155c4c81a4c | BadRequest: resources.Controller.resources[0].resources.Controller: Expecting to find username or userId in passwordCredentials - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be | CREATE_FAILED      | 2016-01-08T08:46:24Z |\n| overcloud              | 58aee2d1-9bfc-455c-8df9-06c4d0562eab | Resource CREATE failed: BadRequest: resources.CephStorage.resources[0].resources.CephStorage: Expecting to find username or userId in passwordCredentials - the server could not comply with the request since it is either malformed or otherwise incorrect. T | CREATE_FAILED      | 2016-01-08T08:46:25Z 

How reproducible:
Deployment done with khaleesi tool using params:

ksgen --config-dir settings generate --provisioner=beaker --provisioner-distro=rhel --provisioner-distro-version=7.2 --provisioner-options=execute_provision --provisioner-site=bkr --provisioner-site-user=rdoci-jenkins --product-version=7_director --product-version-build=latest --product-version-repo=puddle --distro=rhel-7.2 --product=rhos --installer=rdo_manager --installer-deploy=templates --installer-env=baremetal --installer-images=import --installer-network=neutron --installer-network-isolation=none --installer-network-variant=gre --installer-post_action=default --installer-topology=minimal --installer-tempest=all --workarounds=enabled --extra-vars @../khaleesi-settings/hardware_environments/qe/dell_pe_c6220_6-10/network_configs/none/hw_settings.yml --extra-vars @../khaleesi-settings/settings/product/rhos/private_settings/redhat_internal.yml ksgen_settings.yml

Additional info:
https://rhos-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/RHOS/view/RHOS7/job/ospd_qe-7_director-puddle-rhel-7.2-templates-baremetal-dell_pe_c6220_6-10-minimal-none-gre-all/20/consoleFull for example.

Comment 4 Mike Burns 2016-04-07 21:03:37 UTC
This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.

Comment 6 Jaromir Coufal 2016-10-10 04:21:50 UTC
Could not reproduce in our CI. Please re-open if reproducable. Might be issue with specific environment which we would need to investigate on the concrete environment.


Note You need to log in before you can comment on or make changes to this bug.