Bug 1296889

Summary:	OSPd deployment fails with "Expecting to find username or userId in passwordCredentials"
Product:	Red Hat OpenStack	Reporter:	Filip Hubík <fhubik>
Component:	rhosp-director	Assignee:	Hugh Brock <hbrock>
Status:	CLOSED WORKSFORME	QA Contact:	Shai Revivo <srevivo>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	7.0 (Kilo)	CC:	fzdarsky, jcoufal, mburns, rhel-osp-director-maint, stanislav.polasek, tkammer
Target Milestone:	---
Target Release:	10.0 (Newton)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-10-10 04:21:50 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Filip Hubík 2016-01-08 11:05:15 UTC

Description of problem:
CI is unable to deploy OSPd7(8 too with same error) puddle (2016-01-06.1) on baremetals with any topology or network variant on RHEL 7.2 Server. We are deploying on 5 BM nodes, first node assigned for undercloud.

Overcloud deployment failed. OC controllers are not accessible through SSH, they are in ERROR state. Heat-stack list reports overcloud CREATE_FAILED, there are errors in heat resource-list & event-list.

1) Introspection result:
rdomanager_oscplugin.utils.wait_for_node_discovery Discovery didn't finish for nodes e083f04c-1929-4432-9d95-408bc9520f87,f3a5ca66-5352-4a0c-9894-db70063d4af3,00bc9ed0-87d2-406e-b426-ecb90acce257,86a513c4-65b8-4be8-a6b1-04fff7bf64d1", "stdout": "Setting available nodes to manageable...\nStarting introspection of node: e083f04c-1929-4432-9d95-408bc9520f87\nStarting introspection of node: f3a5ca66-5352-4a0c-9894-db70063d4af3\nStarting introspection of node: 00bc9ed0-87d2-406e-b426-ecb90acce257\nStarting introspection of node: 86a513c4-65b8-4be8-a6b1-04fff7bf64d1\nWaiting for discovery to finish...\nSetting manageable nodes to available...\nNode e083f04c-1929-4432-9d95-408bc9520f87 has been set to available.\nNode f3a5ca66-5352-4a0c-9894-db70063d4af3 has been set to available.\nNode 00bc9ed0-87d2-406e-b426-ecb90acce257 has been set to available.\nNode 86a513c4-65b8-4be8-a6b1-04fff7bf64d1 has been set to available.\nDiscovery completed.", "warnings": []}

04:43:50 stdout: Setting available nodes to manageable...
04:43:50 Starting introspection of node: e083f04c-1929-4432-9d95-408bc9520f87
04:43:50 Starting introspection of node: f3a5ca66-5352-4a0c-9894-db70063d4af3
04:43:50 Starting introspection of node: 00bc9ed0-87d2-406e-b426-ecb90acce257
04:43:50 Starting introspection of node: 86a513c4-65b8-4be8-a6b1-04fff7bf64d1
04:43:50 Waiting for discovery to finish...
04:43:50 Setting manageable nodes to available...
04:43:50 Node e083f04c-1929-4432-9d95-408bc9520f87 has been set to available.
04:43:50 Node f3a5ca66-5352-4a0c-9894-db70063d4af3 has been set to available.
04:43:50 Node 00bc9ed0-87d2-406e-b426-ecb90acce257 has been set to available.
04:43:50 Node 86a513c4-65b8-4be8-a6b1-04fff7bf64d1 has been set to available.
04:43:50 Discovery completed.

2) Deploy command:

openstack overcloud deploy --debug --log-file overcloud_deployment_14.log --templates --libvirt-type=kvm --neutron-network-type gre --neutron-tunnel-types gre --ntp-server 10.5.26.10 --control-scale 1 --compute-scale 1 --ceph-storage-scale 1 --block-storage-scale 0 --swift-storage-scale 0 --control-flavor baremetal --compute-flavor baremetal --ceph-storage-flavor baremetal --block-storage-flavor baremetal --swift-storage-flavor baremetal --neutron-public-interface=nic2 --timeout=480 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml

3) nova list; ironic node-list; openstack server list; after deploy command

08:46:46 | ID                                   | Name                    | Status | Task State | Power State | Networks               |
08:46:46 +--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
08:46:46 | 8376ad16-4f01-464d-841e-037d066d5c9b | overcloud-cephstorage-0 | BUILD  | spawning   | NOSTATE     | ctlplane=10.200.200.14 |
08:46:46 | 0f227da2-e1d1-4ec9-9278-b29c88579965 | overcloud-compute-0     | BUILD  | spawning   | NOSTATE     | ctlplane=10.200.200.12 |
08:46:46 | 4f90b498-0e7b-4d35-9204-d92aae0794f7 | overcloud-controller-0  | BUILD  | spawning   | NOSTATE     | ctlplane=10.200.200.13 |
08:46:46 +--------------------------------------+-------------------------+--------+------------+-------------+------------------------+
08:46:46 +--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+
08:46:46 | UUID                                 | Name | Instance UUID                        | Power State | Provision State | Maintenance |
08:46:46 +--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+
08:46:46 | e083f04c-1929-4432-9d95-408bc9520f87 | None | 0f227da2-e1d1-4ec9-9278-b29c88579965 | power on    | wait call-back  | False       |
08:46:46 | f3a5ca66-5352-4a0c-9894-db70063d4af3 | None | 8376ad16-4f01-464d-841e-037d066d5c9b | power on    | wait call-back  | False       |
08:46:46 | 00bc9ed0-87d2-406e-b426-ecb90acce257 | None | 4f90b498-0e7b-4d35-9204-d92aae0794f7 | power on    | wait call-back  | False       |
08:46:46 | 86a513c4-65b8-4be8-a6b1-04fff7bf64d1 | None | None                                 | power off   | available       | False       |
08:46:46 +--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+
08:46:46 +--------------------------------------+-------------------------+--------+------------------------+
08:46:46 | ID                                   | Name                    | Status | Networks               |
08:46:46 +--------------------------------------+-------------------------+--------+------------------------+
08:46:46 | 8376ad16-4f01-464d-841e-037d066d5c9b | overcloud-cephstorage-0 | BUILD  | ctlplane=10.200.200.14 |
08:46:46 | 4f90b498-0e7b-4d35-9204-d92aae0794f7 | overcloud-controller-0  | BUILD  | ctlplane=10.200.200.13 |
08:46:46 | 0f227da2-e1d1-4ec9-9278-b29c88579965 | overcloud-compute-0     | BUILD  | ctlplane=10.200.200.12 |
08:46:46 +--------------------------------------+-------------------------+--------+------------------------+

4) Heat resource-list overcloud
08:46:54 | CephStorage                               | 2a22f3cf-5503-4a99-baee-63af747c7e72          | OS::Heat::ResourceGroup                           | CREATE_FAILED   | 2016-01-08T04:46:27Z |
08:46:54 | Controller                                | 939f672a-6ae4-4575-a71b-1a4c4db5e25d          | OS::Heat::ResourceGroup                           | CREATE_FAILED   | 2016-01-08T04:46:27Z |

5) heat event-list overcloud
...
|\n| CephStorage            | e0b7755e-9cf7-4d33-bcc6-2f723d170a88 | BadRequest: resources.CephStorage.resources[0].resources.CephStorage: Expecting to find username or userId in passwordCredentials - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to  | CREATE_FAILED      | 2016-01-08T08:46:22Z |\n| Compute                | 8db490cc-d77f-42fc-9320-79c67a4a0eb9 | BadRequest: resources.Compute.resources[0].resources.NovaCompute: Expecting to find username or userId in passwordCredentials - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be i | CREATE_FAILED      | 2016-01-08T08:46:23Z |\n| Controller             | 48ef3a0b-9dee-4403-a453-1155c4c81a4c | BadRequest: resources.Controller.resources[0].resources.Controller: Expecting to find username or userId in passwordCredentials - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be | CREATE_FAILED      | 2016-01-08T08:46:24Z |\n| overcloud              | 58aee2d1-9bfc-455c-8df9-06c4d0562eab | Resource CREATE failed: BadRequest: resources.CephStorage.resources[0].resources.CephStorage: Expecting to find username or userId in passwordCredentials - the server could not comply with the request since it is either malformed or otherwise incorrect. T | CREATE_FAILED      | 2016-01-08T08:46:25Z 

How reproducible:
Deployment done with khaleesi tool using params:

ksgen --config-dir settings generate --provisioner=beaker --provisioner-distro=rhel --provisioner-distro-version=7.2 --provisioner-options=execute_provision --provisioner-site=bkr --provisioner-site-user=rdoci-jenkins --product-version=7_director --product-version-build=latest --product-version-repo=puddle --distro=rhel-7.2 --product=rhos --installer=rdo_manager --installer-deploy=templates --installer-env=baremetal --installer-images=import --installer-network=neutron --installer-network-isolation=none --installer-network-variant=gre --installer-post_action=default --installer-topology=minimal --installer-tempest=all --workarounds=enabled --extra-vars @../khaleesi-settings/hardware_environments/qe/dell_pe_c6220_6-10/network_configs/none/hw_settings.yml --extra-vars @../khaleesi-settings/settings/product/rhos/private_settings/redhat_internal.yml ksgen_settings.yml

Additional info:
https://rhos-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/RHOS/view/RHOS7/job/ospd_qe-7_director-puddle-rhel-7.2-templates-baremetal-dell_pe_c6220_6-10-minimal-none-gre-all/20/consoleFull for example.

Comment 2 Filip Hubík 2016-01-08 12:47:41 UTC

Adding link for related log files: https://rhos-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/RHOS/view/RHOS7/job/ospd_qe-7_director-puddle-rhel-7.2-templates-baremetal-dell_pe_c6220_6-10-minimal-none-gre-all/20/artifact/.

Comment 4 Mike Burns 2016-04-07 21:03:37 UTC

This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.

Comment 6 Jaromir Coufal 2016-10-10 04:21:50 UTC

Could not reproduce in our CI. Please re-open if reproducable. Might be issue with specific environment which we would need to investigate on the concrete environment.