Hide Forgot
Description of problem: CI is unable to deploy OSPd7(8 too with same error) puddle (2016-01-06.1) on baremetals with any topology or network variant on RHEL 7.2 Server. We are deploying on 5 BM nodes, first node assigned for undercloud. Overcloud deployment failed. OC controllers are not accessible through SSH, they are in ERROR state. Heat-stack list reports overcloud CREATE_FAILED, there are errors in heat resource-list & event-list. 1) Introspection result: rdomanager_oscplugin.utils.wait_for_node_discovery Discovery didn't finish for nodes e083f04c-1929-4432-9d95-408bc9520f87,f3a5ca66-5352-4a0c-9894-db70063d4af3,00bc9ed0-87d2-406e-b426-ecb90acce257,86a513c4-65b8-4be8-a6b1-04fff7bf64d1", "stdout": "Setting available nodes to manageable...\nStarting introspection of node: e083f04c-1929-4432-9d95-408bc9520f87\nStarting introspection of node: f3a5ca66-5352-4a0c-9894-db70063d4af3\nStarting introspection of node: 00bc9ed0-87d2-406e-b426-ecb90acce257\nStarting introspection of node: 86a513c4-65b8-4be8-a6b1-04fff7bf64d1\nWaiting for discovery to finish...\nSetting manageable nodes to available...\nNode e083f04c-1929-4432-9d95-408bc9520f87 has been set to available.\nNode f3a5ca66-5352-4a0c-9894-db70063d4af3 has been set to available.\nNode 00bc9ed0-87d2-406e-b426-ecb90acce257 has been set to available.\nNode 86a513c4-65b8-4be8-a6b1-04fff7bf64d1 has been set to available.\nDiscovery completed.", "warnings": []} 04:43:50 stdout: Setting available nodes to manageable... 04:43:50 Starting introspection of node: e083f04c-1929-4432-9d95-408bc9520f87 04:43:50 Starting introspection of node: f3a5ca66-5352-4a0c-9894-db70063d4af3 04:43:50 Starting introspection of node: 00bc9ed0-87d2-406e-b426-ecb90acce257 04:43:50 Starting introspection of node: 86a513c4-65b8-4be8-a6b1-04fff7bf64d1 04:43:50 Waiting for discovery to finish... 04:43:50 Setting manageable nodes to available... 04:43:50 Node e083f04c-1929-4432-9d95-408bc9520f87 has been set to available. 04:43:50 Node f3a5ca66-5352-4a0c-9894-db70063d4af3 has been set to available. 04:43:50 Node 00bc9ed0-87d2-406e-b426-ecb90acce257 has been set to available. 04:43:50 Node 86a513c4-65b8-4be8-a6b1-04fff7bf64d1 has been set to available. 04:43:50 Discovery completed. 2) Deploy command: openstack overcloud deploy --debug --log-file overcloud_deployment_14.log --templates --libvirt-type=kvm --neutron-network-type gre --neutron-tunnel-types gre --ntp-server 10.5.26.10 --control-scale 1 --compute-scale 1 --ceph-storage-scale 1 --block-storage-scale 0 --swift-storage-scale 0 --control-flavor baremetal --compute-flavor baremetal --ceph-storage-flavor baremetal --block-storage-flavor baremetal --swift-storage-flavor baremetal --neutron-public-interface=nic2 --timeout=480 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml 3) nova list; ironic node-list; openstack server list; after deploy command 08:46:46 | ID | Name | Status | Task State | Power State | Networks | 08:46:46 +--------------------------------------+-------------------------+--------+------------+-------------+------------------------+ 08:46:46 | 8376ad16-4f01-464d-841e-037d066d5c9b | overcloud-cephstorage-0 | BUILD | spawning | NOSTATE | ctlplane=10.200.200.14 | 08:46:46 | 0f227da2-e1d1-4ec9-9278-b29c88579965 | overcloud-compute-0 | BUILD | spawning | NOSTATE | ctlplane=10.200.200.12 | 08:46:46 | 4f90b498-0e7b-4d35-9204-d92aae0794f7 | overcloud-controller-0 | BUILD | spawning | NOSTATE | ctlplane=10.200.200.13 | 08:46:46 +--------------------------------------+-------------------------+--------+------------+-------------+------------------------+ 08:46:46 +--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+ 08:46:46 | UUID | Name | Instance UUID | Power State | Provision State | Maintenance | 08:46:46 +--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+ 08:46:46 | e083f04c-1929-4432-9d95-408bc9520f87 | None | 0f227da2-e1d1-4ec9-9278-b29c88579965 | power on | wait call-back | False | 08:46:46 | f3a5ca66-5352-4a0c-9894-db70063d4af3 | None | 8376ad16-4f01-464d-841e-037d066d5c9b | power on | wait call-back | False | 08:46:46 | 00bc9ed0-87d2-406e-b426-ecb90acce257 | None | 4f90b498-0e7b-4d35-9204-d92aae0794f7 | power on | wait call-back | False | 08:46:46 | 86a513c4-65b8-4be8-a6b1-04fff7bf64d1 | None | None | power off | available | False | 08:46:46 +--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+ 08:46:46 +--------------------------------------+-------------------------+--------+------------------------+ 08:46:46 | ID | Name | Status | Networks | 08:46:46 +--------------------------------------+-------------------------+--------+------------------------+ 08:46:46 | 8376ad16-4f01-464d-841e-037d066d5c9b | overcloud-cephstorage-0 | BUILD | ctlplane=10.200.200.14 | 08:46:46 | 4f90b498-0e7b-4d35-9204-d92aae0794f7 | overcloud-controller-0 | BUILD | ctlplane=10.200.200.13 | 08:46:46 | 0f227da2-e1d1-4ec9-9278-b29c88579965 | overcloud-compute-0 | BUILD | ctlplane=10.200.200.12 | 08:46:46 +--------------------------------------+-------------------------+--------+------------------------+ 4) Heat resource-list overcloud 08:46:54 | CephStorage | 2a22f3cf-5503-4a99-baee-63af747c7e72 | OS::Heat::ResourceGroup | CREATE_FAILED | 2016-01-08T04:46:27Z | 08:46:54 | Controller | 939f672a-6ae4-4575-a71b-1a4c4db5e25d | OS::Heat::ResourceGroup | CREATE_FAILED | 2016-01-08T04:46:27Z | 5) heat event-list overcloud ... |\n| CephStorage | e0b7755e-9cf7-4d33-bcc6-2f723d170a88 | BadRequest: resources.CephStorage.resources[0].resources.CephStorage: Expecting to find username or userId in passwordCredentials - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to | CREATE_FAILED | 2016-01-08T08:46:22Z |\n| Compute | 8db490cc-d77f-42fc-9320-79c67a4a0eb9 | BadRequest: resources.Compute.resources[0].resources.NovaCompute: Expecting to find username or userId in passwordCredentials - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be i | CREATE_FAILED | 2016-01-08T08:46:23Z |\n| Controller | 48ef3a0b-9dee-4403-a453-1155c4c81a4c | BadRequest: resources.Controller.resources[0].resources.Controller: Expecting to find username or userId in passwordCredentials - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be | CREATE_FAILED | 2016-01-08T08:46:24Z |\n| overcloud | 58aee2d1-9bfc-455c-8df9-06c4d0562eab | Resource CREATE failed: BadRequest: resources.CephStorage.resources[0].resources.CephStorage: Expecting to find username or userId in passwordCredentials - the server could not comply with the request since it is either malformed or otherwise incorrect. T | CREATE_FAILED | 2016-01-08T08:46:25Z How reproducible: Deployment done with khaleesi tool using params: ksgen --config-dir settings generate --provisioner=beaker --provisioner-distro=rhel --provisioner-distro-version=7.2 --provisioner-options=execute_provision --provisioner-site=bkr --provisioner-site-user=rdoci-jenkins --product-version=7_director --product-version-build=latest --product-version-repo=puddle --distro=rhel-7.2 --product=rhos --installer=rdo_manager --installer-deploy=templates --installer-env=baremetal --installer-images=import --installer-network=neutron --installer-network-isolation=none --installer-network-variant=gre --installer-post_action=default --installer-topology=minimal --installer-tempest=all --workarounds=enabled --extra-vars @../khaleesi-settings/hardware_environments/qe/dell_pe_c6220_6-10/network_configs/none/hw_settings.yml --extra-vars @../khaleesi-settings/settings/product/rhos/private_settings/redhat_internal.yml ksgen_settings.yml Additional info: https://rhos-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/RHOS/view/RHOS7/job/ospd_qe-7_director-puddle-rhel-7.2-templates-baremetal-dell_pe_c6220_6-10-minimal-none-gre-all/20/consoleFull for example.
Adding link for related log files: https://rhos-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/RHOS/view/RHOS7/job/ospd_qe-7_director-puddle-rhel-7.2-templates-baremetal-dell_pe_c6220_6-10-minimal-none-gre-all/20/artifact/.
This bug did not make the OSP 8.0 release. It is being deferred to OSP 10.
Could not reproduce in our CI. Please re-open if reproducable. Might be issue with specific environment which we would need to investigate on the concrete environment.