Description of problem: TripleO deployment results in "No valid hosts" during deployment Version-Release number of selected component (if applicable): RHCI 6.0 TP2 How reproducible: Intermittent, but frequent Steps to Reproduce: 1. Install RHCS-I ISO 2. Install TripleO ISO 3. Log in each to run launch-fusor-installer and launch-fusor-undercloud-installer 4. In Sat6, deploy OpenStack Actual results: Fails with "No valid hosts found" during Overstack deployment Expected results: Successful deployment Additional info: Investigation with RHCS-I and OpenStack developers revealed that ironic nodes aren't being properly tagged with property/capabilities. This causes the deployment to fail because there are no corresponding nodes -- this occurs for Controller and/or Compute flavors.
A different way of presenting it was: undercloud flavors were not set to understand the tags in ironic (profile:control) The ironic nodes were not tagged.
heat stack-list --show-nested -f "status=FAILED" +--------------------------------------+--------------------------------------------------+---------------+----------------------+--------------------------------------+ | id | stack_name | stack_status | creation_time | parent | +--------------------------------------+--------------------------------------------------+---------------+----------------------+--------------------------------------+ | 52cd80d6-e377-446c-955d-dc8783361af5 | overcloud | CREATE_FAILED | 2016-01-28T22:22:53Z | None | | 682602b5-10b2-4b70-8570-bac1db15def7 | overcloud-Controller-ggtkpr4oj52w | CREATE_FAILED | 2016-01-28T22:23:03Z | 52cd80d6-e377-446c-955d-dc8783361af5 | | de6dc2e8-b100-4c8d-9bff-cac459bdd960 | overcloud-Compute-ipaafcbjrip3 | CREATE_FAILED | 2016-01-28T22:23:04Z | 52cd80d6-e377-446c-955d-dc8783361af5 | | b2acd7e9-bab5-4d6d-9d75-368d5f71e209 | overcloud-Controller-ggtkpr4oj52w-0-75xoh7qqzvoy | CREATE_FAILED | 2016-01-28T22:23:05Z | 682602b5-10b2-4b70-8570-bac1db15def7 | | dd4ff091-52cb-4e78-8ec6-cee216c288ca | overcloud-Compute-ipaafcbjrip3-0-fb62rjligwmu | CREATE_FAILED | 2016-01-28T22:23:06Z | de6dc2e8-b100-4c8d-9bff-cac459bdd960 | +--------------------------------------+--------------------------------------------------+---------------+----------------------+--------------------------------------+ heat stack-show b2acd7e9-bab5-4d6d-9d75-368d5f71e209 [edited] | template_description | OpenStack controller node configured by Puppet. | | "Flavor": "Flavor-4-x86_64-32768-464", | stack_status | CREATE_FAILED | | stack_status_reason | Resource CREATE failed: ResourceInError: | | | resources.Controller: Went to status ERROR due to | | | "Message: No valid host was found. There are not enough | | | hosts available., Code: 500" | similarly for heat stack-show dd4ff091-52cb-4e78-8ec6-cee216c288ca | description | OpenStack hypervisor node configured via Puppet. | | | "Flavor": "Flavor-12-x86_64-32-930", | stack_status | CREATE_FAILED | | stack_status_reason | Resource CREATE failed: ResourceInError: | | | resources.NovaCompute: Went to status ERROR due to | | "Message: No valid host was found. There are not enough | | | hosts available., Code: 500" |
nova flavor-show Flavor-12-x86_64-32-930 +----------------------------+-------------------------------------------------------------+ | Property | Value | +----------------------------+-------------------------------------------------------------+ | OS-FLV-DISABLED:disabled | False | | OS-FLV-EXT-DATA:ephemeral | 0 | | disk | 930 | | extra_specs | {"capabilities:boot_option": "local", "cpu_arch": "x86_64"} | | id | 1 | | name | Flavor-12-x86_64-32-930 | | os-flavor-access:is_public | True | | ram | 32 | | rxtx_factor | 1.0 | | swap | | | vcpus | 12 | +----------------------------+-------------------------------------------------------------+ nova flavor-show Flavor-4-x86_64-32768-464 +----------------------------+-------------------------------------------------------------+ | Property | Value | +----------------------------+-------------------------------------------------------------+ | OS-FLV-DISABLED:disabled | False | | OS-FLV-EXT-DATA:ephemeral | 0 | | disk | 464 | | extra_specs | {"capabilities:boot_option": "local", "cpu_arch": "x86_64"} | | id | 2 | | name | Flavor-4-x86_64-32768-464 | | os-flavor-access:is_public | True | | ram | 32768 | | rxtx_factor | 1.0 | | swap | | | vcpus | 4 | +----------------------------+-------------------------------------------------------------+ ironic node-list +--------------------------------------+------+---------------+-------------+-----------------+-------------+ | UUID | Name | Instance UUID | Power State | Provision State | Maintenance | +--------------------------------------+------+---------------+-------------+-----------------+-------------+ | 88113f38-a2ef-41e1-9964-0b265cf807a4 | None | None | power off | available | False | | faef6bff-5338-44ce-b45b-83d9fa09410d | None | None | power off | available | False | +--------------------------------------+------+---------------+-------------+-----------------+-------------+ ironic node-show 88113f38-a2ef-41e1-9964-0b265cf807a4 +------------------------+----------------------------------------------------------------------+ | Property | Value | +------------------------+----------------------------------------------------------------------+ | target_power_state | None | | extra | {u'newly_discovered': u'true', u'block_devices': {u'serials': [u'<disk_serial>']}, u'hardware_swift_object': u'extra_hardware-88113f38 | | | -a2ef-41e1-9964-0b265cf807a4'} | | last_error | None | | updated_at | 2016-01-28T22:43:25+00:00 | | maintenance_reason | None | | provision_state | available | | uuid | 88113f38-a2ef-41e1-9964-0b265cf807a4 | | console_enabled | False | | target_provision_state | None | | maintenance | False | | inspection_started_at | None | | inspection_finished_at | None | | power_state | power off | | driver | pxe_ipmitool | | reservation | None | | properties | {u'memory_mb': u'32', u'cpu_arch': u'x86_64', u'local_gb': u'930', | | | u'cpus': u'12', u'capabilities': u'boot_option:local'} | | instance_uuid | None | | name | None | | driver_info | {u'deploy_kernel': u'41395a60-dd7a-46d8-8f66-55f5abf70ebb', | | | u'ipmi_address': u'<bmc_ip_address>', u'ipmi_username': u'<bmc_username>', | | | u'ipmi_password': u'hunter2', u'deploy_ramdisk': u'1003d8ce- | | | 15f6-42a9-a504-e5cd7b82c8f3'} | | created_at | 2016-01-28T21:17:19+00:00 | | driver_internal_info | {u'clean_steps': None} | | chassis_uuid | | | instance_info | {} | +------------------------+----------------------------------------------------------------------+ ironic node-show faef6bff-5338-44ce-b45b-83d9fa09410d +------------------------+-----------------------------------------------------------------------+ | Property | Value | +------------------------+-----------------------------------------------------------------------+ | target_power_state | None | | extra | {u'newly_discovered': u'true', u'block_devices': {u'serials': | | | [u'<disk_serial>']}, u'hardware_swift_object': u'extra_hardware- | | | faef6bff-5338-44ce-b45b-83d9fa09410d'} | | last_error | None | | updated_at | 2016-01-28T22:45:28+00:00 | | maintenance_reason | None | | provision_state | available | | uuid | faef6bff-5338-44ce-b45b-83d9fa09410d | | console_enabled | False | | target_provision_state | None | | maintenance | False | | inspection_started_at | None | | inspection_finished_at | None | | power_state | power off | | driver | pxe_ipmitool | | reservation | None | | properties | {u'memory_mb': u'32768', u'cpu_arch': u'x86_64', u'local_gb': u'464', | | | u'cpus': u'4', u'capabilities': u'boot_option:local'} | | instance_uuid | None | | name | None | | driver_info | {u'deploy_kernel': u'41395a60-dd7a-46d8-8f66-55f5abf70ebb', | | | u'ipmi_address': u'<bmc_ip_address>', u'ipmi_username': u'<bmc_username>', | | | u'ipmi_password': u'hunter2', u'deploy_ramdisk': u'1003d8ce- | | | 15f6-42a9-a504-e5cd7b82c8f3'} | | created_at | 2016-01-28T21:17:21+00:00 | | driver_internal_info | {u'clean_steps': None} | | chassis_uuid | | | instance_info | {} | +------------------------+-----------------------------------------------------------------------+ Note that properties/capabilities is missing the role
nova hypervisor-list shows No hypervisors Potential workaround: systemctl restart openstack-nova-api.service systemctl restart openstack-nova-compute.service systemctl restart openstack-nova-conductor.service systemctl restart openstack-nova-consoleauth.service systemctl restart openstack-nova-scheduler.service systemctl restart openstack-ironic-api.service systemctl restart openstack-ironic-conductor.service systemctl restart openstack-ironic-discoverd-dnsmasq.service systemctl restart openstack-ironic-discoverd.service and nova hypervisor-list shows 2 hypervisors (thanks to @mcornea)
Also, do a heat stack-delete overcloud wait for it to finish then resume foreman task
AFTER the foreman (sub) task finishes successfully, resume the parent foreman task (thanks to @jmontleo)
This bug did not make the OSP 8.0 release. It is being deferred to OSP 10.
I believe this should be in Portfolio
RHCI/QCI product is EOL