Bug 1302858

Summary: [RHCI][QCI] No valid hosts
Product: Red Hat OpenStack Reporter: Thom Carlin <tcarlin>
Component: openstack-tripleoAssignee: Tzu-Mainn Chen <tzumainn>
Status: CLOSED EOL QA Contact: Shai Revivo <srevivo>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 7.0 (Kilo)CC: apevec, jcoufal, jmatthew, jmontleo, jrist, lhh, maufart, mburns, rhel-osp-director-maint, srevivo, tcarlin
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-10-26 12:56:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1273561    

Description Thom Carlin 2016-01-28 19:23:09 UTC
Description of problem:

TripleO deployment results in "No valid hosts" during deployment

Version-Release number of selected component (if applicable):

RHCI 6.0 TP2

How reproducible:

Intermittent, but frequent

Steps to Reproduce:
1. Install RHCS-I ISO
2. Install TripleO ISO
3. Log in each to run launch-fusor-installer and launch-fusor-undercloud-installer
4. In Sat6, deploy OpenStack

Actual results:

Fails with "No valid hosts found" during Overstack deployment

Expected results:

Successful deployment

Additional info:

Investigation with RHCS-I and OpenStack developers revealed that ironic nodes aren't being properly tagged with property/capabilities.  This causes the deployment to fail because there are no corresponding nodes -- this occurs for Controller and/or Compute flavors.

Comment 2 Thom Carlin 2016-01-28 22:18:06 UTC
A different way of presenting it was: 
undercloud flavors were not set to understand the tags in ironic (profile:control)

The ironic nodes were not tagged.

Comment 3 Thom Carlin 2016-01-28 22:41:15 UTC
heat stack-list --show-nested -f "status=FAILED"
+--------------------------------------+--------------------------------------------------+---------------+----------------------+--------------------------------------+
| id                                   | stack_name                                       | stack_status  | creation_time        | parent                               |
+--------------------------------------+--------------------------------------------------+---------------+----------------------+--------------------------------------+
| 52cd80d6-e377-446c-955d-dc8783361af5 | overcloud                                        | CREATE_FAILED | 2016-01-28T22:22:53Z | None                                 |
| 682602b5-10b2-4b70-8570-bac1db15def7 | overcloud-Controller-ggtkpr4oj52w                | CREATE_FAILED | 2016-01-28T22:23:03Z | 52cd80d6-e377-446c-955d-dc8783361af5 |
| de6dc2e8-b100-4c8d-9bff-cac459bdd960 | overcloud-Compute-ipaafcbjrip3                   | CREATE_FAILED | 2016-01-28T22:23:04Z | 52cd80d6-e377-446c-955d-dc8783361af5 |
| b2acd7e9-bab5-4d6d-9d75-368d5f71e209 | overcloud-Controller-ggtkpr4oj52w-0-75xoh7qqzvoy | CREATE_FAILED | 2016-01-28T22:23:05Z | 682602b5-10b2-4b70-8570-bac1db15def7 |
| dd4ff091-52cb-4e78-8ec6-cee216c288ca | overcloud-Compute-ipaafcbjrip3-0-fb62rjligwmu    | CREATE_FAILED | 2016-01-28T22:23:06Z | de6dc2e8-b100-4c8d-9bff-cac459bdd960 |
+--------------------------------------+--------------------------------------------------+---------------+----------------------+--------------------------------------+

 heat stack-show b2acd7e9-bab5-4d6d-9d75-368d5f71e209 [edited]
| template_description  | OpenStack controller node configured by Puppet.  |  
                 |   "Flavor": "Flavor-4-x86_64-32768-464",   
| stack_status          | CREATE_FAILED |
| stack_status_reason   | Resource CREATE failed: ResourceInError: |
|                       | resources.Controller: Went to status ERROR due to |
|                       | "Message: No valid host was found. There are not enough |
|                       | hosts available., Code: 500" |
 similarly for heat stack-show dd4ff091-52cb-4e78-8ec6-cee216c288ca
| description           | OpenStack hypervisor node configured via Puppet. |
|                       |   "Flavor": "Flavor-12-x86_64-32-930",
| stack_status          | CREATE_FAILED |
| stack_status_reason   | Resource CREATE failed: ResourceInError: |
|                       | resources.NovaCompute: Went to status ERROR due to                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |                       | "Message: No valid host was found. There are not enough |
|                       | hosts available., Code: 500" |

Comment 4 Thom Carlin 2016-01-28 22:47:37 UTC
 nova flavor-show Flavor-12-x86_64-32-930
+----------------------------+-------------------------------------------------------------+
| Property                   | Value                                                       |
+----------------------------+-------------------------------------------------------------+
| OS-FLV-DISABLED:disabled   | False                                                       |
| OS-FLV-EXT-DATA:ephemeral  | 0                                                           |
| disk                       | 930                                                         |
| extra_specs                | {"capabilities:boot_option": "local", "cpu_arch": "x86_64"} |
| id                         | 1                                                           |
| name                       | Flavor-12-x86_64-32-930                                     |
| os-flavor-access:is_public | True                                                        |
| ram                        | 32                                                          |
| rxtx_factor                | 1.0                                                         |
| swap                       |                                                             |
| vcpus                      | 12                                                          |
+----------------------------+-------------------------------------------------------------+
 nova flavor-show Flavor-4-x86_64-32768-464
+----------------------------+-------------------------------------------------------------+
| Property                   | Value                                                       |
+----------------------------+-------------------------------------------------------------+
| OS-FLV-DISABLED:disabled   | False                                                       |
| OS-FLV-EXT-DATA:ephemeral  | 0                                                           |
| disk                       | 464                                                         |
| extra_specs                | {"capabilities:boot_option": "local", "cpu_arch": "x86_64"} |
| id                         | 2                                                           |
| name                       | Flavor-4-x86_64-32768-464                                   |
| os-flavor-access:is_public | True                                                        |
| ram                        | 32768                                                       |
| rxtx_factor                | 1.0                                                         |
| swap                       |                                                             |
| vcpus                      | 4                                                           |
+----------------------------+-------------------------------------------------------------+
 ironic node-list
+--------------------------------------+------+---------------+-------------+-----------------+-------------+
| UUID                                 | Name | Instance UUID | Power State | Provision State | Maintenance |
+--------------------------------------+------+---------------+-------------+-----------------+-------------+
| 88113f38-a2ef-41e1-9964-0b265cf807a4 | None | None          | power off   | available       | False       |
| faef6bff-5338-44ce-b45b-83d9fa09410d | None | None          | power off   | available       | False       |
+--------------------------------------+------+---------------+-------------+-----------------+-------------+
 ironic node-show 88113f38-a2ef-41e1-9964-0b265cf807a4
+------------------------+----------------------------------------------------------------------+
| Property               | Value                                                                |
+------------------------+----------------------------------------------------------------------+
| target_power_state     | None                                                                 |
| extra                  | {u'newly_discovered': u'true', u'block_devices': {u'serials': [u'<disk_serial>']}, u'hardware_swift_object': u'extra_hardware-88113f38 |
|                        | -a2ef-41e1-9964-0b265cf807a4'}                                       |
| last_error             | None                                                                 |
| updated_at             | 2016-01-28T22:43:25+00:00                                            |
| maintenance_reason     | None                                                                 |
| provision_state        | available                                                            |
| uuid                   | 88113f38-a2ef-41e1-9964-0b265cf807a4                                 |
| console_enabled        | False                                                                |
| target_provision_state | None                                                                 |
| maintenance            | False                                                                |
| inspection_started_at  | None                                                                 |
| inspection_finished_at | None                                                                 |
| power_state            | power off                                                            |
| driver                 | pxe_ipmitool                                                         |
| reservation            | None                                                                 |
| properties             | {u'memory_mb': u'32', u'cpu_arch': u'x86_64', u'local_gb': u'930',   |
|                        | u'cpus': u'12', u'capabilities': u'boot_option:local'}               |
| instance_uuid          | None                                                                 |
| name                   | None                                                                 |
| driver_info            | {u'deploy_kernel': u'41395a60-dd7a-46d8-8f66-55f5abf70ebb',          |
|                        | u'ipmi_address': u'<bmc_ip_address>', u'ipmi_username': u'<bmc_username>',          |
|                        | u'ipmi_password': u'hunter2', u'deploy_ramdisk': u'1003d8ce-          |
|                        | 15f6-42a9-a504-e5cd7b82c8f3'}                                        |
| created_at             | 2016-01-28T21:17:19+00:00                                            |
| driver_internal_info   | {u'clean_steps': None}                                               |
| chassis_uuid           |                                                                      |
| instance_info          | {}                                                                   |
+------------------------+----------------------------------------------------------------------+
ironic node-show faef6bff-5338-44ce-b45b-83d9fa09410d
+------------------------+-----------------------------------------------------------------------+
| Property               | Value                                                                 |
+------------------------+-----------------------------------------------------------------------+
| target_power_state     | None                                                                  |
| extra                  | {u'newly_discovered': u'true', u'block_devices': {u'serials':         |
|                        | [u'<disk_serial>']}, u'hardware_swift_object': u'extra_hardware-          |
|                        | faef6bff-5338-44ce-b45b-83d9fa09410d'}                                |
| last_error             | None                                                                  |
| updated_at             | 2016-01-28T22:45:28+00:00                                             |
| maintenance_reason     | None                                                                  |
| provision_state        | available                                                             |
| uuid                   | faef6bff-5338-44ce-b45b-83d9fa09410d                                  |
| console_enabled        | False                                                                 |
| target_provision_state | None                                                                  |
| maintenance            | False                                                                 |
| inspection_started_at  | None                                                                  |
| inspection_finished_at | None                                                                  |
| power_state            | power off                                                             |
| driver                 | pxe_ipmitool                                                          |
| reservation            | None                                                                  |
| properties             | {u'memory_mb': u'32768', u'cpu_arch': u'x86_64', u'local_gb': u'464', |
|                        | u'cpus': u'4', u'capabilities': u'boot_option:local'}                 |
| instance_uuid          | None                                                                  |
| name                   | None                                                                  |
| driver_info            | {u'deploy_kernel': u'41395a60-dd7a-46d8-8f66-55f5abf70ebb',           |
|                        | u'ipmi_address': u'<bmc_ip_address>', u'ipmi_username': u'<bmc_username>',           |
|                        | u'ipmi_password': u'hunter2', u'deploy_ramdisk': u'1003d8ce-           |
|                        | 15f6-42a9-a504-e5cd7b82c8f3'}                                         |
| created_at             | 2016-01-28T21:17:21+00:00                                             |
| driver_internal_info   | {u'clean_steps': None}                                                |
| chassis_uuid           |                                                                       |
| instance_info          | {}                                                                    |
+------------------------+-----------------------------------------------------------------------+

Note that properties/capabilities is missing the role

Comment 7 Thom Carlin 2016-02-01 13:05:54 UTC
nova hypervisor-list shows No hypervisors

Potential workaround:
systemctl restart openstack-nova-api.service
 systemctl restart openstack-nova-compute.service
 systemctl restart openstack-nova-conductor.service
 systemctl restart openstack-nova-consoleauth.service
 systemctl restart openstack-nova-scheduler.service
 systemctl restart openstack-ironic-api.service
 systemctl restart openstack-ironic-conductor.service
 systemctl restart openstack-ironic-discoverd-dnsmasq.service
 systemctl restart openstack-ironic-discoverd.service

and
nova hypervisor-list shows 2 hypervisors
(thanks to @mcornea)

Comment 8 Thom Carlin 2016-02-01 13:11:47 UTC
Also, do a heat stack-delete overcloud
wait for it to finish
then resume foreman task

Comment 9 Thom Carlin 2016-02-01 13:57:11 UTC
AFTER the foreman (sub) task finishes successfully, resume the parent foreman task
(thanks to @jmontleo)

Comment 10 Mike Burns 2016-04-07 21:07:13 UTC
This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.

Comment 12 Jason E. Rist 2016-10-14 22:20:06 UTC
I believe this should be in Portfolio

Comment 14 Thom Carlin 2017-10-26 12:56:16 UTC
RHCI/QCI product is EOL