Bug 1302858 - [RHCI][QCI] No valid hosts
[RHCI][QCI] No valid hosts
Status: CLOSED EOL
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
unspecified Severity medium
: ---
: 10.0 (Newton)
Assigned To: Tzu-Mainn Chen
Shai Revivo
: Triaged, ZStream
Depends On:
Blocks: 1273561
  Show dependency treegraph
 
Reported: 2016-01-28 14:23 EST by Thom Carlin
Modified: 2017-10-26 08:56 EDT (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-10-26 08:56:16 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Thom Carlin 2016-01-28 14:23:09 EST
Description of problem:

TripleO deployment results in "No valid hosts" during deployment

Version-Release number of selected component (if applicable):

RHCI 6.0 TP2

How reproducible:

Intermittent, but frequent

Steps to Reproduce:
1. Install RHCS-I ISO
2. Install TripleO ISO
3. Log in each to run launch-fusor-installer and launch-fusor-undercloud-installer
4. In Sat6, deploy OpenStack

Actual results:

Fails with "No valid hosts found" during Overstack deployment

Expected results:

Successful deployment

Additional info:

Investigation with RHCS-I and OpenStack developers revealed that ironic nodes aren't being properly tagged with property/capabilities.  This causes the deployment to fail because there are no corresponding nodes -- this occurs for Controller and/or Compute flavors.
Comment 2 Thom Carlin 2016-01-28 17:18:06 EST
A different way of presenting it was: 
undercloud flavors were not set to understand the tags in ironic (profile:control)

The ironic nodes were not tagged.
Comment 3 Thom Carlin 2016-01-28 17:41:15 EST
heat stack-list --show-nested -f "status=FAILED"
+--------------------------------------+--------------------------------------------------+---------------+----------------------+--------------------------------------+
| id                                   | stack_name                                       | stack_status  | creation_time        | parent                               |
+--------------------------------------+--------------------------------------------------+---------------+----------------------+--------------------------------------+
| 52cd80d6-e377-446c-955d-dc8783361af5 | overcloud                                        | CREATE_FAILED | 2016-01-28T22:22:53Z | None                                 |
| 682602b5-10b2-4b70-8570-bac1db15def7 | overcloud-Controller-ggtkpr4oj52w                | CREATE_FAILED | 2016-01-28T22:23:03Z | 52cd80d6-e377-446c-955d-dc8783361af5 |
| de6dc2e8-b100-4c8d-9bff-cac459bdd960 | overcloud-Compute-ipaafcbjrip3                   | CREATE_FAILED | 2016-01-28T22:23:04Z | 52cd80d6-e377-446c-955d-dc8783361af5 |
| b2acd7e9-bab5-4d6d-9d75-368d5f71e209 | overcloud-Controller-ggtkpr4oj52w-0-75xoh7qqzvoy | CREATE_FAILED | 2016-01-28T22:23:05Z | 682602b5-10b2-4b70-8570-bac1db15def7 |
| dd4ff091-52cb-4e78-8ec6-cee216c288ca | overcloud-Compute-ipaafcbjrip3-0-fb62rjligwmu    | CREATE_FAILED | 2016-01-28T22:23:06Z | de6dc2e8-b100-4c8d-9bff-cac459bdd960 |
+--------------------------------------+--------------------------------------------------+---------------+----------------------+--------------------------------------+

 heat stack-show b2acd7e9-bab5-4d6d-9d75-368d5f71e209 [edited]
| template_description  | OpenStack controller node configured by Puppet.  |  
                 |   "Flavor": "Flavor-4-x86_64-32768-464",   
| stack_status          | CREATE_FAILED |
| stack_status_reason   | Resource CREATE failed: ResourceInError: |
|                       | resources.Controller: Went to status ERROR due to |
|                       | "Message: No valid host was found. There are not enough |
|                       | hosts available., Code: 500" |
 similarly for heat stack-show dd4ff091-52cb-4e78-8ec6-cee216c288ca
| description           | OpenStack hypervisor node configured via Puppet. |
|                       |   "Flavor": "Flavor-12-x86_64-32-930",
| stack_status          | CREATE_FAILED |
| stack_status_reason   | Resource CREATE failed: ResourceInError: |
|                       | resources.NovaCompute: Went to status ERROR due to                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |                       | "Message: No valid host was found. There are not enough |
|                       | hosts available., Code: 500" |
Comment 4 Thom Carlin 2016-01-28 17:47:37 EST
 nova flavor-show Flavor-12-x86_64-32-930
+----------------------------+-------------------------------------------------------------+
| Property                   | Value                                                       |
+----------------------------+-------------------------------------------------------------+
| OS-FLV-DISABLED:disabled   | False                                                       |
| OS-FLV-EXT-DATA:ephemeral  | 0                                                           |
| disk                       | 930                                                         |
| extra_specs                | {"capabilities:boot_option": "local", "cpu_arch": "x86_64"} |
| id                         | 1                                                           |
| name                       | Flavor-12-x86_64-32-930                                     |
| os-flavor-access:is_public | True                                                        |
| ram                        | 32                                                          |
| rxtx_factor                | 1.0                                                         |
| swap                       |                                                             |
| vcpus                      | 12                                                          |
+----------------------------+-------------------------------------------------------------+
 nova flavor-show Flavor-4-x86_64-32768-464
+----------------------------+-------------------------------------------------------------+
| Property                   | Value                                                       |
+----------------------------+-------------------------------------------------------------+
| OS-FLV-DISABLED:disabled   | False                                                       |
| OS-FLV-EXT-DATA:ephemeral  | 0                                                           |
| disk                       | 464                                                         |
| extra_specs                | {"capabilities:boot_option": "local", "cpu_arch": "x86_64"} |
| id                         | 2                                                           |
| name                       | Flavor-4-x86_64-32768-464                                   |
| os-flavor-access:is_public | True                                                        |
| ram                        | 32768                                                       |
| rxtx_factor                | 1.0                                                         |
| swap                       |                                                             |
| vcpus                      | 4                                                           |
+----------------------------+-------------------------------------------------------------+
 ironic node-list
+--------------------------------------+------+---------------+-------------+-----------------+-------------+
| UUID                                 | Name | Instance UUID | Power State | Provision State | Maintenance |
+--------------------------------------+------+---------------+-------------+-----------------+-------------+
| 88113f38-a2ef-41e1-9964-0b265cf807a4 | None | None          | power off   | available       | False       |
| faef6bff-5338-44ce-b45b-83d9fa09410d | None | None          | power off   | available       | False       |
+--------------------------------------+------+---------------+-------------+-----------------+-------------+
 ironic node-show 88113f38-a2ef-41e1-9964-0b265cf807a4
+------------------------+----------------------------------------------------------------------+
| Property               | Value                                                                |
+------------------------+----------------------------------------------------------------------+
| target_power_state     | None                                                                 |
| extra                  | {u'newly_discovered': u'true', u'block_devices': {u'serials': [u'<disk_serial>']}, u'hardware_swift_object': u'extra_hardware-88113f38 |
|                        | -a2ef-41e1-9964-0b265cf807a4'}                                       |
| last_error             | None                                                                 |
| updated_at             | 2016-01-28T22:43:25+00:00                                            |
| maintenance_reason     | None                                                                 |
| provision_state        | available                                                            |
| uuid                   | 88113f38-a2ef-41e1-9964-0b265cf807a4                                 |
| console_enabled        | False                                                                |
| target_provision_state | None                                                                 |
| maintenance            | False                                                                |
| inspection_started_at  | None                                                                 |
| inspection_finished_at | None                                                                 |
| power_state            | power off                                                            |
| driver                 | pxe_ipmitool                                                         |
| reservation            | None                                                                 |
| properties             | {u'memory_mb': u'32', u'cpu_arch': u'x86_64', u'local_gb': u'930',   |
|                        | u'cpus': u'12', u'capabilities': u'boot_option:local'}               |
| instance_uuid          | None                                                                 |
| name                   | None                                                                 |
| driver_info            | {u'deploy_kernel': u'41395a60-dd7a-46d8-8f66-55f5abf70ebb',          |
|                        | u'ipmi_address': u'<bmc_ip_address>', u'ipmi_username': u'<bmc_username>',          |
|                        | u'ipmi_password': u'hunter2', u'deploy_ramdisk': u'1003d8ce-          |
|                        | 15f6-42a9-a504-e5cd7b82c8f3'}                                        |
| created_at             | 2016-01-28T21:17:19+00:00                                            |
| driver_internal_info   | {u'clean_steps': None}                                               |
| chassis_uuid           |                                                                      |
| instance_info          | {}                                                                   |
+------------------------+----------------------------------------------------------------------+
ironic node-show faef6bff-5338-44ce-b45b-83d9fa09410d
+------------------------+-----------------------------------------------------------------------+
| Property               | Value                                                                 |
+------------------------+-----------------------------------------------------------------------+
| target_power_state     | None                                                                  |
| extra                  | {u'newly_discovered': u'true', u'block_devices': {u'serials':         |
|                        | [u'<disk_serial>']}, u'hardware_swift_object': u'extra_hardware-          |
|                        | faef6bff-5338-44ce-b45b-83d9fa09410d'}                                |
| last_error             | None                                                                  |
| updated_at             | 2016-01-28T22:45:28+00:00                                             |
| maintenance_reason     | None                                                                  |
| provision_state        | available                                                             |
| uuid                   | faef6bff-5338-44ce-b45b-83d9fa09410d                                  |
| console_enabled        | False                                                                 |
| target_provision_state | None                                                                  |
| maintenance            | False                                                                 |
| inspection_started_at  | None                                                                  |
| inspection_finished_at | None                                                                  |
| power_state            | power off                                                             |
| driver                 | pxe_ipmitool                                                          |
| reservation            | None                                                                  |
| properties             | {u'memory_mb': u'32768', u'cpu_arch': u'x86_64', u'local_gb': u'464', |
|                        | u'cpus': u'4', u'capabilities': u'boot_option:local'}                 |
| instance_uuid          | None                                                                  |
| name                   | None                                                                  |
| driver_info            | {u'deploy_kernel': u'41395a60-dd7a-46d8-8f66-55f5abf70ebb',           |
|                        | u'ipmi_address': u'<bmc_ip_address>', u'ipmi_username': u'<bmc_username>',           |
|                        | u'ipmi_password': u'hunter2', u'deploy_ramdisk': u'1003d8ce-           |
|                        | 15f6-42a9-a504-e5cd7b82c8f3'}                                         |
| created_at             | 2016-01-28T21:17:21+00:00                                             |
| driver_internal_info   | {u'clean_steps': None}                                                |
| chassis_uuid           |                                                                       |
| instance_info          | {}                                                                    |
+------------------------+-----------------------------------------------------------------------+

Note that properties/capabilities is missing the role
Comment 7 Thom Carlin 2016-02-01 08:05:54 EST
nova hypervisor-list shows No hypervisors

Potential workaround:
systemctl restart openstack-nova-api.service
 systemctl restart openstack-nova-compute.service
 systemctl restart openstack-nova-conductor.service
 systemctl restart openstack-nova-consoleauth.service
 systemctl restart openstack-nova-scheduler.service
 systemctl restart openstack-ironic-api.service
 systemctl restart openstack-ironic-conductor.service
 systemctl restart openstack-ironic-discoverd-dnsmasq.service
 systemctl restart openstack-ironic-discoverd.service

and
nova hypervisor-list shows 2 hypervisors
(thanks to @mcornea)
Comment 8 Thom Carlin 2016-02-01 08:11:47 EST
Also, do a heat stack-delete overcloud
wait for it to finish
then resume foreman task
Comment 9 Thom Carlin 2016-02-01 08:57:11 EST
AFTER the foreman (sub) task finishes successfully, resume the parent foreman task
(thanks to @jmontleo)
Comment 10 Mike Burns 2016-04-07 17:07:13 EDT
This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.
Comment 12 Jason E. Rist 2016-10-14 18:20:06 EDT
I believe this should be in Portfolio
Comment 14 Thom Carlin 2017-10-26 08:56:16 EDT
RHCI/QCI product is EOL

Note You need to log in before you can comment on or make changes to this bug.