Bug 1288192 - Introspection in OSP7 finds the first Virtual Disk at position 0, OSP8 finds Virtual Disk at position 1 [NEEDINFO]
Summary: Introspection in OSP7 finds the first Virtual Disk at position 0, OSP8 finds ...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director
Version: 7.0 (Kilo)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 10.0 (Newton)
Assignee: Dmitry Tantsur
QA Contact: Shai Revivo
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-12-03 19:11 UTC by Ronelle Landy
Modified: 2017-05-05 10:13 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-10-11 10:32:36 UTC
Target Upstream Version:
Embargoed:
pmorey: needinfo?


Attachments (Terms of Use)

Description Ronelle Landy 2015-12-03 19:11:17 UTC
Description of problem:

Deploying the same 1 controller 1 computer overcloud on the same baremetal hardware is successful in OSP7 and fails in OSP8 with 'No available host' error. The host mismatch problem in OSP8 is due the fact that in OSP8, Introspection picks up virtual disk1, which has only 18GB, while virtual disk0 has 1800GB. The baremetal flavor used for the controller and compute nodes requires at least 40GB which virtual disk1 would not meet.

To try mitigate this disk problem, Joe helped us remove one of the virtual disks so that we landed up with 3 machines with RAID5 and two virtual disks, and three machines with RAID0 with a single virtual disk.

*********************************

With this configuration, OSP 8 inspection shows u'2792' for the machines with a single virtual disk and u'17' for machines with two virtual disks:

***********************************

[stack@host15 ~]$ ironic node-show f4c371e0-714c-4203-849b-b928ffc48e39 
+------------------------+-------------------------------------------------------------------------+
| Property               | Value                                                                   |
+------------------------+-------------------------------------------------------------------------+
| target_power_state     | None                                                                    |
| extra                  | {}                                                                      |
| last_error             | None                                                                    |
| updated_at             | 2015-12-02T17:49:07+00:00                                               |
| maintenance_reason     | None                                                                    |
| provision_state        | available                                                               |
| clean_step             | {}                                                                      |
| uuid                   | f4c371e0-714c-4203-849b-b928ffc48e39                                    |
| console_enabled        | False                                                                   |
| target_provision_state | None                                                                    |
| provision_updated_at   | 2015-12-02T17:34:23+00:00                                               |
| maintenance            | False                                                                   |
| inspection_started_at  | None                                                                    |
| inspection_finished_at | None                                                                    |
| power_state            | power off                                                               |
| driver                 | pxe_ipmitool                                                            |
| reservation            | None                                                                    |
| properties             | {u'memory_mb': u'131072', u'cpu_arch': u'x86_64', u'local_gb': u'17',   |
|                        | u'cpus': u'24', u'capabilities': u'boot_option:local'}                  |
| instance_uuid          | None                                                                    |
| name                   | None                                                                    |
| driver_info            | {u'ipmi_password': u'******', u'ipmi_address': u'10.9.10.129',          |
|                        | u'ipmi_username': u'root', u'deploy_kernel': u'45ec3ea9-4cf3-4ade-8dbb- |
|                        | 19d0b16a2b99', u'deploy_ramdisk': u'6ba24b84-afef-477e-                 |
|                        | 8cc6-ad192beadf62'}                                                     |
| created_at             | 2015-12-02T17:28:02+00:00                                               |
| driver_internal_info   | {u'clean_steps': None}                                                  |
| chassis_uuid           |                                                                         |
| instance_info          | {}                                                                      |
+------------------------+-------------------------------------------------------------------------+

[stack@host15 ~]$ ironic node-show 01d83cee-7d81-4579-8dc0-5b04fd7eb478
+------------------------+-------------------------------------------------------------------------+
| Property               | Value                                                                   |
+------------------------+-------------------------------------------------------------------------+
| target_power_state     | None                                                                    |
| extra                  | {}                                                                      |
| last_error             | None                                                                    |
| updated_at             | 2015-12-02T23:17:52+00:00                                               |
| maintenance_reason     | None                                                                    |
| provision_state        | available                                                               |
| clean_step             | {}                                                                      |
| uuid                   | 01d83cee-7d81-4579-8dc0-5b04fd7eb478                                    |
| console_enabled        | False                                                                   |
| target_provision_state | None                                                                    |
| provision_updated_at   | 2015-12-02T17:34:23+00:00                                               |
| maintenance            | True                                                                    |
| inspection_started_at  | None                                                                    |
| inspection_finished_at | None                                                                    |
| power_state            | power off                                                               |
| driver                 | pxe_ipmitool                                                            |
| reservation            | None                                                                    |
| properties             | {u'memory_mb': u'131072', u'cpu_arch': u'x86_64', u'local_gb': u'2792', |
|                        | u'cpus': u'24', u'capabilities': u'boot_option:local'}                  |
| instance_uuid          | None                                                                    |
| name                   | None                                                                    |
| driver_info            | {u'ipmi_password': u'******', u'ipmi_address': u'10.9.10.130',          |
|                        | u'ipmi_username': u'root', u'deploy_kernel': u'45ec3ea9-4cf3-4ade-8dbb- |
|                        | 19d0b16a2b99', u'deploy_ramdisk': u'6ba24b84-afef-477e-                 |
|                        | 8cc6-ad192beadf62'}                                                     |
| created_at             | 2015-12-02T17:28:03+00:00                                               |
| driver_internal_info   | {u'clean_steps': None}                                                  |
| chassis_uuid           |                                                                         |
| instance_info          | {}                                                                      |
+------------------------+-------------------------------------------------------------------------+

********************************

While OSP7 shows u'1842' for the machines with a single virtual disk and u'2792' for machines with two virtual disks:

********************************

[stack@host15 ~]$ ironic node-show e885e2a4-be82-4fd7-b5d0-e0e182823192
+------------------------+-------------------------------------------------------------------------+
| Property               | Value                                                                   |
+------------------------+-------------------------------------------------------------------------+
| target_power_state     | None                                                                    |
| extra                  | {u'newly_discovered': u'true', u'block_devices': {u'serials':           |
|                        | [u'644a8420100f95001c918ee70f3b54de',                                   |
|                        | u'644a8420100f95001c918eeb0f820162']}, u'hardware_swift_object': u      |
|                        | 'extra_hardware-e885e2a4-be82-4fd7-b5d0-e0e182823192'}                  |
| last_error             | None                                                                    |
| updated_at             | 2015-12-03T18:36:44+00:00                                               |
| maintenance_reason     | None                                                                    |
| provision_state        | available                                                               |
| uuid                   | e885e2a4-be82-4fd7-b5d0-e0e182823192                                    |
| console_enabled        | False                                                                   |
| target_provision_state | None                                                                    |
| maintenance            | False                                                                   |
| inspection_started_at  | None                                                                    |
| inspection_finished_at | None                                                                    |
| power_state            | power off                                                               |
| driver                 | pxe_ipmitool                                                            |
| reservation            | None                                                                    |
| properties             | {u'memory_mb': u'131072', u'cpu_arch': u'x86_64', u'local_gb': u'1842', |
|                        | u'cpus': u'24', u'capabilities': u'boot_option:local'}                  |
| instance_uuid          | None                                                                    |
| name                   | None                                                                    |
| driver_info            | {u'ipmi_password': u'******', u'ipmi_address': u'10.9.10.134',          |
|                        | u'ipmi_username': u'root', u'deploy_kernel': u'6b83ae8c-                |
|                        | b8f9-4fdb-a145-87f605f926f4', u'deploy_ramdisk': u'27d2223f-b4c6-41dc-  |
|                        | b2b0-14c9fce942a7'}                                                     |
| created_at             | 2015-12-03T18:26:11+00:00                                               |
| driver_internal_info   | {u'clean_steps': None}                                                  |
| chassis_uuid           |                                                                         |
| instance_info          | {}                                                                      |
+------------------------+-------------------------------------------------------------------------+
[stack@host15 ~]$ ironic node-show  b7180b6b-5173-4488-ae4a-8a7262bae43b
+------------------------+-------------------------------------------------------------------------+
| Property               | Value                                                                   |
+------------------------+-------------------------------------------------------------------------+
| target_power_state     | None                                                                    |
| extra                  | {u'newly_discovered': u'true', u'block_devices': {u'serials':           |
|                        | [u'644a8420100f6d001df2341b06ddefec']}, u'hardware_swift_object': u     |
|                        | 'extra_hardware-b7180b6b-5173-4488-ae4a-8a7262bae43b'}                  |
| last_error             | None                                                                    |
| updated_at             | 2015-12-03T18:36:43+00:00                                               |
| maintenance_reason     | None                                                                    |
| provision_state        | available                                                               |
| uuid                   | b7180b6b-5173-4488-ae4a-8a7262bae43b                                    |
| console_enabled        | False                                                                   |
| target_provision_state | None                                                                    |
| maintenance            | False                                                                   |
| inspection_started_at  | None                                                                    |
| inspection_finished_at | None                                                                    |
| power_state            | power off                                                               |
| driver                 | pxe_ipmitool                                                            |
| reservation            | None                                                                    |
| properties             | {u'memory_mb': u'131072', u'cpu_arch': u'x86_64', u'local_gb': u'2792', |
|                        | u'cpus': u'24', u'capabilities': u'boot_option:local'}                  |
| instance_uuid          | None                                                                    |
| name                   | None                                                                    |
| driver_info            | {u'ipmi_password': u'******', u'ipmi_address': u'10.9.10.129',          |
|                        | u'ipmi_username': u'root', u'deploy_kernel': u'6b83ae8c-                |
|                        | b8f9-4fdb-a145-87f605f926f4', u'deploy_ramdisk': u'27d2223f-b4c6-41dc-  |
|                        | b2b0-14c9fce942a7'}                                                     |
| created_at             | 2015-12-03T18:25:22+00:00                                               |
| driver_internal_info   | {u'clean_steps': None}                                                  |
| chassis_uuid           |                                                                         |
| instance_info          | {}                                                                      |
+------------------------+-------------------------------------------------------------------------+


Version-Release number of selected component (if applicable):

[stack@host15 ~]$ rpm -qa | grep ironic
python-ironicclient-0.5.1-12.el7ost.noarch
openstack-ironic-discoverd-1.1.0-8.el7ost.noarch
python-ironic-discoverd-1.1.0-8.el7ost.noarch
openstack-ironic-common-2015.1.2-2.el7ost.noarch
openstack-ironic-api-2015.1.2-2.el7ost.noarch
openstack-ironic-conductor-2015.1.2-2.el7ost.noarch

How reproducible:


Steps to Reproduce:
1. Install OSP8 on a system with multiple virtual disks
2. Run inspection and view which disks get picked up
3. Run steps 1 and 2 on the same hardware but install OSP7 rather than OSP8
4. Compare the results

Actual results:

The results reported from inspection in OSP7 and OSP8 differ

Expected results:

We should return the same result or document that there will be a difference if there is an intentional change in OSP8

Additional info:

Comment 2 Ronelle Landy 2015-12-03 19:16:31 UTC
Possibly similar to https://bugzilla.redhat.com/show_bug.cgi?id=1256421

Comment 3 Lucas Alvares Gomes 2015-12-03 19:37:12 UTC
Thanks for reporting this bug Ronnelle,

In OSP8 we have changed the ramdisk to IPA[0] so it may be something to do with that. I'm assigning Dmitry to this bug since he have done a lot of work in this part.

[0] https://github.com/openstack/ironic-python-agent

Comment 4 Dmitry Tantsur 2015-12-04 09:50:31 UTC
Hi! In OSPd8 we got root device hints support for inspector, please use them: http://docs.openstack.org/developer/ironic/liberty/deploy/install-guide.html#specifying-the-disk-for-deployment

Let me know if it works.

P.S.
Actually we'll see the problem much more often now, as picking the smallest disk with size > 4GB is how IPA works.

Comment 5 Ronelle Landy 2016-01-13 14:11:13 UTC
Using size hints works for osp -8

 ironic node-update 389db4d4-327b-44f2-ab2c-786bbe090846 add properties/root_device='{"size": 1843}' resulting in ironic picking the larger of two available disks

Comment 6 Mike Burns 2016-04-07 21:00:12 UTC
This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.

Comment 8 Dmitry Tantsur 2016-10-11 10:32:36 UTC
Hi!

It seems like the problem is solved by using root device hints. Please feel free to reopen if you need more help with it.


Note You need to log in before you can comment on or make changes to this bug.