Bug 1518388 - pxe_ucs driver replaces newly imported nodes instead of adding additional ironic nodes
Summary: pxe_ucs driver replaces newly imported nodes instead of adding additional iro...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-common
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z9
: 10.0 (Newton)
Assignee: Ilya Etingof
QA Contact: mlammon
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-28 19:40 UTC by Jeremy
Modified: 2022-08-09 11:47 UTC (History)
7 users (show)

Fixed In Version: openstack-tripleo-common-5.4.7-6.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-09-17 16:54:10 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1739024 0 None None None 2017-12-19 13:36:01 UTC
OpenStack gerrit 528726 0 None None None 2017-12-18 14:27:07 UTC
Red Hat Bugzilla 1596763 0 high CLOSED [RHOSP13] Not able to register baremetal nodes when using pxe_ucs driver 2022-08-04 14:40:35 UTC
Red Hat Issue Tracker OSP-8668 0 None None None 2022-08-09 11:47:19 UTC
Red Hat Product Errata RHBA-2018:2670 0 None None None 2018-09-17 16:55:29 UTC

Internal Links: 1596763

Description Jeremy 2017-11-28 19:40:25 UTC
Description of problem:  

using pxe_ucs driver we import one ironic node and it works. However when we try to add another node it changes the name of the current ironic node and uses the same uuid. See below

***** Note, the original ironic node did not have issues with power state. It's only when we import the new node that the power state goes to none, and it uses the same uuid instead of creating a new ironic node.

Version-Release number of selected component (if applicable):
openstack-ironic-api-6.2.2-3.el7ost.noarch  

How reproducible:
100%

Steps to Reproduce:
1. import cisco_ucs node
2. import anther ucs node
3. notice the new node takes the old ucs node's ironic uuid

Actual results:
new imported ironic nodes takes the same uuid of other cisco ucs ironic node and changes name.

Expected results:
new ironic node added

Additional info:


 
[stack@lshdir101 console]$ cat UCS-test.json
{
    "nodes":[
        {
            "mac":[
                "00:25:B5:21:40:6F"
            ],
            "cpu":"4",
            "memory":"6144",
            "disk":"40",
            "arch":"x86_64",
            "name":"ucstest1",
            "pm_type":"pxe_ucs",
            "pm_user":"rhev",
            "pm_password":"6yhnMJU&",
            "pm_addr":"10.16.34.10",
            "pm_service_profile": "org-root/org-CPC/ls-CPC-test1"
        }
    ]
}
[stack@lshdir101 console]$
 
 
ole]$ ironic node-show 33b76a87-3406-4ce5-ad1c-c4630cc4eadf
+------------------------+--------------------------------------------------------------------------+
| Property               | Value                                                                    |
+------------------------+--------------------------------------------------------------------------+
| chassis_uuid           |                                                                          |
| clean_step             | {}                                                                       |
| console_enabled        | False                                                                    |
| created_at             | 2017-11-06T14:35:03+00:00                                                |
| driver                 | pxe_ucs                                                                  |
| driver_info            | {u'ucs_address': u'10.16.34.10', u'ucs_username': u'rhev',               |
|                        | u'deploy_kernel': u'4c82a068-a3e3-4d47-9917-5e991daf206d',               |
|                        | u'ucs_service_profile': u'org-root/org-CPC/ls-CPC-test1',                |
|                        | u'deploy_ramdisk': u'6c7d5edd-24b5-4b8d-90fd-d65d477756b4',              |
|                        | u'ucs_password': u'******'}                                              |
| driver_internal_info   | {u'agent_url': u'http://10.29.27.220:9999', u'root_uuid_or_disk_id':     |
|                        | u'a69bf0c7-8d41-42c5-b1f0-e64719aa7ffb', u'is_whole_disk_image': False,  |
|                        | u'agent_last_heartbeat': 1509980709}                                     |
| extra                  | {u'hardware_swift_object': u'extra_hardware-33b76a87-3406-4ce5-ad1c-     |
|                        | c4630cc4eadf'}                                                           |
| inspection_finished_at | None                                                                     |
| inspection_started_at  | None                                                                     |
| instance_info          | {u'root_gb': u'10', u'display_name': u'ucstest00', u'image_source':      |
|                        | u'8b0be170-e771-465b-a545-683acc529747', u'capabilities': u'{"profile":  |
|                        | "compute", "boot_option": "local"}', u'memory_mb': u'4096', u'vcpus':    |
|                        | u'1', u'local_gb': u'31', u'configdrive': u'******', u'swap_mb':         |
|                        | u'12288'}                                                                |
| instance_uuid          | 6deb56ba-3219-4614-80a6-3a760071ae30                                     |
| last_error             | During sync_power_state, max retries exceeded for node                   |
|                        | 33b76a87-3406-4ce5-ad1c-c4630cc4eadf, node state None does not match     |
|                        | expected state 'power on'. Updating DB state to 'None' Switching node to |
|                        | maintenance mode. Error: format requires a mapping                       |
| maintenance            | True                                                                     |
| maintenance_reason     | During sync_power_state, max retries exceeded for node                   |
|                        | 33b76a87-3406-4ce5-ad1c-c4630cc4eadf, node state None does not match     |
|                        | expected state 'power on'. Updating DB state to 'None' Switching node to |
|                        | maintenance mode. Error: format requires a mapping                       |
| name                   | ucstest1                                                                 |
| network_interface      |                                                                          |
| power_state            | None                                                                     |
| properties             | {u'memory_mb': u'6144', u'cpu_arch': u'x86_64', u'local_gb': u'40',      |
|                        | u'cpus': u'4', u'capabilities': u'boot_option:local'}                    |
| provision_state        | active                                                                   |
| provision_updated_at   | 2017-11-06T15:06:41+00:00                                                |
| raid_config            |                                                                          |
| reservation            | None                                                                     |
| resource_class         |                                                                          |
| target_power_state     | None                                                                     |
| target_provision_state | None                                                                     |
| target_raid_config     |                                                                          |
| updated_at             | 2017-11-28T18:57:34+00:00                                                |
| uuid                   | 33b76a87-3406-4ce5-ad1c-c4630cc4eadf                                     |
+------------------------+--------------------------------------------------------------------------+
 
 
stack@lshdir101 console]$ cat lshucs01b01.json
{
    "nodes":[
        {
            "mac":[
                "00:25:B5:21:40:4F"
            ],
            "cpu":"4",
            "memory":"6144",
            "disk":"40",
            "arch":"x86_64",
            "name":"lshucs01b01",
            "pm_type":"pxe_ucs",
            "pm_user":"rhev",
            "pm_password":"6yhnMJU/&",
            "pm_addr":"10.16.34.10",
            "pm_service_profile": "org-root/org-CPC/ls-lshucs01b01"
        }
    ]
}
[stack@lshdir101 console]$
 
 
 
 openstack baremetal import --json lshucs01b01.json
Started Mistral Workflow. Execution ID: c230a533-ae5e-4c67-8e02-9490ecb4fb4c
Successfully registered node UUID 33b76a87-3406-4ce5-ad1c-c4630cc4eadf
Started Mistral Workflow. Execution ID: 63ffe13f-18f1-4569-a31e-47d78399b28b
Successfully set all nodes to available.
[stack@lshdir101 console]$ ironic node-list
+--------------------------------------+-------------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name        | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+-------------+--------------------------------------+-------------+--------------------+-------------+
| 5ed361f2-8da2-415b-8ca1-50cdd9a93295 | lshkvm517i  | d4cfb4f0-f745-4f39-bff4-69034aceb062 | power on    | active             | False       |
| 997a4709-8199-4fce-a742-4213e5c062a9 | lshkvm518i  | da583d92-02fb-416c-9647-a85f41b0992e | power on    | active             | False       |
| b0493940-0369-49b9-9790-794c35966bba | lshkvm519i  | 179728c1-e2e2-4b93-8b3c-03f1b0d1927a | power on    | active             | False       |
| f15d8303-ee8f-41f5-98f6-204932921759 | lshkvm520i  | 3846bdb1-c762-440e-b2fb-f1a0072a7e5f | power on    | active             | False       |
| 61cea614-a759-45a4-9075-61f2cffae41a | lshkvm521i  | 76b3502d-bd9f-4a6d-a06f-0b020415d18c | power on    | active             | False       |
| 33b76a87-3406-4ce5-ad1c-c4630cc4eadf | lshucs01b01 | 6deb56ba-3219-4614-80a6-3a760071ae30 | power on    | active             | False       |
+--------------------------------------+-------------+--------------------------------------+-------------+--------------------+-------------+
[stack@lshdir101 console]$
 
stack@lshdir101 console]$ ironic node-show 33b76a87-3406-4ce5-ad1c-c4630cc4eadf
+------------------------+-------------------------------------------------------------------------+
| Property               | Value                                                                   |
+------------------------+-------------------------------------------------------------------------+
| chassis_uuid           |                                                                         |
| clean_step             | {}                                                                      |
| console_enabled        | False                                                                   |
| created_at             | 2017-11-06T14:35:03+00:00                                               |
| driver                 | pxe_ucs                                                                 |
| driver_info            | {u'ucs_address': u'10.16.34.10', u'ucs_username': u'rhev',              |
|                        | u'deploy_kernel': u'4c82a068-a3e3-4d47-9917-5e991daf206d',              |
|                        | u'ucs_service_profile': u'org-root/org-CPC/ls-lshucs01b01',             |
|                        | u'deploy_ramdisk': u'6c7d5edd-24b5-4b8d-90fd-d65d477756b4',             |
|                        | u'ucs_password': u'******'}                                             |
| driver_internal_info   | {u'agent_url': u'http://10.29.27.220:9999', u'root_uuid_or_disk_id':    |
|                        | u'a69bf0c7-8d41-42c5-b1f0-e64719aa7ffb', u'is_whole_disk_image': False, |
|                        | u'agent_last_heartbeat': 1509980709}                                    |
| extra                  | {u'hardware_swift_object': u'extra_hardware-33b76a87-3406-4ce5-ad1c-    |
|                        | c4630cc4eadf'}                                                          |
| inspection_finished_at | None                                                                    |
| inspection_started_at  | None                                                                    |
| instance_info          | {u'root_gb': u'10', u'display_name': u'ucstest00', u'image_source':     |
|                        | u'8b0be170-e771-465b-a545-683acc529747', u'capabilities': u'{"profile": |
|                        | "compute", "boot_option": "local"}', u'memory_mb': u'4096', u'vcpus':   |
|                        | u'1', u'local_gb': u'31', u'configdrive': u'******', u'swap_mb':        |
|                        | u'12288'}                                                               |
| instance_uuid          | 6deb56ba-3219-4614-80a6-3a760071ae30                                    |
| last_error             | None                                                                    |
| maintenance            | False                                                                   |
| maintenance_reason     | None                                                                    |
| name                   | lshucs01b01                                                             |
| network_interface      |                                                                         |
| power_state            | power on                                                                |
| properties             | {u'memory_mb': u'6144', u'cpu_arch': u'x86_64', u'local_gb': u'40',     |
|                        | u'cpus': u'4', u'capabilities': u'boot_option:local'}                   |
| provision_state        | active                                                                  |
| provision_updated_at   | 2017-11-06T15:06:41+00:00                                               |
| raid_config            |                                                                         |
| reservation            | None                                                                    |
| resource_class         |                                                                         |
| target_power_state     | None                                                                    |
| target_provision_state | None                                                                    |
| target_raid_config     |                                                                         |
| updated_at             | 2017-11-28T19:08:37+00:00                                               |
| uuid                   | 33b76a87-3406-4ce5-ad1c-c4630cc4eadf                                    |
+------------------------+-------------------------------------------------------------------------+
[stack@lshdir101 console]$
 
 
 
***** Note, the original ironic node did not have issues with power state. It's only when we import the new node that the power state goes to none, and it uses the same uuid instead of creating a new ironic node.
 ###seen in ironic-conductor.log
2017-11-28 14:12:06.327 1399 ERROR ironic.conductor.manager [req-aa721fca-7c78-4cfc-b1e6-1a6ee0c1c921 - - - - -] During sync_power_state, max retries exceeded for node 33b76a87-3406-4ce5-ad1c-c4630cc4eadf, node state None does not match expected state 'power on'. Updating DB state to 'None' Switching node to maintenance mode. Error: format requires a mapping

Comment 2 Dmitry Tantsur 2017-12-11 15:34:17 UTC
Hi!

Do I get it right that the nodes only differ in the pm_service_profile field (which should be ucs_service_profile, by the way, using pm_ prefix for non-standard fields is deprecated)?

If yes, then I guess the duplicate detection logic of TripleO is not working correctly for these nodes. I can get a fix for it, but I'll need your help testing it, as I don't have access to UCS hardware.

Comment 3 Dmitry Tantsur 2017-12-11 15:38:50 UTC
> should be ucs_service_profile, by the way, using pm_ prefix for non-standard fields is deprecated

correction: fixing this issue may require me to un-deprecate it

Comment 5 Ilya Etingof 2017-12-18 13:54:15 UTC
First take needs review: https://review.openstack.org/528726

Comment 8 Bob Fournier 2018-08-06 17:26:26 UTC
Fix is upstream in master and stable/queens. Is a backport to OSP-10 still needed? if so we will backport it.

Comment 15 Alex McLeod 2018-09-03 08:00:38 UTC
Hi there,

If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field.

The documentation team will review, edit, and approve the text.

If this bug does not require doc text, please set the 'requires_doc_text' flag to -.

Thanks,
Alex

Comment 18 errata-xmlrpc 2018-09-17 16:54:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:2670


Note You need to log in before you can comment on or make changes to this bug.