Bug 1563807

Summary: Mismatch between entries in ironic and nova.
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: openstack-ironicAssignee: RHOS Maint <rhos-maint>
Status: CLOSED NOTABUG QA Contact: mlammon
Severity: high Docs Contact:
Priority: high    
Version: 13.0 (Queens)CC: bfournie, mburns, mcornea, rhel-osp-director-maint, skatlapa, srevivo
Target Milestone: ga   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-10 19:14:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alexander Chuzhoy 2018-04-04 18:33:42 UTC
Mismatch between entries in ironic and nova.

Environment:
openstack-ironic-common-10.1.2-0.20180326121311.ef08927.el7ost.noarch
python-ironic-inspector-client-3.1.1-1.el7ost.noarch
python2-ironic-neutron-agent-1.0.0-1.el7ost.noarch
openstack-ironic-conductor-10.1.2-0.20180326121311.ef08927.el7ost.noarch
python-ironic-lib-2.12.0-1.el7ost.noarch
openstack-ironic-staging-drivers-0.9.0-2.el7ost.noarch
python2-ironicclient-2.2.0-1.el7ost.noarch
instack-undercloud-8.4.0-2.el7ost.noarch
puppet-ironic-12.3.1-0.20180326225303.3956a4c.el7ost.noarch
openstack-ironic-api-10.1.2-0.20180326121311.ef08927.el7ost.noarch
openstack-ironic-inspector-7.2.1-0.20180302142656.397a98a.el7ost.noarch


(undercloud) [stack@undercloud-0 ~]$ openstack server list
openstack baremetal +--------------------------------------+--------------+--------+------------------------+----------------+------------+
| ID                                   | Name         | Status | Networks               | Image          | Flavor     |
+--------------------------------------+--------------+--------+------------------------+----------------+------------+
| 675c7b99-2c7e-4611-b6cc-ceb12c145147 | controller-0 | ACTIVE | ctlplane=192.168.24.13 | overcloud-full | controller |
| 447b47c3-51d8-4fd8-83eb-5489547e47aa | controller-1 | ACTIVE | ctlplane=192.168.24.20 | overcloud-full | controller |
| be0f62e8-a4a1-4a38-9210-ff45ff612b8b | compute-1    | ACTIVE | ctlplane=192.168.24.16 | overcloud-full | compute    |
| f61d9edd-8059-4733-a84c-953319e9310d | compute-0    | ACTIVE | ctlplane=192.168.24.17 | overcloud-full | compute    |
| 623b9c8e-9e89-4918-a8c7-fffbe633b4a8 | compute-2    | ACTIVE | ctlplane=192.168.24.8  | overcloud-full | compute    |
| d92e7ec8-0279-4fbb-8a2f-7f2db6a6ba3b | controller-2 | ACTIVE | ctlplane=192.168.24.7  | overcloud-full | controller |
+--------------------------------------+--------------+--------+------------------------+----------------+------------+




n(undercloud) [stack@undercloud-0 ~]$ openstack baremetal node list
+--------------------------------------+--------------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name         | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+--------------+--------------------------------------+-------------+--------------------+-------------+
| c2d05c90-b3c6-42f8-a386-d6daa2f8f2e3 | compute-0    | be0f62e8-a4a1-4a38-9210-ff45ff612b8b | power on    | active             | False       |
| c4dea6b6-a812-4c57-a902-a0b8f3603812 | compute-1    | 623b9c8e-9e89-4918-a8c7-fffbe633b4a8 | power on    | active             | False       |
| e1a0eaf1-ab4a-499a-94bd-e81ee3845f5a | compute-2    | f61d9edd-8059-4733-a84c-953319e9310d | power on    | active             | False       |
| 2fecf69d-1405-4c07-8606-cf2201114686 | controller-0 | 675c7b99-2c7e-4611-b6cc-ceb12c145147 | power on    | active             | False       |
| 93b8400f-de97-417f-958c-b45700c961ef | controller-1 | d92e7ec8-0279-4fbb-8a2f-7f2db6a6ba3b | power on    | active             | False       |
| ce78a569-b45c-4927-b28a-5981b88937ad | controller-2 | 447b47c3-51d8-4fd8-83eb-5489547e47aa | power on    | active             | False       |
+--------------------------------------+--------------+--------------------------------------+-------------+--------------------+-------------+


Pinging 192.168.24.16, which corresponds to c2d05c90-b3c6-42f8-a386-d6daa2f8f2e3

(undercloud) [stack@undercloud-0 ~]$ ping -c1 192.168.24.16
PING 192.168.24.16 (192.168.24.16) 56(84) bytes of data.
64 bytes from 192.168.24.16: icmp_seq=1 ttl=64 time=0.332 ms

--- 192.168.24.16 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.332/0.332/0.332/0.000 ms

Pinging 192.168.24.8, which corresponds to c4dea6b6-a812-4c57-a902-a0b8f3603812


(undercloud) [stack@undercloud-0 ~]$ ping -c1 192.168.24.8
PING 192.168.24.8 (192.168.24.8) 56(84) bytes of data.
64 bytes from 192.168.24.8: icmp_seq=1 ttl=64 time=0.219 ms

--- 192.168.24.8 ping statistics --- 
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.219/0.219/0.219/0.000 ms


#we want 192.168.24.8 to go down - let's see
ironic node-set-power-state c4dea6b6-a812-4c57-a902-a0b8f3603812 off 


It's still alive:
(undercloud) [stack@undercloud-0 ~]$ ping -c1 192.168.24.8
PING 192.168.24.8 (192.168.24.8) 56(84) bytes of data.
64 bytes from 192.168.24.8: icmp_seq=1 ttl=64 time=0.219 ms

--- 192.168.24.8 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.219/0.219/0.219/0.000 ms


But at the same time:
(undercloud) [stack@undercloud-0 ~]$ ping -c1 192.168.24.16
PING 192.168.24.16 (192.168.24.16) 56(84) bytes of data.
^C
--- 192.168.24.16 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

Comment 4 Alexander Chuzhoy 2018-04-04 18:45:55 UTC
(undercloud) [stack@undercloud-0 ~]$ openstack baremetal node list
openstack baremet+--------------------------------------+--------------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name         | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+--------------+--------------------------------------+-------------+--------------------+-------------+
| c2d05c90-b3c6-42f8-a386-d6daa2f8f2e3 | compute-0    | be0f62e8-a4a1-4a38-9210-ff45ff612b8b | power on    | active             | False       |
| c4dea6b6-a812-4c57-a902-a0b8f3603812 | compute-1    | 623b9c8e-9e89-4918-a8c7-fffbe633b4a8 | power off   | active             | False       |
| e1a0eaf1-ab4a-499a-94bd-e81ee3845f5a | compute-2    | f61d9edd-8059-4733-a84c-953319e9310d | power on    | active             | False       |
| 2fecf69d-1405-4c07-8606-cf2201114686 | controller-0 | 675c7b99-2c7e-4611-b6cc-ceb12c145147 | power on    | active             | False       |
| 93b8400f-de97-417f-958c-b45700c961ef | controller-1 | d92e7ec8-0279-4fbb-8a2f-7f2db6a6ba3b | power on    | active             | False       |
| ce78a569-b45c-4927-b28a-5981b88937ad | controller-2 | 447b47c3-51d8-4fd8-83eb-5489547e47aa | power on    | active             | False       |
+--------------------------------------+--------------+--------------------------------------+-------------+--------------------+-------------+
(undercloud) [stack@undercloud-0 ~]$ openstack baremetal node show c2d05c90-b3c6-42f8-a386-d6daa2f8f2e3
+------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                  | Value                                                                                                                                                                                                                                                                                                                                   |
+------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| boot_interface         | None                                                                                                                                                                                                                                                                                                                                    |
| chassis_uuid           | None                                                                                                                                                                                                                                                                                                                                    |                        
| clean_step             | {}                                                                                                                                                                                                                                                                                                                                      |
| console_enabled        | False                                                                                                                                                                                                                                                                                                                                   |
| console_interface      | None                                                                                                                                                                                                                                                                                                                                    |
| created_at             | 2018-04-03T23:51:13+00:00                                                                                                                                                                                                                                                                                                               |
| deploy_interface       | None                                                                                                                                                                                                                                                                                                                                    |
| driver                 | pxe_ipmitool                                                                                                                                                                                                                                                                                                                            |
| driver_info            | {u'ipmi_port': u'6231', u'ipmi_username': u'admin', u'deploy_kernel': u'9f063fe1-c24d-4fb0-9dd5-2996245ec529', u'ipmi_address': u'172.16.0.1', u'deploy_ramdisk': u'63bac0db-928d-4599-b5c3-a9fc09e7a9cf', u'ipmi_password': u'******'}                                                                                                 |
| driver_internal_info   | {u'agent_url': u'http://192.168.24.16:9999', u'root_uuid_or_disk_id': u'12ef049e-0166-4f49-882f-e25baef53651', u'is_whole_disk_image': False, u'agent_version': u'3.2.1.dev2'}                                                                                                                                                          |
| extra                  | {u'hardware_swift_object': u'extra_hardware-c2d05c90-b3c6-42f8-a386-d6daa2f8f2e3'}                                                                                                                                                                                                                                                      |
| inspect_interface      | None                                                                                                                                                                                                                                                                                                                                    |
| inspection_finished_at | None                                                                                                                                                                                                                                                                                                                                    |
| inspection_started_at  | None                                                                                                                                                                                                                                                                                                                                    |
| instance_info          | {u'root_gb': u'47', u'display_name': u'compute-1', u'image_source': u'9077a374-551b-42a8-8539-9dd76fc19ad7', u'capabilities': u'{"profile": "compute", "boot_option": "local"}', u'memory_mb': u'6144', u'vcpus': u'1', u'local_gb': u'49', u'configdrive': u'******', u'swap_mb': u'0', u'nova_host_id': u'undercloud-0.redhat.local'} |
| instance_uuid          | be0f62e8-a4a1-4a38-9210-ff45ff612b8b                                                                                                                                                                                                                                                                                                    |
| last_error             | None                                                                                                                                                                                                                                                                                                                                    |
| maintenance            | False                                                                                                                                                                                                                                                                                                                                   |
| maintenance_reason     | None                                                                                                                                                                                                                                                                                                                                    |
| management_interface   | None                                                                                                                                                                                                                                                                                                                                    |
| name                   | compute-0                                                                                                                                                                                                                                                                                                                               |
| network_interface      | flat                                                                                                                                                                                                                                                                                                                                    |
| power_interface        | None                                                                                                                                                                                                                                                                                                                                    |
| power_state            | power on                                                                                                                                                                                                                                                                                                                                |
| properties             | {u'memory_mb': u'6144', u'cpu_arch': u'x86_64', u'local_gb': u'49', u'cpus': u'2', u'capabilities': u'profile:compute,boot_option:local'}                                                                                                                                                                                               |
| provision_state        | active                                                                                                                                                                                                                                                                                                                                  |
| provision_updated_at   | 2018-04-04T00:28:49+00:00                                                                                                                                                                                                                                                                                                               |
| raid_config            | {}                                                                                                                                                                                                                                                                                                                                      |
| raid_interface         | None                                                                                                                                                                                                                                                                                                                                    |
| reservation            | None                                                                                                                                                                                                                                                                                                                                    |
| resource_class         | baremetal                                                                                                                                                                                                                                                                                                                               |
| storage_interface      | noop                                                                                                                                                                                                                                                                                                                                    |
| target_power_state     | None                                                                                                                                                                                                                                                                                                                                    |
| target_provision_state | None                                                                                                                                                                                                                                                                                                                                    |
| target_raid_config     | {}                                                                                                                                                                                                                                                                                                                                      |
| updated_at             | 2018-04-04T18:17:51+00:00                                                                                                                                                                                                                                                                                                               |
| uuid                   | c2d05c90-b3c6-42f8-a386-d6daa2f8f2e3                                                                                                                                                                                                                                                                                                    |
| vendor_interface       | None                                                                                                                                                                                                                                                                                                                                    |
+------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ openstack baremetal node show c4dea6b6-a812-4c57-a902-a0b8f3603812
+------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                  | Value                                                                                                                                                                                                                                                                                                                                   |
+------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| boot_interface         | None                                                                                                                                                                                                                                                                                                                                    |
| chassis_uuid           | None                                                                                                                                                                                                                                                                                                                                    |
| clean_step             | {}                                                                                                                                                                                                                                                                                                                                      |
| console_enabled        | False                                                                                                                                                                                                                                                                                                                                   |
| console_interface      | None                                                                                                                                                                                                                                                                                                                                    |
| created_at             | 2018-04-03T23:51:14+00:00                                                                                                                                                                                                                                                                                                               |
| deploy_interface       | None                                                                                                                                                                                                                                                                                                                                    |
| driver                 | pxe_ipmitool                                                                                                                                                                                                                                                                                                                            |
| driver_info            | {u'ipmi_port': u'6235', u'ipmi_username': u'admin', u'deploy_kernel': u'9f063fe1-c24d-4fb0-9dd5-2996245ec529', u'ipmi_address': u'172.16.0.1', u'deploy_ramdisk': u'63bac0db-928d-4599-b5c3-a9fc09e7a9cf', u'ipmi_password': u'******'}                                                                                                 |
| driver_internal_info   | {u'agent_url': u'http://192.168.24.8:9999', u'root_uuid_or_disk_id': u'12ef049e-0166-4f49-882f-e25baef53651', u'is_whole_disk_image': False, u'agent_version': u'3.2.1.dev2'}                                                                                                                                                           |
| extra                  | {u'hardware_swift_object': u'extra_hardware-c4dea6b6-a812-4c57-a902-a0b8f3603812'}                                                                                                                                                                                                                                                      |
| inspect_interface      | None                                                                                                                                                                                                                                                                                                                                    |
| inspection_finished_at | None                                                                                                                                                                                                                                                                                                                                    |
| inspection_started_at  | None                                                                                                                                                                                                                                                                                                                                    |
| instance_info          | {u'root_gb': u'47', u'display_name': u'compute-2', u'image_source': u'9077a374-551b-42a8-8539-9dd76fc19ad7', u'capabilities': u'{"profile": "compute", "boot_option": "local"}', u'memory_mb': u'6144', u'vcpus': u'1', u'local_gb': u'49', u'configdrive': u'******', u'swap_mb': u'0', u'nova_host_id': u'undercloud-0.redhat.local'} |
| instance_uuid          | 623b9c8e-9e89-4918-a8c7-fffbe633b4a8                                                                                                                                                                                                                                                                                                    |
| last_error             | None                                                                                                                                                                                                                                                                                                                                    |
| maintenance            | False                                                                                                                                                                                                                                                                                                                                   |
| maintenance_reason     | None                                                                                                                                                                                                                                                                                                                                    |
| management_interface   | None                                                                                                                                                                                                                                                                                                                                    |
| name                   | compute-1                                                                                                                                                                                                                                                                                                                               |
| network_interface      | flat                                                                                                                                                                                                                                                                                                                                    |
| power_interface        | None                                                                                                                                                                                                                                                                                                                                    |
| power_state            | power off                                                                                                                                                                                                                                                                                                                               |
| properties             | {u'memory_mb': u'6144', u'cpu_arch': u'x86_64', u'local_gb': u'49', u'cpus': u'2', u'capabilities': u'profile:compute,boot_option:local'}                                                                                                                                                                                               |
| provision_state        | active                                                                                                                                                                                                                                                                                                                                  |
| provision_updated_at   | 2018-04-04T00:28:49+00:00                                                                                                                                                                                                                                                                                                               |
| raid_config            | {}                                                                                                                                                                                                                                                                                                                                      |
| raid_interface         | None                                                                                                                                                                                                                                                                                                                                    |
| reservation            | None                                                                                                                                                                                                                                                                                                                                    |
| resource_class         | baremetal                                                                                                                                                                                                                                                                                                                               |
| storage_interface      | noop                                                                                                                                                                                                                                                                                                                                    |
| target_power_state     | None                                                                                                                                                                                                                                                                                                                                    |
| target_provision_state | None                                                                                                                                                                                                                                                                                                                                    |
| target_raid_config     | {}                                                                                                                                                                                                                                                                                                                                      |
| updated_at             | 2018-04-04T18:25:24+00:00                                                                                                                                                                                                                                                                                                               |
| uuid                   | c4dea6b6-a812-4c57-a902-a0b8f3603812                                                                                                                                                                                                                                                                                                    |
| vendor_interface       | None                                                                                                                                                                                                                                                                                                                                    |
+------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ 
(undercloud) [stack@undercloud-0 ~]$ openstack baremetal port list
+--------------------------------------+-------------------+
| UUID                                 | Address           |
+--------------------------------------+-------------------+
| f735a36c-3033-4026-9700-d4aaef223b37 | 52:54:00:cd:0a:71 |
| 73e0ca9c-e424-46f0-8395-5af242f443f0 | 52:54:00:4a:99:83 |
| c4a7e053-3115-4a2d-8a62-64bd978850d7 | 52:54:00:e7:d0:67 |
| 54cdb1cb-495c-470c-b2e5-03b467769ddb | 52:54:00:0c:1a:f8 |
| 3479c12f-4411-42ce-9a76-881073c1c34a | 52:54:00:4d:ab:60 |
| ac8cb556-eb9b-4155-9934-37b9889a768e | 52:54:00:00:71:54 |
+--------------------------------------+-------------------+

Comment 6 Dmitry Tantsur 2018-04-04 18:51:52 UTC
Could you also list neutron ports? And ideally the details of the ports belonging to the two nodes in question on the ctlplane network?

Comment 7 Alexander Chuzhoy 2018-04-04 18:54:46 UTC
(undercloud) [stack@undercloud-0 ~]$ neutron port-list
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+--------------------------------------+-------------------------------+----------------------------------+-------------------+--------------------------------------------------------------------------------------+
| id                                   | name                          | tenant_id                        | mac_address       | fixed_ips                                                                            |
+--------------------------------------+-------------------------------+----------------------------------+-------------------+--------------------------------------------------------------------------------------+
| 0a166408-cfeb-4805-8363-666bcdcf23c1 |                               | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:c4:cf:2d | {"subnet_id": "a9887623-affa-446b-b51b-b4e4d195414d", "ip_address": "172.17.1.15"}   |
| 0de3fb3e-c3ee-44d2-a598-74ff855803ad |                               | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:27:38:93 | {"subnet_id": "4d00cf11-3ab3-4af7-a158-f90cbb152ab4", "ip_address": "192.168.24.5"}  |
| 15d3a6f9-a1c4-4371-9735-5cbc962109c1 | control_virtual_ip            | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:49:f0:88 | {"subnet_id": "4d00cf11-3ab3-4af7-a158-f90cbb152ab4", "ip_address": "192.168.24.12"} |
| 16f2b401-b94f-40af-932a-fdc07d712125 |                               | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:08:23:2e | {"subnet_id": "b7d33482-4020-4daf-8c58-93f59be7fb68", "ip_address": "10.0.0.109"}    |
| 17415d3a-2447-441c-a178-374c76e27e99 |                               | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:64:47:6f | {"subnet_id": "600d267b-ceba-477b-b264-79003fe84f26", "ip_address": "172.17.2.21"}   |
| 17e9a8ac-cb72-4123-bfb8-b93f99f5f1bb |                               | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:f2:44:a8 | {"subnet_id": "600d267b-ceba-477b-b264-79003fe84f26", "ip_address": "172.17.2.24"}   |
| 18763c53-01d5-48e2-903a-bab8998dd45d | public_virtual_ip             | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:2f:b1:c8 | {"subnet_id": "b7d33482-4020-4daf-8c58-93f59be7fb68", "ip_address": "10.0.0.110"}    |
| 1a1fc093-50f3-4348-b9ef-612e829ba40b | storage_management_virtual_ip | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:21:bd:68 | {"subnet_id": "6c24da90-a3dd-4808-8b88-fa7467868a1b", "ip_address": "172.17.4.16"}   |
| 2567c139-7a8e-461d-be25-f07bdf11c599 |                               | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:6a:29:bb | {"subnet_id": "81295411-873f-4abb-896c-a151a091479d", "ip_address": "172.17.3.11"}   |
| 25f01aa4-00c4-47a7-9457-0ae8edbfb767 |                               | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:f9:30:4c | {"subnet_id": "600d267b-ceba-477b-b264-79003fe84f26", "ip_address": "172.17.2.19"}   |
| 37715b1b-2411-44cc-8a62-4253c548d5d2 | NovaCompute-port-0            | 91c4f0d2de58436fbb433ccce37aa345 | 52:54:00:cd:0a:71 | {"subnet_id": "4d00cf11-3ab3-4af7-a158-f90cbb152ab4", "ip_address": "192.168.24.16"} |
| 38651cbe-506f-47c7-9905-ead45cbea6a4 | redis_virtual_ip              | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:c3:28:b1 | {"subnet_id": "a9887623-affa-446b-b51b-b4e4d195414d", "ip_address": "172.17.1.13"}   |
| 47404bd5-319f-429b-be2f-1879361834a9 |                               | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:b2:38:ab | {"subnet_id": "81295411-873f-4abb-896c-a151a091479d", "ip_address": "172.17.3.17"}   |
| 6337ff82-5633-4d4e-9830-44a66bbaf2da | NovaCompute-port-0            | 91c4f0d2de58436fbb433ccce37aa345 | 52:54:00:4a:99:83 | {"subnet_id": "4d00cf11-3ab3-4af7-a158-f90cbb152ab4", "ip_address": "192.168.24.8"}  |
| 66e3e629-ef32-4336-9269-a2447ec5704a | internal_api_virtual_ip       | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:19:57:14 | {"subnet_id": "a9887623-affa-446b-b51b-b4e4d195414d", "ip_address": "172.17.1.12"}   |
| 67add457-e24d-4da7-9554-8064227de2bc |                               | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:1b:a9:d2 | {"subnet_id": "81295411-873f-4abb-896c-a151a091479d", "ip_address": "172.17.3.19"}   |
| 6a22aa80-fdfa-4311-8201-e665e4552115 | storage_virtual_ip            | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:38:42:25 | {"subnet_id": "81295411-873f-4abb-896c-a151a091479d", "ip_address": "172.17.3.18"}   |
| 75582ccb-3606-4b51-a9e5-dddaf6e8cf66 |                               | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:fb:60:b6 | {"subnet_id": "81295411-873f-4abb-896c-a151a091479d", "ip_address": "172.17.3.24"}   |
| 85643cff-5c92-4d2b-8d6d-d602e33e7b35 |                               | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:9d:03:32 | {"subnet_id": "b7d33482-4020-4daf-8c58-93f59be7fb68", "ip_address": "10.0.0.112"}    |
| 8760fc48-56ec-47f0-a47e-2abf527c7b47 |                               | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:f1:1c:19 | {"subnet_id": "6c24da90-a3dd-4808-8b88-fa7467868a1b", "ip_address": "172.17.4.15"}   |
| 939654b2-969a-4c2e-9e07-d6055bf71811 |                               | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:57:ed:43 | {"subnet_id": "6c24da90-a3dd-4808-8b88-fa7467868a1b", "ip_address": "172.17.4.19"}   |
| 9baea895-0649-4682-81d8-f15b796e49ba |                               | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:7e:e4:b1 | {"subnet_id": "a9887623-affa-446b-b51b-b4e4d195414d", "ip_address": "172.17.1.10"}   |
| 9bdd5941-5473-4d77-a0b8-9ff3244d0bae |                               | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:3d:17:bd | {"subnet_id": "a9887623-affa-446b-b51b-b4e4d195414d", "ip_address": "172.17.1.14"}   |
| a81fa82f-6ddc-4232-8c70-984fcac10e95 | Controller-port-0             | 91c4f0d2de58436fbb433ccce37aa345 | 52:54:00:00:71:54 | {"subnet_id": "4d00cf11-3ab3-4af7-a158-f90cbb152ab4", "ip_address": "192.168.24.20"} |
| ac46a533-97e1-4fa0-b68a-aa1f08712fa6 |                               | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:a8:08:49 | {"subnet_id": "a9887623-affa-446b-b51b-b4e4d195414d", "ip_address": "172.17.1.18"}   |
| ac9d872b-53df-4e51-9dc8-f954b81719a9 |                               | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:6e:de:eb | {"subnet_id": "a9887623-affa-446b-b51b-b4e4d195414d", "ip_address": "172.17.1.23"}   |
| b01519d5-1d5a-42da-85cf-6e771bf2df0b | Controller-port-0             | 91c4f0d2de58436fbb433ccce37aa345 | 52:54:00:4d:ab:60 | {"subnet_id": "4d00cf11-3ab3-4af7-a158-f90cbb152ab4", "ip_address": "192.168.24.7"}  |
| b7215d3e-afe3-40d6-8736-10301974baa7 |                               | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:c6:23:a8 | {"subnet_id": "600d267b-ceba-477b-b264-79003fe84f26", "ip_address": "172.17.2.22"}   |
| bdf55f5d-85c2-479d-9c4e-6538fbbc7e1b |                               | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:41:59:a9 | {"subnet_id": "a9887623-affa-446b-b51b-b4e4d195414d", "ip_address": "172.17.1.20"}   |
| c36f9162-306e-49ef-8c0e-5569ba2fee09 |                               | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:d4:ec:14 | {"subnet_id": "6c24da90-a3dd-4808-8b88-fa7467868a1b", "ip_address": "172.17.4.10"}   |
| c9c1b427-8a02-4d7e-b3a9-47a026525724 |                               | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:f0:09:6a | {"subnet_id": "600d267b-ceba-477b-b264-79003fe84f26", "ip_address": "172.17.2.10"}   |
| ccf4f73e-30a5-4a79-bd59-9337e1d7e643 |                               | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:de:d9:1d | {"subnet_id": "81295411-873f-4abb-896c-a151a091479d", "ip_address": "172.17.3.10"}   |
| cfc58436-ef35-489a-9da5-f52bf9730eb6 | Controller-port-0             | 91c4f0d2de58436fbb433ccce37aa345 | 52:54:00:0c:1a:f8 | {"subnet_id": "4d00cf11-3ab3-4af7-a158-f90cbb152ab4", "ip_address": "192.168.24.13"} |
| d3e95877-89bf-43b6-8fd0-302899bfd8aa | NovaCompute-port-0            | 91c4f0d2de58436fbb433ccce37aa345 | 52:54:00:e7:d0:67 | {"subnet_id": "4d00cf11-3ab3-4af7-a158-f90cbb152ab4", "ip_address": "192.168.24.17"} |
| d5392a9b-4e1a-49a4-b61e-a168267218cd |                               | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:39:aa:5a | {"subnet_id": "600d267b-ceba-477b-b264-79003fe84f26", "ip_address": "172.17.2.12"}   |
| d88b7436-8aab-4db7-919d-121377e066f4 |                               | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:44:52:58 | {"subnet_id": "81295411-873f-4abb-896c-a151a091479d", "ip_address": "172.17.3.20"}   |
| f20f28b0-ec84-465c-a2d6-8e3a797180a8 |                               | 91c4f0d2de58436fbb433ccce37aa345 | fa:16:3e:10:22:05 | {"subnet_id": "b7d33482-4020-4daf-8c58-93f59be7fb68", "ip_address": "10.0.0.111"}    |
+--------------------------------------+-------------------------------+----------------------------------+-------------------+--------------------------------------------------------------------------------------+

Comment 8 Alexander Chuzhoy 2018-04-04 18:57:06 UTC
(undercloud) [stack@undercloud-0 ~]$ openstack baremetal port list
+--------------------------------------+-------------------+
| UUID                                 | Address           |
+--------------------------------------+-------------------+
| f735a36c-3033-4026-9700-d4aaef223b37 | 52:54:00:cd:0a:71 |
| 73e0ca9c-e424-46f0-8395-5af242f443f0 | 52:54:00:4a:99:83 |
| c4a7e053-3115-4a2d-8a62-64bd978850d7 | 52:54:00:e7:d0:67 |
| 54cdb1cb-495c-470c-b2e5-03b467769ddb | 52:54:00:0c:1a:f8 |
| 3479c12f-4411-42ce-9a76-881073c1c34a | 52:54:00:4d:ab:60 |
| ac8cb556-eb9b-4155-9934-37b9889a768e | 52:54:00:00:71:54 |
+--------------------------------------+-------------------+
(undercloud) [stack@undercloud-0 ~]$ openstack baremetal port show f735a36c-3033-4026-9700-d4aaef223b37
+-----------------------+------------------------------------------------------------------+
| Field                 | Value                                                            |
+-----------------------+------------------------------------------------------------------+
| address               | 52:54:00:cd:0a:71                                                |
| created_at            | 2018-04-03T23:51:13+00:00                                        |
| extra                 | {}                                                               |
| internal_info         | {u'tenant_vif_port_id': u'37715b1b-2411-44cc-8a62-4253c548d5d2'} |
| local_link_connection | {}                                                               |
| node_uuid             | c2d05c90-b3c6-42f8-a386-d6daa2f8f2e3                             |
| physical_network      | ctlplane                                                         |
| portgroup_uuid        | None                                                             |
| pxe_enabled           | True                                                             |
| updated_at            | 2018-04-04T00:25:44+00:00                                        |
| uuid                  | f735a36c-3033-4026-9700-d4aaef223b37                             |
+-----------------------+------------------------------------------------------------------+
(undercloud) [stack@undercloud-0 ~]$ openstack baremetal port show 73e0ca9c-e424-46f0-8395-5af242f443f0
+-----------------------+------------------------------------------------------------------+
| Field                 | Value                                                            |
+-----------------------+------------------------------------------------------------------+
| address               | 52:54:00:4a:99:83                                                |
| created_at            | 2018-04-03T23:51:15+00:00                                        |
| extra                 | {}                                                               |
| internal_info         | {u'tenant_vif_port_id': u'6337ff82-5633-4d4e-9830-44a66bbaf2da'} |
| local_link_connection | {}                                                               |
| node_uuid             | c4dea6b6-a812-4c57-a902-a0b8f3603812                             |
| physical_network      | ctlplane                                                         |
| portgroup_uuid        | None                                                             |
| pxe_enabled           | True                                                             |
| updated_at            | 2018-04-04T00:25:42+00:00                                        |
| uuid                  | 73e0ca9c-e424-46f0-8395-5af242f443f0                             |
+-----------------------+------------------------------------------------------------------+

Comment 10 mlammon 2018-04-09 20:18:46 UTC
This has been reproduced three times. Constantly reproducing..

Comment 11 Bob Fournier 2018-04-09 20:52:11 UTC
Please update instackenv.json per comment 9, and recreate nodes.

Comment 12 Alexander Chuzhoy 2018-04-10 19:14:21 UTC
This is environmental, The vbmc ports were reconfigured.