2227740 – We cannot detach interfaces from a VM if the VM has multiple interfaces with same mac address

Bug 2227740 - We cannot detach interfaces from a VM if the VM has multiple interfaces with same mac address [NEEDINFO]

Summary: We cannot detach interfaces from a VM if the VM has multiple interfaces with ...

Keywords:
Status:	NEW
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-nova
Sub Component:
Version:	16.2 (Train)
Hardware:	x86_64
OS:	Linux
Priority:	medium
Severity:	medium
Target Milestone:	---
Target Release:	---
Assignee:	OSP DFG:Compute
QA Contact:	OSP DFG:Compute
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2023-07-31 08:55 UTC by yatanaka
Modified:	2023-08-15 03:06 UTC (History)
CC List:	8 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:
Embargoed:
Flags:	ifrangs: needinfo? (osp-dfg-compute)

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	OSP-27036	0	None	None	None	2023-07-31 08:58:17 UTC
Red Hat Knowledge Base (Solution)	7028682	0	None	None	None	2023-08-15 03:06:35 UTC

Description yatanaka 2023-07-31 08:55:01 UTC

Description of problem:

We cannot detach interfaces from a VM if the VM has multiple interfaces with same mac address.

Crate two ports with the same MAC address, and then create an instance with these two interfaces.
Port creations and instance creation succeeds without error.

~~~
openstack port create --network yatanaka_network0 port0-0 --mac-address fa:16:3e:b5:98:48
openstack port create --network yatanaka_network00 port00-0 --mac-address fa:16:3e:b5:98:48

openstack server create --flavor yatanaka_flavor --image rhel-8.2-290 --security-group yatanaka_sg --key-name yatanaka_keypair yatanaka0 --port port0-0 --port port00-0
~~~

However, when I try to detach of these two ports, it fails with the following error.

~~~
$ openstack server remove port yatanaka0  port0-0

<nova-compute.log>
2023-07-31 08:23:26.459 7 DEBUG nova.virt.libvirt.guest [req-63961267-8749-475c-bfbb-fc27fdf9a9ce ed4ba9ffd70144a4a0261b95538b0097 377c5c5a46a144b08f43da86033d9c8d - d2aaf66db48e428c94e03f8462d39f98 d2aaf66db48e428c94e03f8462d39f98] Attempting initial detach for device tapaa6b7058-64 detach_device_with_retry /usr/lib/python3.6/site-packages/nova/virt/libvirt/guest.py:470
2023-07-31 08:23:26.459 7 DEBUG nova.virt.libvirt.guest [req-63961267-8749-475c-bfbb-fc27fdf9a9ce ed4ba9ffd70144a4a0261b95538b0097 377c5c5a46a144b08f43da86033d9c8d - d2aaf66db48e428c94e03f8462d39f98 d2aaf66db48e428c94e03f8462d39f98] detach device xml: <interface type="bridge">
  <mac address="fa:16:3e:b5:98:48"/>
  <model type="virtio"/>
  <driver name="vhost" rx_queue_size="512"/>
  <source bridge="br-int"/>
  <mtu size="1442"/>
  <target dev="tapaa6b7058-64"/>
  <virtualport type="openvswitch">
    <parameters interfaceid="aa6b7058-64ca-4e26-9aef-7c4adb6fa1bf"/>
  </virtualport>
</interface>
 detach_device /usr/lib/python3.6/site-packages/nova/virt/libvirt/guest.py:521
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [req-63961267-8749-475c-bfbb-fc27fdf9a9ce ed4ba9ffd70144a4a0261b95538b0097 377c5c5a46a144b08f43da86033d9c8d - d2aaf66db48e428c94e03f8462d39f98 d2aaf66db48e428c94e03f8462d39f98] [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0] detaching network adapter failed.: libvirt.libvirtError: operation failed: multiple devices matching MAC address fa:16:3e:b5:98:48 found
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0] Traceback (most recent call last):
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0]   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2409, in detach_interface
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0]     supports_device_missing_error_code=supports_device_missing)
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0]   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", line 472, in detach_device_with_retry
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0]     _try_detach_device(conf, persistent, live)
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0]   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", line 461, in _try_detach_device
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0]     ctx.reraise = True
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0]   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0]     self.force_reraise()
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0]   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0]     six.reraise(self.type_, self.value, self.tb)
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0]   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0]     raise value
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0]   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", line 414, in _try_detach_device
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0]     self.detach_device(conf, persistent=persistent, live=live)
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0]   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", line 522, in detach_device
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0]     self._domain.detachDeviceFlags(device_xml, flags=flags)
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0]   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0]   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0]     rv = execute(f, *args, **kwargs)
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0]   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0]     six.reraise(c, e, tb)
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0]   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0]     raise value
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0]   File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0]     rv = meth(*args, **kwargs)
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0]   File "/usr/lib64/python3.6/site-packages/libvirt.py", line 1534, in detachDeviceFlags
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0]     raise libvirtError('virDomainDetachDeviceFlags() failed')
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0] libvirt.libvirtError: operation failed: multiple devices matching MAC address fa:16:3e:b5:98:48 found
2023-07-31 08:23:26.462 7 ERROR nova.virt.libvirt.driver [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0]
2023-07-31 08:23:26.463 7 WARNING nova.compute.manager [req-63961267-8749-475c-bfbb-fc27fdf9a9ce ed4ba9ffd70144a4a0261b95538b0097 377c5c5a46a144b08f43da86033d9c8d - d2aaf66db48e428c94e03f8462d39f98 d2aaf66db48e428c94e03f8462d39f98] [instance: 9ad4b50d-0a78-42d0-9a33-3eb3403871f0] Detach interface failed, port_id=aa6b7058-64ca-4e26-9aef-7c4adb6fa1bf, reason: Failed to detach network adapter device from 9ad4b50d-0a78-42d0-9a33-3eb3403871f0: nova.exception.InterfaceDetachFailed: Failed to detach network adapter device from 9ad4b50d-0a78-42d0-9a33-3eb3403871f0
~~~

The failed point is here:
  - https://github.com/openstack/nova/blob/stable/train/nova/virt/libvirt/guest.py#L505

The same issue has been reported and fixed in the upstream mater in 2017 below.
  - https://bugs.launchpad.net/nova/+bug/1621076
  - https://review.opendev.org/c/openstack/nova/+/372243

I'm not sure why this fix doesn't work in RHOSP 16.2.
Is it possible to fix this issue in RHOSP 16.2?

Additionally, please let me know if there is a workaround to detach the ports which have the same MAC address in the current version.
(I guess there are no workarounds other than deleting/recreating the insntance)


Version-Release number of selected component (if applicable):
RHOSP 16.2.5


How reproducible:

Steps to Reproduce:
1. Create two tenant networks
2. Create two ports with the same MAC address
3. Create an instance with the two ports
4. Try to detach either of the ports from the VM


Actual results:
Failed to detach the port

Expected results:
Succeed to detach the port

Note You need to log in before you can comment on or make changes to this bug.