Bug 1911214 - libvirt.libvirtError: internal error: unable to execute QEMU command 'device_del': Device net1 is already in the process of unplug
Summary: libvirt.libvirtError: internal error: unable to execute QEMU command 'device_...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova
Version: 17.0 (Wallaby)
Hardware: x86_64
OS: Unspecified
medium
medium
Target Milestone: beta
: 17.0
Assignee: Lee Yarwood
QA Contact: OSP DFG:Compute
URL:
Whiteboard: libvirt_OSP_INT
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-12-28 07:45 UTC by mxie@redhat.com
Modified: 2023-03-21 19:38 UTC (History)
16 users (show)

Fixed In Version: openstack-nova-23.0.3-0.20210908140341.e39bbdc.el9ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-09-21 12:13:24 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
network-interfaces-on-OSP-GUI.png (60.44 KB, image/png)
2020-12-28 07:45 UTC, mxie@redhat.com 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1923206 0 None None None 2021-04-09 14:36:36 UTC
OpenStack gerrit 785682 0 None NEW libvirt: Ignore device already in the process of unplug errors 2021-04-09 17:14:43 UTC
Red Hat Issue Tracker OSP-2124 0 None None None 2022-04-13 20:03:00 UTC
Red Hat Product Errata RHEA-2022:6543 0 None None None 2022-09-21 12:14:31 UTC

Description mxie@redhat.com 2020-12-28 07:45:04 UTC 
Created attachment 1742520 [details]
network-interfaces-on-OSP-GUI.png

Description of problem:
Can't detach the interface from instance by nova interface-detach command

Version-Release number of selected component (if applicable):
RHOS-16.2-RHEL-8-20201203.n.1

How reproducible:
100%

Steps to Reproduce:
1.Prepare a instance and attach a network interface to instance
1.1 (overcloud) [stack@undercloud-0 ~]$ nova interface-list rhel8.3-mxie
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+-----+
| Port State | Port ID                              | Net ID                               | IP addresses | MAC Addr          | Tag |
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+-----+
| ACTIVE     | bef8b879-b145-49a9-9fa5-f6d1c6ce3c7f | 4971b045-11a7-40a5-acbb-a8dd530167d1 | 192.168.0.64 | fa:16:3e:83:f0:6e | -   |
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+-----+

1.2 (overcloud) [stack@undercloud-0 ~]$ nova interface-attach --net-id 2a4de1c9-4b14-4c38-99f2-081e6a05d19d rhel8.3-mxie
+------------+--------------------------------------+
| Property   | Value                                |
+------------+--------------------------------------+
| ip_address | 192.168.0.70                         |
| mac_addr   | fa:16:3e:bb:55:1b                    |
| net_id     | 2a4de1c9-4b14-4c38-99f2-081e6a05d19d |
| port_id    | b6a1b399-5a1f-43a5-8b57-811b50e0c91f |
| port_state | DOWN                                 |
| tag        | -                                    |
+------------+--------------------------------------+

1.3 (overcloud) [stack@undercloud-0 ~]$ nova interface-list rhel8.3-mxie
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+-----+
| Port State | Port ID                              | Net ID                               | IP addresses | MAC Addr          | Tag |
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+-----+
| ACTIVE     | b6a1b399-5a1f-43a5-8b57-811b50e0c91f | 2a4de1c9-4b14-4c38-99f2-081e6a05d19d | 192.168.0.70 | fa:16:3e:bb:55:1b | -   |
| ACTIVE     | bef8b879-b145-49a9-9fa5-f6d1c6ce3c7f | 4971b045-11a7-40a5-acbb-a8dd530167d1 | 192.168.0.64 | fa:16:3e:83:f0:6e | -   |
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+-----+


2.Detach the network interface from instance by nova command
2.1 (overcloud) [stack@undercloud-0 ~]$ nova boot --flavor small --snapshot 1e12f1e4-b791-4e48-acdd-1ca56b2c2a97 --nic net-id=4971b045-11a7-40a5-acbb-a8dd530167d1 rhel8.3-raw-volume-snapshot-instance

2.2 (overcloud) [stack@undercloud-0 ~]$ openstack server list
+--------------------------------------+--------------------------------------+--------+-----------------------+------------------------+--------+
| ID                                   | Name                                 | Status | Networks              | Image                  | Flavor |
+--------------------------------------+--------------------------------------+--------+-----------------------+------------------------+--------+
| 0238c9f8-833f-4185-8aa3-4a4ec71b1855 | rhel8.3-raw-volume-snapshot-instance | ACTIVE | private=192.168.0.45  |                    


2.3 (overcloud) [stack@undercloud-0 ~]$ nova interface-detach rhel8.3-mxie b6a1b399-5a1f-43a5-8b57-811b50e0c91f
(overcloud) [stack@undercloud-0 ~]$ nova interface-list rhel8.3-mxie
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+-----+
| Port State | Port ID                              | Net ID                               | IP addresses | MAC Addr          | Tag |
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+-----+
| DOWN       | b6a1b399-5a1f-43a5-8b57-811b50e0c91f | 2a4de1c9-4b14-4c38-99f2-081e6a05d19d | 192.168.0.70 | fa:16:3e:bb:55:1b | -   |
| ACTIVE     | bef8b879-b145-49a9-9fa5-f6d1c6ce3c7f | 4971b045-11a7-40a5-acbb-a8dd530167d1 | 192.168.0.64 | fa:16:3e:83:f0:6e | -   |
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+-----+

2.4 (overcloud) [stack@undercloud-0 ~]$ nova interface-detach rhel8.3-mxie bef8b879-b145-49a9-9fa5-f6d1c6ce3c7f

2.5 (overcloud) [stack@undercloud-0 ~]$ nova interface-list rhel8.3-mxie
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+-----+
| Port State | Port ID                              | Net ID                               | IP addresses | MAC Addr          | Tag |
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+-----+
| DOWN       | b6a1b399-5a1f-43a5-8b57-811b50e0c91f | 2a4de1c9-4b14-4c38-99f2-081e6a05d19d | 192.168.0.70 | fa:16:3e:bb:55:1b | -   |
| DOWN       | bef8b879-b145-49a9-9fa5-f6d1c6ce3c7f | 4971b045-11a7-40a5-acbb-a8dd530167d1 | 192.168.0.64 | fa:16:3e:83:f0:6e | -   |
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+-----+

Actual results:
The network interface can't be detached from instance by nova interface-detach command

Expected results:
According to the nova interface-detach info, should detach the interface from instance by nova interface-detach command and can't see the interface by nova interface-list command

(overcloud) [stack@undercloud-0 ~]$ nova --help |grep interface-detach
    interface-detach            Detach a network interface from a server.


Additional info：
Comment 1 smooney 2021-01-08 15:40:29 UTC 
please provide sosreports container the logs for the neutron server, nova-api and compute agent where the instance is running.
without that information we cannot properly triage this.
Comment 2 mxie@redhat.com 2021-01-11 06:05:47 UTC 
(In reply to smooney from comment #1)
> please provide sosreports container the logs for the neutron server,
> nova-api and compute agent where the instance is running.
> without that information we cannot properly triage this.

Our OSP16.2 environment has been destroyed, could you please let OSP QE reproduce the bug and provide logs?
Comment 3 smooney 2021-01-11 15:07:39 UTC 
we have integration testing in both upstream and downstream ci that covers detaching interfaces.

https://github.com/openstack/tempest/blob/32e71b06206d8b65712e80429ed3637b8b8b5f73/tempest/api/compute/servers/test_attach_interfaces.py#L294-L345
the behaviour you are describing is not consistent with that test, i have confirm this with our qe and they are not seeing a failur of that test in ci.

if you are able to reproduce this and can capture sosreports for the environment then feel free to reopen
since it not appear to be broken generally i am closing for insufficient data.
Comment 4 mxie@redhat.com 2021-03-04 04:38:11 UTC 
Need to execute command 'nova interface-detach' two times to detach a interface from instance on OSP16.2， the bug should be caused by bug1878659

1.(overcloud) [stack@undercloud-0 ~]$ nova interface-list rhel8.4-volume-instance
+------------+--------------------------------------+--------------------------------------+---------------+-------------------+-----+
| Port State | Port ID                              | Net ID                               | IP addresses  | MAC Addr          | Tag |
+------------+--------------------------------------+--------------------------------------+---------------+-------------------+-----+
| ACTIVE     | 5b0d79c6-8ac0-45f6-a385-7b22dfd82397 | 6305ff53-70fd-4999-b84c-9f6d53ba4834 | 192.168.0.162 | fa:16:3e:92:2c:fc | -   |
| ACTIVE     | f29e9500-0cc2-4998-b1fb-41d09ba4530d | c88aae2f-fad8-4115-b446-e8d7a4d8d25a | 192.168.0.167 | fa:16:3e:ab:56:a1 | -   |
+------------+--------------------------------------+--------------------------------------+---------------+-------------------+-----+


2.(overcloud) [stack@undercloud-0 ~]$ nova interface-detach rhel8.4-volume-instance f29e9500-0cc2-4998-b1fb-41d09ba4530d

3.(overcloud) [stack@undercloud-0 ~]$ nova interface-list rhel8.4-volume-instance
+------------+--------------------------------------+--------------------------------------+---------------+-------------------+-----+
| Port State | Port ID                              | Net ID                               | IP addresses  | MAC Addr          | Tag |
+------------+--------------------------------------+--------------------------------------+---------------+-------------------+-----+
| ACTIVE     | 5b0d79c6-8ac0-45f6-a385-7b22dfd82397 | 6305ff53-70fd-4999-b84c-9f6d53ba4834 | 192.168.0.162 | fa:16:3e:92:2c:fc | -   |
| DOWN       | f29e9500-0cc2-4998-b1fb-41d09ba4530d | c88aae2f-fad8-4115-b446-e8d7a4d8d25a | 192.168.0.167 | fa:16:3e:ab:56:a1 | -   |
+------------+--------------------------------------+--------------------------------------+---------------+-------------------+-----+

4.(overcloud) [stack@undercloud-0 ~]$ nova interface-detach rhel8.4-volume-instance f29e9500-0cc2-4998-b1fb-41d09ba4530d

5.(overcloud) [stack@undercloud-0 ~]$ nova interface-list rhel8.4-volume-instance
+------------+--------------------------------------+--------------------------------------+---------------+-------------------+-----+
| Port State | Port ID                              | Net ID                               | IP addresses  | MAC Addr          | Tag |
+------------+--------------------------------------+--------------------------------------+---------------+-------------------+-----+
| ACTIVE     | 5b0d79c6-8ac0-45f6-a385-7b22dfd82397 | 6305ff53-70fd-4999-b84c-9f6d53ba4834 | 192.168.0.162 | fa:16:3e:92:2c:fc | -   |
+------------+--------------------------------------+--------------------------------------+---------------+-------------------+-----+
Comment 13 errata-xmlrpc 2022-09-21 12:13:24 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:6543
Note You need to log in before you can comment on or make changes to this bug.