Bug 1911214

Summary: libvirt.libvirtError: internal error: unable to execute QEMU command 'device_del': Device net1 is already in the process of unplug
Product: Red Hat OpenStack Reporter: mxie <mxie>
Component: openstack-novaAssignee: Lee Yarwood <lyarwood>
Status: CLOSED ERRATA QA Contact: OSP DFG:Compute <osp-dfg-compute>
Severity: medium Docs Contact:
Priority: medium    
Version: 17.0 (Wallaby)CC: chhu, dasmith, eglynn, jhakimra, jparker, juzhou, kchamart, lyarwood, mzhan, sbauza, sgordon, smooney, tyan, tzheng, vromanso, xiaodwan
Target Milestone: betaKeywords: Patch, Reopened, Triaged
Target Release: 17.0   
Hardware: x86_64   
OS: Unspecified   
Whiteboard: libvirt_OSP_INT
Fixed In Version: openstack-nova-23.0.3-0.20210908140341.e39bbdc.el9ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-09-21 12:13:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
network-interfaces-on-OSP-GUI.png none

Description mxie@redhat.com 2020-12-28 07:45:04 UTC
Created attachment 1742520 [details]
network-interfaces-on-OSP-GUI.png

Description of problem:
Can't detach the interface from instance by nova interface-detach command

Version-Release number of selected component (if applicable):
RHOS-16.2-RHEL-8-20201203.n.1

How reproducible:
100%

Steps to Reproduce:
1.Prepare a instance and attach a network interface to instance
1.1 (overcloud) [stack@undercloud-0 ~]$ nova interface-list rhel8.3-mxie
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+-----+
| Port State | Port ID                              | Net ID                               | IP addresses | MAC Addr          | Tag |
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+-----+
| ACTIVE     | bef8b879-b145-49a9-9fa5-f6d1c6ce3c7f | 4971b045-11a7-40a5-acbb-a8dd530167d1 | 192.168.0.64 | fa:16:3e:83:f0:6e | -   |
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+-----+

1.2 (overcloud) [stack@undercloud-0 ~]$ nova interface-attach --net-id 2a4de1c9-4b14-4c38-99f2-081e6a05d19d rhel8.3-mxie
+------------+--------------------------------------+
| Property   | Value                                |
+------------+--------------------------------------+
| ip_address | 192.168.0.70                         |
| mac_addr   | fa:16:3e:bb:55:1b                    |
| net_id     | 2a4de1c9-4b14-4c38-99f2-081e6a05d19d |
| port_id    | b6a1b399-5a1f-43a5-8b57-811b50e0c91f |
| port_state | DOWN                                 |
| tag        | -                                    |
+------------+--------------------------------------+

1.3 (overcloud) [stack@undercloud-0 ~]$ nova interface-list rhel8.3-mxie
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+-----+
| Port State | Port ID                              | Net ID                               | IP addresses | MAC Addr          | Tag |
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+-----+
| ACTIVE     | b6a1b399-5a1f-43a5-8b57-811b50e0c91f | 2a4de1c9-4b14-4c38-99f2-081e6a05d19d | 192.168.0.70 | fa:16:3e:bb:55:1b | -   |
| ACTIVE     | bef8b879-b145-49a9-9fa5-f6d1c6ce3c7f | 4971b045-11a7-40a5-acbb-a8dd530167d1 | 192.168.0.64 | fa:16:3e:83:f0:6e | -   |
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+-----+


2.Detach the network interface from instance by nova command
2.1 (overcloud) [stack@undercloud-0 ~]$ nova boot --flavor small --snapshot 1e12f1e4-b791-4e48-acdd-1ca56b2c2a97 --nic net-id=4971b045-11a7-40a5-acbb-a8dd530167d1 rhel8.3-raw-volume-snapshot-instance

2.2 (overcloud) [stack@undercloud-0 ~]$ openstack server list
+--------------------------------------+--------------------------------------+--------+-----------------------+------------------------+--------+
| ID                                   | Name                                 | Status | Networks              | Image                  | Flavor |
+--------------------------------------+--------------------------------------+--------+-----------------------+------------------------+--------+
| 0238c9f8-833f-4185-8aa3-4a4ec71b1855 | rhel8.3-raw-volume-snapshot-instance | ACTIVE | private=192.168.0.45  |                    


2.3 (overcloud) [stack@undercloud-0 ~]$ nova interface-detach rhel8.3-mxie b6a1b399-5a1f-43a5-8b57-811b50e0c91f
(overcloud) [stack@undercloud-0 ~]$ nova interface-list rhel8.3-mxie
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+-----+
| Port State | Port ID                              | Net ID                               | IP addresses | MAC Addr          | Tag |
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+-----+
| DOWN       | b6a1b399-5a1f-43a5-8b57-811b50e0c91f | 2a4de1c9-4b14-4c38-99f2-081e6a05d19d | 192.168.0.70 | fa:16:3e:bb:55:1b | -   |
| ACTIVE     | bef8b879-b145-49a9-9fa5-f6d1c6ce3c7f | 4971b045-11a7-40a5-acbb-a8dd530167d1 | 192.168.0.64 | fa:16:3e:83:f0:6e | -   |
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+-----+

2.4 (overcloud) [stack@undercloud-0 ~]$ nova interface-detach rhel8.3-mxie bef8b879-b145-49a9-9fa5-f6d1c6ce3c7f

2.5 (overcloud) [stack@undercloud-0 ~]$ nova interface-list rhel8.3-mxie
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+-----+
| Port State | Port ID                              | Net ID                               | IP addresses | MAC Addr          | Tag |
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+-----+
| DOWN       | b6a1b399-5a1f-43a5-8b57-811b50e0c91f | 2a4de1c9-4b14-4c38-99f2-081e6a05d19d | 192.168.0.70 | fa:16:3e:bb:55:1b | -   |
| DOWN       | bef8b879-b145-49a9-9fa5-f6d1c6ce3c7f | 4971b045-11a7-40a5-acbb-a8dd530167d1 | 192.168.0.64 | fa:16:3e:83:f0:6e | -   |
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+-----+

Actual results:
The network interface can't be detached from instance by nova interface-detach command

Expected results:
According to the nova interface-detach info, should detach the interface from instance by nova interface-detach command and can't see the interface by nova interface-list command

(overcloud) [stack@undercloud-0 ~]$ nova --help |grep interface-detach
    interface-detach            Detach a network interface from a server.


Additional info:

Comment 1 smooney 2021-01-08 15:40:29 UTC
please provide sosreports container the logs for the neutron server, nova-api and compute agent where the instance is running.
without that information we cannot properly triage this.

Comment 2 mxie@redhat.com 2021-01-11 06:05:47 UTC
(In reply to smooney from comment #1)
> please provide sosreports container the logs for the neutron server,
> nova-api and compute agent where the instance is running.
> without that information we cannot properly triage this.

Our OSP16.2 environment has been destroyed, could you please let OSP QE reproduce the bug and provide logs?

Comment 3 smooney 2021-01-11 15:07:39 UTC
we have integration testing in both upstream and downstream ci that covers detaching interfaces.

https://github.com/openstack/tempest/blob/32e71b06206d8b65712e80429ed3637b8b8b5f73/tempest/api/compute/servers/test_attach_interfaces.py#L294-L345
the behaviour you are describing is not consistent with that test, i have confirm this with our qe and they are not seeing a failur of that test in ci.

if you are able to reproduce this and can capture sosreports for the environment then feel free to reopen
since it not appear to be broken generally i am closing for insufficient data.

Comment 4 mxie@redhat.com 2021-03-04 04:38:11 UTC
Need to execute command 'nova interface-detach' two times to detach a interface from instance on OSP16.2, the bug should be caused by bug1878659

1.(overcloud) [stack@undercloud-0 ~]$ nova interface-list rhel8.4-volume-instance
+------------+--------------------------------------+--------------------------------------+---------------+-------------------+-----+
| Port State | Port ID                              | Net ID                               | IP addresses  | MAC Addr          | Tag |
+------------+--------------------------------------+--------------------------------------+---------------+-------------------+-----+
| ACTIVE     | 5b0d79c6-8ac0-45f6-a385-7b22dfd82397 | 6305ff53-70fd-4999-b84c-9f6d53ba4834 | 192.168.0.162 | fa:16:3e:92:2c:fc | -   |
| ACTIVE     | f29e9500-0cc2-4998-b1fb-41d09ba4530d | c88aae2f-fad8-4115-b446-e8d7a4d8d25a | 192.168.0.167 | fa:16:3e:ab:56:a1 | -   |
+------------+--------------------------------------+--------------------------------------+---------------+-------------------+-----+


2.(overcloud) [stack@undercloud-0 ~]$ nova interface-detach rhel8.4-volume-instance f29e9500-0cc2-4998-b1fb-41d09ba4530d

3.(overcloud) [stack@undercloud-0 ~]$ nova interface-list rhel8.4-volume-instance
+------------+--------------------------------------+--------------------------------------+---------------+-------------------+-----+
| Port State | Port ID                              | Net ID                               | IP addresses  | MAC Addr          | Tag |
+------------+--------------------------------------+--------------------------------------+---------------+-------------------+-----+
| ACTIVE     | 5b0d79c6-8ac0-45f6-a385-7b22dfd82397 | 6305ff53-70fd-4999-b84c-9f6d53ba4834 | 192.168.0.162 | fa:16:3e:92:2c:fc | -   |
| DOWN       | f29e9500-0cc2-4998-b1fb-41d09ba4530d | c88aae2f-fad8-4115-b446-e8d7a4d8d25a | 192.168.0.167 | fa:16:3e:ab:56:a1 | -   |
+------------+--------------------------------------+--------------------------------------+---------------+-------------------+-----+

4.(overcloud) [stack@undercloud-0 ~]$ nova interface-detach rhel8.4-volume-instance f29e9500-0cc2-4998-b1fb-41d09ba4530d

5.(overcloud) [stack@undercloud-0 ~]$ nova interface-list rhel8.4-volume-instance
+------------+--------------------------------------+--------------------------------------+---------------+-------------------+-----+
| Port State | Port ID                              | Net ID                               | IP addresses  | MAC Addr          | Tag |
+------------+--------------------------------------+--------------------------------------+---------------+-------------------+-----+
| ACTIVE     | 5b0d79c6-8ac0-45f6-a385-7b22dfd82397 | 6305ff53-70fd-4999-b84c-9f6d53ba4834 | 192.168.0.162 | fa:16:3e:92:2c:fc | -   |
+------------+--------------------------------------+--------------------------------------+---------------+-------------------+-----+

Comment 13 errata-xmlrpc 2022-09-21 12:13:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:6543