Bug 2001387

Summary: [WRB][QEMU6.1]hotplug and hot-unplug all can not plug/un-plug the virtio device successfully
Product: Red Hat Enterprise Linux 9 Reporter: John Ferlan <jferlan>
Component: qemu-kvmAssignee: Igor Mammedov <imammedo>
qemu-kvm sub component: Devices QA Contact: Yanan Fu <yfu>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: ailan, chayang, coli, imammedo, jinzhao, juzhang, leiyang, menli, mrezanin, nanliu, pezhang, virt-bugs, virt-maint, yanghliu, yfu
Version: 9.0Keywords: Regression, Triaged
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: qem-kvm-6.1.0-1.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1988986 Environment:
Last Closed: 2022-05-17 12:24:17 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1988986, 1997408    
Bug Blocks:    

Description John Ferlan 2021-09-05 23:18:11 UTC
+++ This bug was initially created as a clone of Bug #1988986 +++

Description of problem:
This is a regression issue with upstream weeklyrebase build, for QEMu 6.1.0 rc1.
It include 2 issues:
1. hotplug virtio-net-pci can not plug the device
2. hot-unplug virtio-net-pci can not unplug the device

Version-Release number of selected component (if applicable):
Host:
qemu-kvm-core-6.1.0-1.rc1.scrmod+el8.5.0+12016+049b55fd.wrb210728.x86_64
kernel-4.18.0-324.el8.x86_64

Guest kernel version:
kernel-4.18.0-323.el8.x86_64


How reproducible:
100%

Steps to Reproduce:
scenario 1, hotplug
1. Boot vm
2. Hotplug nic
{"execute": "netdev_add", "arguments": {"type": "tap", "id": "netdev0"}}
{"return": {}}
{"execute": "device_add", "arguments": {"id": "nic0", "driver": "virtio-net-pci", "netdev": "netdev0", "mac": "9a:45:bb:a8:b1:90", "bus": "pcie_extra_root_port_0", "addr": "0x0"}}
{"return": {}}


3. can get the devive with monitor but can not get it from guest os:
(qemu) info network 
nic0: index=0,type=nic,model=virtio-net-pci,macaddr=9a:45:bb:a8:b1:90
 \ netdev0: index=0,type=tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown

guest os: lspci can not get the device, and no nic available


4. hotplug the nic can not take effect in qemu monitor:
(qemu) device_del nic0
(qemu) info network 
nic0: index=0,type=nic,model=virtio-net-pci,macaddr=9a:45:bb:a8:b1:90
 \ netdev0: index=0,type=tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown




scenario 2, hot-unplug:
1. Boot vm with virtio-net-pci
    -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x1.0x4,bus=pcie.0,chassis=5 \
    -device virtio-net-pci,mac=9a:2e:6d:4c:1b:b2,id=nic0,netdev=netdev0,bus=pcie-root-port-4,addr=0x0  \
    -netdev tap,id=netdev0 \


2. check network device in qemu monitor and guest os, it is ok
(qemu) info network 
nic0: index=0,type=nic,model=virtio-net-pci,macaddr=9a:2e:6d:4c:1b:b2
 \ netdev0: index=0,type=tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown


guest os: lspci can get the virtio-net-pci device, and the nic work well:
# lspci
...
05:00.0 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01)


3. hot unplug virtio-net-pci, still can get the device with both monitor and guest os
(qemu) device_del nic0 
(qemu) info network 
nic0: index=0,type=nic,model=virtio-net-pci,macaddr=9a:2e:6d:4c:1b:b2
 \ netdev0: index=0,type=tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown

# lspci
...
05:00.0 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01)




Actual results:
hotplug and hot-unplug nic does not work


Expected results:
hotplug and hot-unplug nic can work

Additional info:

--- Additional comment from Yanan Fu on 2021-08-02 09:56:17 UTC ---

Update~   It is not a virtio-net-pci only issue,   hotplug virtio-blk-pci does not work too.

1. qemu command line (except the system disk):
    -blockdev node-name=file_stg0,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/images/storage0.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_stg0,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_stg0 \
    -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x1.0x4,bus=pcie.0,chassis=5 \
    -device virtio-blk-pci,id=stg0,drive=drive_stg0,bootindex=1,write-cache=on,bus=pcie-root-port-4,addr=0x0 \

2. device still exist after hot-unplug in both qemu monitor and guest os
(qemu) info block
drive_image1: /home/kvm_autotest_root/images/rhel850-64-virtio.qcow2 (qcow2)
    Attached to:      /machine/peripheral/image1/virtio-backend
    Cache mode:       writeback, direct

drive_stg0: /home/kvm_autotest_root/images/storage0.qcow2 (qcow2)
    Attached to:      /machine/peripheral/stg0/virtio-backend
    Cache mode:       writeback, direct

(qemu) device_del stg0 
(qemu) info block
drive_image1: /home/kvm_autotest_root/images/rhel850-64-virtio.qcow2 (qcow2)
    Attached to:      /machine/peripheral/image1/virtio-backend
    Cache mode:       writeback, direct

drive_stg0: /home/kvm_autotest_root/images/storage0.qcow2 (qcow2)
    Attached to:      /machine/peripheral/stg0/virtio-backend
    Cache mode:       writeback, direct

--- Additional comment from Pei Zhang on 2021-08-04 10:01:09 UTC ---

Update:

1. Hot plug/unplug PF (x540-AT2, ixgbe driver, 10G) also fails, should be same issue with this one.

2. Hot plug/unplug vhost-user also fails, should be same issue with this one. 

3. After hot plug/unplug fails, guest kernel shows error:

# dmesg
...
[   42.929429] ACPI BIOS Error (bug): Could not resolve symbol [^S16.PCNT], AE_NOT_FOUND (20210105/psargs-330)
[   42.930959] ACPI Error: Aborting method \_SB.PCI0.PCNT due to previous error (AE_NOT_FOUND) (20210105/psparse-531)
[   42.932564] ACPI Error: Aborting method \_GPE._E01 due to previous error (AE_NOT_FOUND) (20210105/psparse-531)
[   42.934120] ACPI Error: AE_NOT_FOUND, while evaluating GPE method [_E01] (20210105/evgpe-515)
[   50.581454] ACPI BIOS Error (bug): Could not resolve symbol [^S16.PCNT], AE_NOT_FOUND (20210105/psargs-330)
[   50.582971] ACPI Error: Aborting method \_SB.PCI0.PCNT due to previous error (AE_NOT_FOUND) (20210105/psparse-531)
[   50.584696] ACPI Error: Aborting method \_GPE._E01 due to previous error (AE_NOT_FOUND) (20210105/psparse-531)
[   50.586304] ACPI Error: AE_NOT_FOUND, while evaluating GPE method [_E01] (20210105/evgpe-515)
...

--- Additional comment from menli on 2021-08-05 02:52:31 UTC ---

hit the same issue when  Hot-unplug virtio-rng device.

--- Additional comment from Yanan Fu on 2021-08-05 03:02:56 UTC ---

It is not virtual network only, reset the qa-contact, and feature name in QA whiteboard, thanks!

--- Additional comment from Yanan Fu on 2021-08-06 07:29:04 UTC ---

This issue gone with:
qemu-kvm-6.1.0-1.rc2.scrmod+el8.5.0+12133+c45b5bc2.wrb210804.x86_64

Tested with virtio-net-pci, virtio-blk-pci, virtio-scsi-pci, both hotplug and hot-unplug can work well.

I guess it is related with commit:
https://gitlab.com/qemu-project/qemu/-/commit/e2a6290aab578b2170c1f5909fa556385dc0d820

--- Additional comment from John Ferlan on 2021-08-06 15:49:02 UTC ---

Based on comment 5, moving to POST.  Assigned to Marcel since he authored the patch.

Used DTM=5 similar to bug 1989338 which is another issue fixed in rc2 for qemu-6.1.  Once QE sets the ITM, release+ will be granted.

--- Additional comment from Marcel Apfelbaum on 2021-08-06 16:00:43 UTC ---

(In reply to John Ferlan from comment #6)
> Based on comment 5, moving to POST.  Assigned to Marcel since he authored
> the patch.
> 

Can't take credit for this one, it was all Igor :)

https://gitlab.com/qemu-project/qemu/-/commit/d7346e614f4ec353f9b24bb69bfeaf1ade287e07



> Used DTM=5 similar to bug 1989338 which is another issue fixed in rc2 for
> qemu-6.1.  Once QE sets the ITM, release+ will be granted.

Comment 2 Yanan Fu 2021-10-08 01:19:59 UTC
QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass.

Comment 3 Yanan Fu 2021-10-08 01:23:22 UTC
Hi Mirek,

This bz is for RHEL9, the fixed in version is for RHEL8.
I think it is qemu-kvm-6.1.0-1.el9 ? Could you help confirm ?
Thanks!

Best regards
Yanan Fu

Comment 6 Yanan Fu 2021-10-11 12:50:10 UTC
This scenario already be covered by gating test.

virtio nic hotplug/unplug,  virtio disk hotplug/unplug all works as expected.
Move to VERIFIED accordingly, thanks!

Comment 8 errata-xmlrpc 2022-05-17 12:24:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: qemu-kvm), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2307