Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2219224

Summary: VFIO: qemu-kvm core dumped after hot-plugging a xres=1 VF into the vm
Product: Red Hat Enterprise Linux 9 Reporter: Yanghang Liu <yanghliu>
Component: qemu-kvmAssignee: Cédric Le Goater <clegoate>
qemu-kvm sub component: PCI QA Contact: Yanghang Liu <yanghliu>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: chayang, clegoate, coli, jinzhao, juzhang, virt-maint, yanghliu, yfu
Version: 9.3Keywords: Triaged
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: 9.3   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: qemu-kvm-8.0.0-8.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-11-07 08:28:05 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version: 8.1
Embargoed:

Description Yanghang Liu 2023-07-03 03:33:29 UTC 
Description of problem:
The qemu core dump after hot-plug a xres=1 VF into the vm

Version-Release number of selected component (if applicable):
qemu-kvm-8.0.0-6.el9.x86_64

How reproducible:
100%

Steps to Reproduce:
1. check the vfio-pci options
# /usr/libexec/qemu-kvm -device vfio-pci,?
vfio-pci options:
  xres=<uint32>          -  (default: 0)
2. create a vf and bind the vf to vfio-pci

# echo 1 > /sys/bus/pci/devices/0000\:3b\:00.0/sriov_numvfs 
# virsh nodedev-detach pci_0000_3b_01_0
Device pci_0000_3b_01_0 detached
# readlink -f /sys/bus/pci/devices/0000\:3b\:01.0/driver
/sys/bus/pci/drivers/vfio-pci

3. start a vm

/usr/libexec/qemu-kvm \
-name guest=rhel93,debug-threads=on \
-machine pc-q35-rhel9.2.0,usb=off,dump-guest-core=off,memory-backend=pc.ram,hpet=off,acpi=on \
-accel kvm \
-cpu host,migratable=on \
-m 8192 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":8589934592}' \
-overcommit mem-lock=off \
-smp 4,sockets=4,dies=1,cores=1,threads=1 \
-uuid ce70e79f-8854-490a-8b0b-f5261a9b8bad \
-no-user-config \
-nodefaults \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device '{"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \
-device '{"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"}' \
-device '{"driver":"pcie-root-port","port":18,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x2.0x2"}' \
-device '{"driver":"pcie-root-port","port":19,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x2.0x3"}' \
-device '{"driver":"pcie-root-port","port":20,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x2.0x4"}' \
-device '{"driver":"pcie-root-port","port":21,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x2.0x5"}' \
-device '{"driver":"pcie-root-port","port":22,"chassis":7,"id":"pci.7","bus":"pcie.0","addr":"0x2.0x6"}' \
-device '{"driver":"pcie-root-port","port":23,"chassis":8,"id":"pci.8","bus":"pcie.0","addr":"0x2.0x7"}' \
-device '{"driver":"pcie-root-port","port":24,"chassis":9,"id":"pci.9","bus":"pcie.0","multifunction":true,"addr":"0x3"}' \
-device '{"driver":"pcie-root-port","port":25,"chassis":10,"id":"pci.10","bus":"pcie.0","addr":"0x3.0x1"}' \
-device '{"driver":"pcie-root-port","port":26,"chassis":11,"id":"pci.11","bus":"pcie.0","addr":"0x3.0x2"}' \
-device '{"driver":"pcie-root-port","port":27,"chassis":12,"id":"pci.12","bus":"pcie.0","addr":"0x3.0x3"}' \
-device '{"driver":"pcie-root-port","port":28,"chassis":13,"id":"pci.13","bus":"pcie.0","addr":"0x3.0x4"}' \
-device '{"driver":"pcie-root-port","port":29,"chassis":14,"id":"pci.14","bus":"pcie.0","addr":"0x3.0x5"}' \
-blockdev '{"node-name": "file_image1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/images/migration/RHEL93.qcow2", "cache": {"direct": true, "no-flush": false}}' \
-blockdev '{"node-name": "drive_image1", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_image1"}' \
-device '{"driver": "virtio-blk-pci", "id": "image1", "drive": "drive_image1", "bootindex": 1, "write-cache": "on", "bus": "pci.2", "addr": "0x0"}' \
-netdev '{"type":"tap","vhost":true,"id":"hostnet0"}' \
-device '{"driver":"virtio-net-pci","netdev":"hostnet0","id":"net0","mac":"52:54:00:41:5b:56","bus":"pci.1","addr":"0x0"}' \
-vnc 0.0.0.0:93 \
-device '{"driver":"virtio-vga","id":"video0","max_outputs":1,"bus":"pcie.0","addr":"0x1"}' \
-device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.6","addr":"0x0"}' \
-object '{"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"}' \
-device '{"driver":"virtio-rng-pci","rng":"objrng0","id":"rng0","bus":"pci.7","addr":"0x0"}' \
-monitor stdio \
-qmp tcp:0:5555,server,nowait \


4. hot-plug a xres=1 VF into the vm
(qemu) device_add vfio-pci,host=0000:3b:01.0,id=hostdev0,bus=pci.5,xres=1



Actual results:
qemu-kvm core dumped

93_without_vf.sh: line 44:  8425 Segmentation fault      (core dumped) 

# dmesg
[240886.863971] qemu-kvm[8425]: segfault at 0 ip 000055b0d16fdd8d sp 00007fffa24ea890 error 6 in qemu-kvm[55b0d13c3000+6ca000] likely on CPU 8 (core 3, socket 0)
[240886.863992] Code: 00 49 8b 8e c0 0d 00 00 48 85 c9 74 14 49 8b 96 c8 0d 00 00 48 89 51 10 49 8b 8e c0 0d 00 00 eb 02 31 c9 49 8b 96 c8 0d 00 00 <48> 89 0a 66 0f ef c0 f3 0f 7f 00 4c 89 f7 e8 d0 27 00 00 4c 89 f7



Expected results:
The xres=1 VF can be hot-plugged into the vm successfully

Additional info:
(1) The vf *without* xres=1 can be hot-plugged into vm successfully

The related cmd: 
(qemu) device_add vfio-pci,host=0000:3b:01.0,id=hostdev0,bus=pci.5


The vm works well and the vf info in the vm is as following:
# ifconfig 
enp5s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::c87:fb5c:3a4b:a259  prefixlen 64  scopeid 0x20<link>
        ether 96:2f:b2:0c:94:75  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 9  bytes 1610 (1.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


# dmesg
[  952.525203] pci 0000:05:00.0: [8086:1889] type 00 class 0x020000
[  952.525834] pci 0000:05:00.0: reg 0x10: [mem 0x00000000-0x0001ffff 64bit pref]
[  952.526501] pci 0000:05:00.0: reg 0x1c: [mem 0x00000000-0x00003fff 64bit pref]
[  952.527044] pci 0000:05:00.0: enabling Extended Tags
[  952.532186] pci 0000:05:00.0: BAR 0: assigned [mem 0xfc200000-0xfc21ffff 64bit pref]
[  952.532472] pci 0000:05:00.0: BAR 3: assigned [mem 0xfc220000-0xfc223fff 64bit pref]
[  952.575262] iavf: Intel(R) Ethernet Adaptive Virtual Function Network Driver
[  952.575265] Copyright (c) 2013 - 2018 Intel Corporation.
[  952.575394] iavf 0000:05:00.0: enabling device (0000 -> 0002)
[  952.650399] iavf 0000:05:00.0: Multiqueue Enabled: Queue pair count = 4
[  952.652114] iavf 0000:05:00.0: MAC address: 96:2f:b2:0c:94:75
[  952.652121] iavf 0000:05:00.0: GRO is enabled
[  952.665329] iavf 0000:05:00.0 enp5s0: renamed from eth0
[  952.793831] iavf 0000:05:00.0 enp5s0: NIC Link is Up Speed is 100 Gbps Full Duplex
[  952.793861] IPv6: ADDRCONF(NETDEV_CHANGE): enp5s0: link becomes ready

(2) Related patch : https://lore.kernel.org/all/20230629084042.86502-1-zhenzhong.duan@intel.com/

The qemu-kvm core dump issue is fixed via upstream v8.0.0-2316-gd145c0da22 qemu-kvm. 

After hot-plug the xres=1 VF into the vm , the qemu-kvm will throw an error instead of qemu-kvm core dump :

(qemu) device_add vfio-pci,host=0000:b1:01.0,id=hostdev0,bus=pci.5,xres=1,enable-migration=on
Error: vfio 0000:b1:01.0: xres and yres properties require display=on
(qemu)
Comment 1 Cédric Le Goater 2023-07-06 06:40:06 UTC 
This is fixed by :

https://gitlab.com/qemu-project/qemu/-/commit/357bd7932a136613d700ee8bc83e9165f059d1f7
Comment 3 Yanan Fu 2023-08-02 08:10:22 UTC 
QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass.
Comment 6 Yanghang Liu 2023-08-02 09:06:44 UTC 
Final Verification:


Test env:
qemu-kvm-8.0.0-9.el9.x86_64


Test step:
1. check the vfio-pci options
# /usr/libexec/qemu-kvm -device vfio-pci,?
vfio-pci options:
  xres=<uint32>          -  (default: 0)
2. create a vf and bind the vf's driver to vfio-pci

# echo 1 > /sys/bus/pci/devices/0000\:60\:00.0/sriov_numvfs 
# virsh nodedev-detach pci_0000_60_00_2
Device pci_0000_60_00_2 detached
# readlink -f /sys/bus/pci/devices/0000\:60\:00.2/driver
/sys/bus/pci/drivers/vfio-pci

3. start a vm

/usr/libexec/qemu-kvm \
-name guest=rhel93,debug-threads=on \
-machine pc-q35-rhel9.2.0,usb=off,dump-guest-core=off,memory-backend=pc.ram,hpet=off,acpi=on \
-accel kvm \
-cpu host,migratable=on \
-m 8192 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":8589934592}' \
-overcommit mem-lock=off \
-smp 4,sockets=4,dies=1,cores=1,threads=1 \
-uuid ce70e79f-8854-490a-8b0b-f5261a9b8bad \
-no-user-config \
-nodefaults \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device '{"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \
-device '{"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"}' \
-device '{"driver":"pcie-root-port","port":18,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x2.0x2"}' \
-device '{"driver":"pcie-root-port","port":19,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x2.0x3"}' \
-device '{"driver":"pcie-root-port","port":20,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x2.0x4"}' \
-device '{"driver":"pcie-root-port","port":21,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x2.0x5"}' \
-device '{"driver":"pcie-root-port","port":22,"chassis":7,"id":"pci.7","bus":"pcie.0","addr":"0x2.0x6"}' \
-device '{"driver":"pcie-root-port","port":23,"chassis":8,"id":"pci.8","bus":"pcie.0","addr":"0x2.0x7"}' \
-device '{"driver":"pcie-root-port","port":24,"chassis":9,"id":"pci.9","bus":"pcie.0","multifunction":true,"addr":"0x3"}' \
-device '{"driver":"pcie-root-port","port":25,"chassis":10,"id":"pci.10","bus":"pcie.0","addr":"0x3.0x1"}' \
-device '{"driver":"pcie-root-port","port":26,"chassis":11,"id":"pci.11","bus":"pcie.0","addr":"0x3.0x2"}' \
-device '{"driver":"pcie-root-port","port":27,"chassis":12,"id":"pci.12","bus":"pcie.0","addr":"0x3.0x3"}' \
-device '{"driver":"pcie-root-port","port":28,"chassis":13,"id":"pci.13","bus":"pcie.0","addr":"0x3.0x4"}' \
-device '{"driver":"pcie-root-port","port":29,"chassis":14,"id":"pci.14","bus":"pcie.0","addr":"0x3.0x5"}' \
-blockdev '{"node-name": "file_image1", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/images/migration/RHEL93.qcow2", "cache": {"direct": true, "no-flush": false}}' \
-blockdev '{"node-name": "drive_image1", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_image1"}' \
-device '{"driver": "virtio-blk-pci", "id": "image1", "drive": "drive_image1", "bootindex": 1, "write-cache": "on", "bus": "pci.2", "addr": "0x0"}' \
-netdev '{"type":"tap","vhost":true,"id":"hostnet0"}' \
-device '{"driver":"virtio-net-pci","netdev":"hostnet0","id":"net0","mac":"52:54:00:41:5b:56","bus":"pci.1","addr":"0x0"}' \
-vnc 0.0.0.0:93 \
-device '{"driver":"virtio-vga","id":"video0","max_outputs":1,"bus":"pcie.0","addr":"0x1"}' \
-device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.6","addr":"0x0"}' \
-object '{"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"}' \
-device '{"driver":"virtio-rng-pci","rng":"objrng0","id":"rng0","bus":"pci.7","addr":"0x0"}' \
-monitor stdio \
-qmp tcp:0:5555,server,nowait \


4. hot-plug a xres=1 VF into the vm
(qemu) device_add vfio-pci,host=0000:60:00.2,id=hostdev0,bus=pci.5,xres=1
Error: vfio 0000:60:00.2: xres and yres properties require display=on  <-- After hot-plug the xres=1 VF into the vm , the qemu-kvm will throw an error instead of qemu-kvm core dump
Comment 8 errata-xmlrpc 2023-11-07 08:28:05 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: qemu-kvm security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6368