RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2007129 - pcie hotplug emulation has various problems due to insufficient state tracking
Summary: pcie hotplug emulation has various problems due to insufficient state tracking
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: qemu-kvm
Version: 8.2
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: rc
: 8.5
Assignee: Amnon Ilan
QA Contact: jingzhao
URL:
Whiteboard:
: 2024662 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-09-23 08:10 UTC by Gerd Hoffmann
Modified: 2022-05-10 13:34 UTC (History)
23 users (show)

Fixed In Version: qemu-kvm-6.2.0-1.module+el8.6.0+13725+61ae1949
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-05-10 13:21:40 UTC
Type: Bug
Target Upstream Version:
Embargoed:
jinzhao: needinfo-
jinzhao: needinfo-

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-98000 0 None None None 2021-09-23 08:13:40 UTC
Red Hat Product Errata RHSA-2022:1759 0 None None None 2022-05-10 13:23:06 UTC

Description Gerd Hoffmann 2021-09-23 08:10:17 UTC 
So, I think the underlying problem for this is that qemu simply doesn't
track whenever pci devices are enabled or not.  But that is rather
essential state to correctly emulate pcie hotplug.

How we handle pcie hotplug (resulting in the race seen by David Gibson):
  (1) pci device is plugged in (and visiable to the guest).
  (2) guest is signaled.

How we should handle pcie hotplug:
  (1) pci device is plugged in disabled state (address spaces disabled,
      config space access blocked).
  (2) guest is signaled.
  (3) pci device is enabled in response to the guest turning on
      slot power.

pcie hot-unplug should greatly benefit from enabled/disabled state too.
When a device is in disabled state we should be able to simply unplug it
right away, without round-trip to the guest.  For enabled devices we
press the virtual attention button to nicely ask the guest.  Maybe only
after checking the power led doesn't blink (which indicates the guest is
still busy processing our previous request).

enabled/disabled state tracking should also allow to remove the
multifunction hotplug hacks.  Well, at least in the pcie code paths,
acpi hotplug probably continues to need them.

Also:  Unplugging the device unconditionally in case the guest turns off
slot power looks wrong.  I think qemu should do that only in case there
is a pending unplug request.  Otherwise simply disable the device.  The
guest can virtually power-cycle individual devices then using slot power.
Comment 1 John Ferlan 2021-10-01 19:51:36 UTC 
Assigned to Amnon for initial triage per bz process and age of bug created or assigned to virt-maint without triage.
Comment 2 John Ferlan 2021-10-01 19:59:01 UTC 
Moving from RHEL-AV to RHEL8 since starting with 8.6, there is not a plan for a separate package. If we need to create a clone eventually to generate AV 8.5.z (and earlier) patches, then we can do so once a fix is available.
Comment 3 Gerd Hoffmann 2021-10-11 09:49:48 UTC 
https://gitlab.com/kraxel/qemu/-/commits/sirius/pcie-hotplug
Comment 4 Gerd Hoffmann 2021-10-11 12:09:56 UTC 
https://patchwork.ozlabs.org/project/qemu-devel/list/?series=266393
Comment 5 Gerd Hoffmann 2021-10-19 05:59:55 UTC 
https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=40435368
Comment 6 Gerd Hoffmann 2021-10-19 06:52:51 UTC 
http://brew-task-repos.usersys.redhat.com/repos/scratch/ghoffman/qemu-kvm/6.1.0/3.el8.pcie.4/
Comment 7 Michael S. Tsirkin 2021-10-20 06:22:00 UTC 
For testing, the following flag should be set:
-global PIIX4_PM.acpi-pci-hotplug-with-bridge-support=off
Comment 12 Michael S. Tsirkin 2021-10-21 06:52:27 UTC 
Sorry my bad. I made a mistake in the flag this is what causes the warning,
the actual flag should be:

-global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off

tests are good but please repeat several times in a cycle, and with a couple more guest
guests.

Thanks!
Comment 13 Gerd Hoffmann 2021-10-21 09:39:32 UTC 
> tests are good but please repeat several times in a cycle, and with a couple
> more guest guests.

Are there autotest testcases for pcie hotplug?
If so, can you just run them all?
Comment 26 Sandro Bonazzola 2021-11-19 09:55:00 UTC 
The issue landed in CentOS Stream 8 (bug #2024605) and is now hitting oVirt community users.
Comment 27 Lee Yarwood 2021-11-22 13:27:49 UTC 
*** Bug 2024662 has been marked as a duplicate of this bug. ***
Comment 32 xiagao 2022-01-12 08:44:12 UTC 
Tested hotplug/unplug vhost-user-fs-pci device, guest works well, virtiofs function works well. 

qemu-kvm-6.2.0-1.module+el8.6.0+13725+61ae1949.x86_64
kernel-4.18.0-358.el8.x86_64
Guest: rhel9.0.0 ,win2022 - q35+seabios

Steps:

1. Boot guest with '-global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off'.
    -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x3 \
    -device pcie-root-port,port=0x11,chassis=2,id=pcie-root-port-3,bus=pcie.0,addr=0x3.0x1 \
    -global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off \
    -chardev socket,id=char_virtiofs_fs1,path=/tmp/avocado_d0ojq02p/avocado-vt-vm1-fs1-virtiofsd.sock \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device vhost-user-fs-pci,id=vufs_virtiofs_fs1,chardev=char_virtiofs_fs1,tag=myfs1,queue-size=1024,bus=pcie-root-port-3,addr=0x0 \

2. Hotunplug vhost-user-fs-pci device
{"execute": "device_del","arguments":{"id":"vufs_virtiofs_fs1"}}
{"return": {}}

{"timestamp": {"seconds": 1641975399, "microseconds": 81035}, "event": "DEVICE_DELETED", "data": {"path": "/machine/peripheral/vufs_virtiofs_fs1/virtio-backend"}}
{"timestamp": {"seconds": 1641975399, "microseconds": 81221}, "event": "DEVICE_DELETED", "data": {"device": "vufs_virtiofs_fs1", "path": "/machine/peripheral/vufs_virtiofs_fs1"}}

3. hotplug it.
{"execute": "device_add", "arguments": {"id": "vufs_virtiofs_fs1", "driver": "vhost-user-fs-pci", "bus": "pcie-root-port-3", "tag": "myfs1", "queue-size": "1024", "chardev": "char_virtiofs_fs1"}}
{"return": {}}

Results: 
Pass.
After hotunplug vhost-user-fs-pci device, device is not in guest, and after hotplug it, it will come back.
Comment 33 liunana 2022-01-13 03:46:13 UTC 
Tested hotplug/unplug the virtio-serial-pci device, guest works well, device function works well.

Test Env:
Host:
    4.18.0-358.el8.x86_64
    qemu-kvm-6.2.0-2.module+el8.6.0+13738+17338784.x86_64
    edk2-ovmf-20210527gite1999b264f1f-3.el8.noarch

Guest: RHEL8.6.0/Win2022/Win2019

Steps:
1. boot guests with '-global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off'.

 - rhel8.6.0/win2019 guests:
    -machine q35 \
    -smp 8,maxcpus=8,cores=4,threads=1,dies=1,sockets=2  \
    -cpu 'EPYC-Rome',+kvm_pv_unhalt \
    -global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off  \
    -device pcie-root-port,id=root5,port=0x5,addr=0x1.0x5,bus=pcie.0,chassis=6 \
    -device virtio-serial-pci,id=virtio-serial1,max_ports=31,bus=root5,addr=0x0 \
    -chardev socket,id=channel1,host=127.0.0.1,port=2222,server=on,wait=off \
    -device virtserialport,bus=virtio-serial1.0,chardev=channel1,name=port1,id=port1 \

 - Win2022 guest:
    -blockdev node-name=file_ovmf_code,driver=file,filename=/usr/share/OVMF/OVMF_CODE.secboot.fd,auto-read-only=on,discard=unmap \
    -blockdev node-name=drive_ovmf_code,driver=raw,read-only=on,file=file_ovmf_code \
    -blockdev node-name=file_ovmf_vars,driver=file,filename=/home/win2022-64-virtio-scsi.qcow2_VARS.fd,auto-read-only=on,discard=unmap \
    -blockdev node-name=drive_ovmf_vars,driver=raw,read-only=off,file=file_ovmf_vars \
    -machine q35,memory-backend=mem-machine_mem,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
    -global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off  \
    -object memory-backend-ram,size=30720M,id=mem-machine_mem  \
    -smp 16,maxcpus=16,cores=8,threads=1,dies=1,sockets=2  \
    -cpu 'EPYC-Rome',hv_stimer,hv_synic,hv_vpindex,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_frequencies,hv_runtime,hv_tlbflush,hv_reenlightenment,hv_stimer_direct,hv_ipi,+kvm_pv_unhalt \
    -device pcie-root-port,id=root5,port=0x5,addr=0x1.0x5,bus=pcie.0,chassis=6 \
    -device virtio-serial-pci,id=virtio-serial1,max_ports=31,bus=root5,addr=0x0 \
    -chardev socket,id=channel1,host=127.0.0.1,port=2222,server=on,wait=off \
    -device virtserialport,bus=virtio-serial1.0,chardev=channel1,name=port1,id=port1 \


2. hot-unplug/hot-plug virtio-serial-pci device many times.

{ 'execute': 'device_del', 'arguments': {'id': 'port1' }}
{"timestamp": {"seconds": 1642045423, "microseconds": 642601}, "event": "DEVICE_DELETED", "data": {"device": "port1", "path": "/machine/peripheral/port1"}}
{"return": {}}

{"execute":"device_del","arguments":{"id":"virtio-serial1"}}

{"timestamp": {"seconds": 1642044532, "microseconds": 183677}, "event": "DEVICE_DELETED", "data": {"path": "/machine/peripheral/virtio-serial1/virtio-backend"}}
{"timestamp": {"seconds": 1642044532, "microseconds": 183821}, "event": "DEVICE_DELETED", "data": {"device": "virtio-serial1", "path": "/machine/peripheral/virtio-serial1"}}
{"return": {}}

{"execute":"device_add", "arguments":{"driver":"virtio-serial-pci", "id":"virtio-serial1","max_ports":"31","bus":"root5","addr":"0x0"}}
{"return": {}}

{"execute":"device_add","arguments":{"driver":"virtserialport","bus":"virtio-serial1.0","chardev":"channel1","name":"port1","id":"port1","nr":"1"}}
{"return": {}}


3. Transfer data. PASS.
Comment 40 Yanan Fu 2022-01-27 10:38:34 UTC 
Add 'Verified:Tested,SanityOnly' as gating test with qemu-kvm-6.2.0-1.module+el8.6.0+13725+61ae1949 PASS
Comment 43 qing.wang 2022-01-29 02:26:53 UTC 
It uses default configuration in qemu-kvm-6.2 without 
"-global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off"
No new relevant issues found on block device testing.
Comment 44 xiagao 2022-01-29 03:28:52 UTC 
virtiofs works without "-global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off".
Test hotplug/unplug on rhel860 guest.
pkg:
qemu-kvm-6.2.0-5.module+el8.6.0+14025+ca131e0a.x86_64
kernel-4.18.0-360.el8
Comment 45 Lei Yang 2022-01-29 03:45:57 UTC 
Virtual Network test pass without "-global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off"

Test hotplug/unplug rhtl8139 nic, e1000e nic and virtio-net nic

Test Version:
qemu-kvm-6.2.0-5.module+el8.6.0+14025+ca131e0a.x86_64
kernel-4.18.0-360.el8.mr1880_220122_0148.x86_64
Comment 46 liunana 2022-01-29 03:54:37 UTC 
Test hotplug/unplug PASS with virtio-serial-pci device without "-global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off".

Test Env:
    4.18.0-362.el8.x86_64
    qemu-kvm-6.2.0-5.module+el8.6.0+14025+ca131e0a.x86_64
Comment 47 Yanghang Liu 2022-01-29 07:27:44 UTC 
vfio-vf/vfio-pf/nvme-vfio Test result : PASS
Comment 48 Mario Casquero 2022-02-01 11:09:20 UTC 
Virtio balloon test passed on rhel8.6 + Win2022 guests without -global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off option

qemu-kvm-6.2.0-2.module+el8.6.0+13738+17338784
kernel-4.18.0-358.el8.x86_64
Comment 59 errata-xmlrpc 2022-05-10 13:21:40 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: virt:rhel and virt-devel:rhel security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:1759
Note You need to log in before you can comment on or make changes to this bug.