Bug 2004829

Summary: [ovmf] The guest does not present hot-plugged disk
Product: Red Hat Enterprise Linux 9 Reporter: qing.wang <qinwang>
Component: qemu-kvmAssignee: Gerd Hoffmann <kraxel>
qemu-kvm sub component: virtio-blk,scsi QA Contact: qing.wang <qinwang>
Status: CLOSED CURRENTRELEASE Docs Contact:
Severity: high    
Priority: high CC: ani, coli, jinzhao, jusual, juzhang, kkiwi, kraxel, lijin, lkotek, menli, mst, phou, qzhang, virt-maint, xfu, xuwei, yanghliu, yfu, zhencliu
Version: 9.0Keywords: Regression, Triaged
Target Milestone: rcFlags: kraxel: needinfo-
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-01-20 02:18:49 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
patch: flip back default none

Description qing.wang 2021-09-16 07:28:56 UTC 
Description of problem:
there is no hot-plugged disk after execute hot-plug action in qmp.

Version-Release number of selected component (if applicable):
Red Hat Enterprise Linux release 9.0Beta (Plow)
5.14.0-1.el9.x86_64
qemu-kvm-6.1.0-2.el9.x86_64
seabios-bin-1.14.0-6.el9.noarch
edk2-ovmf-20210527gite1999b264f1f-6.el9.noarch


How reproducible:
100%

Steps to Reproduce:
1.boot vm
/usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -blockdev node-name=file_ovmf_code,driver=file,filename=/usr/share/OVMF/OVMF_CODE.secboot.fd,auto-read-only=on,discard=unmap \
    -blockdev node-name=drive_ovmf_code,driver=raw,read-only=on,file=file_ovmf_code \
    -blockdev node-name=file_ovmf_vars,driver=file,filename=/home/kvm_autotest_root/images/avocado-vt-vm1_rhel850-64-virtio.qcow2_VARS.fd,auto-read-only=on,discard=unmap \
    -blockdev node-name=drive_ovmf_vars,driver=raw,read-only=off,file=file_ovmf_vars \
    -machine q35,memory-backend=mem-machine_mem,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 8g \
    -object memory-backend-ram,size=8G,id=mem-machine_mem  \
    -smp 12,maxcpus=12,cores=6,threads=1,dies=1,sockets=2  \
    -cpu 'Haswell-noTSX',+kvm_pv_unhalt \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/images/rhel850-64-virtio-ovmf.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,write-cache=on,bus=pcie-root-port-2,addr=0x0 \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:1b:4e:74:10:e6,id=idIQPPEw,netdev=idANAOp6,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=idANAOp6,vhost=on  \
    -vnc :5 \
  -qmp tcp:0:5955,server,nowait \
  -monitor stdio \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5


2. hotplug disk
qemu-img create -f qcow2 /home/kvm_autotest_root/images/storage0.qcow2 1G

{"execute": "qmp_capabilities", "id": "RRgekEIw"}
{"execute": "blockdev-add", "arguments": {"node-name": "file_stg0", "driver": "file", "auto-read-only": true, "discard": "unmap", "aio": "threads", "filename": "/home/kvm_autotest_root/images/storage0.qcow2", "cache": {"direct": true, "no-flush": false}}, "id": "WECzE2mI"}
{"execute": "blockdev-add", "arguments": {"node-name": "drive_stg0", "driver": "qcow2", "read-only": false, "cache": {"direct": true, "no-flush": false}, "file": "file_stg0"}, "id": "jD3uXHy5"}
{"execute": "device_add", "arguments": {"driver": "virtio-blk-pci", "id": "stg0", "drive": "drive_stg0", "write-cache": "on", "bus": "pcie_extra_root_port_0", "addr": "0x0"}, "id": "TIH2qZu5"}

3.check disk in guest
lsblk

Actual results:
No new disk present in guest

Expected results:
New disk present in guest

Additional info:

No issue on:
Red Hat Enterprise Linux release 8.5 Beta (Ootpa)
4.18.0-339.el8.x86_64
qemu-kvm-6.0.0-30.module+el8.5.0+12586+476da3e1.x86_64
seabios-bin-1.14.0-1.module+el8.4.0+8855+a9e237a9.noarch
edk2-ovmf-20210527gite1999b264f1f-3.el8.noarch


Automation:
python ConfigTest.py --testcase=block_hotplug.block_virtio.fmt_qcow2.default.with_plug.default.one_pci --guestname=RHEL.8.5.0 --driveformat=virtio_blk --machine=q35 --firmware=ovmf --clone=no
Comment 1 CongLi 2021-09-16 07:39:19 UTC 
*** Bug 2004830 has been marked as a duplicate of this bug. ***
Comment 2 Klaus Heinrich Kiwi 2021-09-16 17:44:53 UTC 
Gerd,

 this looks important - care to take a look?
Comment 3 Gerd Hoffmann 2021-09-17 07:28:39 UTC 
I suspect that is more fallout coming from the pcie -> acpi hotplug switch.
I don't see how we can fix this given that edk2 has no AML interpreter.

Does it still happen with "-global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off" added to the qemu command line?
Comment 4 qing.wang 2021-09-17 07:35:35 UTC 
(In reply to Gerd Hoffmann from comment #3)
> I suspect that is more fallout coming from the pcie -> acpi hotplug switch.
> I don't see how we can fix this given that edk2 has no AML interpreter.
> 
> Does it still happen with "-global
> ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off" added to the qemu command
> line?

qemu-kvm: can't apply global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off: Property 'ICH9-LPC.acpi-pci-hotplug-with-bridge-support' not found
Comment 5 Gerd Hoffmann 2021-09-17 07:42:22 UTC 
> qemu-kvm: can't apply global
> ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off: Property
> 'ICH9-LPC.acpi-pci-hotplug-with-bridge-support' not found

Hmm?  Working fine here with the version given in the initial report:

kraxel@xeni ~# rpm -q qemu-kvm
qemu-kvm-6.1.0-1.el9.x86_64
kraxel@xeni ~# /usr/libexec/qemu-kvm -global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off
VNC server running on ::1:5900
^Cqemu-kvm: terminating on signal 2
kraxel@xeni ~# 

Please double-check.
Comment 6 qing.wang 2021-09-17 10:54:02 UTC 
(In reply to Gerd Hoffmann from comment #5)
> > qemu-kvm: can't apply global
> > ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off: Property
> > 'ICH9-LPC.acpi-pci-hotplug-with-bridge-support' not found
> 
> Hmm?  Working fine here with the version given in the initial report:
> 
> kraxel@xeni ~# rpm -q qemu-kvm
> qemu-kvm-6.1.0-1.el9.x86_64
> kraxel@xeni ~# /usr/libexec/qemu-kvm -global
> ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off
> VNC server running on ::1:5900
> ^Cqemu-kvm: terminating on signal 2
> kraxel@xeni ~# 
> 
> Please double-check.
work for very simple qemu line 
/usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off \



but it still have same issue on full qemu line:

/usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off \
    -blockdev node-name=file_ovmf_code,driver=file,filename=/usr/share/OVMF/OVMF_CODE.secboot.fd,auto-read-only=on,discard=unmap \
    -blockdev node-name=drive_ovmf_code,driver=raw,read-only=on,file=file_ovmf_code \
    -blockdev node-name=file_ovmf_vars,driver=file,filename=/home/kvm_autotest_root/images/avocado-vt-vm1_rhel850-64-virtio.qcow2_VARS.fd,auto-read-only=on,discard=unmap \
    -blockdev node-name=drive_ovmf_vars,driver=raw,read-only=off,file=file_ovmf_vars \
    -machine q35,memory-backend=mem-machine_mem,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 8g \
    -object memory-backend-ram,size=8G,id=mem-machine_mem  \
    -smp 12,maxcpus=12,cores=6,threads=1,dies=1,sockets=2  \
    -cpu 'Haswell-noTSX',+kvm_pv_unhalt \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/images/rhel850-64-virtio-ovmf.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,write-cache=on,bus=pcie-root-port-2,addr=0x0 \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:1b:4e:74:10:e6,id=idIQPPEw,netdev=idANAOp6,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=idANAOp6,vhost=on  \
    -vnc :5 \
  -qmp tcp:0:5955,server,nowait \
  -monitor stdio \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5
Comment 7 Gerd Hoffmann 2021-09-17 12:04:42 UTC 
> work for very simple qemu line 
> but it still have same issue on full qemu line:

Not reproducable here with full qemu cmd line either.

I still think your avocado runs use an older qemu binary.
The option was added in qemu 6.1, so when using a 6.0 (or older)
qemu build it'll complain the option doesn't exist.
Comment 8 Gerd Hoffmann 2021-09-21 08:24:54 UTC 
(In reply to Gerd Hoffmann from comment #3)
> I suspect that is more fallout coming from the pcie -> acpi hotplug switch.
> I don't see how we can fix this given that edk2 has no AML interpreter.

Essential information is only available in the DSDT (specifically which pci
bridge got which bsel value).  So writing a native efi driver -- talking
directly to the hotplug registers instead of using the ACPI methods -- is
impossible without host-side changes in qemu.
Comment 9 qing.wang 2021-11-04 07:56:10 UTC 
Tested on 
Red Hat Enterprise Linux release 9.0 Beta (Plow)
5.14.0-11.el9.x86_64
qemu-kvm-6.1.0-6.el9.x86_64
seabios-bin-1.14.0-7.el9.noarch
edk2-ovmf-20210527gite1999b264f1f-6.el9.noarch

It does not reproduce after adding following option on VM command.
-global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off 

 
The question is  "ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off" is necessary for the OVMF ?


/usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off \
    -blockdev node-name=file_ovmf_code,driver=file,filename=/usr/share/OVMF/OVMF_CODE.secboot.fd,auto-read-only=on,discard=unmap \
    -blockdev node-name=drive_ovmf_code,driver=raw,read-only=on,file=file_ovmf_code \
    -blockdev node-name=file_ovmf_vars,driver=file,filename=/home/kvm_autotest_root/images/avocado-vt-vm1_rhel850-64-virtio-scsi.qcow2_VARS.fd,auto-read-only=on,discard=unmap \
    -blockdev node-name=drive_ovmf_vars,driver=raw,read-only=off,file=file_ovmf_vars \
    -machine q35,memory-backend=mem-machine_mem,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 8g \
    -object memory-backend-ram,size=8G,id=mem-machine_mem  \
    -smp 12,maxcpus=12,cores=6,threads=1,dies=1,sockets=2  \
    -cpu host,+kvm_pv_unhalt \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/images/rhel850-ovmf-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,write-cache=on,bus=pcie-root-port-2,addr=0x0 \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:1b:4e:74:10:e6,id=idIQPPEw,netdev=idANAOp6,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=idANAOp6,vhost=on  \
    -vnc :5 \
  -qmp tcp:0:5955,server,nowait \
  -monitor stdio \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5
Comment 10 Gerd Hoffmann 2021-11-04 08:43:36 UTC 
> The question is  "ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off" is
> necessary for the OVMF ?

That is the current state of affairs, yes.
Unlikely to change for 6.1.

The 6.2 rebase should make hotplug work
again with the default configuration.
Comment 11 CongLi 2021-11-04 08:46:52 UTC 
*** Bug 2019314 has been marked as a duplicate of this bug. ***
Comment 12 qing.wang 2021-11-04 08:57:23 UTC 
(In reply to Gerd Hoffmann from comment #10)
> > The question is  "ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off" is
> > necessary for the OVMF ?
> 
> That is the current state of affairs, yes.
> Unlikely to change for 6.1.
> 
> The 6.2 rebase should make hotplug work
> again with the default configuration.
Hi,Gerd, 
Do you mean it need to add "-global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off \" on qemu command for 6.1 .


But the option does not need to be added for qemu 6.2 or Future version?
Comment 13 Gerd Hoffmann 2021-11-04 10:07:38 UTC 
> Hi,Gerd, 
> Do you mean it need to add "-global
> ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off \" on qemu command for 6.1
> .

Yes.

> But the option does not need to be added for qemu 6.2 or Future version?

I hope so.
Comment 14 Ani Sinha 2021-11-09 07:11:16 UTC 
(In reply to Gerd Hoffmann from comment #10)
> > The question is  "ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off" is
> > necessary for the OVMF ?
> 
> That is the current state of affairs, yes.
> Unlikely to change for 6.1.
> 
> The 6.2 rebase should make hotplug work
> again with the default configuration.

Gerd,
I am curious, how is this going to be addressed in 6.2? I have not seen any discussions in the qemu-devel@ list. I am in the MAINTAINERS file for acpi/smbios subsystem.
Comment 15 Gerd Hoffmann 2021-11-09 07:48:21 UTC 
> Gerd,
> I am curious, how is this going to be addressed in 6.2? I have not seen any
> discussions in the qemu-devel@ list. I am in the MAINTAINERS file for
> acpi/smbios subsystem.

Well, there are basically two options.

(1) first improve native pcie hotplug (see bug 2007129),
    then flip the default back to native pcie hotplug.
(2) acpi people think it is possible to have both native pcie hotplug
    and acpi hotplug enabled and make have the guest os prefer acpi
    hotplug in that case somehow.  no idea how this works in detail,
    also can't remember having seen any patches for that on qemu-devel,
    but this should also solve the problems we see because the firmware
    thinks the pcie root ports do not support hotplug.

Not sure how realistic (2) is given that there are no (public) patches
for this yet and we are in soft freeze already.

The native pcie patches are not yet merged either, but at they are out
for review for a while and QE hasn't found any issues so far.

mst?
Comment 16 Michael S. Tsirkin 2021-11-09 13:26:33 UTC 
I think it's reasonable to apply your patches and switch the default back for 6.2.
The challenge would be to address all the compatibility things with 6.1.
I did hope we can fix ACPI in 6.2 but I am not sure it is feasible at this point ...
Comment 17 Gerd Hoffmann 2021-11-09 14:52:43 UTC 
> The challenge would be to address all the compatibility things with 6.1.

Isn't that just ICH9-LPC.acpi-pci-hotplug-with-bridge-support=on for 6.1
and off for everything else?
Comment 18 Gerd Hoffmann 2021-11-09 15:04:29 UTC 
Created attachment 1840901 [details]
patch: flip back default
Comment 19 Ani Sinha 2021-11-10 05:17:29 UTC 
Comment on attachment 1840901 [details]
patch: flip back default

could you please post this patch on the mailing list for upstream folks?
Comment 22 Klaus Heinrich Kiwi 2022-01-19 17:07:29 UTC 
Gerd, any chance this was fixed on Qemu 6.2?

Submitter, do you have any more repros with more recent builds?
Comment 23 qing.wang 2022-01-20 02:18:49 UTC 
Not hit issue in testing as #0 on 
Red Hat Enterprise Linux release 8.6 Beta (Ootpa)
4.18.0-358.el8.x86_64
qemu-kvm-6.2.0-2.module+el8.6.0+13738+17338784.x86_64
seabios-bin-1.15.0-1.module+el8.6.0+13725+61ae1949.noarch
edk2-ovmf-20210527gite1999b264f1f-3.el8.noarch
virtio-win-prewhql-0.1-215.iso