Implement VCPU hotplug suport in QEMU to work with OVMF (edk2). This bug was initially created as a copy of Bug #1454803
Proposed logic in QEMU: - bit#0: SMI broadcast [existent bit] - bit#1: hotplug [new bit] - bit#2: hot-unplug [new bit] The bits should be handled as follows, then: - feature negotiation: if (!bit#0 && (bit#1 || bit#2)) { fail feature negotiation (invalid feature set requested by firmware) } else { feature negotiation OK } - device_add: if (bit#0) { if (bit#1) { plug permitted, raise SMI in AML } else { plug rejected } } else { plug permitted, no SMI } - device_del: if (bit#0) { if (bit#2) { unplug permitted, raise SMI in AML } else { unplug rejected } } else { unplug permitted, no SMI }
Patches from Igor on qemu-devel: [RFC 0/3] x86: fix cpu hotplug with secure boot https://lists.gnu.org/archive/html/qemu-devel/2020-07/msg03746.html http://mid.mail-archive.com/20200710161704.309824-1-imammedo@redhat.com
Patches from Igor on qemu-devel: [PATCH 0/6] x86: fix cpu hotplug with secure boot https://lists.gnu.org/archive/html/qemu-devel/2020-07/msg05850.html http://mid.mail-archive.com/20200720141610.574308-1-imammedo@redhat.com
Further versions: - [PATCH v2 0/7] x86: fix cpu hotplug with secure boot http://mid.mail-archive.com/20200818122208.1243901-1-imammedo@redhat.com https://lists.gnu.org/archive/html/qemu-devel/2020-08/msg03892.html - [PATCH v3 1/7] x86: lpc9: let firmware negotiate 'CPU hotplug with SMI' features (update for patch#1 only, the other patches remain at v2) http://mid.mail-archive.com/20200820145635.1305276-1-imammedo@redhat.com https://lists.gnu.org/archive/html/qemu-devel/2020-08/msg04772.html - [PATCH v4 1/6] x86: lpc9: let firmware negotiate 'CPU hotplug with SMI' features (update for patch#1 only, the other patches remain at v2) http://mid.mail-archive.com/20200824110038.1365178-1-imammedo@redhat.com https://lists.gnu.org/archive/html/qemu-devel/2020-08/msg05712.html
[PATCH v5 00/10] x86: fix cpu hotplug with secure boot http://mid.mail-archive.com/20200907112348.530921-1-imammedo@redhat.com https://lists.gnu.org/archive/html/qemu-devel/2020-09/msg02500.html
[PATCH v6 00/11] x86: fix cpu hotplug with secure boot http://mid.mail-archive.com/20200923094650.1301166-1-imammedo@redhat.com https://lists.gnu.org/archive/html/qemu-devel/2020-09/msg08285.html (the last patch of the v6 series was included in error; it has been correctly ignored subsequently) Merged upstream as commit range 2d69eba5fe52..6e2e2e8a4220.
Hi Igor, Seems the bug has been POST for more than 6 weeks, would you please help set Current Deadline? Then QE will set ITM accordingly. Thanks.
Hi Igor, I hit error "cpu hotplug with SMI wasn't enabled by firmware" when testing cpu hotplug under ovmf, would you please help check? Thanks. Packages: qemu-kvm-5.2.0-1.module+el8.4.0+9091+650b220a edk2-ovmf-20200602gitca407c7246bf-4.el8.noarch host kernel: 4.18.0-269.el8.x86_64 guest kernel: 4.18.0-259.el8.x86_64 QEMU cli: # /usr/libexec/qemu-kvm \ -S \ -name 'avocado-vt-vm1' \ -sandbox on \ -blockdev node-name=file_ovmf_code,driver=file,filename=/usr/share/OVMF/OVMF_CODE.secboot.fd,auto-read-only=on,discard=unmap \ -blockdev node-name=drive_ovmf_code,driver=raw,read-only=on,file=file_ovmf_code \ -blockdev node-name=file_ovmf_vars,driver=file,filename=/var/lib/avocado/data/avocado-vt/images/avocado-vt-vm1_rhel840-64-virtio-scsi.qcow2_VARS.fd,auto-read-only=on,discard=unmap \ -blockdev node-name=drive_ovmf_vars,driver=raw,read-only=off,file=file_ovmf_vars \ -machine q35,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \ -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \ -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0 \ -nodefaults \ -device VGA,bus=pcie.0,addr=0x2 \ -m 8192 \ -smp 15,maxcpus=16,cores=8,threads=1,dies=1,sockets=2 \ -cpu 'EPYC-Rome',+kvm_pv_unhalt \ -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/avocado_was0uatt/monitor-qmpmonitor1-20210104-004755-WTiHYCH3,nowait,server \ -mon chardev=qmp_id_qmpmonitor1,mode=control \ -chardev socket,id=qmp_id_catch_monitor,path=/tmp/avocado_was0uatt/monitor-catch_monitor-20210104-004755-WTiHYCH3,nowait,server \ -mon chardev=qmp_id_catch_monitor,mode=control \ -device pvpanic,ioport=0x505,id=idty5uab \ -chardev socket,id=chardev_serial0,path=/tmp/avocado_was0uatt/serial-serial0-20210104-004755-WTiHYCH3,nowait,server \ -device isa-serial,id=serial0,chardev=chardev_serial0 \ -chardev socket,id=seabioslog_id_20210104-004755-WTiHYCH3,path=/tmp/avocado_was0uatt/seabios-20210104-004755-WTiHYCH3,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20210104-004755-WTiHYCH3,iobase=0x402 \ -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \ -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \ -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \ -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/images/rhel840-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \ -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \ -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \ -device virtio-net-pci,mac=9a:16:b1:28:8e:07,id=idZjH9im,netdev=idrRAtUP,bus=pcie-root-port-3,addr=0x0 \ -netdev tap,id=idrRAtUP,vhost=on,vhostfd=20,fd=14 \ -vnc :0 \ -rtc base=utc,clock=host,driftfix=slew \ -boot menu=off,order=cdn,once=c,strict=off \ -enable-kvm \ -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5 QMP log: 2021-01-04 00:48:26: {"execute": "device_add", "arguments": {"id": "vcpu1", "driver": "EPYC-Rome-x86_64-cpu", "socket-id": 1, "die-id": 0, "core-id": 7, "thread-id": 0}, "id": "5OlM3X42"} 2021-01-04 00:48:26: {"id": "5OlM3X42", "error": {"class": "GenericError", "desc": "cpu hotplug with SMI wasn't enabled by firmware"}}
(In reply to Yumei Huang from comment #15) > Hi Igor, > > I hit error "cpu hotplug with SMI wasn't enabled by firmware" when testing > cpu hotplug under ovmf, would you please help check? Thanks. you need a firmware that supports it, from message it looks like firmware patches weren't backported yet > > Packages: > > qemu-kvm-5.2.0-1.module+el8.4.0+9091+650b220a > edk2-ovmf-20200602gitca407c7246bf-4.el8.noarch > host kernel: 4.18.0-269.el8.x86_64 > guest kernel: 4.18.0-259.el8.x86_64 > > > QEMU cli: > > # /usr/libexec/qemu-kvm \ > -S \ > -name 'avocado-vt-vm1' \ > -sandbox on \ > -blockdev > node-name=file_ovmf_code,driver=file,filename=/usr/share/OVMF/OVMF_CODE. > secboot.fd,auto-read-only=on,discard=unmap \ > -blockdev > node-name=drive_ovmf_code,driver=raw,read-only=on,file=file_ovmf_code \ > -blockdev > node-name=file_ovmf_vars,driver=file,filename=/var/lib/avocado/data/avocado- > vt/images/avocado-vt-vm1_rhel840-64-virtio-scsi.qcow2_VARS.fd,auto-read- > only=on,discard=unmap \ > -blockdev > node-name=drive_ovmf_vars,driver=raw,read-only=off,file=file_ovmf_vars \ > -machine q35,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \ > -device > pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1, > chassis=1 \ > -device > pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0 \ > -nodefaults \ > -device VGA,bus=pcie.0,addr=0x2 \ > -m 8192 \ > -smp 15,maxcpus=16,cores=8,threads=1,dies=1,sockets=2 \ > -cpu 'EPYC-Rome',+kvm_pv_unhalt \ > -chardev > socket,id=qmp_id_qmpmonitor1,path=/tmp/avocado_was0uatt/monitor-qmpmonitor1- > 20210104-004755-WTiHYCH3,nowait,server \ > -mon chardev=qmp_id_qmpmonitor1,mode=control \ > -chardev > socket,id=qmp_id_catch_monitor,path=/tmp/avocado_was0uatt/monitor- > catch_monitor-20210104-004755-WTiHYCH3,nowait,server \ > -mon chardev=qmp_id_catch_monitor,mode=control \ > -device pvpanic,ioport=0x505,id=idty5uab \ > -chardev > socket,id=chardev_serial0,path=/tmp/avocado_was0uatt/serial-serial0-20210104- > 004755-WTiHYCH3,nowait,server \ > -device isa-serial,id=serial0,chardev=chardev_serial0 \ > -chardev > socket,id=seabioslog_id_20210104-004755-WTiHYCH3,path=/tmp/avocado_was0uatt/ > seabios-20210104-004755-WTiHYCH3,server,nowait \ > -device > isa-debugcon,chardev=seabioslog_id_20210104-004755-WTiHYCH3,iobase=0x402 \ > -device > pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0, > chassis=2 \ > -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \ > -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ > -device > pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0, > chassis=3 \ > -device > virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \ > -blockdev > node-name=file_image1,driver=file,auto-read-only=on,discard=unmap, > aio=threads,filename=/home/kvm_autotest_root/images/rhel840-64-virtio-scsi. > qcow2,cache.direct=on,cache.no-flush=off \ > -blockdev > node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no- > flush=off,file=file_image1 \ > -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \ > -device > pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0, > chassis=4 \ > -device > virtio-net-pci,mac=9a:16:b1:28:8e:07,id=idZjH9im,netdev=idrRAtUP,bus=pcie- > root-port-3,addr=0x0 \ > -netdev tap,id=idrRAtUP,vhost=on,vhostfd=20,fd=14 \ > -vnc :0 \ > -rtc base=utc,clock=host,driftfix=slew \ > -boot menu=off,order=cdn,once=c,strict=off \ > -enable-kvm \ > -device > pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0, > addr=0x3,chassis=5 > > > QMP log: > > 2021-01-04 00:48:26: {"execute": "device_add", "arguments": {"id": "vcpu1", > "driver": "EPYC-Rome-x86_64-cpu", "socket-id": 1, "die-id": 0, "core-id": 7, > "thread-id": 0}, "id": "5OlM3X42"} > 2021-01-04 00:48:26: {"id": "5OlM3X42", "error": {"class": "GenericError", > "desc": "cpu hotplug with SMI wasn't enabled by firmware"}}
At tag "qemu-kvm-5.2.0-1.module+el8.4.0+9091+650b220a", the downstream QEMU code does not seem correct. We have this, in "hw/i386/pc.c": > /* This macro is for changes to properties that are RHEL specific, > * different to the current upstream and to be applied to the latest > * machine type. > */ > GlobalProperty pc_rhel_compat[] = { > { TYPE_X86_CPU, "host-phys-bits", "on" }, > { TYPE_X86_CPU, "host-phys-bits-limit", "48" }, > { TYPE_X86_CPU, "vmx-entry-load-perf-global-ctrl", "off" }, > { TYPE_X86_CPU, "vmx-exit-load-perf-global-ctrl", "off" }, > /* bz 1508330 */ > { "vfio-pci", "x-no-geforce-quirks", "on" }, > /* BZ 1846886 */ > { "ICH9-LPC", "x-smi-cpu-hotplug", "off" }, > }; See the last entry in the array. It comes from downstream commit 004d31cf0e8b ("Add x86_64 machine types", 2020-12-09), on the rhel-av-8.4.0 branch. Originally, this compat setting comes from downstream commit e2d32096071d ("x86: lpc9: let firmware negotiate 'CPU hotplug with SMI' features", 2020-10-08), on the rhel-av-8.3.0 branch. Please note Igor's comments on that commit (e2d32096071d): > Conflicts: > hw/i386/pc.c > move x-smi-cpu-hotplug chunk from missing pc_compat_5_1[] compat props > to pc_rhel_compat[] to disable cpu hotplug for [ovmf+smi] config > (should be moved to versioned q35 machine type later, when RHEL gets > complete feature and we decide to support it downstream) The parenthesized action ("move to versioned q35 machine type later") has not been implemented as a part of the QEMU rebase to upstream 5.2.0, for RHEL-8.4. In other words, it is a rebase bug -- it is a bug (or omission) in commit 004d31cf0e8b ("Add x86_64 machine types", 2020-12-09), on the rhel-av-8.4.0 branch. As a consequence, the compat property which disables "x-smi-cpu-hotplug" continues affecting the latest "q35" machine type, not just "pc-q35-rhel8.3.0" and earlier. The solution is the following (I didn't try to build it): > diff --git a/hw/i386/pc.c b/hw/i386/pc.c > index f3fc695fe2f8..d5ea5b634c53 100644 > --- a/hw/i386/pc.c > +++ b/hw/i386/pc.c > @@ -362,13 +362,17 @@ GlobalProperty pc_rhel_compat[] = { > { TYPE_X86_CPU, "vmx-entry-load-perf-global-ctrl", "off" }, > { TYPE_X86_CPU, "vmx-exit-load-perf-global-ctrl", "off" }, > /* bz 1508330 */ > { "vfio-pci", "x-no-geforce-quirks", "on" }, > - /* BZ 1846886 */ > - { "ICH9-LPC", "x-smi-cpu-hotplug", "off" }, > }; > const size_t pc_rhel_compat_len = G_N_ELEMENTS(pc_rhel_compat); > > +GlobalProperty pc_rhel_8_3_compat[] = { > + /* pc_rhel_8_3_compat from pc_compat_5_1 */ > + { "ICH9-LPC", "x-smi-cpu-hotplug", "off" }, > +}; > +const size_t pc_rhel_8_3_compat_len = G_N_ELEMENTS(pc_rhel_8_3_compat); > + > GlobalProperty pc_rhel_8_2_compat[] = { > /* pc_rhel_8_2_compat from pc_compat_4_2 */ > { "mch", "smbase-smram", "off" }, > }; > diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c > index 815da791081f..acd243e2e3b9 100644 > --- a/hw/i386/pc_piix.c > +++ b/hw/i386/pc_piix.c > @@ -1044,8 +1044,12 @@ static void pc_machine_rhel760_options(MachineClass *m) > m->async_pf_vmexit_disable = true; > m->smbus_no_migration_support = true; > pcmc->pvh_enabled = false; > pcmc->default_cpu_version = CPU_VERSION_LEGACY; > + compat_props_add(m->compat_props, hw_compat_rhel_8_3, > + hw_compat_rhel_8_3_len); > + compat_props_add(m->compat_props, pc_rhel_8_3_compat, > + pc_rhel_8_3_compat_len); > compat_props_add(m->compat_props, hw_compat_rhel_8_2, > hw_compat_rhel_8_2_len); > compat_props_add(m->compat_props, pc_rhel_8_2_compat, > pc_rhel_8_2_compat_len); > diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c > index 3340008c0010..a7739a169dc4 100644 > --- a/hw/i386/pc_q35.c > +++ b/hw/i386/pc_q35.c > @@ -604,8 +604,12 @@ static void pc_q35_machine_rhel830_options(MachineClass *m) > pc_q35_machine_rhel_options(m); > m->desc = "RHEL-8.3.0 PC (Q35 + ICH9, 2009)"; > pcmc->smbios_stream_product = "RHEL-AV"; > pcmc->smbios_stream_version = "8.3.0"; > + compat_props_add(m->compat_props, hw_compat_rhel_8_3, > + hw_compat_rhel_8_3_len); > + compat_props_add(m->compat_props, pc_rhel_8_3_compat, > + pc_rhel_8_3_compat_len); > } > > DEFINE_PC_MACHINE(q35_rhel830, "pc-q35-rhel8.3.0", pc_q35_init_rhel830, > pc_q35_machine_rhel830_options); Notes: - "pc_compat_5_0" is empty, so there's nothing to take from that, into "pc_rhel_8_3_compat". - the above patch (which I have not tested BTW) will *only* build on top of tag "qemu-kvm-5.2.0-2.module+el8.4.0+9186+ec44380f" -- not "qemu-kvm-5.2.0-1.module+el8.4.0+9091+650b220a"! -- because "hw_compat_rhel_8_3" is introduced in downstream commit fa0063ba6707 ("redhat: Define hw_compat_8_3", 2020-12-15), which is only part of tag "qemu-kvm-5.2.0-2.module+el8.4.0+9186+ec44380f". I suggest opening a new RHBZ with this analysis, and then making the present RHBZ (1849170) dependent on that new RHBZ. (Alternatively, I guess we can use this BZ as well, for fixing up the compat properties -- and then this one should be moved back to ASSIGNED status.) Thanks Laszlo
Laszlo, thanks for analysis. I'll fix it as part of this BZ, moving it back to assigned.
it is being fixed as part of https://bugzilla.redhat.com/show_bug.cgi?id=1885555 "[RHEL-av-8.4.0 qemu-kvm PATCH v2 3/3] 8.4 x86 machine type" so once that is merged, we should retest this BZ.
Verify: qemu-kvm-5.2.0-3.module+el8.4.0+9499+42e58f08 kernel-4.18.0-275.el8.x86_64 edk2-ovmf-20200602gitca407c7246bf-4.el8.noarch Ran vcpu hotplug test with RHEL8.4 and Win2019 guest under ovmf, guest works well, vcpu hotplug succeed without error.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:2098