Bug 1846886
| Summary: | Guest hit soft lockup or reboots if hotplug vcpu under ovmf | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux Advanced Virtualization | Reporter: | Yumei Huang <yuhuang> |
| Component: | qemu-kvm | Assignee: | Igor Mammedov <imammedo> |
| qemu-kvm sub component: | Devices | QA Contact: | Yumei Huang <yuhuang> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | high | ||
| Priority: | high | CC: | chayang, imammedo, jen, juzhang, lersek, mtessun, virt-maint, ymankad |
| Version: | 8.3 | Keywords: | Reopened |
| Target Milestone: | rc | Flags: | pm-rhel:
mirror+
|
| Target Release: | 8.3 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | qemu-kvm-5.1.0-13.module+el8.3.0+8382+afc3bbea | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-11-17 17:49:16 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1834250 | ||
Tested with win2019 guest, guest hang after hotplug 4 vcpus. Due to bug 1835330, the test always hotplug 4 vcpus to guest at same time. "qemu-kvm-5.0.0-0.module+el8.3.0+6620+5d5e1420" is not suitable for testing VCPU hotplug with OVMF; see <https://bugzilla.redhat.com/show_bug.cgi?id=1454803#c70>. (Independently, only individual device_add QMP commands should be used -- or put differently, the "virsh setvcpu" (singular) command should be used, not the "virsh setvcpus" (plural) command. But that's not why I'm closing this as NOTABUG -- the reason I'm closing this as NOTABUG is because "qemu-kvm-5.0.0-0.module+el8.3.0+6620+5d5e1420" is not usable for this test.) (In reply to Laszlo Ersek from comment #3) > "qemu-kvm-5.0.0-0.module+el8.3.0+6620+5d5e1420" is not suitable for testing > VCPU hotplug with OVMF; see > <https://bugzilla.redhat.com/show_bug.cgi?id=1454803#c70>. > > (Independently, only individual device_add QMP commands should be used -- or > put differently, the "virsh setvcpu" (singular) command should be used, not > the "virsh setvcpus" (plural) command. May I know why only singular command should be used? Is it only for ovmf? Can plural command be used for seabios? If only singular command should be used, should we disable the "virsh setvcpus" (plural) command? > But that's not why I'm closing this > as NOTABUG -- the reason I'm closing this as NOTABUG is because > "qemu-kvm-5.0.0-0.module+el8.3.0+6620+5d5e1420" is not usable for this test.) Do we have a plan to support it with qemu? It looks like the qemu I tested doesn't support it yet, instead of closing it as NOTBUG, maybe we could use this bug to track the progress, what do you think? (In reply to Yumei Huang from comment #4) > (In reply to Laszlo Ersek from comment #3) > > "qemu-kvm-5.0.0-0.module+el8.3.0+6620+5d5e1420" is not suitable for testing > > VCPU hotplug with OVMF; see > > <https://bugzilla.redhat.com/show_bug.cgi?id=1454803#c70>. > > > > (Independently, only individual device_add QMP commands should be used -- or > > put differently, the "virsh setvcpu" (singular) command should be used, not > > the "virsh setvcpus" (plural) command. > > May I know why only singular command should be used? Is it only for ovmf? > Can plural command be used for seabios? If only singular command should be > used, should we disable the "virsh setvcpus" (plural) command? Igor seems to have found the cause of this problem: """ 1. QEMU device_add 1 -> SCI 2. guest on SCI from CPU_SCAN_METHOD sends SMI 3. FW finds 1 new CPU, does its job and returns control to guest 4. QEMU device_add 2 -> SCI -> pending on ctrl_lock 5. CPU_SCAN_METHOD scans for new CPUs, finds 2 cpus, sends notifications and clears flags and exits 6. pending SCI (4) gets released lock, CPU_SCAN_METHOD sends SMI but events are already cleared """ I've proposed for libvirtd to wait for the ACPI_DEVICE_OST event (qemuProcessHandleAcpiOstInfo) in the qemuDomainSetVcpusLive() loop, in order to delay the above step 4 until after step 5 ("sends notifications" = ACPI_DEVICE_OST). We'll likely track this in a separate ticket. Until then, please only use "setvcpu" (singular). And this applies to OVMF only; SeaBIOS is not affected. Feel free to continue using "setvcpus" (plural) with SeaBIOS. > > But that's not why I'm closing this > > as NOTABUG -- the reason I'm closing this as NOTABUG is because > > "qemu-kvm-5.0.0-0.module+el8.3.0+6620+5d5e1420" is not usable for this test.) > > Do we have a plan to support it with qemu? It looks like the qemu I tested > doesn't support it yet, instead of closing it as NOTBUG, maybe we could use > this bug to track the progress, what do you think? Please see <https://bugzilla.redhat.com/show_bug.cgi?id=1454803#c73>. Thanks! (In reply to Laszlo Ersek from comment #6) > (In reply to Yumei Huang from comment #4) > > (In reply to Laszlo Ersek from comment #3) > > > "qemu-kvm-5.0.0-0.module+el8.3.0+6620+5d5e1420" is not suitable for testing > > > VCPU hotplug with OVMF; see > > > <https://bugzilla.redhat.com/show_bug.cgi?id=1454803#c70>. > > > > > > (Independently, only individual device_add QMP commands should be used -- or > > > put differently, the "virsh setvcpu" (singular) command should be used, not > > > the "virsh setvcpus" (plural) command. > > > > May I know why only singular command should be used? Is it only for ovmf? > > Can plural command be used for seabios? If only singular command should be > > used, should we disable the "virsh setvcpus" (plural) command? > > Igor seems to have found the cause of this problem: > > """ > 1. QEMU device_add 1 -> SCI > 2. guest on SCI from CPU_SCAN_METHOD sends SMI > 3. FW finds 1 new CPU, does its job and returns control to guest > 4. QEMU device_add 2 -> SCI -> pending on ctrl_lock > 5. CPU_SCAN_METHOD scans for new CPUs, finds 2 cpus, sends > notifications and clears flags and exits > 6. pending SCI (4) gets released lock, CPU_SCAN_METHOD sends SMI but > events are already cleared > """ > > I've proposed for libvirtd to wait for the ACPI_DEVICE_OST event > (qemuProcessHandleAcpiOstInfo) in the qemuDomainSetVcpusLive() loop, in > order to delay the above step 4 until after step 5 ("sends notifications" = > ACPI_DEVICE_OST). > > We'll likely track this in a separate ticket. Until then, please only use > "setvcpu" (singular). > Thank you very much for all the information. It would be appreciated if you could cc me in the ticket so we qe could keep track of the progress and adjust test strategy timely. > And this applies to OVMF only; SeaBIOS is not affected. Feel free to > continue using "setvcpus" (plural) with SeaBIOS. Thanks, for SeaBIOS, we will cover both the singular and plural commands from qemu level by testing hotplug vcpus one by one and hotplug multiple vcpus at same time. Feel free to let me know if anything changed, thanks! > > > > But that's not why I'm closing this > > > as NOTABUG -- the reason I'm closing this as NOTABUG is because > > > "qemu-kvm-5.0.0-0.module+el8.3.0+6620+5d5e1420" is not usable for this test.) > > > > Do we have a plan to support it with qemu? It looks like the qemu I tested > > doesn't support it yet, instead of closing it as NOTBUG, maybe we could use > > this bug to track the progress, what do you think? > > Please see <https://bugzilla.redhat.com/show_bug.cgi?id=1454803#c73>. > > Thanks! Igor, would you please set ITR to 8.4 since bug 1849170 has moved to 8.4? Thanks. (In reply to Yumei Huang from comment #10) > Igor, would you please set ITR to 8.4 since bug 1849170 has moved to 8.4? > Thanks. done The first two patches in Igor's series noted at <https://bugzilla.redhat.com/show_bug.cgi?id=1849170#c6> reject CPU hotplug gracefully. [RFC 0/3] x86: fix cpu hotplug with secure boot https://lists.gnu.org/archive/html/qemu-devel/2020-07/msg03746.html http://mid.mail-archive.com/20200710161704.309824-1-imammedo@redhat.com Verify:
qemu-kvm-5.1.0-13.module+el8.3.0+8382+afc3bbea
guest kernel: 4.18.0-240.el8.x86_64
host kernel: 4.18.0-240.el8.x86_64
Boot guest with ovmf firmware, and hotplug one vcpu, got error message, guest works well, no crash.
2020-10-09 04:21:56: {"execute": "device_add", "arguments": {"id": "vcpu1", "driver": "Cascadelake-Server-noTSX-x86_64-cpu", "socket-id": 1, "die-id": 0, "core-id": 9, "thread-id": 0}, "id": "bK2aEF20"}
2020-10-09 04:21:56: {"id": "bK2aEF20", "error": {"class": "GenericError", "desc": "cpu hotplug with SMI wasn't enabled by firmware"}}
Boot guest with ovmf firmware and a cpu device, hot-unplug the cpu device, got error message, guest works well, no crash.
2020-10-09 04:22:39: {"execute": "device_del", "arguments": {"id": "vcpu1"}, "id": "gknUKWGs"}
2020-10-09 04:22:39: {"id": "gknUKWGs", "error": {"class": "GenericError", "desc": "cpu hot-unplug with SMI wasn't enabled by firmware"}}
Dropping exception flag now that BZ 1886889 has been fixed, and clearing needinfos because they are no longer relevant. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (virt:8.3 bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:5137 |
Description of problem: Boot guest with ovmf, hotplug vcpu, guest hit soft lockup. Version-Release number of selected component (if applicable): qemu-kvm-5.0.0-0.module+el8.3.0+6620+5d5e1420 guest kernel: 4.18.0-211.el8.x86_64 host kernel: 4.18.0-212.el8.x86_64 edk2-ovmf-20200602gitca407c7246bf-1.el8.noarch How reproducible: always Steps to Reproduce: 1. Boot guest with ovmf 2. Hotplug 4 vcpus to guest {"execute": "device_add", "arguments": {"id": "vcpu1", "driver": "Broadwell-x86_64-cpu", "socket-id": 1, "die-id": 0, "core-id": 4, "thread-id": 0}, "id": "G9wrn582"} {"execute": "device_add", "arguments": {"id": "vcpu2", "driver": "Broadwell-x86_64-cpu", "socket-id": 1, "die-id": 0, "core-id": 5, "thread-id": 0}, "id": "1Qv9rYHg"} {"execute": "device_add", "arguments": {"id": "vcpu3", "driver": "Broadwell-x86_64-cpu", "socket-id": 1, "die-id": 0, "core-id": 6, "thread-id": 0}, "id": "WtWnvCIR"} {"execute": "device_add", "arguments": {"id": "vcpu4", "driver": "Broadwell-x86_64-cpu", "socket-id": 1, "die-id": 0, "core-id": 7, "thread-id": 0}, "id": "xOyopbHs"} Actual results: Guest hit soft lockup. Expected results: Guest works well. Additional info: 1. QEMU cli: /usr/libexec/qemu-kvm \ -S \ -name 'avocado-vt-vm1' \ -sandbox on \ -machine q35 \ -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \ -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0 \ -nodefaults \ -device VGA,bus=pcie.0,addr=0x2 \ -m 8192 \ -smp 12,maxcpus=16,cores=8,threads=1,dies=1,sockets=2 \ -cpu 'Broadwell',+kvm_pv_unhalt \ -chardev socket,path=/var/tmp/avocado_vpasgpc7/monitor-qmpmonitor1-20200615-035414-DWS1VFbF,id=qmp_id_qmpmonitor1,server,nowait \ -mon chardev=qmp_id_qmpmonitor1,mode=control \ -chardev socket,path=/var/tmp/avocado_vpasgpc7/monitor-catch_monitor-20200615-035414-DWS1VFbF,id=qmp_id_catch_monitor,server,nowait \ -mon chardev=qmp_id_catch_monitor,mode=control \ -device pvpanic,ioport=0x505,id=idsQbQiW \ -chardev socket,path=/var/tmp/avocado_vpasgpc7/serial-serial0-20200615-035414-DWS1VFbF,id=chardev_serial0,server,nowait \ -device isa-serial,id=serial0,chardev=chardev_serial0 \ -chardev socket,id=seabioslog_id_20200615-035414-DWS1VFbF,path=/var/tmp/avocado_vpasgpc7/seabios-20200615-035414-DWS1VFbF,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20200615-035414-DWS1VFbF,iobase=0x402 \ -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \ -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \ -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \ -blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \ -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \ -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \ -device virtio-net-pci,mac=9a:3a:88:f6:23:5f,id=id2wATmi,netdev=id0zh0Kj,bus=pcie-root-port-3,addr=0x0 \ -netdev tap,id=id0zh0Kj,vhost=on,vhostfd=20,fd=14 \ -vnc :0 \ -rtc base=utc,clock=host,driftfix=slew \ -boot menu=off,order=cdn,once=c,strict=off \ -blockdev node-name=file_ovmf_code,driver=file,read-only=on,filename=/usr/share/OVMF/OVMF_CODE.secboot.fd \ -blockdev node-name=file_ovmf_vars,driver=file,filename=/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2.fd \ -machine pflash0=file_ovmf_code,pflash1=file_ovmf_vars \ -enable-kvm \ -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5 2. Guest dmesg: 2020-06-15 03:55:49: [ 74.690301] CPU12 has been hot-added 2020-06-15 03:55:49: [ 74.691984] CPU13 has been hot-added 2020-06-15 03:55:49: [ 74.693201] CPU14 has been hot-added 2020-06-15 03:55:49: [ 74.693984] CPU15 has been hot-added 2020-06-15 03:55:49: [ 74.702706] smpboot: Booting Node 0 Processor 12 APIC 0xc 2020-06-15 03:55:49: [ 74.704243] kvm-clock: cpu 12, msr 1a7601301, secondary cpu clock 2020-06-15 03:55:49: [ 74.706548] KVM setup async PF for cpu 12 2020-06-15 03:55:49: [ 74.708120] kvm-stealtime: cpu 12, msr 277b2c080 2020-06-15 03:55:49: [ 74.709657] Will online and init hotplugged CPU: 12 2020-06-15 03:55:49: [ 74.734687] smpboot: Booting Node 0 Processor 13 APIC 0xd 2020-06-15 03:55:49: [ 74.735706] kvm-clock: cpu 13, msr 1a7601341, secondary cpu clock 2020-06-15 03:55:49: [ 74.737947] KVM setup async PF for cpu 13 2020-06-15 03:55:49: [ 74.739679] kvm-stealtime: cpu 13, msr 277b6c080 2020-06-15 03:55:49: [ 74.740716] Will online and init hotplugged CPU: 13 2020-06-15 03:55:49: [ 74.765638] smpboot: Booting Node 0 Processor 14 APIC 0xe 2020-06-15 03:55:49: [ 74.766626] kvm-clock: cpu 14, msr 1a7601381, secondary cpu clock 2020-06-15 03:55:49: [ 74.768884] KVM setup async PF for cpu 14 2020-06-15 03:55:49: [ 74.770455] kvm-stealtime: cpu 14, msr 277bac080 2020-06-15 03:55:49: [ 74.771532] Will online and init hotplugged CPU: 14 2020-06-15 03:55:49: [ 74.782748] smpboot: Booting Node 0 Processor 15 APIC 0xf 2020-06-15 03:55:49: [ 74.783774] kvm-clock: cpu 15, msr 1a76013c1, secondary cpu clock 2020-06-15 03:55:49: [ 74.786130] KVM setup async PF for cpu 15 2020-06-15 03:55:49: [ 74.787700] kvm-stealtime: cpu 15, msr 277bec080 2020-06-15 03:55:49: [ 74.788632] Will online and init hotplugged CPU: 15 2020-06-15 03:56:25: [ 111.262848] watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [kworker/7:2:673] 2020-06-15 03:56:25: [ 111.264538] Modules linked in: fuse xt_CHECKSUM ipt_MASQUERADE xt_conntrack ipt_REJECT nf_nat_tftp nft_objref nf_conntrack_tftp nft_counter tun bridge stp llc nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nf_tables_set nft_chain_nat_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nft_chain_route_ipv6 nft_chain_nat_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack nft_chain_route_ipv4 ip6_tables nft_compat ip_set nf_tables nfnetlink rpcrdma ib_isert iscsi_target_mod ib_iser ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_umad iw_cxgb4 ib_uverbs rdma_cm iw_cm ib_cm ib_core vfat fat bochs_drm drm_vram_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm iTCO_wdt intel_rapl_msr iTCO_vendor_support intel_rapl_common joydev lpc_ich i2c_i801 pcspkr ip_tables xfs libcrc32c sd_mod sg crct10dif_pclmul crc32_pclmul crc32c_intel virtio_net ghash_clmulni_intel 2020-06-15 03:56:25: [ 111.264589] net_failover ahci serio_raw failover libahci libata virtio_scsi dm_multipath sunrpc dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic uio cxgb4i cxgb4 libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi 2020-06-15 03:56:25: [ 111.281093] CPU: 7 PID: 673 Comm: kworker/7:2 Kdump: loaded Not tainted 4.18.0-211.el8.x86_64 #1 2020-06-15 03:56:25: [ 111.282418] Hardware name: Red Hat KVM/RHEL-AV, BIOS 0.0.0 02/06/2015 2020-06-15 03:56:25: [ 111.283409] Workqueue: events bpf_prog_free_deferred 2020-06-15 03:56:25: [ 111.284170] RIP: 0010:smp_call_function_many+0x1ec/0x250 2020-06-15 03:56:25: [ 111.284982] Code: c7 e8 18 21 74 00 3b 05 66 d1 2b 01 0f 83 99 fe ff ff 48 63 d0 48 8b 0b 48 03 0c d5 60 38 f5 90 8b 51 18 83 e2 01 74 0a f3 90 <8b> 51 18 83 e2 01 75 f6 eb c7 48 c7 c2 00 12 21 91 4c 89 f6 89 df 2020-06-15 03:56:25: [ 111.287802] RSP: 0018:ffffb1bf015c7d48 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff13 2020-06-15 03:56:25: [ 111.288946] RAX: 000000000000000c RBX: ffff9011f79eae00 RCX: ffff9011f7b31620 2020-06-15 03:56:25: [ 111.290021] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff901087c41210 2020-06-15 03:56:25: [ 111.291102] RBP: ffffffff8fe72860 R08: 000000000002f040 R09: ffffffff8fe4f25a 2020-06-15 03:56:25: [ 111.292182] R10: ffffef26041a7580 R11: 0000000000000000 R12: 0000000000000000 2020-06-15 03:56:25: [ 111.293261] R13: 0000000000000001 R14: 0000000000000010 R15: 0000000000000001 2020-06-15 03:56:25: [ 111.294363] FS: 0000000000000000(0000) GS:ffff9011f79c0000(0000) knlGS:0000000000000000 2020-06-15 03:56:25: [ 111.295598] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2020-06-15 03:56:25: [ 111.296474] CR2: 00005607dc59de70 CR3: 00000001a600a002 CR4: 00000000003606e0 2020-06-15 03:56:25: [ 111.297560] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 2020-06-15 03:56:25: [ 111.298650] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 2020-06-15 03:56:25: [ 111.299735] Call Trace: 2020-06-15 03:56:25: [ 111.300140] ? load_new_mm_cr3+0xe0/0xe0 2020-06-15 03:56:25: [ 111.300748] on_each_cpu+0x28/0x60 2020-06-15 03:56:25: [ 111.301274] flush_tlb_kernel_range+0x48/0x90 2020-06-15 03:56:25: [ 111.301945] __purge_vmap_area_lazy+0x79/0x150 2020-06-15 03:56:25: [ 111.302627] _vm_unmap_aliases+0xe5/0x120 2020-06-15 03:56:25: [ 111.303239] __vunmap+0x126/0x220 2020-06-15 03:56:25: [ 111.303753] bpf_jit_binary_free+0xd/0x20 2020-06-15 03:56:25: [ 111.304367] bpf_jit_free+0x28/0x60 2020-06-15 03:56:25: [ 111.304908] process_one_work+0x1a7/0x360 2020-06-15 03:56:25: [ 111.305525] worker_thread+0x30/0x390 2020-06-15 03:56:25: [ 111.306086] ? create_worker+0x1a0/0x1a0 2020-06-15 03:56:25: [ 111.306691] kthread+0x112/0x130 2020-06-15 03:56:25: [ 111.307189] ? kthread_flush_work_fn+0x10/0x10 2020-06-15 03:56:25: [ 111.307874] ret_from_fork+0x35/0x40