Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2203821

Summary: L2 kernel crash after hotplugging a pc-dimm device to L2 guest on a AMD EPYC-Naples L0 host
Product: Red Hat Enterprise Linux 9 Reporter: yduan
Component: qemu-kvmAssignee: Virtualization Maintenance <virt-maint>
qemu-kvm sub component: General QA Contact: Mario Casquero <mcasquer>
Status: CLOSED DUPLICATE Docs Contact:
Severity: high    
Priority: unspecified CC: chayang, coli, imammedo, jinzhao, juzhang, virt-maint
Version: 9.3Flags: pm-rhel: mirror+
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-05-17 15:18:03 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description yduan 2023-05-15 10:56:52 UTC
Description of problem:
L2 kernel crash after hotplugging a pc-dimm device to L2 guest on a AMD EPYC-Naples L0 host

Version-Release number of selected component (if applicable):
L0 host:
5.14.0-306.el9.x86_64
qemu-kvm-8.0.0-1.el9.x86_64
hp-dl385g10-02.lab.eng.pek2.redhat.com
Model name:            AMD EPYC 7251 8-Core Processor
L1 VM:
5.14.0-306.el9.x86_64
qemu-kvm-8.0.0-1.el9.x86_64
L2 VM:
5.14.0-306.el9.x86_64

How reproducible:
2/2

Steps to Reproduce:
1.Boot a L1 VM on L0 host:
...
 -machine q35,memory-backend=mem-machine_mem,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
 -m 14336 \
 -object '{"size": 15032385536, "id": "mem-machine_mem", "qom-type": "memory-backend-ram"}'  \
 -smp 16,maxcpus=16,cores=8,threads=1,dies=1,sockets=2  \
 -cpu 'host',x2apic=on,tsc-deadline=on,hypervisor=on,tsc-adjust=on,arch-capabilities=on,xsaves=on,cmp-legacy=on,perfctr-core=on,clzero=on,xsaveerptr=on,virt-ssbd=on,npt=on,lbrv=on,nrip-save=on,tsc-scale=on,vmcb-clean=on,pause-filter=on,pfthreshold=on,v-vmsave-vmload=on,vgif=on,svme-addr-chk=on,rdctl-no=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on,monitor=off,svm=on,kvm_pv_unhalt=on \
...
2.On L1:
$ python3 ConfigTest.py --testcase=hotplug_memory.after.guest_reboot.hotplug.backend_file.policy_default.two --platform=x86_64 --guestname=RHEL.9.3.0 --clone=no
...
[stdlog]     -m 4096,maxmem=32G,slots=4 \
[stdlog]     -object '{"size": 1073741824, "mem-path": "/mnt/kvm_hugepage", "policy": "default", "id": "mem-mem1", "qom-type": "memory-backend-file"}' \
[stdlog]     -device '{"node": 1, "driver": "pc-dimm", "id": "dimm-mem1", "memdev": "mem-mem1"}' \
[stdlog]     -object '{"size": 1073741824, "mem-path": "/mnt/kvm_hugepage", "policy": "default", "id": "mem-mem2", "qom-type": "memory-backend-file"}' \
[stdlog]     -device '{"node": 0, "driver": "pc-dimm", "id": "dimm-mem2", "memdev": "mem-mem2"}' \
[stdlog]     -object '{"size": 2147483648, "policy": "default", "id": "mem-memN0", "qom-type": "memory-backend-ram"}' \
[stdlog]     -object '{"size": 2147483648, "policy": "default", "id": "mem-memN1", "qom-type": "memory-backend-ram"}'  \
[stdlog]     -smp 8,maxcpus=8,cores=4,threads=1,dies=1,sockets=2  \
[stdlog]     -numa node,memdev=mem-memN0,nodeid=0  \
[stdlog]     -numa node,memdev=mem-memN1,nodeid=1  \
[stdlog]     -cpu 'EPYC-IBPB',x2apic=on,tsc-deadline=on,hypervisor=on,tsc-adjust=on,arch-capabilities=on,xsaves=on,cmp-legacy=on,perfctr-core=on,clzero=on,xsaveerptr=on,virt-ssbd=on,npt=on,lbrv=on,nrip-save=on,tsc-scale=on,vmcb-clean=on,pause-filter=on,pfthreshold=on,v-vmsave-vmload=on,vgif=on,svme-addr-chk=on,rdctl-no=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on,monitor=off,kvm_pv_unhalt=on \
...

Actual results:
[stdlog] 2023-05-15 15:00:05,522 avocado.virttest.utils_test.qemu INFO | Context: (run) --> Hotplug pc-dimm 'dimm-plug1' to VM
[stdlog] 2023-05-15 15:00:05,523 avocado.virttest.qemu_monitor DEBUG| (monitor avocado-vt-vm1.qmpmonitor1) Sending command 'device_add'
[stdlog] 2023-05-15 15:00:05,523 avocado.virttest.qemu_monitor DEBUG| Send command: {'execute': 'device_add', 'arguments': OrderedDict([('node', '0'), ('driver', 'pc-dimm'), ('id', 'dimm-plug1'), ('memdev', 'mem-plug1')]), 'id': 'Pau1Xxaf'}
[stdlog] 2023-05-15 15:00:05,581 avocado.virttest.utils_test.qemu INFO | Context: (run) --> Verify memory info
[stdlog] 2023-05-15 15:00:05,589 avocado.virttest.virt_vm DEBUG| ----------[ cut here ]------------
[stdlog] [    0.997028] sched: CPU #4's llc-sibling CPU #0 is not on the same node! [node: 1 != 0]. Ignoring dependency.
[stdlog] [    0.997028] WARNING: CPU: 4 PID: 0 at arch/x86/kernel/smpboot.c:424 topology_sane.isra.0+0x6b/0x80
[stdlog] [    0.997028] Modules linked in:
[stdlog] [    0.997028] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 5.14.0-306.el9.x86_64 #1
[stdlog] [    0.997028] Hardware name: Red Hat KVM/RHEL, BIOS edk2-20230301gitf80f052277c8-2.el9 03/01/2023
[stdlog] [    0.997028] RIP: 0010:topology_sane.isra.0+0x6b/0x80
[stdlog] [    0.997028] Code: 80 3d 41 4e 0c 02 00 75 f2 48 83 ec 08 4c 89 da 44 89 d6 48 c7 c7 48 03 7f a1 88 44 24 07 c6 05 23 4e 0c 02 01 e8 25 7e 09 00 <0f> 0b 0f b6 44 24 07 48 83 c4 08 e9 d5 1f da 00 0f 1f 44 00 00 0f
[stdlog] [    0.997028] RSP: 0000:ffffb2e48010fec8 EFLAGS: 00010086
[stdlog] [    0.997028] RAX: 0000000000000000 RBX: 0000000000000004 RCX: 0000000000000027
[stdlog] [    0.997028] RDX: 0000000000000027 RSI: ffffffffa2067380 RDI: ffff8d153dc1f8e8
[stdlog] [    0.997028] RBP: ffff8d153dc198c0 R08: ffff8d153dc1f8e0 R09: 0000000000000000
[stdlog] [    0.997028] R10: ffffffffffffffff R11: ffffffffa2d40771 R12: 0000000000000000
[stdlog] [    0.997028] R13: 0000000000000000 R14: ffff8d141b0198c0 R15: 0000000000000000
[stdlog] [    0.997028] FS:  0000000000000000(0000) GS:ffff8d153dc00000(0000) knlGS:0000000000000000
[stdlog] [    0.997028] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[stdlog] [    0.997028] CR2: 0000000000000000 CR3: 0000000106810000 CR4: 00000000003506e0
[stdlog] [    0.997028] Call Trace:
[stdlog] [    0.997028]  <TASK>
[stdlog] [    0.997028]  set_cpu_sibling_map+0x18e/0x600
[stdlog] [    0.997028]  start_secondary+0x5b/0x140
[stdlog] [    0.997028]  secondary_startup_64_no_verify+0xe5/0xeb
[stdlog] [    0.997028]  </TASK>
[stdlog] [    0.997028] ---[ end trace 6b8b2c6c3f12349e ]---
[stdlog] 2023-05-15 15:00:05,594 avocado.test ERROR|
[stdlog] 2023-05-15 15:00:05,595 avocado.test ERROR| Reproduced traceback from: /home/kar/workspace/avocado-vt/avocado_vt/test.py:274
[stdlog] 2023-05-15 15:00:05,703 avocado.test ERROR| Traceback (most recent call last):
[stdlog] 2023-05-15 15:00:05,703 avocado.test ERROR|   File "/home/kar/workspace/avocado-vt/virttest/error_context.py", line 135, in new_fn
[stdlog] 2023-05-15 15:00:05,703 avocado.test ERROR|     return fn(*args, **kwargs)
[stdlog] 2023-05-15 15:00:05,703 avocado.test ERROR|   File "/root/avocado/data/avocado-vt/virttest/test-providers.d/downloads/io-github-autotest-qemu/qemu/tests/hotplug_mem.py", line 133, in run
[stdlog] 2023-05-15 15:00:05,703 avocado.test ERROR|     hotplug_test.start_test()
[stdlog] 2023-05-15 15:00:05,703 avocado.test ERROR|   File "/root/avocado/data/avocado-vt/virttest/test-providers.d/downloads/io-github-autotest-qemu/qemu/tests/hotplug_mem.py", line 99, in start_test
[stdlog] 2023-05-15 15:00:05,703 avocado.test ERROR|     self.check_memory(vm)
[stdlog] 2023-05-15 15:00:05,703 avocado.test ERROR|   File "/home/kar/workspace/avocado-vt/virttest/error_context.py", line 135, in new_fn
[stdlog] 2023-05-15 15:00:05,703 avocado.test ERROR|     return fn(*args, **kwargs)
[stdlog] 2023-05-15 15:00:05,703 avocado.test ERROR|   File "/home/kar/workspace/avocado-vt/virttest/utils_test/qemu/__init__.py", line 724, in check_memory
[stdlog] 2023-05-15 15:00:05,703 avocado.test ERROR|     vm.verify_alive()
[stdlog] 2023-05-15 15:00:05,704 avocado.test ERROR|   File "/home/kar/workspace/avocado-vt/virttest/qemu_vm.py", line 298, in verify_alive
[stdlog] 2023-05-15 15:00:05,704 avocado.test ERROR|     self.verify_kernel_crash()
[stdlog] 2023-05-15 15:00:05,704 avocado.test ERROR|   File "/home/kar/workspace/avocado-vt/virttest/virt_vm.py", line 1005, in verify_kernel_crash
[stdlog] 2023-05-15 15:00:05,704 avocado.test ERROR|     raise VMDeadKernelCrashError(match.group(0))
[stdlog] 2023-05-15 15:00:05,704 avocado.test ERROR| virttest.virt_vm.VMDeadKernelCrashError: VM is dead due to a kernel crash, see debug/serial log for details
Expected results:
no kernel crash
Additional info:
This issue cannot be reproduced on L1 VM.

Comment 1 Mario Casquero 2023-05-15 11:15:36 UTC
This is the same bug as https://bugzilla.redhat.com/show_bug.cgi?id=2176010

Comment 2 John Ferlan 2023-05-16 14:00:13 UTC
Igor - do you concur with Mario here. If so, then please close duplicate to your bug.

Comment 3 Igor Mammedov 2023-05-17 15:18:03 UTC
(In reply to John Ferlan from comment #2)
> Igor - do you concur with Mario here. If so, then please close duplicate to
> your bug.

Yes, it's the same.
It's not a crash, just a warning and it's not related to memory hotplug.

*** This bug has been marked as a duplicate of bug 2176010 ***