Bug 2224509
| Summary: | During hotplug 416 vcpus hit "watchdog: BUG: soft lockup - CPU#9 stuck for 44s! [swapper/9:0]" issues | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | Yiqian Wei <yiwei> |
| Component: | kernel | Assignee: | Igor Mammedov <imammedo> |
| kernel sub component: | KVM | QA Contact: | Yiqian Wei <yiwei> |
| Status: | CLOSED MIGRATED | Docs Contact: | |
| Severity: | unspecified | ||
| Priority: | unspecified | CC: | akoutsou, atodorov, coli, imammedo, jen, jinzhao, nanliu, nilal, virt-maint, vkuznets, xuwei |
| Version: | 9.3 | Keywords: | MigratedToJIRA |
| Target Milestone: | rc | Flags: | pm-rhel:
mirror+
|
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2024-06-03 13:17:35 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 2118240 | ||
| Bug Blocks: | |||
|
Description
Yiqian Wei
2023-07-21 09:25:41 UTC
Can reproduce this bug with qemu-kvm-8.0.0-10.el9.x86_64 Reproduce steps: Boot a q35 + ovmf guest with 448 vcpus host version: qemu-kvm-8.0.0-10.el9.x86_64 kernel-5.14.0-349.el9.x86_64 edk2-ovmf-20230524-2.el9.noarch guest: rhel9.3.0 Yiqian Wei, what on host are running this test? (I'm interested in lscpu command output) and also lscpu from guest (I see mwait_idle in trace but don't see in script an option that enables mwait feature) Hi Yiqian, I see nothing suspicious about host. Lets try to narrow down possible places to look at. Can you try to following variants (only one thing at a time): 1. try find maxcpus at which problem starts to appear 2. try with 'idle=nomwait' in guest's kernel command line 3. try older (9.2) host/guest kernels 4. try with older QEMU (from RHEL.9.2) Hi Yiqian, Like Vitaly said bisection is the next logical step to finding out the offending patch. Can you lend me machine you reproduce this issue on? (In reply to Igor Mammedov from comment #14) > Hi Yiqian, > > I see nothing suspicious about host. > Lets try to narrow down possible places to look at. > > Can you try to following variants (only one thing at a time): > 1. try find maxcpus at which problem starts to appear It is not possible to determine the specific cpu, just the process of hotplug 416 vcpus. > 2. try with 'idle=nomwait' in guest's kernel command line With 'idle=nowait' in guest's kernel, can reproduce this bug with hotplug 416 vcpus. > 3. try older (9.2) host/guest kernels NOT reproduce this bug when hotplug 416 vcpus with kernel-5.14.0-284.28.1.el9_2. x86_64, but hit Bug 2118240 when reboot after hotplug 416 vcpus. > 4. try with older QEMU (from RHEL.9.2) Can reproduce this bug with qemu-kvm-7.2.0-14.el9_2.4.x86_64 due to Bug 2118240 it's hard to say is definitely if it's a separate issue. Lets retest once above dependency is fixed. Igor - This bug is in a strange state - if ON_QA I'd expect there to be an ITR or FixVersion set or the TestOnly keyword being set. My concern / fear especially because of the Jira transition is that this will be lost. Can you elaborate a bit more or perhaps connect to https://issues.redhat.com/browse/RHEL-7173 (new Jira issue from the previous bug 2118240). Maybe we just need to migrate this bug manually (hint, use the "MigratedToJira" keyword - a bot will do the rest, then link that to the other Jira issue). |