Bug 2157958
| Summary: | kcompactd0 runs indefinitely at 100% CPU with inter.ference from VMware | ||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | David Nadle <david> | ||||||||||||||||||||
| Component: | kernel | Assignee: | Rafael Aquini <aquini> | ||||||||||||||||||||
| kernel sub component: | Memory Management | QA Contact: | Ping Fang <pifang> | ||||||||||||||||||||
| Status: | CLOSED CURRENTRELEASE | Docs Contact: | |||||||||||||||||||||
| Severity: | medium | ||||||||||||||||||||||
| Priority: | unspecified | CC: | aquini, boyang, bstinson, chuhu, jwboyer, ldu, liwan, llong, pm-rhel, shshang, yacao | ||||||||||||||||||||
| Version: | CentOS Stream | Keywords: | TestOnly, Triaged | ||||||||||||||||||||
| Target Milestone: | rc | ||||||||||||||||||||||
| Target Release: | --- | ||||||||||||||||||||||
| Hardware: | x86_64 | ||||||||||||||||||||||
| OS: | Linux | ||||||||||||||||||||||
| Whiteboard: | |||||||||||||||||||||||
| Fixed In Version: | kernel-5.14.0-264.el9 | Doc Type: | No Doc Update | ||||||||||||||||||||
| Doc Text: | Story Points: | --- | |||||||||||||||||||||
| Clone Of: | Environment: | ||||||||||||||||||||||
| Last Closed: | 2023-02-21 19:50:12 UTC | Type: | Bug | ||||||||||||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||||||||||||
| Documentation: | --- | CRM: | |||||||||||||||||||||
| Verified Versions: | Category: | --- | |||||||||||||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||||||||||
| Embargoed: | |||||||||||||||||||||||
| Bug Depends On: | 2166447 | ||||||||||||||||||||||
| Bug Blocks: | |||||||||||||||||||||||
| Attachments: |
|
||||||||||||||||||||||
|
Description
David Nadle
2023-01-03 17:44:43 UTC
We don't have VMware Workstation Pro and a Windows 11 VM guest on hand to reproduce the problem. Would you mind running the perf-record command to record a trace of the 100% cpu processes and then attach the perf-report output of them so that we have a better idea of what the kernel is doing to consume all those CPU cycles? It would be nice to have two sets with and without the --call-graph perf-record option. David, When kcompactd0 and vmware-vmx processes are seen to go on burning CPU at 100%, please capture some snapshots of their stacks and attach them here. The following bash one-liner can do the trick: # for n in $(seq 1 10); do echo $n; cat /proc/$(pidof -w kcompactd0)/stack; echo ""; cat /proc/$(pidof -w vmware-vmx)/stack; echo ""; sleep 1; done -- Rafael David, While you are at the data collection (comment #3), please, also grab a snapshot of these files: /proc/modules /proc/sys/kernel/tainted Thanks in advance! -- Rafael kcompactd0 is pid 144 vmware-vmx is pid 81390 [dnadle@dln perf]$ sudo cat /proc/144/stack [dnadle@dln perf]$ sudo cat /proc/81390/stack [<0>] do_poll.constprop.0+0x225/0x390 [<0>] do_sys_poll+0x1c8/0x260 [<0>] __x64_sys_ppoll+0xbc/0x150 [<0>] do_syscall_64+0x59/0x90 [<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd [dnadle@dln perf]$ sudo cat /proc/modules dm_zero 16384 4 - Live 0xffffffffc0902000 vsock_loopback 16384 0 - Live 0xffffffffc07d4000 vmw_vsock_virtio_transport_common 49152 1 vsock_loopback, Live 0xffffffffc0d81000 vmw_vsock_vmci_transport 40960 0 - Live 0xffffffffc0884000 vsock 53248 3 vsock_loopback,vmw_vsock_virtio_transport_common,vmw_vsock_vmci_transport, Live 0xffffffffc07f3000 tls 118784 0 - Live 0xffffffffc0866000 binfmt_misc 28672 1 - Live 0xffffffffc0780000 snd_seq_dummy 16384 0 - Live 0xffffffffc0772000 snd_hrtimer 16384 1 - Live 0xffffffffc0742000 rfcomm 94208 4 - Live 0xffffffffc0751000 nft_objref 16384 1 - Live 0xffffffffc0732000 nf_conntrack_netbios_ns 16384 1 - Live 0xffffffffc072d000 nf_conntrack_broadcast 16384 1 nf_conntrack_netbios_ns, Live 0xffffffffc040b000 nft_fib_inet 16384 1 - Live 0xffffffffc03b6000 nft_fib_ipv4 16384 1 nft_fib_inet, Live 0xffffffffc038a000 nft_fib_ipv6 16384 1 nft_fib_inet, Live 0xffffffffc0385000 nft_fib 16384 3 nft_fib_inet,nft_fib_ipv4,nft_fib_ipv6, Live 0xffffffffc0380000 nft_reject_inet 16384 6 - Live 0xffffffffc037b000 nf_reject_ipv4 16384 1 nft_reject_inet, Live 0xffffffffc0369000 nf_reject_ipv6 20480 1 nft_reject_inet, Live 0xffffffffc034d000 nft_reject 16384 1 nft_reject_inet, Live 0xffffffffc0342000 nft_ct 24576 15 - Live 0xffffffffc0362000 nft_chain_nat 16384 3 - Live 0xffffffffc0358000 nf_nat 57344 1 nft_chain_nat, Live 0xffffffffc039f000 nf_conntrack 180224 4 nf_conntrack_netbios_ns,nf_conntrack_broadcast,nft_ct,nf_nat, Live 0xffffffffc078d000 nf_defrag_ipv6 24576 1 nf_conntrack, Live 0xffffffffc031d000 nf_defrag_ipv4 16384 1 nf_conntrack, Live 0xffffffffc02d8000 ip_set 61440 0 - Live 0xffffffffc02b0000 vmw_vmci 118784 4 vmw_vsock_vmci_transport, Live 0xffffffffc070f000 vmnet 73728 17 - Live 0xffffffffc032f000 (OE) vmmon 147456 3 - Live 0xffffffffc02f8000 (OE) nf_tables 270336 221 nft_objref,nft_fib_inet,nft_fib_ipv4,nft_fib_ipv6,nft_fib,nft_reject_inet,nft_reject,nft_ct,nft_chain_nat, Live 0xffffffffc081a000 nfnetlink 20480 3 ip_set,nf_tables, Live 0xffffffffc027a000 qrtr 53248 4 - Live 0xffffffffc026c000 bnep 32768 2 - Live 0xffffffffc022a000 sunrpc 696320 1 - Live 0xffffffffc3cee000 dm_crypt 61440 5 - Live 0xffffffffc0252000 vfat 20480 1 - Live 0xffffffffc01e7000 fat 86016 1 vfat, Live 0xffffffffc0214000 squashfs 77824 19 - Live 0xffffffffc0237000 loop 36864 38 - Live 0xffffffffc01dd000 intel_rapl_msr 20480 0 - Live 0xffffffffc024c000 iTCO_wdt 16384 0 - Live 0xffffffffc07cf000 iTCO_vendor_support 16384 1 iTCO_wdt, Live 0xffffffffc07ca000 i2c_designware_platform 16384 0 - Live 0xffffffffc07d9000 i2c_designware_core 36864 1 i2c_designware_platform, Live 0xffffffffc0747000 pmt_telemetry 16384 0 - Live 0xffffffffc020f000 pmt_class 16384 1 pmt_telemetry, Live 0xffffffffc0201000 intel_rapl_common 32768 1 intel_rapl_msr, Live 0xffffffffc02a7000 x86_pkg_temp_thermal 20480 0 - Live 0xffffffffc0769000 intel_powerclamp 20480 0 - Live 0xffffffffc0cc2000 coretemp 20480 0 - Live 0xffffffffc02e4000 kvm_intel 409600 0 - Live 0xffffffffc3c89000 kvm 1130496 1 kvm_intel, Live 0xffffffffc3b74000 irqbypass 16384 1 kvm, Live 0xffffffffc0dbd000 rapl 24576 0 - Live 0xffffffffc3b6d000 intel_cstate 24576 0 - Live 0xffffffffc02dd000 intel_uncore 225280 0 - Live 0xffffffffc3b35000 pcspkr 16384 0 - Live 0xffffffffc0de3000 eeepc_wmi 16384 0 - Live 0xffffffffc0dc3000 asus_wmi 53248 1 eeepc_wmi, Live 0xffffffffc3b02000 sparse_keymap 16384 1 asus_wmi, Live 0xffffffffc07e6000 wmi_bmof 16384 0 - Live 0xffffffffc07e1000 nvidia_drm 73728 12 - Live 0xffffffffc3b22000 (POE) nvidia_modeset 1155072 16 nvidia_drm, Live 0xffffffffc39e7000 (POE) nvidia_uvm 2777088 0 - Live 0xffffffffc3740000 (POE) nvidia 40886272 1055 nvidia_modeset,nvidia_uvm, Live 0xffffffffc1041000 (POE) iwlmvm 475136 0 - Live 0xffffffffc0fcc000 mac80211 1167360 1 iwlmvm, Live 0xffffffffc0eae000 snd_sof_pci_intel_tgl 16384 0 - Live 0xffffffffc0cf6000 snd_sof_intel_hda_common 147456 1 snd_sof_pci_intel_tgl, Live 0xffffffffc0e89000 soundwire_intel 49152 1 snd_sof_intel_hda_common, Live 0xffffffffc0d74000 soundwire_generic_allocation 16384 1 soundwire_intel, Live 0xffffffffc0cec000 soundwire_cadence 40960 1 soundwire_intel, Live 0xffffffffc0ce1000 snd_sof_intel_hda 20480 1 snd_sof_intel_hda_common, Live 0xffffffffc0cdb000 snd_sof_pci 24576 2 snd_sof_pci_intel_tgl,snd_sof_intel_hda_common, Live 0xffffffffc0cd4000 libarc4 16384 1 mac80211, Live 0xffffffffc08fd000 snd_sof_xtensa_dsp 16384 1 snd_sof_intel_hda_common, Live 0xffffffffc0788000 snd_sof 249856 2 snd_sof_intel_hda_common,snd_sof_pci, Live 0xffffffffc0e4b000 snd_sof_utils 20480 1 snd_sof, Live 0xffffffffc0b58000 snd_soc_hdac_hda 24576 1 snd_sof_intel_hda_common, Live 0xffffffffc0a64000 snd_hda_ext_core 36864 3 snd_sof_intel_hda_common,snd_sof_intel_hda,snd_soc_hdac_hda, Live 0xffffffffc0b98000 snd_soc_acpi_intel_match 69632 2 snd_sof_pci_intel_tgl,snd_sof_intel_hda_common, Live 0xffffffffc08eb000 snd_soc_acpi 16384 2 snd_sof_intel_hda_common,snd_soc_acpi_intel_match, Live 0xffffffffc03d6000 iwlwifi 385024 1 iwlmvm, Live 0xffffffffc0dec000 soundwire_bus 98304 3 soundwire_intel,soundwire_generic_allocation,soundwire_cadence, Live 0xffffffffc0dca000 snd_hda_codec_realtek 172032 1 - Live 0xffffffffc0d92000 snd_soc_core 344064 4 snd_sof_intel_hda_common,soundwire_intel,snd_sof,snd_soc_hdac_hda, Live 0xffffffffc0d1f000 snd_hda_codec_generic 98304 1 snd_hda_codec_realtek, Live 0xffffffffc0cfd000 i2c_i801 36864 0 - Live 0xffffffffc08e1000 mei_me 49152 0 - Live 0xffffffffc08d4000 i2c_smbus 20480 1 i2c_i801, Live 0xffffffffc0347000 cfg80211 1052672 3 iwlmvm,mac80211,iwlwifi, Live 0xffffffffc0bc0000 snd_compress 28672 1 snd_soc_core, Live 0xffffffffc0bb8000 ledtrig_audio 16384 1 snd_hda_codec_generic, Live 0xffffffffc09f8000 snd_hda_codec_hdmi 77824 1 - Live 0xffffffffc0ba4000 mei 155648 1 mei_me, Live 0xffffffffc0b71000 btusb 65536 0 - Live 0xffffffffc0b60000 btrtl 28672 1 btusb, Live 0xffffffffc09f0000 btbcm 20480 1 btusb, Live 0xffffffffc091c000 snd_hda_intel 57344 6 - Live 0xffffffffc0b40000 btintel 49152 1 btusb, Live 0xffffffffc09b4000 btmtk 16384 1 btusb, Live 0xffffffffc0917000 snd_intel_dspcfg 32768 3 snd_sof_intel_hda_common,snd_sof,snd_hda_intel, Live 0xffffffffc0908000 intel_lpss_pci 28672 0 - Live 0xffffffffc0327000 snd_intel_sdw_acpi 20480 2 snd_sof_intel_hda_common,snd_intel_dspcfg, Live 0xffffffffc0860000 intel_lpss 16384 1 intel_lpss_pci, Live 0xffffffffc0289000 idma64 20480 0 - Live 0xffffffffc0283000 bluetooth 864256 36 rfcomm,bnep,btusb,btrtl,btbcm,btintel,btmtk, Live 0xffffffffc0a6c000 snd_hda_codec 176128 5 snd_soc_hdac_hda,snd_hda_codec_realtek,snd_hda_codec_generic,snd_hda_codec_hdmi,snd_hda_intel, Live 0xffffffffc0a2f000 uas 32768 1 - Live 0xffffffffc0706000 usb_storage 81920 1 uas, Live 0xffffffffc0a1a000 snd_hda_core 110592 9 snd_sof_intel_hda_common,snd_sof_intel_hda,snd_soc_hdac_hda,snd_hda_ext_core,snd_hda_codec_realtek,snd_hda_codec_generic,snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec, Live 0xffffffffc09fe000 snd_hwdep 16384 1 snd_hda_codec, Live 0xffffffffc02f0000 snd_seq 94208 7 snd_seq_dummy, Live 0xffffffffc092c000 snd_seq_device 16384 1 snd_seq, Live 0xffffffffc02eb000 rfkill 36864 8 asus_wmi,iwlmvm,cfg80211,bluetooth, Live 0xffffffffc0922000 snd_pcm 151552 11 snd_sof_intel_hda_common,soundwire_intel,snd_sof,snd_sof_utils,snd_soc_core,snd_hda_codec_hdmi,snd_compress,snd_hda_intel,snd_hda_codec,snd_hda_core, Live 0xffffffffc09c4000 drm_kms_helper 192512 1 nvidia_drm, Live 0xffffffffc097e000 snd_timer 49152 3 snd_hrtimer,snd_seq,snd_pcm, Live 0xffffffffc0971000 snd 122880 26 snd_sof,snd_hda_codec_realtek,snd_soc_core,snd_hda_codec_generic,snd_compress,snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec,snd_hwdep,snd_seq,snd_seq_device,snd_pcm,snd_timer, Live 0xffffffffc0946000 intel_vsec 20480 0 - Live 0xffffffffc0375000 syscopyarea 16384 1 drm_kms_helper, Live 0xffffffffc0370000 sysfillrect 16384 1 drm_kms_helper, Live 0xffffffffc029b000 sysimgblt 16384 1 drm_kms_helper, Live 0xffffffffc0296000 soundcore 16384 1 snd, Live 0xffffffffc0291000 fb_sys_fops 16384 1 drm_kms_helper, Live 0xffffffffc035d000 intel_pmc_core 53248 0 - Live 0xffffffffc08c6000 acpi_tad 20480 0 - Live 0xffffffffc0414000 acpi_pad 184320 0 - Live 0xffffffffc0898000 raid1 57344 1 - Live 0xffffffffc080b000 drm 581632 16 nvidia_drm,nvidia,drm_kms_helper, Live 0xffffffffc0677000 xfs 2043904 4 - Live 0xffffffffc0483000 libcrc32c 16384 4 nf_nat,nf_conntrack,nf_tables,xfs, Live 0xffffffffc01b6000 sd_mod 69632 3 - Live 0xffffffffc01ef000 sg 49152 0 - Live 0xffffffffc01d0000 ahci 49152 2 - Live 0xffffffffc01bb000 libahci 53248 1 ahci, Live 0xffffffffc01a8000 nvme 57344 3 - Live 0xffffffffc0390000 crct10dif_pclmul 16384 1 - Live 0xffffffffc0353000 crc32_pclmul 16384 0 - Live 0xffffffffc03b1000 crc32c_intel 24576 1 - Live 0xffffffffc015e000 libata 425984 2 ahci,libahci, Live 0xffffffffc041a000 nvme_core 180224 6 nvme, Live 0xffffffffc03de000 r8169 106496 0 - Live 0xffffffffc03bb000 ghash_clmulni_intel 16384 0 - Live 0xffffffffc02c1000 nvme_common 24576 1 nvme_core, Live 0xffffffffc02d1000 realtek 36864 1 - Live 0xffffffffc02c7000 t10_pi 16384 2 sd_mod,nvme_core, Live 0xffffffffc02a2000 wmi 36864 2 asus_wmi,wmi_bmof, Live 0xffffffffc0262000 video 61440 1 asus_wmi, Live 0xffffffffc0198000 pinctrl_alderlake 36864 0 - Live 0xffffffffc018e000 serio_raw 20480 0 - Live 0xffffffffc0158000 dm_mirror 28672 0 - Live 0xffffffffc017e000 dm_region_hash 24576 1 dm_mirror, Live 0xffffffffc0174000 dm_log 24576 2 dm_mirror,dm_region_hash, Live 0xffffffffc0169000 dm_mod 204800 24 dm_zero,dm_crypt,dm_mirror,dm_log, Live 0xffffffffc0125000 fuse 176128 7 - Live 0xffffffffc00f9000 [dnadle@dln perf]$ sudo cat /proc/sys/kernel/tainted 12289 Created attachment 1935619 [details]
perf report
Output of sudo perf record -p 144,81390 sleep 10
Created attachment 1935620 [details]
perf report with call graph
Output of sudo perf record -p 144,81390 --call-graph lbr sleep 10
(In reply to David Nadle from comment #5) > kcompactd0 is pid 144 > vmware-vmx is pid 81390 > > [dnadle@dln perf]$ sudo cat /proc/144/stack > [dnadle@dln perf]$ sudo cat /proc/81390/stack > [<0>] do_poll.constprop.0+0x225/0x390 > [<0>] do_sys_poll+0x1c8/0x260 > [<0>] __x64_sys_ppoll+0xbc/0x150 > [<0>] do_syscall_64+0x59/0x90 > [<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd > David, It would be better to have a sequence of stacks over time, just like the script in comment #3 would do it. I noticed, though, there's no stack for kcompactd0 which seems weird given that kernel thread, while not working, remains sleeping and I would expect to (at least) see something similar to: [<0>] kcompactd+0x17c/0x340 [<0>] kthread+0x127/0x150 [<0>] ret_from_fork+0x22/0x30 as a result of your cat. > [dnadle@dln perf]$ sudo cat /proc/sys/kernel/tainted > 12289 that taint value means that we have taint bits 0, 12 and 13 set which means PROPRIETARY_MODULE (P), OOT_MODULE (O) and UNSIGNED_MODULE (E) and given the list of loaded modules, these are the ones causing the taints: > vmnet 73728 17 - Live 0xffffffffc032f000 (OE) > vmmon 147456 3 - Live 0xffffffffc02f8000 (OE) > nvidia_drm 73728 12 - Live 0xffffffffc3b22000 (POE) > nvidia_modeset 1155072 16 nvidia_drm, Live 0xffffffffc39e7000 (POE) > nvidia_uvm 2777088 0 - Live 0xffffffffc3740000 (POE) > nvidia 40886272 1055 nvidia_modeset,nvidia_uvm, Live 0xffffffffc1041000 (POE) I'd suggest attempting to unload the NVIDIA proprietary modules for a test run. You mentioned that kcompactd keeps running for hours burning 100% CPU, so I don't know how significant a 10 second perf sampling can be to characterize where it spends most of its time, but from your perf sampling it seems that it's the wait for remote TLB flushing that is holding it back: 20.19% 0.00% kcompactd0 [kernel.kallsyms] [k] kthread | ---kthread kcompactd kcompactd_do_work compact_zone | --20.14%--migrate_pages | --20.13%--unmap_and_move | --20.11%--__unmap_and_move.constprop.0 | --19.70%--try_to_migrate | --19.68%--rmap_walk_file | --19.63%--try_to_migrate_one | --19.43%--ptep_clear_flush | --19.42%--flush_tlb_mm_range | --19.31%--on_each_cpu_cond_mask | --19.31%--smp_call_function_many_cond | --0.82%--__x2apic_send_IPI_mask | --0.51%--__x2apic_send_IPI_dest Also from what is collected in the perf data, it seems that there might be an interaction between VMWare's (vmmon) vmx-vcpu-1 thread and the kcompactd0 kernel thread, but that is hard to follow up by the call-graph as it appears you are missing the debug symbols for the running kernel and for the 3rd party module: 19.98% 0.00% vmx-vcpu-1 [unknown] [k] 0xffffffff9ec0009b | ---0xffffffff9ec0009b 0xffffffff9eb14159 | --19.98%--0xffffffff9e3d6fb7 | --19.98%--0xffffffffc02f9325 | --19.78%--0xffffffffc02fdc73 | --19.66%--0xffffffffc03017d5 | --19.64%--0xffffffffc02fbe86 This one is starting to smell like a problem with these 3rd party modules, to be honest. Have you approached VMWare support on this one? -- Rafael Rafael, Next time it happens I'll run some version of the script. The reason I didn't is that there is more than one VM running and therefore more than one vmware-vmx process. I'm not sure which nvidia modules I can disable given that it's the graphics card driver. How long would you suggest I collect perf data for? I'm going to see if I can build a debug version of vmmon. Yes, I have a support ticket open with VMware. Update: kernel updated to 5.14.0-226. vmmon built with -g. Took a few days to reencounter the kcompatd0 issue. kcompactd still does not output any stacks: 1 [<0>] do_poll.constprop.0+0x225/0x390 [<0>] do_sys_poll+0x1c8/0x260 [<0>] __x64_sys_ppoll+0xbc/0x150 [<0>] do_syscall_64+0x59/0x90 [<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd 2 [<0>] do_poll.constprop.0+0x225/0x390 [<0>] do_sys_poll+0x1c8/0x260 [<0>] __x64_sys_ppoll+0xbc/0x150 [<0>] do_syscall_64+0x59/0x90 [<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd 3 [<0>] do_poll.constprop.0+0x225/0x390 [<0>] do_sys_poll+0x1c8/0x260 [<0>] __x64_sys_ppoll+0xbc/0x150 [<0>] do_syscall_64+0x59/0x90 [<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd 4 [<0>] do_poll.constprop.0+0x225/0x390 [<0>] do_sys_poll+0x1c8/0x260 [<0>] __x64_sys_ppoll+0xbc/0x150 [<0>] do_syscall_64+0x59/0x90 [<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd 5 [<0>] do_poll.constprop.0+0x225/0x390 [<0>] do_sys_poll+0x1c8/0x260 [<0>] __x64_sys_ppoll+0xbc/0x150 [<0>] do_syscall_64+0x59/0x90 [<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd 6 [<0>] do_poll.constprop.0+0x225/0x390 [<0>] do_sys_poll+0x1c8/0x260 [<0>] __x64_sys_ppoll+0xbc/0x150 [<0>] do_syscall_64+0x59/0x90 [<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd 7 [<0>] do_poll.constprop.0+0x225/0x390 [<0>] do_sys_poll+0x1c8/0x260 [<0>] __x64_sys_ppoll+0xbc/0x150 [<0>] do_syscall_64+0x59/0x90 [<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd 8 [<0>] do_poll.constprop.0+0x225/0x390 [<0>] do_sys_poll+0x1c8/0x260 [<0>] __x64_sys_ppoll+0xbc/0x150 [<0>] do_syscall_64+0x59/0x90 [<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd 9 [<0>] do_poll.constprop.0+0x225/0x390 [<0>] do_sys_poll+0x1c8/0x260 [<0>] __x64_sys_ppoll+0xbc/0x150 [<0>] do_syscall_64+0x59/0x90 [<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd 10 [<0>] do_poll.constprop.0+0x225/0x390 [<0>] do_sys_poll+0x1c8/0x260 [<0>] __x64_sys_ppoll+0xbc/0x150 [<0>] do_syscall_64+0x59/0x90 [<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd Created attachment 1936654 [details]
perf report 60s capture with vmmon symbols
Created attachment 1936655 [details]
perf report 60s capture with call graph
Not sure how useful this capture is. Seems most of the VMware CPU time is spent in sections for which I have no debug symbols.
David, If you didn't reboot the system after the last perf collectioin, can you attach here the output of "cat /proc/kallsyms" ? Thanks in advance! -- Rafael Created attachment 1936706 [details]
output of cat /proc/kallsyms
I did not reboot the host but I did suspend and restore the Windows VM to allow kcompactd0 to complete.
(In reply to David Nadle from comment #14) > Created attachment 1936706 [details] > output of cat /proc/kallsyms > > I did not reboot the host but I did suspend and restore the Windows VM to > allow kcompactd0 to complete. David, the kallsysms dump has to be taken under a privileged account (root), otherwise no addresses for the symbols are available: ---8<--- 0000000000000000 t do_boot_cpu 0000000000000000 T common_cpu_up 0000000000000000 T native_cpu_up 0000000000000000 T arch_disable_smp_support 0000000000000000 T arch_thaw_secondary_cpus_begin 0000000000000000 T arch_thaw_secondary_cpus_end 0000000000000000 T cpu_disable_common 0000000000000000 T native_cpu_disable 0000000000000000 T common_cpu_die 0000000000000000 T native_cpu_die --->8--- Created attachment 1936791 [details]
output of sudo cat /proc/kallsyms
Sorry about that...
(In reply to David Nadle from comment #16) > Created attachment 1936791 [details] > output of sudo cat /proc/kallsyms > Thank you, David! By comparing the symbol's base addresses and the call graph dumped by perf, we can see that all those vmx-vcpu-# threads are spending ~20% of their time (in the sampled period) at vmmon's APIC_Write routine: here's one example. 19.96% 0.00% vmx-vcpu-1 [unknown] [k] 0xffffffff8be0009b | ---0xffffffff8be0009b (entry_SYSCALL_64_after_hwframe + 0x63) | --19.96%--0xffffffff8bd1b159 (do_syscall_64 + 0x59) | --19.96%--0xffffffff8b5da9a7 (__x64_sys_ioctl + 0x87) | --19.96%--0xffffffffc0b0e325 (LinuxDriver_Ioctl ?) | --19.76%--0xffffffffc0b12c73 (Vmx86_RunVM + 0xb3) | --19.65%--0xffffffffc0b167d5 (Task_Switch + 0x1845) | --19.63%--0xffffffffc0b10e86 (APIC_Write + 0x16) it seems to have a relationship with kcompactd also working on APIC writes for the TLB shootdown, although this is just a conjecture of mine at the moment. I'd suggest you pushing this through VMWare support to see if they have any knwon issue registered. I also would like to see if you can get it reproducing without loading your NVIDIA modules (disable your GPU for a test run, if possible). Also, do you happen to notice any lockup messages in the kernel logs when these processes are running? Can you attach the output of "dmesg" here, please? Last, but not least, in order to give us a better idea on what might be preventing cat /proc/$(pidof -w kcompactd0)/stack to produce output, would you mind running it again, but this way instead: "cat /proc/$(pidof -w kcompactd0)/stack; echo $?" Thanks in advance -- Rafael I did some research on this bug, and I found that there are already some similar issues, please refer to below links: https://bugzilla.redhat.com/show_bug.cgi?id=1694305 https://communities.vmware.com/t5/VMware-Workstation-Pro/VMWare-workstation-in-a-fistfight-with-Linux-Memory-Compactor/td-p/2876992 https://communities.vmware.com/t5/VMware-Workstation-Pro/kcompacd0-using-100-CPU-with-VMware-Workstation-16/td-p/2896972 Update: kernel updated to 5.14.0-229. Removed NVIDIA drivers, running X11 session with nouveau. kcompactd0 still does not list any stacks: 1 0 -- [<0>] do_poll.constprop.0+0x225/0x390 [<0>] do_sys_poll+0x1c8/0x260 [<0>] __x64_sys_ppoll+0xbc/0x150 [<0>] do_syscall_64+0x59/0x90 [<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd 2 0 -- [<0>] do_poll.constprop.0+0x225/0x390 [<0>] do_sys_poll+0x1c8/0x260 [<0>] __x64_sys_ppoll+0xbc/0x150 [<0>] do_syscall_64+0x59/0x90 [<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd 3 0 -- [<0>] do_poll.constprop.0+0x225/0x390 [<0>] do_sys_poll+0x1c8/0x260 [<0>] __x64_sys_ppoll+0xbc/0x150 [<0>] do_syscall_64+0x59/0x90 [<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd 4 0 -- [<0>] do_poll.constprop.0+0x225/0x390 [<0>] do_sys_poll+0x1c8/0x260 [<0>] __x64_sys_ppoll+0xbc/0x150 [<0>] do_syscall_64+0x59/0x90 [<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd 5 0 -- [<0>] do_poll.constprop.0+0x225/0x390 [<0>] do_sys_poll+0x1c8/0x260 [<0>] __x64_sys_ppoll+0xbc/0x150 [<0>] do_syscall_64+0x59/0x90 [<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd 6 0 -- [<0>] do_poll.constprop.0+0x225/0x390 [<0>] do_sys_poll+0x1c8/0x260 [<0>] __x64_sys_ppoll+0xbc/0x150 [<0>] do_syscall_64+0x59/0x90 [<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd 7 0 -- [<0>] do_poll.constprop.0+0x225/0x390 [<0>] do_sys_poll+0x1c8/0x260 [<0>] __x64_sys_ppoll+0xbc/0x150 [<0>] do_syscall_64+0x59/0x90 [<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd 8 0 -- [<0>] do_poll.constprop.0+0x225/0x390 [<0>] do_sys_poll+0x1c8/0x260 [<0>] __x64_sys_ppoll+0xbc/0x150 [<0>] do_syscall_64+0x59/0x90 [<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd 9 0 -- [<0>] do_poll.constprop.0+0x225/0x390 [<0>] do_sys_poll+0x1c8/0x260 [<0>] __x64_sys_ppoll+0xbc/0x150 [<0>] do_syscall_64+0x59/0x90 [<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd 10 0 -- [<0>] do_poll.constprop.0+0x225/0x390 [<0>] do_sys_poll+0x1c8/0x260 [<0>] __x64_sys_ppoll+0xbc/0x150 [<0>] do_syscall_64+0x59/0x90 [<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd Nothing in dmesg during the incident. /proc/modules: dm_zero 16384 4 - Live 0xffffffffc0478000 ppdev 24576 0 - Live 0xffffffffc093f000 parport_pc 49152 0 - Live 0xffffffffc08f7000 parport 77824 2 ppdev,parport_pc, Live 0xffffffffc084c000 vmw_vsock_vmci_transport 45056 1 - Live 0xffffffffc0832000 vsock 53248 3 vmw_vsock_vmci_transport, Live 0xffffffffc07d5000 vmw_vmci 118784 4 vmw_vsock_vmci_transport, Live 0xffffffffc13b2000 vmnet 73728 19 - Live 0xffffffffc081f000 (OE) vmmon 147456 3 - Live 0xffffffffc07aa000 (OE) tls 131072 0 - Live 0xffffffffc048a000 uinput 20480 0 - Live 0xffffffffc0414000 binfmt_misc 28672 1 - Live 0xffffffffc03ec000 rfcomm 94208 4 - Live 0xffffffffc042d000 snd_seq_dummy 16384 0 - Live 0xffffffffc0397000 snd_hrtimer 16384 1 - Live 0xffffffffc0389000 nft_objref 16384 1 - Live 0xffffffffc0382000 nf_conntrack_netbios_ns 16384 1 - Live 0xffffffffc0374000 nf_conntrack_broadcast 16384 1 nf_conntrack_netbios_ns, Live 0xffffffffc036f000 nft_fib_inet 16384 1 - Live 0xffffffffc036a000 nft_fib_ipv4 16384 1 nft_fib_inet, Live 0xffffffffc0315000 nft_fib_ipv6 16384 1 nft_fib_inet, Live 0xffffffffc0306000 nft_fib 16384 3 nft_fib_inet,nft_fib_ipv4,nft_fib_ipv6, Live 0xffffffffc0301000 nft_reject_inet 16384 6 - Live 0xffffffffc02fc000 nf_reject_ipv4 16384 1 nft_reject_inet, Live 0xffffffffc02f7000 nf_reject_ipv6 20480 1 nft_reject_inet, Live 0xffffffffc02f1000 nft_reject 16384 1 nft_reject_inet, Live 0xffffffffc02e0000 nft_ct 24576 15 - Live 0xffffffffc02ce000 nft_chain_nat 16384 3 - Live 0xffffffffc02d8000 nf_nat 57344 1 nft_chain_nat, Live 0xffffffffc0326000 nf_conntrack 180224 4 nf_conntrack_netbios_ns,nf_conntrack_broadcast,nft_ct,nf_nat, Live 0xffffffffc086f000 nf_defrag_ipv6 24576 1 nf_conntrack, Live 0xffffffffc02c7000 nf_defrag_ipv4 16384 1 nf_conntrack, Live 0xffffffffc02c2000 ip_set 61440 0 - Live 0xffffffffc02b2000 nf_tables 274432 221 nft_objref,nft_fib_inet,nft_fib_ipv4,nft_fib_ipv6,nft_fib,nft_reject_inet,nft_reject,nft_ct,nft_chain_nat, Live 0xffffffffc0e9b000 nfnetlink 20480 3 ip_set,nf_tables, Live 0xffffffffc02ac000 qrtr 53248 4 - Live 0xffffffffc029e000 bnep 32768 2 - Live 0xffffffffc0295000 sunrpc 696320 1 - Live 0xffffffffc1551000 dm_crypt 61440 5 - Live 0xffffffffc023a000 vfat 20480 1 - Live 0xffffffffc0234000 fat 86016 1 vfat, Live 0xffffffffc0217000 squashfs 77824 20 - Live 0xffffffffc0274000 loop 36864 40 - Live 0xffffffffc020d000 intel_rapl_msr 20480 0 - Live 0xffffffffc0460000 iTCO_wdt 16384 0 - Live 0xffffffffc07a5000 iTCO_vendor_support 16384 1 iTCO_wdt, Live 0xffffffffc07a0000 i2c_designware_platform 16384 0 - Live 0xffffffffc0359000 i2c_designware_core 36864 1 i2c_designware_platform, Live 0xffffffffc0345000 pmt_telemetry 16384 0 - Live 0xffffffffc0290000 pmt_class 16384 1 pmt_telemetry, Live 0xffffffffc028b000 intel_rapl_common 32768 1 intel_rapl_msr, Live 0xffffffffc0481000 x86_pkg_temp_thermal 20480 0 - Live 0xffffffffc0353000 intel_powerclamp 20480 0 - Live 0xffffffffc035e000 coretemp 20480 0 - Live 0xffffffffc0364000 kvm_intel 409600 0 - Live 0xffffffffc14ec000 kvm 1130496 1 kvm_intel, Live 0xffffffffc13d7000 irqbypass 16384 1 kvm, Live 0xffffffffc07d0000 rapl 24576 0 - Live 0xffffffffc13d0000 intel_cstate 24576 0 - Live 0xffffffffc031f000 intel_uncore 225280 0 - Live 0xffffffffc137a000 eeepc_wmi 16384 0 - Live 0xffffffffc0f60000 pcspkr 16384 0 - Live 0xffffffffc0473000 asus_wmi 53248 1 eeepc_wmi, Live 0xffffffffc107b000 sparse_keymap 16384 1 asus_wmi, Live 0xffffffffc031a000 wmi_bmof 16384 0 - Live 0xffffffffc046e000 iwlmvm 475136 0 - Live 0xffffffffc1305000 mac80211 1224704 1 iwlmvm, Live 0xffffffffc11d9000 snd_sof_pci_intel_tgl 16384 0 - Live 0xffffffffc0bc8000 snd_sof_intel_hda_common 147456 1 snd_sof_pci_intel_tgl, Live 0xffffffffc11b4000 soundwire_intel 49152 1 snd_sof_intel_hda_common, Live 0xffffffffc0f53000 libarc4 16384 1 mac80211, Live 0xffffffffc0807000 soundwire_generic_allocation 16384 1 soundwire_intel, Live 0xffffffffc0802000 soundwire_cadence 40960 1 soundwire_intel, Live 0xffffffffc1099000 snd_sof_intel_hda 20480 1 snd_sof_intel_hda_common, Live 0xffffffffc1093000 snd_sof_pci 24576 2 snd_sof_pci_intel_tgl,snd_sof_intel_hda_common, Live 0xffffffffc108c000 snd_sof_xtensa_dsp 16384 1 snd_sof_intel_hda_common, Live 0xffffffffc0b94000 snd_sof 253952 2 snd_sof_intel_hda_common,snd_sof_pci, Live 0xffffffffc1175000 snd_sof_utils 20480 1 snd_sof, Live 0xffffffffc0969000 snd_soc_hdac_hda 24576 1 snd_sof_intel_hda_common, Live 0xffffffffc090b000 snd_hda_ext_core 36864 3 snd_sof_intel_hda_common,snd_sof_intel_hda,snd_soc_hdac_hda, Live 0xffffffffc116b000 iwlwifi 385024 1 iwlmvm, Live 0xffffffffc110c000 snd_hda_codec_realtek 172032 1 - Live 0xffffffffc10e1000 snd_soc_acpi_intel_match 69632 2 snd_sof_pci_intel_tgl,snd_sof_intel_hda_common, Live 0xffffffffc0c09000 snd_soc_acpi 16384 2 snd_sof_intel_hda_common,snd_soc_acpi_intel_match, Live 0xffffffffc0840000 soundwire_bus 98304 3 soundwire_intel,soundwire_generic_allocation,soundwire_cadence, Live 0xffffffffc10c8000 snd_hda_codec_generic 98304 1 snd_hda_codec_realtek, Live 0xffffffffc10af000 cfg80211 1085440 3 iwlmvm,mac80211,iwlwifi, Live 0xffffffffc0f68000 ledtrig_audio 16384 1 snd_hda_codec_generic, Live 0xffffffffc09bf000 snd_soc_core 348160 4 snd_sof_intel_hda_common,soundwire_intel,snd_sof,snd_soc_hdac_hda, Live 0xffffffffc0ef4000 nouveau 2469888 47 - Live 0xffffffffc0c3f000 snd_hda_codec_hdmi 77824 1 - Live 0xffffffffc0c2b000 snd_compress 28672 1 snd_soc_core, Live 0xffffffffc0a57000 i2c_i801 36864 0 - Live 0xffffffffc0c21000 snd_hda_intel 57344 6 - Live 0xffffffffc0b85000 i2c_smbus 20480 1 i2c_i801, Live 0xffffffffc0846000 snd_intel_dspcfg 32768 3 snd_sof_intel_hda_common,snd_sof,snd_hda_intel, Live 0xffffffffc0975000 mei_me 49152 0 - Live 0xffffffffc08e2000 snd_intel_sdw_acpi 20480 2 snd_sof_intel_hda_common,snd_intel_dspcfg, Live 0xffffffffc019f000 mei 155648 1 mei_me, Live 0xffffffffc0bd9000 snd_hda_codec 176128 5 snd_soc_hdac_hda,snd_hda_codec_realtek,snd_hda_codec_generic,snd_hda_codec_hdmi,snd_hda_intel, Live 0xffffffffc0b9c000 btusb 65536 0 - Live 0xffffffffc0b74000 btrtl 28672 1 btusb, Live 0xffffffffc09c5000 btbcm 20480 1 btusb, Live 0xffffffffc0963000 snd_hda_core 110592 9 snd_sof_intel_hda_common,snd_sof_intel_hda,snd_soc_hdac_hda,snd_hda_ext_core,snd_hda_codec_realtek,snd_hda_codec_generic,snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec, Live 0xffffffffc0b4b000 btintel 49152 1 btusb, Live 0xffffffffc0956000 btmtk 16384 1 btusb, Live 0xffffffffc0951000 mxm_wmi 16384 1 nouveau, Live 0xffffffffc094c000 snd_hwdep 16384 1 snd_hda_codec, Live 0xffffffffc0947000 i2c_algo_bit 16384 1 nouveau, Live 0xffffffffc038f000 intel_lpss_pci 28672 0 - Live 0xffffffffc07e6000 intel_lpss 16384 1 intel_lpss_pci, Live 0xffffffffc037d000 bluetooth 937984 36 rfcomm,bnep,btusb,btrtl,btbcm,btintel,btmtk, Live 0xffffffffc0a65000 idma64 20480 0 - Live 0xffffffffc0a5f000 drm_display_helper 172032 1 nouveau, Live 0xffffffffc0a23000 snd_seq 94208 7 snd_seq_dummy, Live 0xffffffffc0a0b000 snd_seq_device 16384 1 snd_seq, Live 0xffffffffc07ee000 cec 61440 1 drm_display_helper, Live 0xffffffffc09fb000 snd_pcm 151552 11 snd_sof_intel_hda_common,soundwire_intel,snd_sof,snd_sof_utils,snd_hda_codec_hdmi,snd_soc_core,snd_compress,snd_hda_intel,snd_hda_codec,snd_hda_core, Live 0xffffffffc09d5000 drm_ttm_helper 16384 1 nouveau, Live 0xffffffffc0428000 ttm 90112 2 nouveau,drm_ttm_helper, Live 0xffffffffc08cb000 rfkill 36864 8 asus_wmi,iwlmvm,cfg80211,bluetooth, Live 0xffffffffc0865000 drm_kms_helper 192512 2 nouveau,drm_display_helper, Live 0xffffffffc098f000 snd_timer 49152 3 snd_hrtimer,snd_seq,snd_pcm, Live 0xffffffffc0932000 snd 122880 26 snd_sof,snd_hda_codec_realtek,snd_hda_codec_generic,snd_soc_core,snd_hda_codec_hdmi,snd_compress,snd_hda_intel,snd_hda_codec,snd_hwdep,snd_seq,snd_seq_device,snd_pcm,snd_timer, Live 0xffffffffc0913000 intel_vsec 20480 0 - Live 0xffffffffc02eb000 syscopyarea 16384 1 drm_kms_helper, Live 0xffffffffc0208000 sysfillrect 16384 1 drm_kms_helper, Live 0xffffffffc0203000 sysimgblt 16384 1 drm_kms_helper, Live 0xffffffffc01fe000 soundcore 16384 1 snd, Live 0xffffffffc01f9000 fb_sys_fops 16384 1 drm_kms_helper, Live 0xffffffffc01ca000 intel_pmc_core 53248 0 - Live 0xffffffffc07f4000 acpi_tad 20480 0 - Live 0xffffffffc02e5000 acpi_pad 184320 0 - Live 0xffffffffc089d000 raid1 57344 1 - Live 0xffffffffc0810000 drm 581632 24 nouveau,drm_display_helper,drm_ttm_helper,ttm,drm_kms_helper, Live 0xffffffffc070c000 xfs 2043904 4 - Live 0xffffffffc0518000 libcrc32c 16384 4 nf_nat,nf_conntrack,nf_tables,xfs, Live 0xffffffffc0146000 uas 32768 1 - Live 0xffffffffc01aa000 usb_storage 81920 1 uas, Live 0xffffffffc01e4000 sd_mod 69632 3 - Live 0xffffffffc01d2000 sg 49152 0 - Live 0xffffffffc01b8000 ahci 49152 2 - Live 0xffffffffc0338000 libahci 53248 1 ahci, Live 0xffffffffc0191000 nvme 57344 3 - Live 0xffffffffc0402000 crct10dif_pclmul 16384 1 - Live 0xffffffffc01b3000 libata 425984 2 ahci,libahci, Live 0xffffffffc04af000 crc32_pclmul 16384 0 - Live 0xffffffffc044d000 crc32c_intel 24576 1 - Live 0xffffffffc026d000 nvme_core 180224 6 nvme, Live 0xffffffffc03bb000 r8169 106496 0 - Live 0xffffffffc03a0000 ghash_clmulni_intel 16384 0 - Live 0xffffffffc0265000 realtek 36864 1 - Live 0xffffffffc025b000 nvme_common 24576 1 nvme_core, Live 0xffffffffc030e000 t10_pi 16384 2 sd_mod,nvme_core, Live 0xffffffffc014d000 wmi 36864 4 asus_wmi,wmi_bmof,nouveau,mxm_wmi, Live 0xffffffffc024b000 video 61440 2 asus_wmi,nouveau, Live 0xffffffffc0181000 pinctrl_alderlake 36864 0 - Live 0xffffffffc0177000 serio_raw 20480 0 - Live 0xffffffffc0140000 dm_mirror 28672 0 - Live 0xffffffffc0167000 dm_region_hash 24576 1 dm_mirror, Live 0xffffffffc015d000 dm_log 24576 2 dm_mirror,dm_region_hash, Live 0xffffffffc0152000 dm_mod 204800 24 dm_zero,dm_crypt,dm_mirror,dm_log, Live 0xffffffffc010d000 fuse 176128 7 - Live 0xffffffffc00e1000 Created attachment 1937599 [details]
perf report
Created attachment 1937600 [details]
perf report call graph
Created attachment 1937601 [details]
kallsyms
Interestingly, this time kcompactd0 completed on its own soon after I ran perf, without my having to suspend the Windows VM. I estimate kcompactd0 ran for about 50 minutes. (In reply to David Nadle from comment #23) > Interestingly, this time kcompactd0 completed on its own soon after I ran > perf, without my having to suspend the Windows VM. I estimate kcompactd0 ran > for about 50 minutes. Did you happen to notice any lockup/hang messages in the kernel logs while kcompactd0 was running? Can you attach the output of "dmesg" here, please? There were no lines output in dmesg during the incident. One interesting wrinkle that happened this morning is that it seemed to be the Fedora guest that was affected and not the Windows guest. The Fedora guest was slow but not as completely frozen up as the Windows guest was. Dragging windows around didn't work but you could see the display updating. So the issue appears to be between VMware and the kernel and not necessarily related to the guest OS. kcompactd0 did not complete on its own while I was observing it. It may have been running all night. I suspended the VM to get it to complete. David, Would you mind checking this test kernel against your reproducer? https://s3.upshift.redhat.com/DH-PROD-CKI/index.html?prefix=internal-artifacts/764655335/publish%20x86_64/3699711725/artifacts/repo/5.14.0-253.1996_764655335.el9.x86_64/ Upstream has recently identified a problem with commit 7efc3b726103 ("mm/compaction: fix set skip in fast_find_migrateblock") that caused issues in a similar fashion to those you are observing, and the aforementioned test build reverts it. Thanks in advance, -- Rafael Rafael, The link does not work for me. Is that server in public DNS? [dnadle@dln ~]$ host s3.upshift.redhat.com Host s3.upshift.redhat.com not found: 3(NXDOMAIN) Regards, David Nadle (In reply to David Nadle from comment #30) > Rafael, > > The link does not work for me. Is that server in public DNS? > UGH! my bad, sorry! I moved the RPMs to: https://people.redhat.com/~raquini/bz2157958/ -- Rafael (In reply to Rafael Aquini from comment #31) > I moved the RPMs to: > > https://people.redhat.com/~raquini/bz2157958/ I have the files but I will need some guidance on how to install the kernel. For example: [dnadle@dln test-kernel]$ rpm -i --test kernel-5.14.0-253.1996_764655335.el9.x86_64.rpm error: Failed dependencies: kernel-core-uname-r = 5.14.0-253.1996_764655335.el9.x86_64 is needed by kernel-5.14.0-253.1996_764655335.el9.x86_64 kernel-modules-uname-r = 5.14.0-253.1996_764655335.el9.x86_64 is needed by kernel-5.14.0-253.1996_764655335.el9.x86_64 Do I manually install the dependencies first? (In reply to David Nadle from comment #32) > (In reply to Rafael Aquini from comment #31) > > I moved the RPMs to: > > > > https://people.redhat.com/~raquini/bz2157958/ > > I have the files but I will need some guidance on how to install the kernel. > For example: > > [dnadle@dln test-kernel]$ rpm -i --test > kernel-5.14.0-253.1996_764655335.el9.x86_64.rpm > error: Failed dependencies: > kernel-core-uname-r = 5.14.0-253.1996_764655335.el9.x86_64 is needed by > kernel-5.14.0-253.1996_764655335.el9.x86_64 > kernel-modules-uname-r = 5.14.0-253.1996_764655335.el9.x86_64 is needed by > kernel-5.14.0-253.1996_764655335.el9.x86_64 > > Do I manually install the dependencies first? David, You can do it one by one, or just tell RPM to do it in one single transaction: # rpm -ihv --test kernel-5.14.0-253.1996_764655335.el9.x86_64.rpm kernel-core-5.14.0-253.1996_764655335.el9.x86_64.rpm kernel-modules-5.14.0-253.1996_764655335.el9.x86_64.rpm kernel-devel-5.14.0-253.1996_764655335.el9.x86_64.rpm warning: Signature not supported. Hash algorithm SHA1 not available. Verifying... ################################# [100%] Preparing... ################################# [100%] (then to install, just drop the --test) -- Rafael I have been running this test kernel since the morning of Feb. 8. Later that morning I experienced a brief run of kcompactd0 that affected the Win11 VM UI performance, but by the time I could prepare to capture process info kcompactd0 has completed. I have not seen another issue since. kcompactd0 had been locking up on average every 4 days so I think no incidents in 12 days is a good sign. Kernel 5.14.0-267.el9 is now available for me to install. I will move on to that kernel and see if the problem returns. I would appreciate it if someone can report in this thread if or when the reverted commit that fixed the test kernel makes it into the mainstream Centos9 kernel. (In reply to David Nadle from comment #34) > I would appreciate it if someone can report in this thread > if or when the reverted commit that fixed the test > kernel makes it into the mainstream Centos9 kernel. David, First: thank you for testing that scratch-build kernel and reporting back it really addressed your issue with kcompactd. The patch you tested in that scratch-build is now part of CentOS-Stream-9 kernel-5.14.0-264.el9 interim build. -- Rafael The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days |