Bug 2157958

Summary: kcompactd0 runs indefinitely at 100% CPU with inter.ference from VMware
Product: Red Hat Enterprise Linux 9 Reporter: David Nadle <david>
Component: kernelAssignee: Rafael Aquini <aquini>
kernel sub component: Memory Management QA Contact: Ping Fang <pifang>
Status: CLOSED CURRENTRELEASE Docs Contact:
Severity: medium    
Priority: unspecified CC: aquini, boyang, bstinson, chuhu, jwboyer, ldu, liwan, llong, pm-rhel, shshang, yacao
Version: CentOS StreamKeywords: TestOnly, Triaged
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: kernel-5.14.0-264.el9 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-02-21 19:50:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2166447    
Bug Blocks:    
Attachments:
Description Flags
perf report
none
perf report with call graph
none
perf report 60s capture with vmmon symbols
none
perf report 60s capture with call graph
none
output of cat /proc/kallsyms
none
output of sudo cat /proc/kallsyms
none
perf report
none
perf report call graph
none
kallsyms none

Description David Nadle 2023-01-03 17:44:43 UTC
Description of problem:
When memory compaction occurs while running VMWare Workstation Pro with a Windows 11 guest, both the kcompactd0 and vmware-vmx processes run at 100% CPU for a long time (hours), and the Windows VM is frozen. It's not clear that compaction would ever complete without manual intervention such as suspending the virtual machine. Other VM guests and applications on the host appear to work normally.

Version-Release number of selected component (if applicable):
CentOS Stream 9 kernel 5.14.0-214.el9.x86_64 (and earlier)

How reproducible:
Unclear, but it continues to happen despite all my efforts to eliminate it, including disabling transparent huge pages and setting vm.compaction_proactiveness=0.

Steps to Reproduce:
1. Start a Windows 11 guest on VMWare Workstation Pro 17.0.0 build-20800274 
2. Wait for or trigger memory compaction (kcompactd0).

Actual results:
The vmware and kcompactd0 process run indefinitely at 100% CPU until something is done to stop it. The Windows UI is unresponsive, with brief intervals once or twice a minute where queued up mouse/keyboard events are executed.

Expected results:
kcompactd0 completes quickly, or detects the spinning caused by a possibly misbehaving VMWare, and aborts.

Additional info:

Issues with VMware, transparent huge pages,and memory compaction have apparently been around for a long time:

1. https://unix.stackexchange.com/questions/458893/vmware-on-linux-host-causes-regular-freezes
2. https://unix.stackexchange.com/questions/161858/arch-linux-becomes-unresponsive-from-khugepaged/
3. https://bugzilla.redhat.com/show_bug.cgi?id=879801
4. https://bugzilla.redhat.com/show_bug.cgi?id=1694305

Comment 2 Waiman Long 2023-01-03 18:25:02 UTC
We don't have VMware Workstation Pro and a Windows 11 VM guest on hand to reproduce the problem. Would you mind running the perf-record command to record a trace of the 100% cpu processes and then attach the perf-report output of them so that we have a better idea of what the kernel is doing to consume all those CPU cycles? It would be nice to have two sets with and without the --call-graph perf-record option.

Comment 3 Rafael Aquini 2023-01-03 20:19:43 UTC
David,

When kcompactd0 and vmware-vmx processes are seen to go on burning CPU at 100%, 
please capture some snapshots of their stacks and attach them here.

The following bash one-liner can do the trick:

# for n in $(seq 1 10); do echo $n; cat /proc/$(pidof -w kcompactd0)/stack; echo ""; cat /proc/$(pidof -w vmware-vmx)/stack; echo ""; sleep 1; done

-- Rafael

Comment 4 Rafael Aquini 2023-01-03 20:32:37 UTC
David, 

While you are at the data collection (comment #3),
please, also grab a snapshot of  these files:

/proc/modules
/proc/sys/kernel/tainted 

Thanks in advance!
-- Rafael

Comment 5 David Nadle 2023-01-04 05:24:52 UTC
kcompactd0 is pid 144
vmware-vmx is pid 81390

[dnadle@dln perf]$ sudo cat /proc/144/stack
[dnadle@dln perf]$ sudo cat /proc/81390/stack
[<0>] do_poll.constprop.0+0x225/0x390
[<0>] do_sys_poll+0x1c8/0x260
[<0>] __x64_sys_ppoll+0xbc/0x150
[<0>] do_syscall_64+0x59/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd

[dnadle@dln perf]$ sudo cat /proc/modules
dm_zero 16384 4 - Live 0xffffffffc0902000
vsock_loopback 16384 0 - Live 0xffffffffc07d4000
vmw_vsock_virtio_transport_common 49152 1 vsock_loopback, Live 0xffffffffc0d81000
vmw_vsock_vmci_transport 40960 0 - Live 0xffffffffc0884000
vsock 53248 3 vsock_loopback,vmw_vsock_virtio_transport_common,vmw_vsock_vmci_transport, Live 0xffffffffc07f3000
tls 118784 0 - Live 0xffffffffc0866000
binfmt_misc 28672 1 - Live 0xffffffffc0780000
snd_seq_dummy 16384 0 - Live 0xffffffffc0772000
snd_hrtimer 16384 1 - Live 0xffffffffc0742000
rfcomm 94208 4 - Live 0xffffffffc0751000
nft_objref 16384 1 - Live 0xffffffffc0732000
nf_conntrack_netbios_ns 16384 1 - Live 0xffffffffc072d000
nf_conntrack_broadcast 16384 1 nf_conntrack_netbios_ns, Live 0xffffffffc040b000
nft_fib_inet 16384 1 - Live 0xffffffffc03b6000
nft_fib_ipv4 16384 1 nft_fib_inet, Live 0xffffffffc038a000
nft_fib_ipv6 16384 1 nft_fib_inet, Live 0xffffffffc0385000
nft_fib 16384 3 nft_fib_inet,nft_fib_ipv4,nft_fib_ipv6, Live 0xffffffffc0380000
nft_reject_inet 16384 6 - Live 0xffffffffc037b000
nf_reject_ipv4 16384 1 nft_reject_inet, Live 0xffffffffc0369000
nf_reject_ipv6 20480 1 nft_reject_inet, Live 0xffffffffc034d000
nft_reject 16384 1 nft_reject_inet, Live 0xffffffffc0342000
nft_ct 24576 15 - Live 0xffffffffc0362000
nft_chain_nat 16384 3 - Live 0xffffffffc0358000
nf_nat 57344 1 nft_chain_nat, Live 0xffffffffc039f000
nf_conntrack 180224 4 nf_conntrack_netbios_ns,nf_conntrack_broadcast,nft_ct,nf_nat, Live 0xffffffffc078d000
nf_defrag_ipv6 24576 1 nf_conntrack, Live 0xffffffffc031d000
nf_defrag_ipv4 16384 1 nf_conntrack, Live 0xffffffffc02d8000
ip_set 61440 0 - Live 0xffffffffc02b0000
vmw_vmci 118784 4 vmw_vsock_vmci_transport, Live 0xffffffffc070f000
vmnet 73728 17 - Live 0xffffffffc032f000 (OE)
vmmon 147456 3 - Live 0xffffffffc02f8000 (OE)
nf_tables 270336 221 nft_objref,nft_fib_inet,nft_fib_ipv4,nft_fib_ipv6,nft_fib,nft_reject_inet,nft_reject,nft_ct,nft_chain_nat, Live 0xffffffffc081a000
nfnetlink 20480 3 ip_set,nf_tables, Live 0xffffffffc027a000
qrtr 53248 4 - Live 0xffffffffc026c000
bnep 32768 2 - Live 0xffffffffc022a000
sunrpc 696320 1 - Live 0xffffffffc3cee000
dm_crypt 61440 5 - Live 0xffffffffc0252000
vfat 20480 1 - Live 0xffffffffc01e7000
fat 86016 1 vfat, Live 0xffffffffc0214000
squashfs 77824 19 - Live 0xffffffffc0237000
loop 36864 38 - Live 0xffffffffc01dd000
intel_rapl_msr 20480 0 - Live 0xffffffffc024c000
iTCO_wdt 16384 0 - Live 0xffffffffc07cf000
iTCO_vendor_support 16384 1 iTCO_wdt, Live 0xffffffffc07ca000
i2c_designware_platform 16384 0 - Live 0xffffffffc07d9000
i2c_designware_core 36864 1 i2c_designware_platform, Live 0xffffffffc0747000
pmt_telemetry 16384 0 - Live 0xffffffffc020f000
pmt_class 16384 1 pmt_telemetry, Live 0xffffffffc0201000
intel_rapl_common 32768 1 intel_rapl_msr, Live 0xffffffffc02a7000
x86_pkg_temp_thermal 20480 0 - Live 0xffffffffc0769000
intel_powerclamp 20480 0 - Live 0xffffffffc0cc2000
coretemp 20480 0 - Live 0xffffffffc02e4000
kvm_intel 409600 0 - Live 0xffffffffc3c89000
kvm 1130496 1 kvm_intel, Live 0xffffffffc3b74000
irqbypass 16384 1 kvm, Live 0xffffffffc0dbd000
rapl 24576 0 - Live 0xffffffffc3b6d000
intel_cstate 24576 0 - Live 0xffffffffc02dd000
intel_uncore 225280 0 - Live 0xffffffffc3b35000
pcspkr 16384 0 - Live 0xffffffffc0de3000
eeepc_wmi 16384 0 - Live 0xffffffffc0dc3000
asus_wmi 53248 1 eeepc_wmi, Live 0xffffffffc3b02000
sparse_keymap 16384 1 asus_wmi, Live 0xffffffffc07e6000
wmi_bmof 16384 0 - Live 0xffffffffc07e1000
nvidia_drm 73728 12 - Live 0xffffffffc3b22000 (POE)
nvidia_modeset 1155072 16 nvidia_drm, Live 0xffffffffc39e7000 (POE)
nvidia_uvm 2777088 0 - Live 0xffffffffc3740000 (POE)
nvidia 40886272 1055 nvidia_modeset,nvidia_uvm, Live 0xffffffffc1041000 (POE)
iwlmvm 475136 0 - Live 0xffffffffc0fcc000
mac80211 1167360 1 iwlmvm, Live 0xffffffffc0eae000
snd_sof_pci_intel_tgl 16384 0 - Live 0xffffffffc0cf6000
snd_sof_intel_hda_common 147456 1 snd_sof_pci_intel_tgl, Live 0xffffffffc0e89000
soundwire_intel 49152 1 snd_sof_intel_hda_common, Live 0xffffffffc0d74000
soundwire_generic_allocation 16384 1 soundwire_intel, Live 0xffffffffc0cec000
soundwire_cadence 40960 1 soundwire_intel, Live 0xffffffffc0ce1000
snd_sof_intel_hda 20480 1 snd_sof_intel_hda_common, Live 0xffffffffc0cdb000
snd_sof_pci 24576 2 snd_sof_pci_intel_tgl,snd_sof_intel_hda_common, Live 0xffffffffc0cd4000
libarc4 16384 1 mac80211, Live 0xffffffffc08fd000
snd_sof_xtensa_dsp 16384 1 snd_sof_intel_hda_common, Live 0xffffffffc0788000
snd_sof 249856 2 snd_sof_intel_hda_common,snd_sof_pci, Live 0xffffffffc0e4b000
snd_sof_utils 20480 1 snd_sof, Live 0xffffffffc0b58000
snd_soc_hdac_hda 24576 1 snd_sof_intel_hda_common, Live 0xffffffffc0a64000
snd_hda_ext_core 36864 3 snd_sof_intel_hda_common,snd_sof_intel_hda,snd_soc_hdac_hda, Live 0xffffffffc0b98000
snd_soc_acpi_intel_match 69632 2 snd_sof_pci_intel_tgl,snd_sof_intel_hda_common, Live 0xffffffffc08eb000
snd_soc_acpi 16384 2 snd_sof_intel_hda_common,snd_soc_acpi_intel_match, Live 0xffffffffc03d6000
iwlwifi 385024 1 iwlmvm, Live 0xffffffffc0dec000
soundwire_bus 98304 3 soundwire_intel,soundwire_generic_allocation,soundwire_cadence, Live 0xffffffffc0dca000
snd_hda_codec_realtek 172032 1 - Live 0xffffffffc0d92000
snd_soc_core 344064 4 snd_sof_intel_hda_common,soundwire_intel,snd_sof,snd_soc_hdac_hda, Live 0xffffffffc0d1f000
snd_hda_codec_generic 98304 1 snd_hda_codec_realtek, Live 0xffffffffc0cfd000
i2c_i801 36864 0 - Live 0xffffffffc08e1000
mei_me 49152 0 - Live 0xffffffffc08d4000
i2c_smbus 20480 1 i2c_i801, Live 0xffffffffc0347000
cfg80211 1052672 3 iwlmvm,mac80211,iwlwifi, Live 0xffffffffc0bc0000
snd_compress 28672 1 snd_soc_core, Live 0xffffffffc0bb8000
ledtrig_audio 16384 1 snd_hda_codec_generic, Live 0xffffffffc09f8000
snd_hda_codec_hdmi 77824 1 - Live 0xffffffffc0ba4000
mei 155648 1 mei_me, Live 0xffffffffc0b71000
btusb 65536 0 - Live 0xffffffffc0b60000
btrtl 28672 1 btusb, Live 0xffffffffc09f0000
btbcm 20480 1 btusb, Live 0xffffffffc091c000
snd_hda_intel 57344 6 - Live 0xffffffffc0b40000
btintel 49152 1 btusb, Live 0xffffffffc09b4000
btmtk 16384 1 btusb, Live 0xffffffffc0917000
snd_intel_dspcfg 32768 3 snd_sof_intel_hda_common,snd_sof,snd_hda_intel, Live 0xffffffffc0908000
intel_lpss_pci 28672 0 - Live 0xffffffffc0327000
snd_intel_sdw_acpi 20480 2 snd_sof_intel_hda_common,snd_intel_dspcfg, Live 0xffffffffc0860000
intel_lpss 16384 1 intel_lpss_pci, Live 0xffffffffc0289000
idma64 20480 0 - Live 0xffffffffc0283000
bluetooth 864256 36 rfcomm,bnep,btusb,btrtl,btbcm,btintel,btmtk, Live 0xffffffffc0a6c000
snd_hda_codec 176128 5 snd_soc_hdac_hda,snd_hda_codec_realtek,snd_hda_codec_generic,snd_hda_codec_hdmi,snd_hda_intel, Live 0xffffffffc0a2f000
uas 32768 1 - Live 0xffffffffc0706000
usb_storage 81920 1 uas, Live 0xffffffffc0a1a000
snd_hda_core 110592 9 snd_sof_intel_hda_common,snd_sof_intel_hda,snd_soc_hdac_hda,snd_hda_ext_core,snd_hda_codec_realtek,snd_hda_codec_generic,snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec, Live 0xffffffffc09fe000
snd_hwdep 16384 1 snd_hda_codec, Live 0xffffffffc02f0000
snd_seq 94208 7 snd_seq_dummy, Live 0xffffffffc092c000
snd_seq_device 16384 1 snd_seq, Live 0xffffffffc02eb000
rfkill 36864 8 asus_wmi,iwlmvm,cfg80211,bluetooth, Live 0xffffffffc0922000
snd_pcm 151552 11 snd_sof_intel_hda_common,soundwire_intel,snd_sof,snd_sof_utils,snd_soc_core,snd_hda_codec_hdmi,snd_compress,snd_hda_intel,snd_hda_codec,snd_hda_core, Live 0xffffffffc09c4000
drm_kms_helper 192512 1 nvidia_drm, Live 0xffffffffc097e000
snd_timer 49152 3 snd_hrtimer,snd_seq,snd_pcm, Live 0xffffffffc0971000
snd 122880 26 snd_sof,snd_hda_codec_realtek,snd_soc_core,snd_hda_codec_generic,snd_compress,snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec,snd_hwdep,snd_seq,snd_seq_device,snd_pcm,snd_timer, Live 0xffffffffc0946000
intel_vsec 20480 0 - Live 0xffffffffc0375000
syscopyarea 16384 1 drm_kms_helper, Live 0xffffffffc0370000
sysfillrect 16384 1 drm_kms_helper, Live 0xffffffffc029b000
sysimgblt 16384 1 drm_kms_helper, Live 0xffffffffc0296000
soundcore 16384 1 snd, Live 0xffffffffc0291000
fb_sys_fops 16384 1 drm_kms_helper, Live 0xffffffffc035d000
intel_pmc_core 53248 0 - Live 0xffffffffc08c6000
acpi_tad 20480 0 - Live 0xffffffffc0414000
acpi_pad 184320 0 - Live 0xffffffffc0898000
raid1 57344 1 - Live 0xffffffffc080b000
drm 581632 16 nvidia_drm,nvidia,drm_kms_helper, Live 0xffffffffc0677000
xfs 2043904 4 - Live 0xffffffffc0483000
libcrc32c 16384 4 nf_nat,nf_conntrack,nf_tables,xfs, Live 0xffffffffc01b6000
sd_mod 69632 3 - Live 0xffffffffc01ef000
sg 49152 0 - Live 0xffffffffc01d0000
ahci 49152 2 - Live 0xffffffffc01bb000
libahci 53248 1 ahci, Live 0xffffffffc01a8000
nvme 57344 3 - Live 0xffffffffc0390000
crct10dif_pclmul 16384 1 - Live 0xffffffffc0353000
crc32_pclmul 16384 0 - Live 0xffffffffc03b1000
crc32c_intel 24576 1 - Live 0xffffffffc015e000
libata 425984 2 ahci,libahci, Live 0xffffffffc041a000
nvme_core 180224 6 nvme, Live 0xffffffffc03de000
r8169 106496 0 - Live 0xffffffffc03bb000
ghash_clmulni_intel 16384 0 - Live 0xffffffffc02c1000
nvme_common 24576 1 nvme_core, Live 0xffffffffc02d1000
realtek 36864 1 - Live 0xffffffffc02c7000
t10_pi 16384 2 sd_mod,nvme_core, Live 0xffffffffc02a2000
wmi 36864 2 asus_wmi,wmi_bmof, Live 0xffffffffc0262000
video 61440 1 asus_wmi, Live 0xffffffffc0198000
pinctrl_alderlake 36864 0 - Live 0xffffffffc018e000
serio_raw 20480 0 - Live 0xffffffffc0158000
dm_mirror 28672 0 - Live 0xffffffffc017e000
dm_region_hash 24576 1 dm_mirror, Live 0xffffffffc0174000
dm_log 24576 2 dm_mirror,dm_region_hash, Live 0xffffffffc0169000
dm_mod 204800 24 dm_zero,dm_crypt,dm_mirror,dm_log, Live 0xffffffffc0125000
fuse 176128 7 - Live 0xffffffffc00f9000

[dnadle@dln perf]$ sudo cat /proc/sys/kernel/tainted
12289

Comment 6 David Nadle 2023-01-04 05:26:17 UTC
Created attachment 1935619 [details]
perf report

Output of sudo perf record -p 144,81390 sleep 10

Comment 7 David Nadle 2023-01-04 05:30:53 UTC
Created attachment 1935620 [details]
perf report with call graph

Output of sudo perf record -p 144,81390 --call-graph lbr sleep 10

Comment 8 Rafael Aquini 2023-01-04 14:53:54 UTC
(In reply to David Nadle from comment #5)
> kcompactd0 is pid 144
> vmware-vmx is pid 81390
> 
> [dnadle@dln perf]$ sudo cat /proc/144/stack
> [dnadle@dln perf]$ sudo cat /proc/81390/stack
> [<0>] do_poll.constprop.0+0x225/0x390
> [<0>] do_sys_poll+0x1c8/0x260
> [<0>] __x64_sys_ppoll+0xbc/0x150
> [<0>] do_syscall_64+0x59/0x90
> [<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd
> 
David,

It would be better to have a sequence of stacks over time, just like the script in comment #3
would do it.

I noticed, though, there's no stack for kcompactd0 which seems weird given that kernel thread,
while not working, remains sleeping and I would expect to (at least) see something similar to:

[<0>] kcompactd+0x17c/0x340
[<0>] kthread+0x127/0x150
[<0>] ret_from_fork+0x22/0x30

as a result of your cat.


> [dnadle@dln perf]$ sudo cat /proc/sys/kernel/tainted
> 12289

that taint value means that we have taint bits 0, 12 and 13 set
which means PROPRIETARY_MODULE (P), OOT_MODULE (O) and UNSIGNED_MODULE (E)

and given the list of loaded modules, these are the ones causing the taints:
> vmnet 73728 17 - Live 0xffffffffc032f000 (OE)
> vmmon 147456 3 - Live 0xffffffffc02f8000 (OE)

> nvidia_drm 73728 12 - Live 0xffffffffc3b22000 (POE)
> nvidia_modeset 1155072 16 nvidia_drm, Live 0xffffffffc39e7000 (POE)
> nvidia_uvm 2777088 0 - Live 0xffffffffc3740000 (POE)
> nvidia 40886272 1055 nvidia_modeset,nvidia_uvm, Live 0xffffffffc1041000 (POE)
 
I'd suggest attempting to unload the NVIDIA proprietary modules for a test run.


You mentioned that kcompactd keeps running for hours burning 100% CPU, so I don't
know how significant a 10 second perf sampling can be to characterize where it
spends most of its time, but from your perf sampling it seems that it's the wait
for remote TLB flushing that is holding it back:

    20.19%     0.00%  kcompactd0       [kernel.kallsyms]  [k] kthread
            |
            ---kthread
               kcompactd
               kcompactd_do_work
               compact_zone
               |          
                --20.14%--migrate_pages
                          |          
                           --20.13%--unmap_and_move
                                     |          
                                      --20.11%--__unmap_and_move.constprop.0
                                                |          
                                                 --19.70%--try_to_migrate
                                                           |          
                                                            --19.68%--rmap_walk_file
                                                                      |          
                                                                       --19.63%--try_to_migrate_one
                                                                                 |          
                                                                                  --19.43%--ptep_clear_flush
                                                                                            |          
                                                                                             --19.42%--flush_tlb_mm_range
                                                                                                       |          
                                                                                                        --19.31%--on_each_cpu_cond_mask
                                                                                                                  |          
                                                                                                                   --19.31%--smp_call_function_many_cond
                                                                                                                             |          
                                                                                                                              --0.82%--__x2apic_send_IPI_mask
                                                                                                                                        |          
                                                                                                                                         --0.51%--__x2apic_send_IPI_dest


Also from what is collected in the perf data, it seems that there might be an interaction 
between VMWare's (vmmon) vmx-vcpu-1 thread and the kcompactd0 kernel thread, but that
is hard to follow up by the call-graph as it appears you are missing the debug symbols
for the running kernel and for the 3rd party module:


    19.98%     0.00%  vmx-vcpu-1       [unknown]          [k] 0xffffffff9ec0009b
            |
            ---0xffffffff9ec0009b
               0xffffffff9eb14159
               |          
                --19.98%--0xffffffff9e3d6fb7
                          |          
                           --19.98%--0xffffffffc02f9325
                                     |          
                                      --19.78%--0xffffffffc02fdc73
                                                |          
                                                 --19.66%--0xffffffffc03017d5
                                                           |          
                                                            --19.64%--0xffffffffc02fbe86


This one is starting to smell like a problem with these 3rd party modules, to be honest.
Have you approached VMWare support on this one?

-- Rafael

Comment 9 David Nadle 2023-01-04 15:37:45 UTC
Rafael,

Next time it happens I'll run some version of the script. The reason I didn't is that there is more than one VM running and therefore more than one vmware-vmx process.

I'm not sure which nvidia modules I can disable given that it's the graphics card driver.

How long would you suggest I collect perf data for?

I'm going to see if I can build a debug version of vmmon.

Yes, I have a support ticket open with VMware.

Comment 10 David Nadle 2023-01-08 16:30:59 UTC
Update: kernel updated to 5.14.0-226. vmmon built with -g. Took a few days to reencounter the kcompatd0 issue.

kcompactd still does not output any stacks:

1

[<0>] do_poll.constprop.0+0x225/0x390
[<0>] do_sys_poll+0x1c8/0x260
[<0>] __x64_sys_ppoll+0xbc/0x150
[<0>] do_syscall_64+0x59/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd

2

[<0>] do_poll.constprop.0+0x225/0x390
[<0>] do_sys_poll+0x1c8/0x260
[<0>] __x64_sys_ppoll+0xbc/0x150
[<0>] do_syscall_64+0x59/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd

3

[<0>] do_poll.constprop.0+0x225/0x390
[<0>] do_sys_poll+0x1c8/0x260
[<0>] __x64_sys_ppoll+0xbc/0x150
[<0>] do_syscall_64+0x59/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd

4

[<0>] do_poll.constprop.0+0x225/0x390
[<0>] do_sys_poll+0x1c8/0x260
[<0>] __x64_sys_ppoll+0xbc/0x150
[<0>] do_syscall_64+0x59/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd

5

[<0>] do_poll.constprop.0+0x225/0x390
[<0>] do_sys_poll+0x1c8/0x260
[<0>] __x64_sys_ppoll+0xbc/0x150
[<0>] do_syscall_64+0x59/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd

6

[<0>] do_poll.constprop.0+0x225/0x390
[<0>] do_sys_poll+0x1c8/0x260
[<0>] __x64_sys_ppoll+0xbc/0x150
[<0>] do_syscall_64+0x59/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd

7

[<0>] do_poll.constprop.0+0x225/0x390
[<0>] do_sys_poll+0x1c8/0x260
[<0>] __x64_sys_ppoll+0xbc/0x150
[<0>] do_syscall_64+0x59/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd

8

[<0>] do_poll.constprop.0+0x225/0x390
[<0>] do_sys_poll+0x1c8/0x260
[<0>] __x64_sys_ppoll+0xbc/0x150
[<0>] do_syscall_64+0x59/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd

9

[<0>] do_poll.constprop.0+0x225/0x390
[<0>] do_sys_poll+0x1c8/0x260
[<0>] __x64_sys_ppoll+0xbc/0x150
[<0>] do_syscall_64+0x59/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd

10

[<0>] do_poll.constprop.0+0x225/0x390
[<0>] do_sys_poll+0x1c8/0x260
[<0>] __x64_sys_ppoll+0xbc/0x150
[<0>] do_syscall_64+0x59/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd

Comment 11 David Nadle 2023-01-08 16:32:52 UTC
Created attachment 1936654 [details]
perf report 60s capture with vmmon symbols

Comment 12 David Nadle 2023-01-08 16:39:28 UTC
Created attachment 1936655 [details]
perf report 60s capture with call graph

Not sure how useful this capture is. Seems most of the VMware CPU time is spent in sections for which I have no debug symbols.

Comment 13 Rafael Aquini 2023-01-09 00:28:22 UTC
David,

If you didn't reboot the system after the last perf collectioin, can you attach here the output of "cat /proc/kallsyms" ? 

Thanks in advance!
-- Rafael

Comment 14 David Nadle 2023-01-09 01:35:34 UTC
Created attachment 1936706 [details]
output of cat /proc/kallsyms

I did not reboot the host but I did suspend and restore the Windows VM to allow kcompactd0 to complete.

Comment 15 Rafael Aquini 2023-01-09 14:20:13 UTC
(In reply to David Nadle from comment #14)
> Created attachment 1936706 [details]
> output of cat /proc/kallsyms
> 
> I did not reboot the host but I did suspend and restore the Windows VM to
> allow kcompactd0 to complete.

David,

the kallsysms dump has to be taken under a privileged account (root), 
otherwise no addresses for the symbols are available:

---8<---
0000000000000000 t do_boot_cpu
0000000000000000 T common_cpu_up
0000000000000000 T native_cpu_up
0000000000000000 T arch_disable_smp_support
0000000000000000 T arch_thaw_secondary_cpus_begin
0000000000000000 T arch_thaw_secondary_cpus_end
0000000000000000 T cpu_disable_common
0000000000000000 T native_cpu_disable
0000000000000000 T common_cpu_die
0000000000000000 T native_cpu_die
--->8---

Comment 16 David Nadle 2023-01-09 14:27:00 UTC
Created attachment 1936791 [details]
output of sudo cat /proc/kallsyms

Sorry about that...

Comment 17 Rafael Aquini 2023-01-10 18:32:03 UTC
(In reply to David Nadle from comment #16)
> Created attachment 1936791 [details]
> output of sudo cat /proc/kallsyms
> 

Thank you, David!

By comparing the symbol's base addresses and the call graph dumped by perf,
we can see that all those vmx-vcpu-# threads are spending ~20% of their time
(in the sampled period) at vmmon's APIC_Write routine:

here's one example.
19.96%     0.00%  vmx-vcpu-1       [unknown]          [k] 0xffffffff8be0009b
        |
        ---0xffffffff8be0009b  (entry_SYSCALL_64_after_hwframe + 0x63)
           |
            --19.96%--0xffffffff8bd1b159  (do_syscall_64 + 0x59)
                      |
                       --19.96%--0xffffffff8b5da9a7 (__x64_sys_ioctl + 0x87)
                                 |
                                  --19.96%--0xffffffffc0b0e325 (LinuxDriver_Ioctl ?) 
                                            |
                                             --19.76%--0xffffffffc0b12c73  (Vmx86_RunVM + 0xb3)
                                                       |
                                                        --19.65%--0xffffffffc0b167d5  (Task_Switch + 0x1845)
                                                                  |
                                                                   --19.63%--0xffffffffc0b10e86 (APIC_Write + 0x16)

it seems to have a relationship with kcompactd also working on APIC writes for the TLB shootdown,
although this is just a conjecture of mine at the moment.

I'd suggest you pushing this through VMWare support to see if they have any knwon issue
registered. I also would like to see if you can get it reproducing without loading your 
NVIDIA modules (disable your GPU for a test run, if possible).

Also, do you happen to notice any lockup messages in the kernel logs when these
processes are running? Can you attach the output of "dmesg" here, please?

Last, but not least, in order to give us a better idea on what might be preventing
cat /proc/$(pidof -w kcompactd0)/stack to produce output, would you mind running it
again, but this way instead: "cat /proc/$(pidof -w kcompactd0)/stack; echo $?"

Thanks in advance

-- Rafael

Comment 19 David Nadle 2023-01-12 14:42:29 UTC
Update: kernel updated to 5.14.0-229. Removed NVIDIA drivers, running X11 session with nouveau.

kcompactd0 still does not list any stacks:

1
0
--
[<0>] do_poll.constprop.0+0x225/0x390
[<0>] do_sys_poll+0x1c8/0x260
[<0>] __x64_sys_ppoll+0xbc/0x150
[<0>] do_syscall_64+0x59/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd

2
0
--
[<0>] do_poll.constprop.0+0x225/0x390
[<0>] do_sys_poll+0x1c8/0x260
[<0>] __x64_sys_ppoll+0xbc/0x150
[<0>] do_syscall_64+0x59/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd

3
0
--
[<0>] do_poll.constprop.0+0x225/0x390
[<0>] do_sys_poll+0x1c8/0x260
[<0>] __x64_sys_ppoll+0xbc/0x150
[<0>] do_syscall_64+0x59/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd

4
0
--
[<0>] do_poll.constprop.0+0x225/0x390
[<0>] do_sys_poll+0x1c8/0x260
[<0>] __x64_sys_ppoll+0xbc/0x150
[<0>] do_syscall_64+0x59/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd

5
0
--
[<0>] do_poll.constprop.0+0x225/0x390
[<0>] do_sys_poll+0x1c8/0x260
[<0>] __x64_sys_ppoll+0xbc/0x150
[<0>] do_syscall_64+0x59/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd

6
0
--
[<0>] do_poll.constprop.0+0x225/0x390
[<0>] do_sys_poll+0x1c8/0x260
[<0>] __x64_sys_ppoll+0xbc/0x150
[<0>] do_syscall_64+0x59/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd

7
0
--
[<0>] do_poll.constprop.0+0x225/0x390
[<0>] do_sys_poll+0x1c8/0x260
[<0>] __x64_sys_ppoll+0xbc/0x150
[<0>] do_syscall_64+0x59/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd

8
0
--
[<0>] do_poll.constprop.0+0x225/0x390
[<0>] do_sys_poll+0x1c8/0x260
[<0>] __x64_sys_ppoll+0xbc/0x150
[<0>] do_syscall_64+0x59/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd

9
0
--
[<0>] do_poll.constprop.0+0x225/0x390
[<0>] do_sys_poll+0x1c8/0x260
[<0>] __x64_sys_ppoll+0xbc/0x150
[<0>] do_syscall_64+0x59/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd

10
0
--
[<0>] do_poll.constprop.0+0x225/0x390
[<0>] do_sys_poll+0x1c8/0x260
[<0>] __x64_sys_ppoll+0xbc/0x150
[<0>] do_syscall_64+0x59/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd

Nothing in dmesg during the incident.

/proc/modules:
dm_zero 16384 4 - Live 0xffffffffc0478000
ppdev 24576 0 - Live 0xffffffffc093f000
parport_pc 49152 0 - Live 0xffffffffc08f7000
parport 77824 2 ppdev,parport_pc, Live 0xffffffffc084c000
vmw_vsock_vmci_transport 45056 1 - Live 0xffffffffc0832000
vsock 53248 3 vmw_vsock_vmci_transport, Live 0xffffffffc07d5000
vmw_vmci 118784 4 vmw_vsock_vmci_transport, Live 0xffffffffc13b2000
vmnet 73728 19 - Live 0xffffffffc081f000 (OE)
vmmon 147456 3 - Live 0xffffffffc07aa000 (OE)
tls 131072 0 - Live 0xffffffffc048a000
uinput 20480 0 - Live 0xffffffffc0414000
binfmt_misc 28672 1 - Live 0xffffffffc03ec000
rfcomm 94208 4 - Live 0xffffffffc042d000
snd_seq_dummy 16384 0 - Live 0xffffffffc0397000
snd_hrtimer 16384 1 - Live 0xffffffffc0389000
nft_objref 16384 1 - Live 0xffffffffc0382000
nf_conntrack_netbios_ns 16384 1 - Live 0xffffffffc0374000
nf_conntrack_broadcast 16384 1 nf_conntrack_netbios_ns, Live 0xffffffffc036f000
nft_fib_inet 16384 1 - Live 0xffffffffc036a000
nft_fib_ipv4 16384 1 nft_fib_inet, Live 0xffffffffc0315000
nft_fib_ipv6 16384 1 nft_fib_inet, Live 0xffffffffc0306000
nft_fib 16384 3 nft_fib_inet,nft_fib_ipv4,nft_fib_ipv6, Live 0xffffffffc0301000
nft_reject_inet 16384 6 - Live 0xffffffffc02fc000
nf_reject_ipv4 16384 1 nft_reject_inet, Live 0xffffffffc02f7000
nf_reject_ipv6 20480 1 nft_reject_inet, Live 0xffffffffc02f1000
nft_reject 16384 1 nft_reject_inet, Live 0xffffffffc02e0000
nft_ct 24576 15 - Live 0xffffffffc02ce000
nft_chain_nat 16384 3 - Live 0xffffffffc02d8000
nf_nat 57344 1 nft_chain_nat, Live 0xffffffffc0326000
nf_conntrack 180224 4 nf_conntrack_netbios_ns,nf_conntrack_broadcast,nft_ct,nf_nat, Live 0xffffffffc086f000
nf_defrag_ipv6 24576 1 nf_conntrack, Live 0xffffffffc02c7000
nf_defrag_ipv4 16384 1 nf_conntrack, Live 0xffffffffc02c2000
ip_set 61440 0 - Live 0xffffffffc02b2000
nf_tables 274432 221 nft_objref,nft_fib_inet,nft_fib_ipv4,nft_fib_ipv6,nft_fib,nft_reject_inet,nft_reject,nft_ct,nft_chain_nat, Live 0xffffffffc0e9b000
nfnetlink 20480 3 ip_set,nf_tables, Live 0xffffffffc02ac000
qrtr 53248 4 - Live 0xffffffffc029e000
bnep 32768 2 - Live 0xffffffffc0295000
sunrpc 696320 1 - Live 0xffffffffc1551000
dm_crypt 61440 5 - Live 0xffffffffc023a000
vfat 20480 1 - Live 0xffffffffc0234000
fat 86016 1 vfat, Live 0xffffffffc0217000
squashfs 77824 20 - Live 0xffffffffc0274000
loop 36864 40 - Live 0xffffffffc020d000
intel_rapl_msr 20480 0 - Live 0xffffffffc0460000
iTCO_wdt 16384 0 - Live 0xffffffffc07a5000
iTCO_vendor_support 16384 1 iTCO_wdt, Live 0xffffffffc07a0000
i2c_designware_platform 16384 0 - Live 0xffffffffc0359000
i2c_designware_core 36864 1 i2c_designware_platform, Live 0xffffffffc0345000
pmt_telemetry 16384 0 - Live 0xffffffffc0290000
pmt_class 16384 1 pmt_telemetry, Live 0xffffffffc028b000
intel_rapl_common 32768 1 intel_rapl_msr, Live 0xffffffffc0481000
x86_pkg_temp_thermal 20480 0 - Live 0xffffffffc0353000
intel_powerclamp 20480 0 - Live 0xffffffffc035e000
coretemp 20480 0 - Live 0xffffffffc0364000
kvm_intel 409600 0 - Live 0xffffffffc14ec000
kvm 1130496 1 kvm_intel, Live 0xffffffffc13d7000
irqbypass 16384 1 kvm, Live 0xffffffffc07d0000
rapl 24576 0 - Live 0xffffffffc13d0000
intel_cstate 24576 0 - Live 0xffffffffc031f000
intel_uncore 225280 0 - Live 0xffffffffc137a000
eeepc_wmi 16384 0 - Live 0xffffffffc0f60000
pcspkr 16384 0 - Live 0xffffffffc0473000
asus_wmi 53248 1 eeepc_wmi, Live 0xffffffffc107b000
sparse_keymap 16384 1 asus_wmi, Live 0xffffffffc031a000
wmi_bmof 16384 0 - Live 0xffffffffc046e000
iwlmvm 475136 0 - Live 0xffffffffc1305000
mac80211 1224704 1 iwlmvm, Live 0xffffffffc11d9000
snd_sof_pci_intel_tgl 16384 0 - Live 0xffffffffc0bc8000
snd_sof_intel_hda_common 147456 1 snd_sof_pci_intel_tgl, Live 0xffffffffc11b4000
soundwire_intel 49152 1 snd_sof_intel_hda_common, Live 0xffffffffc0f53000
libarc4 16384 1 mac80211, Live 0xffffffffc0807000
soundwire_generic_allocation 16384 1 soundwire_intel, Live 0xffffffffc0802000
soundwire_cadence 40960 1 soundwire_intel, Live 0xffffffffc1099000
snd_sof_intel_hda 20480 1 snd_sof_intel_hda_common, Live 0xffffffffc1093000
snd_sof_pci 24576 2 snd_sof_pci_intel_tgl,snd_sof_intel_hda_common, Live 0xffffffffc108c000
snd_sof_xtensa_dsp 16384 1 snd_sof_intel_hda_common, Live 0xffffffffc0b94000
snd_sof 253952 2 snd_sof_intel_hda_common,snd_sof_pci, Live 0xffffffffc1175000
snd_sof_utils 20480 1 snd_sof, Live 0xffffffffc0969000
snd_soc_hdac_hda 24576 1 snd_sof_intel_hda_common, Live 0xffffffffc090b000
snd_hda_ext_core 36864 3 snd_sof_intel_hda_common,snd_sof_intel_hda,snd_soc_hdac_hda, Live 0xffffffffc116b000
iwlwifi 385024 1 iwlmvm, Live 0xffffffffc110c000
snd_hda_codec_realtek 172032 1 - Live 0xffffffffc10e1000
snd_soc_acpi_intel_match 69632 2 snd_sof_pci_intel_tgl,snd_sof_intel_hda_common, Live 0xffffffffc0c09000
snd_soc_acpi 16384 2 snd_sof_intel_hda_common,snd_soc_acpi_intel_match, Live 0xffffffffc0840000
soundwire_bus 98304 3 soundwire_intel,soundwire_generic_allocation,soundwire_cadence, Live 0xffffffffc10c8000
snd_hda_codec_generic 98304 1 snd_hda_codec_realtek, Live 0xffffffffc10af000
cfg80211 1085440 3 iwlmvm,mac80211,iwlwifi, Live 0xffffffffc0f68000
ledtrig_audio 16384 1 snd_hda_codec_generic, Live 0xffffffffc09bf000
snd_soc_core 348160 4 snd_sof_intel_hda_common,soundwire_intel,snd_sof,snd_soc_hdac_hda, Live 0xffffffffc0ef4000
nouveau 2469888 47 - Live 0xffffffffc0c3f000
snd_hda_codec_hdmi 77824 1 - Live 0xffffffffc0c2b000
snd_compress 28672 1 snd_soc_core, Live 0xffffffffc0a57000
i2c_i801 36864 0 - Live 0xffffffffc0c21000
snd_hda_intel 57344 6 - Live 0xffffffffc0b85000
i2c_smbus 20480 1 i2c_i801, Live 0xffffffffc0846000
snd_intel_dspcfg 32768 3 snd_sof_intel_hda_common,snd_sof,snd_hda_intel, Live 0xffffffffc0975000
mei_me 49152 0 - Live 0xffffffffc08e2000
snd_intel_sdw_acpi 20480 2 snd_sof_intel_hda_common,snd_intel_dspcfg, Live 0xffffffffc019f000
mei 155648 1 mei_me, Live 0xffffffffc0bd9000
snd_hda_codec 176128 5 snd_soc_hdac_hda,snd_hda_codec_realtek,snd_hda_codec_generic,snd_hda_codec_hdmi,snd_hda_intel, Live 0xffffffffc0b9c000
btusb 65536 0 - Live 0xffffffffc0b74000
btrtl 28672 1 btusb, Live 0xffffffffc09c5000
btbcm 20480 1 btusb, Live 0xffffffffc0963000
snd_hda_core 110592 9 snd_sof_intel_hda_common,snd_sof_intel_hda,snd_soc_hdac_hda,snd_hda_ext_core,snd_hda_codec_realtek,snd_hda_codec_generic,snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec, Live 0xffffffffc0b4b000
btintel 49152 1 btusb, Live 0xffffffffc0956000
btmtk 16384 1 btusb, Live 0xffffffffc0951000
mxm_wmi 16384 1 nouveau, Live 0xffffffffc094c000
snd_hwdep 16384 1 snd_hda_codec, Live 0xffffffffc0947000
i2c_algo_bit 16384 1 nouveau, Live 0xffffffffc038f000
intel_lpss_pci 28672 0 - Live 0xffffffffc07e6000
intel_lpss 16384 1 intel_lpss_pci, Live 0xffffffffc037d000
bluetooth 937984 36 rfcomm,bnep,btusb,btrtl,btbcm,btintel,btmtk, Live 0xffffffffc0a65000
idma64 20480 0 - Live 0xffffffffc0a5f000
drm_display_helper 172032 1 nouveau, Live 0xffffffffc0a23000
snd_seq 94208 7 snd_seq_dummy, Live 0xffffffffc0a0b000
snd_seq_device 16384 1 snd_seq, Live 0xffffffffc07ee000
cec 61440 1 drm_display_helper, Live 0xffffffffc09fb000
snd_pcm 151552 11 snd_sof_intel_hda_common,soundwire_intel,snd_sof,snd_sof_utils,snd_hda_codec_hdmi,snd_soc_core,snd_compress,snd_hda_intel,snd_hda_codec,snd_hda_core, Live 0xffffffffc09d5000
drm_ttm_helper 16384 1 nouveau, Live 0xffffffffc0428000
ttm 90112 2 nouveau,drm_ttm_helper, Live 0xffffffffc08cb000
rfkill 36864 8 asus_wmi,iwlmvm,cfg80211,bluetooth, Live 0xffffffffc0865000
drm_kms_helper 192512 2 nouveau,drm_display_helper, Live 0xffffffffc098f000
snd_timer 49152 3 snd_hrtimer,snd_seq,snd_pcm, Live 0xffffffffc0932000
snd 122880 26 snd_sof,snd_hda_codec_realtek,snd_hda_codec_generic,snd_soc_core,snd_hda_codec_hdmi,snd_compress,snd_hda_intel,snd_hda_codec,snd_hwdep,snd_seq,snd_seq_device,snd_pcm,snd_timer, Live 0xffffffffc0913000
intel_vsec 20480 0 - Live 0xffffffffc02eb000
syscopyarea 16384 1 drm_kms_helper, Live 0xffffffffc0208000
sysfillrect 16384 1 drm_kms_helper, Live 0xffffffffc0203000
sysimgblt 16384 1 drm_kms_helper, Live 0xffffffffc01fe000
soundcore 16384 1 snd, Live 0xffffffffc01f9000
fb_sys_fops 16384 1 drm_kms_helper, Live 0xffffffffc01ca000
intel_pmc_core 53248 0 - Live 0xffffffffc07f4000
acpi_tad 20480 0 - Live 0xffffffffc02e5000
acpi_pad 184320 0 - Live 0xffffffffc089d000
raid1 57344 1 - Live 0xffffffffc0810000
drm 581632 24 nouveau,drm_display_helper,drm_ttm_helper,ttm,drm_kms_helper, Live 0xffffffffc070c000
xfs 2043904 4 - Live 0xffffffffc0518000
libcrc32c 16384 4 nf_nat,nf_conntrack,nf_tables,xfs, Live 0xffffffffc0146000
uas 32768 1 - Live 0xffffffffc01aa000
usb_storage 81920 1 uas, Live 0xffffffffc01e4000
sd_mod 69632 3 - Live 0xffffffffc01d2000
sg 49152 0 - Live 0xffffffffc01b8000
ahci 49152 2 - Live 0xffffffffc0338000
libahci 53248 1 ahci, Live 0xffffffffc0191000
nvme 57344 3 - Live 0xffffffffc0402000
crct10dif_pclmul 16384 1 - Live 0xffffffffc01b3000
libata 425984 2 ahci,libahci, Live 0xffffffffc04af000
crc32_pclmul 16384 0 - Live 0xffffffffc044d000
crc32c_intel 24576 1 - Live 0xffffffffc026d000
nvme_core 180224 6 nvme, Live 0xffffffffc03bb000
r8169 106496 0 - Live 0xffffffffc03a0000
ghash_clmulni_intel 16384 0 - Live 0xffffffffc0265000
realtek 36864 1 - Live 0xffffffffc025b000
nvme_common 24576 1 nvme_core, Live 0xffffffffc030e000
t10_pi 16384 2 sd_mod,nvme_core, Live 0xffffffffc014d000
wmi 36864 4 asus_wmi,wmi_bmof,nouveau,mxm_wmi, Live 0xffffffffc024b000
video 61440 2 asus_wmi,nouveau, Live 0xffffffffc0181000
pinctrl_alderlake 36864 0 - Live 0xffffffffc0177000
serio_raw 20480 0 - Live 0xffffffffc0140000
dm_mirror 28672 0 - Live 0xffffffffc0167000
dm_region_hash 24576 1 dm_mirror, Live 0xffffffffc015d000
dm_log 24576 2 dm_mirror,dm_region_hash, Live 0xffffffffc0152000
dm_mod 204800 24 dm_zero,dm_crypt,dm_mirror,dm_log, Live 0xffffffffc010d000
fuse 176128 7 - Live 0xffffffffc00e1000

Comment 20 David Nadle 2023-01-12 14:43:40 UTC
Created attachment 1937599 [details]
perf report

Comment 21 David Nadle 2023-01-12 14:44:51 UTC
Created attachment 1937600 [details]
perf report call graph

Comment 22 David Nadle 2023-01-12 14:46:29 UTC
Created attachment 1937601 [details]
kallsyms

Comment 23 David Nadle 2023-01-12 14:50:59 UTC
Interestingly, this time kcompactd0 completed on its own soon after I ran perf, without my having to suspend the Windows VM. I estimate kcompactd0 ran for about 50 minutes.

Comment 24 Rafael Aquini 2023-01-14 01:37:35 UTC
(In reply to David Nadle from comment #23)
> Interestingly, this time kcompactd0 completed on its own soon after I ran
> perf, without my having to suspend the Windows VM. I estimate kcompactd0 ran
> for about 50 minutes.

Did you happen to notice any lockup/hang messages in the kernel logs while
kcompactd0 was running? Can you attach the output of "dmesg" here, please?

Comment 25 David Nadle 2023-01-14 02:04:21 UTC
There were no lines output in dmesg during the incident. One interesting wrinkle that happened this morning is that it seemed to be the Fedora guest that was affected and not the Windows guest. The Fedora guest was slow but not as completely frozen up as the Windows guest was. Dragging windows around didn't work but you could see the display updating. So the issue appears to be between VMware and the kernel and not necessarily related to the guest OS. kcompactd0 did not complete on its own while I was observing it. It may have been running all night. I suspended the VM to get it to complete.

Comment 26 Rafael Aquini 2023-02-02 15:23:52 UTC
David, 

Would you mind checking this test kernel against your reproducer?

https://s3.upshift.redhat.com/DH-PROD-CKI/index.html?prefix=internal-artifacts/764655335/publish%20x86_64/3699711725/artifacts/repo/5.14.0-253.1996_764655335.el9.x86_64/

Upstream has recently identified a problem with commit
7efc3b726103 ("mm/compaction: fix set skip in fast_find_migrateblock")
that caused issues in a similar fashion to those you are observing,
and the aforementioned test build reverts it.

Thanks in advance,
-- Rafael

Comment 30 David Nadle 2023-02-02 17:32:42 UTC
Rafael,

The link does not work for me. Is that server in public DNS?

[dnadle@dln ~]$ host s3.upshift.redhat.com
Host s3.upshift.redhat.com not found: 3(NXDOMAIN)

Regards,
David Nadle

Comment 31 Rafael Aquini 2023-02-02 19:08:10 UTC
(In reply to David Nadle from comment #30)
> Rafael,
> 
> The link does not work for me. Is that server in public DNS?
> 

UGH! my bad, sorry!

I moved the RPMs to:

https://people.redhat.com/~raquini/bz2157958/

-- Rafael

Comment 32 David Nadle 2023-02-07 13:39:16 UTC
(In reply to Rafael Aquini from comment #31)
> I moved the RPMs to:
> 
> https://people.redhat.com/~raquini/bz2157958/

I have the files but I will need some guidance on how to install the kernel. For example:

[dnadle@dln test-kernel]$ rpm -i --test kernel-5.14.0-253.1996_764655335.el9.x86_64.rpm
error: Failed dependencies:
	kernel-core-uname-r = 5.14.0-253.1996_764655335.el9.x86_64 is needed by kernel-5.14.0-253.1996_764655335.el9.x86_64
	kernel-modules-uname-r = 5.14.0-253.1996_764655335.el9.x86_64 is needed by kernel-5.14.0-253.1996_764655335.el9.x86_64

Do I manually install the dependencies first?

Comment 33 Rafael Aquini 2023-02-07 15:40:42 UTC
(In reply to David Nadle from comment #32)
> (In reply to Rafael Aquini from comment #31)
> > I moved the RPMs to:
> > 
> > https://people.redhat.com/~raquini/bz2157958/
> 
> I have the files but I will need some guidance on how to install the kernel.
> For example:
> 
> [dnadle@dln test-kernel]$ rpm -i --test
> kernel-5.14.0-253.1996_764655335.el9.x86_64.rpm
> error: Failed dependencies:
> 	kernel-core-uname-r = 5.14.0-253.1996_764655335.el9.x86_64 is needed by
> kernel-5.14.0-253.1996_764655335.el9.x86_64
> 	kernel-modules-uname-r = 5.14.0-253.1996_764655335.el9.x86_64 is needed by
> kernel-5.14.0-253.1996_764655335.el9.x86_64
> 
> Do I manually install the dependencies first?

David,

You can do it one by one, or just tell RPM to do it in one single transaction:

# rpm -ihv --test kernel-5.14.0-253.1996_764655335.el9.x86_64.rpm kernel-core-5.14.0-253.1996_764655335.el9.x86_64.rpm kernel-modules-5.14.0-253.1996_764655335.el9.x86_64.rpm kernel-devel-5.14.0-253.1996_764655335.el9.x86_64.rpm 
warning: Signature not supported. Hash algorithm SHA1 not available.
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]


(then to install, just drop the --test)


-- Rafael

Comment 34 David Nadle 2023-02-20 15:44:20 UTC
I have been running this test kernel since the morning of Feb. 8. Later that morning I experienced a brief run of kcompactd0 that affected the Win11 VM UI performance, but by the time I could prepare to capture process info kcompactd0 has completed. I have not seen another issue since. kcompactd0 had been locking up on average every 4 days so I think no incidents in 12 days is a good sign.

Kernel 5.14.0-267.el9 is now available for me to install. I will move on to that kernel and see if the problem returns. I would appreciate it if someone can report in this thread if or when the reverted commit that fixed the test kernel makes it into the mainstream Centos9 kernel.

Comment 35 Rafael Aquini 2023-02-20 20:24:36 UTC
(In reply to David Nadle from comment #34)
> I would appreciate it if someone can report in this thread 
> if or when the reverted commit that fixed the test
> kernel makes it into the mainstream Centos9 kernel.

David, 

First: thank you for testing that scratch-build kernel and reporting back
it really addressed your issue with kcompactd.

The patch you tested in that scratch-build is now part of CentOS-Stream-9
kernel-5.14.0-264.el9 interim build.

-- Rafael

Comment 37 Red Hat Bugzilla 2023-09-19 04:32:06 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days