Created attachment 1549716 [details] kernel log 1. Please describe the problem: The process kcompactd0 is using 100% cpu on one core during performance demanding tasks. In my case this is happening when I'm running applications using much resources in a virtual machine. 2. What is the Version-Release number of the kernel: 5.0.4-200.fc29.x86_64 3. Did it work previously in Fedora? If so, what kernel version did the issue *first* appear? Old kernels are available for download at https://koji.fedoraproject.org/koji/packageinfo?packageID=8 : The issue started after upgrading to the 5.0.4 kernel. The issue was not present in kernel-core-4.20.14-200 4. Can you reproduce this issue? If so, please provide the steps to reproduce the issue below: Yes. 1. Run performance demanding tasks such as running virtual machines, compiling a large source tree or similar. 5. Does this problem occur with the latest Rawhide kernel? To install the Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by ``sudo dnf update --enablerepo=rawhide kernel``: 6. Are you running any modules that not shipped with directly Fedora's kernel?: Yes. zfs and Nvidia 7. Please attach the kernel logs. You can get the complete kernel log for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the issue occurred on a previous boot, use the journalctl ``-b`` flag. The issue is known upstream and is discussed in the following LKML-thread: https://lore.kernel.org/lkml/20190126200005.GB27513@amd/T/#u A workaround is to drop the caches which mitigates the issue for a short while: # echo 3 > /proc/sys/vm/drop_caches
Created attachment 1549717 [details] trace.dat.gz A trace while kcompactd is pegged at 100% is attached, and created by using the following command: # trace-cmd record -a -e compaction -e migrate -e kmem:mm_page_alloc -e vmscan:mm_vmscan_kswapd_wake -e vmscan:mm_vmscan_kswapd_sleep sleep 10
Created attachment 1549718 [details] transparent_hugepage As requested upstream[1] the output from # grep -r . /sys/kernel/mm/transparent_hugepage/* is attached. [1]: https://lore.kernel.org/lkml/20190130104020.GE9565@techsingularity.net/
Created attachment 1550794 [details] kcompatd-trace Another datapoint requested from upstream dev.
Reply from maintainer at kernel-mm: What I'm seeing from the trace is that vmware in particular is making a large number of high-order allocation requests and waking kcompactd at high frequency to try and allocate them and failing due to memory fragmentation issues. ZFS is also doing the same although NVIDIA seems to be ok. With the presense of proprietary modules, there is no much I can do, it's bad in general for kernel drivers to rely so heavily on high-order allocations and it can cause fragmentation issues like this and it would be something that would be heavily frowned upon or even NAKed in an in-tree driver. You'll be able to relieve some of the pressure by disabling transparent huge pages but it won't be perfect by a long shot. You may also be able to relieve some of the pressure by increase min_free_kbytes although you'll find that kswapd is a bit more active as a result but you may be able to side-step the fragmentation issues. You may also be able to relieve some of the pressure by specifying slub_max_order=0 on the kernel command line assuming the Fedora kernel uses CONFIG_SLUB but this is all mitigation of problems caused by the proprietary modules, not a solution. If this worked perfectly in the past, then I would suggest bisection and brining it to the linux-mm list with me cc'd if it points to anything mm/ related. In general though, the proprietary modules will mean that this gets limited to no traction upstream.
*********** MASS BUG UPDATE ************** We apologize for the inconvenience. There are a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 29 kernel bugs. Fedora 29 has now been rebased to 5.2.9-100.fc29. Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel. If you have moved on to Fedora 30, and are still experiencing this issue, please change the version to Fedora 30. If you experience different issues, please open a new bug report for those.
*********** MASS BUG UPDATE ************** This bug is being closed with INSUFFICIENT_DATA as there has not been a response in 3 weeks. If you are still experiencing this issue, please reopen and attach the relevant data from the latest kernel you are running and any data that might have been requested previously.
I have the exact same symptom and configuration: kernel 5.0.0-31-generic, Nvidia gpu, ZFSOnLinux 0.8.2 and VMware Workstation 15, except it is running on Ubuntu 18.04LTS distro instead of RedHat flavour. kcompactd0 goes away only with drop_caches. when it goes 100%, the vmware virtual machine guest is completely unresponsive (windows 10 ltsc vm) Please let me know what commands you want me to run and upload the output.
I don't have any additional information to provide and will let this issue be closed.
Removing myself from NEEDINFO
** Adding this information here for searchability, but leaving the BZ closed. ** This has been an ongoing issue that VMware Workstation triggers for many years. Disabling hugepages and dropping caches often does not help. As a non-ideal workaround disabling Linux memory compaction entirely, or just not using VMware Workstation, may help. https://communities.vmware.com/t5/VMware-Workstation-Pro/VMWare-workstation-in-a-fistfight-with-Linux-Memory-Compactor/td-p/2876992 Temporarily "echo 0 > /proc/sys/vm/compaction_proactiveness" while running VMware Workstation... Or you can set it permanently in the following, but that might lead to other issues: /etc/sysctl.conf: vm.compaction_proactiveness=0
I have experienced the 100% system cpu usage by kcompactd0 on "6.1.6-200.fc37.x86_64" kernel (fedora 37), without nvidia, zfs and vmware. I am using hugepages and zram (default fedora swap). Good thing is that workaround to drop the caches (from this bug) helped. Can't find any clues that might point at what triggered the issue.
I have since determined how to resolve this completely for VMware Workstation but it is not ideal. Turn off memory page trimming in VMware, this appears to be done on a per VM basis: VM -> Settings -> Options -> Advanced -> Settings -> memory page trimming is not allowed This appears to stop VMware and the Linux kernel kcompactd from fighting each other. This same type of issue probably could happen with anything that tries to compact memory at the same time that kcompactd is also trying to do so.