We want to enable/support CMA in RHEL9 on x86-64 (and eventually aarch64). Enabling CMA mostly involves enabling CONFIG_CMA and adding RHEL-specific warnings that CMA areas were defined / CMA allocations happened. As one example CMA will be required to eventually support RDMA<->GPU p2pdma via dma-buf. As another example, CMA will be useful useful for more reliable runtime allocation of gigantic pages. CMA is already enabled in ARK for s390x and ppc64le. For aarch64, 64k base page size currently implies a minimum CMA area size of 512 MiB (due to large pageblock order), which could be problematic on smaller machines.
Related MR https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1023
Change has been merged into kernel-ark: https://gitlab.com/cki-project/kernel-ark/-/commit/4c454e8be716ad8b7b529afd221f380447ea0f9a Closing as RAWHIDE as the change is available in upstream kernel-ark.
@Ping Fang I remember closing as RAWHIDE this is the right procedure for kernel-ark, but I wonder how to to handle QE for such things. Do you know what the right target state is once on upstream kernel-ark?
Okay, thanks. Reopening this one as "TestOnly" for simplicity.
@Ping Fang who'd be the right QE person to tackle testing this feature (hoping that there is capacity :) )? I can give some guidance regarding what/how to test.
We can also test with gigantic page allocation via CMA. I assume a system with 64G memory, 1. Define "hugetlb_cma=16G" on the kernel cmdline 2. Boot the system and observe how much free memory there is (free -g), like 62G 3. Run some workload (e.g., memtester 62G) that consumes most of the free memory in the system to try shuffling free page lists. 4. Stop the workload. 5. Run a workload that leaves roughly 4G in the system free (e.g., memtester 58G) 6. While the workload is running, try allocating some gigantic pages. echo 4 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages 7. Observe if allocation succeeded cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages Might have to retry 6/7 a couple of times if such an extreme workload (memtester) is running concurrently. memtester might be an extreme workload as in mlocks all memory. I'll try out myself later if that workload is suitable for testing CMA here or if we need another one (like memhog).
(In reply to David Hildenbrand from comment #18) > We can also test with gigantic page allocation via CMA. I assume a system > with 64G memory, > > 1. Define "hugetlb_cma=16G" on the kernel cmdline > > 2. Boot the system and observe how much free memory there is (free -g), like > 62G > > 3. Run some workload (e.g., memtester 62G) that consumes most of the free > memory in the system to try shuffling free page lists. > > 4. Stop the workload. > > 5. Run a workload that leaves roughly 4G in the system free (e.g., memtester > 58G) > > 6. While the workload is running, try allocating some gigantic pages. > > echo 4 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages > > 7. Observe if allocation succeeded > > cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages Sorry, both paths should target /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages