Red Hat Bugzilla – Bug 143472
hugetlb mmap failed in compatibility mode in em64t
Last modified: 2013-03-06 00:58:01 EST
Description of problem:
This kernel enables unmapped area topdown search sequence at
compatibility mode of EM64T, but forgets to enable macro
Version-Release number of selected component (if applicable):
OS Version: RHEL4-pre-rc1
kernel version: 2.6.9-1.849_EL
Steps to Reproduce:
1. echo "100">/proc/sys/vm/nr_hugepages
2. gcc -m32 rhel4prerc1_hugtlb_mmap_fail.c
Segment fault happens
No segment fault happens
Created attachment 108937 [details]
test case to reproduce bug
so it seems like we need the following patch:
--- linux-2.6.9/include/asm-x86_64/page.h.bak Wed Dec 22 15:33:35 2004
+++ linux-2.6.9/include/asm-x86_64/page.h Wed Dec 22 15:34:56 2004
@@ -24,10 +24,13 @@
#define LARGE_PAGE_MASK (~(LARGE_PAGE_SIZE-1))
#define LARGE_PAGE_SIZE (1UL << PMD_SHIFT)
#define HPAGE_SHIFT PMD_SHIFT
#define HPAGE_SIZE ((1UL) << HPAGE_SHIFT)
#define HPAGE_MASK (~(HPAGE_SIZE - 1))
#define HUGETLB_PAGE_ORDER (HPAGE_SHIFT - PAGE_SHIFT)
i'll build a test kernel with that patch
This bug still exists in RHEL4-RC
Has the above patch been tested? If not, i'll build a kernel for us to
Yes. It works. But we didn't see this patch has been integrated into
If we apply the patch at comment 2, another bug will be triggered.
hugetlb_get_unmapped_area_topdown in file arch/i386/mm/hugetlbpage.c
might cause vma overflow out of 4GB address space of IA32 processes
on x86_64. Then, the third bug will be triggered that a memory leak
will happen when the IA32 process exit because kernel just releases
the page table below 4GB.
I will add the test case and a patch to fix these bugs.
Created attachment 110234 [details]
The test case to show the unmapped area overflow beyond 4GB for IA32 processes
1) Apply the patch of comment 2 to kernel;
2) #mkdir /root/test/mmap/
3) #echo "100">/proc/sys/vm/nr_hugepages
4) #mount -t hugetlbfs none /root/test/mmap
5) #gcc -o test4 -m32 test4.c
7) #cat /proc/test4_pid/maps
If we delete the getchar() in test4.c and use '#while true; do ./test4; done'
to execute test4 looply, the memory will be used up soon.
Created attachment 110235 [details]
The patch to fix these bugs
Pls. don't apply the patch of comment #2 before applying
includes the patch of comment #2.
Ok. thanks. I don't see this patch posted to upstream?
I guess its not upstream b/c flexmap isn't there for x86_64
This bug still exists in RHEL4
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.