Bug 143472 - hugetlb mmap failed in compatibility mode in em64t
hugetlb mmap failed in compatibility mode in em64t
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: kernel (Show other bugs)
x86_64 Linux
medium Severity medium
: ---
: ---
Assigned To: Jason Baron
Depends On:
  Show dependency treegraph
Reported: 2004-12-21 06:29 EST by Racing Guo
Modified: 2013-03-06 00:58 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2005-06-08 11:13:14 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
test case to reproduce bug (901 bytes, text/plain)
2004-12-21 06:31 EST, Racing Guo
no flags Details
The test case to show the unmapped area overflow beyond 4GB for IA32 processes (2.29 KB, text/plain)
2005-01-25 20:16 EST, Zhang Yanmin
no flags Details
The patch to fix these bugs (12.19 KB, patch)
2005-01-25 20:20 EST, Zhang Yanmin
no flags Details | Diff

  None (edit)
Description Racing Guo 2004-12-21 06:29:05 EST
Description of problem:
 This kernel enables unmapped area topdown search sequence at 
compatibility mode of EM64T, but forgets to enable macro 

Version-Release number of selected component (if applicable):
OS Version: RHEL4-pre-rc1
kernel version: 2.6.9-1.849_EL

How reproducible:

Steps to Reproduce:
1.  echo "100">/proc/sys/vm/nr_hugepages
2.  gcc -m32 rhel4prerc1_hugtlb_mmap_fail.c
3. a.out
Actual results:
   Segment fault happens

Expected results:
  No segment fault happens

Additional info:
Comment 1 Racing Guo 2004-12-21 06:31:49 EST
Created attachment 108937 [details]
test case to reproduce bug
Comment 2 Jason Baron 2004-12-22 15:44:05 EST
so it seems like we need the following patch:

--- linux-2.6.9/include/asm-x86_64/page.h.bak	Wed Dec 22 15:33:35 2004
+++ linux-2.6.9/include/asm-x86_64/page.h	Wed Dec 22 15:34:56 2004
@@ -24,10 +24,13 @@
 #define HPAGE_SIZE	((1UL) << HPAGE_SHIFT)
 #define HPAGE_MASK	(~(HPAGE_SIZE - 1))
 #ifdef __KERNEL__
 #ifndef __ASSEMBLY__

Comment 3 Jason Baron 2004-12-22 15:45:15 EST
i'll build a test kernel with that patch
Comment 4 Racing Guo 2005-01-17 04:56:21 EST
This bug still exists in RHEL4-RC
Comment 5 Jason Baron 2005-01-17 17:06:02 EST
Has the above patch been tested? If not, i'll build a kernel for us to
test. thanks.
Comment 6 Racing Guo 2005-01-17 19:50:33 EST
Yes. It works. But we didn't see this patch has been integrated into 
Comment 7 Zhang Yanmin 2005-01-25 20:10:12 EST
If we apply the patch at comment 2, another bug will be triggered. 
hugetlb_get_unmapped_area_topdown in file arch/i386/mm/hugetlbpage.c 
might cause vma overflow out of 4GB address space of IA32 processes 
on x86_64. Then, the third bug will be triggered that a memory leak 
will happen when the IA32 process exit because kernel just releases 
the page table below 4GB.

I will add the test case and a patch to fix these bugs.
Comment 8 Zhang Yanmin 2005-01-25 20:16:55 EST
Created attachment 110234 [details]
The test case to show the unmapped area overflow beyond 4GB for IA32 processes

Reproduce it:
1) Apply the patch of comment 2 to kernel;
2) #mkdir /root/test/mmap/
3) #echo "100">/proc/sys/vm/nr_hugepages
4) #mount -t hugetlbfs none /root/test/mmap
5) #gcc -o test4 -m32 test4.c
6) #./test4
7) #cat /proc/test4_pid/maps

If we delete the getchar() in test4.c and use '#while true; do ./test4; done'
to execute test4 looply, the memory will be used up soon.
Comment 9 Zhang Yanmin 2005-01-25 20:20:07 EST
Created attachment 110235 [details]
The patch to fix these bugs

Pls. don't apply the patch of comment #2 before applying
rhel4-rc-hugetlb_mmap_fail2.patch.diff. rhel4-rc-hugetlb_mmap_fail2.patch.diff
includes the patch of comment #2.
Comment 10 Jason Baron 2005-02-17 11:44:02 EST
Ok. thanks. I don't see this patch posted to upstream?
Comment 11 Jason Baron 2005-02-17 18:01:40 EST
I guess its not upstream b/c flexmap isn't there for x86_64
Comment 13 Racing Guo 2005-02-24 00:20:11 EST
This bug still exists in RHEL4
Comment 14 Tim Powers 2005-06-08 11:13:14 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.