Bug 143472 - hugetlb mmap failed in compatibility mode in em64t
hugetlb mmap failed in compatibility mode in em64t
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: kernel (Show other bugs)
4.0
x86_64 Linux
medium Severity medium
: ---
: ---
Assigned To: Jason Baron
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2004-12-21 06:29 EST by Racing Guo
Modified: 2013-03-06 00:58 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2005-06-08 11:13:14 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
test case to reproduce bug (901 bytes, text/plain)
2004-12-21 06:31 EST, Racing Guo
no flags Details
The test case to show the unmapped area overflow beyond 4GB for IA32 processes (2.29 KB, text/plain)
2005-01-25 20:16 EST, Zhang Yanmin
no flags Details
The patch to fix these bugs (12.19 KB, patch)
2005-01-25 20:20 EST, Zhang Yanmin
no flags Details | Diff

  None (edit)
Description Racing Guo 2004-12-21 06:29:05 EST
Description of problem:
 This kernel enables unmapped area topdown search sequence at 
compatibility mode of EM64T, but forgets to enable macro 
HAVE_ARCH_HUGETLB_UNMAPPED_AREA

Version-Release number of selected component (if applicable):
OS Version: RHEL4-pre-rc1
kernel version: 2.6.9-1.849_EL

How reproducible:

Steps to Reproduce:
1.  echo "100">/proc/sys/vm/nr_hugepages
2.  gcc -m32 rhel4prerc1_hugtlb_mmap_fail.c
3. a.out
  
Actual results:
   Segment fault happens

Expected results:
  No segment fault happens

Additional info:
Comment 1 Racing Guo 2004-12-21 06:31:49 EST
Created attachment 108937 [details]
test case to reproduce bug
Comment 2 Jason Baron 2004-12-22 15:44:05 EST
so it seems like we need the following patch:

--- linux-2.6.9/include/asm-x86_64/page.h.bak	Wed Dec 22 15:33:35 2004
+++ linux-2.6.9/include/asm-x86_64/page.h	Wed Dec 22 15:34:56 2004
@@ -24,10 +24,13 @@
 #define LARGE_PAGE_MASK (~(LARGE_PAGE_SIZE-1))
 #define LARGE_PAGE_SIZE (1UL << PMD_SHIFT)
 
+#ifdef CONFIG_HUGETLB_PAGE
 #define HPAGE_SHIFT PMD_SHIFT
 #define HPAGE_SIZE	((1UL) << HPAGE_SHIFT)
 #define HPAGE_MASK	(~(HPAGE_SIZE - 1))
 #define HUGETLB_PAGE_ORDER	(HPAGE_SHIFT - PAGE_SHIFT)
+#define HAVE_ARCH_HUGETLB_UNMAPPED_AREA
+#endif
 
 #ifdef __KERNEL__
 #ifndef __ASSEMBLY__



Comment 3 Jason Baron 2004-12-22 15:45:15 EST
i'll build a test kernel with that patch
Comment 4 Racing Guo 2005-01-17 04:56:21 EST
This bug still exists in RHEL4-RC
Comment 5 Jason Baron 2005-01-17 17:06:02 EST
Has the above patch been tested? If not, i'll build a kernel for us to
test. thanks.
Comment 6 Racing Guo 2005-01-17 19:50:33 EST
Yes. It works. But we didn't see this patch has been integrated into 
RHEL4-RC.
Comment 7 Zhang Yanmin 2005-01-25 20:10:12 EST
If we apply the patch at comment 2, another bug will be triggered. 
hugetlb_get_unmapped_area_topdown in file arch/i386/mm/hugetlbpage.c 
might cause vma overflow out of 4GB address space of IA32 processes 
on x86_64. Then, the third bug will be triggered that a memory leak 
will happen when the IA32 process exit because kernel just releases 
the page table below 4GB.

I will add the test case and a patch to fix these bugs.
Comment 8 Zhang Yanmin 2005-01-25 20:16:55 EST
Created attachment 110234 [details]
The test case to show the unmapped area overflow beyond 4GB for IA32 processes

Reproduce it:
1) Apply the patch of comment 2 to kernel;
2) #mkdir /root/test/mmap/
3) #echo "100">/proc/sys/vm/nr_hugepages
4) #mount -t hugetlbfs none /root/test/mmap
5) #gcc -o test4 -m32 test4.c
6) #./test4
7) #cat /proc/test4_pid/maps

If we delete the getchar() in test4.c and use '#while true; do ./test4; done'
to execute test4 looply, the memory will be used up soon.
Comment 9 Zhang Yanmin 2005-01-25 20:20:07 EST
Created attachment 110235 [details]
The patch to fix these bugs

Pls. don't apply the patch of comment #2 before applying
rhel4-rc-hugetlb_mmap_fail2.patch.diff. rhel4-rc-hugetlb_mmap_fail2.patch.diff
includes the patch of comment #2.
Comment 10 Jason Baron 2005-02-17 11:44:02 EST
Ok. thanks. I don't see this patch posted to upstream?
Comment 11 Jason Baron 2005-02-17 18:01:40 EST
I guess its not upstream b/c flexmap isn't there for x86_64
Comment 13 Racing Guo 2005-02-24 00:20:11 EST
This bug still exists in RHEL4
Comment 14 Tim Powers 2005-06-08 11:13:14 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2005-420.html

Note You need to log in before you can comment on or make changes to this bug.