Bug 143472 - hugetlb mmap failed in compatibility mode in em64t
Summary: hugetlb mmap failed in compatibility mode in em64t
Alias: None
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: kernel   
(Show other bugs)
Version: 4.0
Hardware: x86_64 Linux
Target Milestone: ---
: ---
Assignee: Jason Baron
QA Contact:
Depends On:
TreeView+ depends on / blocked
Reported: 2004-12-21 11:29 UTC by Racing Guo
Modified: 2013-03-06 05:58 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2005-06-08 15:13:14 UTC
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
test case to reproduce bug (901 bytes, text/plain)
2004-12-21 11:31 UTC, Racing Guo
no flags Details
The test case to show the unmapped area overflow beyond 4GB for IA32 processes (2.29 KB, text/plain)
2005-01-26 01:16 UTC, Zhang Yanmin
no flags Details
The patch to fix these bugs (12.19 KB, patch)
2005-01-26 01:20 UTC, Zhang Yanmin
no flags Details | Diff

External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2005:420 normal SHIPPED_LIVE Important: Updated kernel packages available for Red Hat Enterprise Linux 4 Update 1 2005-06-08 04:00:00 UTC

Description Racing Guo 2004-12-21 11:29:05 UTC
Description of problem:
 This kernel enables unmapped area topdown search sequence at 
compatibility mode of EM64T, but forgets to enable macro 

Version-Release number of selected component (if applicable):
OS Version: RHEL4-pre-rc1
kernel version: 2.6.9-1.849_EL

How reproducible:

Steps to Reproduce:
1.  echo "100">/proc/sys/vm/nr_hugepages
2.  gcc -m32 rhel4prerc1_hugtlb_mmap_fail.c
3. a.out
Actual results:
   Segment fault happens

Expected results:
  No segment fault happens

Additional info:

Comment 1 Racing Guo 2004-12-21 11:31:49 UTC
Created attachment 108937 [details]
test case to reproduce bug

Comment 2 Jason Baron 2004-12-22 20:44:05 UTC
so it seems like we need the following patch:

--- linux-2.6.9/include/asm-x86_64/page.h.bak	Wed Dec 22 15:33:35 2004
+++ linux-2.6.9/include/asm-x86_64/page.h	Wed Dec 22 15:34:56 2004
@@ -24,10 +24,13 @@
 #define HPAGE_SIZE	((1UL) << HPAGE_SHIFT)
 #define HPAGE_MASK	(~(HPAGE_SIZE - 1))
 #ifdef __KERNEL__
 #ifndef __ASSEMBLY__

Comment 3 Jason Baron 2004-12-22 20:45:15 UTC
i'll build a test kernel with that patch

Comment 4 Racing Guo 2005-01-17 09:56:21 UTC
This bug still exists in RHEL4-RC

Comment 5 Jason Baron 2005-01-17 22:06:02 UTC
Has the above patch been tested? If not, i'll build a kernel for us to
test. thanks.

Comment 6 Racing Guo 2005-01-18 00:50:33 UTC
Yes. It works. But we didn't see this patch has been integrated into 

Comment 7 Zhang Yanmin 2005-01-26 01:10:12 UTC
If we apply the patch at comment 2, another bug will be triggered. 
hugetlb_get_unmapped_area_topdown in file arch/i386/mm/hugetlbpage.c 
might cause vma overflow out of 4GB address space of IA32 processes 
on x86_64. Then, the third bug will be triggered that a memory leak 
will happen when the IA32 process exit because kernel just releases 
the page table below 4GB.

I will add the test case and a patch to fix these bugs.

Comment 8 Zhang Yanmin 2005-01-26 01:16:55 UTC
Created attachment 110234 [details]
The test case to show the unmapped area overflow beyond 4GB for IA32 processes

Reproduce it:
1) Apply the patch of comment 2 to kernel;
2) #mkdir /root/test/mmap/
3) #echo "100">/proc/sys/vm/nr_hugepages
4) #mount -t hugetlbfs none /root/test/mmap
5) #gcc -o test4 -m32 test4.c
6) #./test4
7) #cat /proc/test4_pid/maps

If we delete the getchar() in test4.c and use '#while true; do ./test4; done'
to execute test4 looply, the memory will be used up soon.

Comment 9 Zhang Yanmin 2005-01-26 01:20:07 UTC
Created attachment 110235 [details]
The patch to fix these bugs

Pls. don't apply the patch of comment #2 before applying
rhel4-rc-hugetlb_mmap_fail2.patch.diff. rhel4-rc-hugetlb_mmap_fail2.patch.diff
includes the patch of comment #2.

Comment 10 Jason Baron 2005-02-17 16:44:02 UTC
Ok. thanks. I don't see this patch posted to upstream?

Comment 11 Jason Baron 2005-02-17 23:01:40 UTC
I guess its not upstream b/c flexmap isn't there for x86_64

Comment 13 Racing Guo 2005-02-24 05:20:11 UTC
This bug still exists in RHEL4

Comment 14 Tim Powers 2005-06-08 15:13:14 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.