Bug 143472 - hugetlb mmap failed in compatibility mode in em64t
Summary: hugetlb mmap failed in compatibility mode in em64t
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: kernel   
(Show other bugs)
Version: 4.0
Hardware: x86_64 Linux
medium
medium
Target Milestone: ---
: ---
Assignee: Jason Baron
QA Contact:
URL:
Whiteboard:
Keywords:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2004-12-21 11:29 UTC by Racing Guo
Modified: 2013-03-06 05:58 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2005-06-08 15:13:14 UTC
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
test case to reproduce bug (901 bytes, text/plain)
2004-12-21 11:31 UTC, Racing Guo
no flags Details
The test case to show the unmapped area overflow beyond 4GB for IA32 processes (2.29 KB, text/plain)
2005-01-26 01:16 UTC, Zhang Yanmin
no flags Details
The patch to fix these bugs (12.19 KB, patch)
2005-01-26 01:20 UTC, Zhang Yanmin
no flags Details | Diff


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2005:420 normal SHIPPED_LIVE Important: Updated kernel packages available for Red Hat Enterprise Linux 4 Update 1 2005-06-08 04:00:00 UTC

Description Racing Guo 2004-12-21 11:29:05 UTC
Description of problem:
 This kernel enables unmapped area topdown search sequence at 
compatibility mode of EM64T, but forgets to enable macro 
HAVE_ARCH_HUGETLB_UNMAPPED_AREA

Version-Release number of selected component (if applicable):
OS Version: RHEL4-pre-rc1
kernel version: 2.6.9-1.849_EL

How reproducible:

Steps to Reproduce:
1.  echo "100">/proc/sys/vm/nr_hugepages
2.  gcc -m32 rhel4prerc1_hugtlb_mmap_fail.c
3. a.out
  
Actual results:
   Segment fault happens

Expected results:
  No segment fault happens

Additional info:

Comment 1 Racing Guo 2004-12-21 11:31:49 UTC
Created attachment 108937 [details]
test case to reproduce bug

Comment 2 Jason Baron 2004-12-22 20:44:05 UTC
so it seems like we need the following patch:

--- linux-2.6.9/include/asm-x86_64/page.h.bak	Wed Dec 22 15:33:35 2004
+++ linux-2.6.9/include/asm-x86_64/page.h	Wed Dec 22 15:34:56 2004
@@ -24,10 +24,13 @@
 #define LARGE_PAGE_MASK (~(LARGE_PAGE_SIZE-1))
 #define LARGE_PAGE_SIZE (1UL << PMD_SHIFT)
 
+#ifdef CONFIG_HUGETLB_PAGE
 #define HPAGE_SHIFT PMD_SHIFT
 #define HPAGE_SIZE	((1UL) << HPAGE_SHIFT)
 #define HPAGE_MASK	(~(HPAGE_SIZE - 1))
 #define HUGETLB_PAGE_ORDER	(HPAGE_SHIFT - PAGE_SHIFT)
+#define HAVE_ARCH_HUGETLB_UNMAPPED_AREA
+#endif
 
 #ifdef __KERNEL__
 #ifndef __ASSEMBLY__





Comment 3 Jason Baron 2004-12-22 20:45:15 UTC
i'll build a test kernel with that patch


Comment 4 Racing Guo 2005-01-17 09:56:21 UTC
This bug still exists in RHEL4-RC

Comment 5 Jason Baron 2005-01-17 22:06:02 UTC
Has the above patch been tested? If not, i'll build a kernel for us to
test. thanks.


Comment 6 Racing Guo 2005-01-18 00:50:33 UTC
Yes. It works. But we didn't see this patch has been integrated into 
RHEL4-RC.

Comment 7 Zhang Yanmin 2005-01-26 01:10:12 UTC
If we apply the patch at comment 2, another bug will be triggered. 
hugetlb_get_unmapped_area_topdown in file arch/i386/mm/hugetlbpage.c 
might cause vma overflow out of 4GB address space of IA32 processes 
on x86_64. Then, the third bug will be triggered that a memory leak 
will happen when the IA32 process exit because kernel just releases 
the page table below 4GB.

I will add the test case and a patch to fix these bugs.


Comment 8 Zhang Yanmin 2005-01-26 01:16:55 UTC
Created attachment 110234 [details]
The test case to show the unmapped area overflow beyond 4GB for IA32 processes

Reproduce it:
1) Apply the patch of comment 2 to kernel;
2) #mkdir /root/test/mmap/
3) #echo "100">/proc/sys/vm/nr_hugepages
4) #mount -t hugetlbfs none /root/test/mmap
5) #gcc -o test4 -m32 test4.c
6) #./test4
7) #cat /proc/test4_pid/maps

If we delete the getchar() in test4.c and use '#while true; do ./test4; done'
to execute test4 looply, the memory will be used up soon.

Comment 9 Zhang Yanmin 2005-01-26 01:20:07 UTC
Created attachment 110235 [details]
The patch to fix these bugs

Pls. don't apply the patch of comment #2 before applying
rhel4-rc-hugetlb_mmap_fail2.patch.diff. rhel4-rc-hugetlb_mmap_fail2.patch.diff
includes the patch of comment #2.

Comment 10 Jason Baron 2005-02-17 16:44:02 UTC
Ok. thanks. I don't see this patch posted to upstream?

Comment 11 Jason Baron 2005-02-17 23:01:40 UTC
I guess its not upstream b/c flexmap isn't there for x86_64

Comment 13 Racing Guo 2005-02-24 05:20:11 UTC
This bug still exists in RHEL4

Comment 14 Tim Powers 2005-06-08 15:13:14 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2005-420.html



Note You need to log in before you can comment on or make changes to this bug.