Bug 236662 - Sun 32-bit JVM unable to allocate large pages memory greater than 2147483648 bytes
Sun 32-bit JVM unable to allocate large pages memory greater than 2147483648 ...
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: kernel (Show other bugs)
x86_64 Linux
medium Severity medium
: ---
: ---
Assigned To: Larry Woodman
Martin Jenner
Depends On:
  Show dependency treegraph
Reported: 2007-04-16 21:33 EDT by Shrinivas Joshi
Modified: 2007-11-16 20:14 EST (History)
5 users (show)

See Also:
Fixed In Version: RHBA-2007-0791
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2007-11-15 11:25:10 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Shrinivas Joshi 2007-04-16 21:33:26 EDT
Description of problem:
Sun JVM exits with error:
Java HotSpot(TM) Server VM warning: Failed to reserve shared memory (errno = 22)
when it tries to allocate heap > 1980M and is configured to use large page
support. The OS trace generated using strace command shows that the cause of the
problem is call to shmget syscall.

Version-Release number of selected component (if applicable):
Dual-core AMD Opteron 870 4 processor system (8 cores). 8G system memory.
The problem is probably reproducible on other x86_64 systems with memory greater
than 4G
RHEL 4 update 3, 2.6.9-34.ELsmp, x86_64 
Sun 32-bit JVM 1.5 build 6

How reproducible:
Try to initialize JVM process on above mentioned system configuration.

Steps to Reproduce:
1. Install Sun JVM 32-bit 1.5 build 6 or above
2. Configure the system to enable large pages after OS reboot (to avoid
 a. echo 8589934591 > /proc/sys/kernel/shmmax //All 8G shareable
 b. echo 3072 > /proc/sys/vm/nr_hugepages //6G for large pages of 2048K each
 c. verify whether large pages are allocated using cat /proc/meminfo;
HugePagesTotal and HugePagesFree should be equal to 3072
3. Start installed JVM process using:
<java-install-path>/bin/java -Xms1984M -Xmx1984M -XX:+UseLargePages -version
Problem is in fact reproducible for any -Xms and -Xmx values greater than 1980M,
however make sure that the requested memory aligns to 2M pages (1980M, 1982M,
1984M ...)
Actual results:
JVM exits with error:
Java HotSpot(TM) Server VM warning: Failed to reserve shared memory (errno = 22)

Expected results:
JVM should successfully allocate requested memory using large pages support

Additional info: It appears that the call to shmget fails due to max permissible
value of its second parameter "size" in 32-bit compatibility mode. However even
in 32-bit compatibility mode the "size" parameter which is of type unsigned long
should allow values <= 2^32 (4G) rather than 2^31 (2G, signed long max value)
Comment 1 Jakub Jelinek 2007-04-16 22:37:41 EDT
This has nothing to do with compat-libs (which have been only shipped in RHEL2.1
and never since then).
Comment 2 Larry Woodman 2007-04-17 13:16:54 EDT
I doubt we can change this without breaking the ABI.  Even worse, this is the
syscall interface so changing this will break user applications.  Basically
32-bit compatibility mode limits system V shared memory regions to 231-1 because
sys32_ipc() uses ints for args.

asmlinkage long
sys32_ipc(u32 call, int first, int second, int third,
               compat_uptr_t ptr, u32 fifth)
       int version;
       version = call >> 16; /* hack for backward compatibility */
       call &= 0xffff;
       switch (call) {
             case SHMGET:
               return sys_shmget(first, second, third);

Larry Woodman

Comment 3 Larry Woodman 2007-04-18 11:40:35 EDT
I made this upstream change to RHEL4 and it seems to fix the problem:

--- linux-2.6.9/arch/x86_64/ia32/ipc32.c.orig
+++ linux-2.6.9/arch/x86_64/ia32/ipc32.c
@@ -49,7 +49,7 @@ sys32_ipc(u32 call, int first, int secon
              case SHMDT:
                return sys_shmdt(compat_ptr(ptr));
              case SHMGET:
-               return sys_shmget(first, second, third);
+               return sys_shmget(first, (unsigned)second, third);
              case SHMCTL:
                return compat_sys_shmctl(first, second, compat_ptr(ptr));

I built a new kernel located here can you try it out and let me know if its OK
with you???


Larry Woodman
Comment 4 Bhavna Sarathy 2007-05-15 09:12:33 EDT
This fix results in a 5-10% increase improvement in SPECjAppServer benchmark and
AMD would like customers to have this fix before RHEL4.6 which will release in
Oct 24.  AMD would like to request a kernel errata with this fix made available
to customers.  The actual performance gain will depend on the SUT configuration
and with injection rates of > 450 the performance will improve considerably.

Comment 5 Russell Doty 2007-07-10 17:52:10 EDT
Peter M. indicates that it was ACK'd by Rik and Alan on 4/19/2007. Larry Woodman
says he will make sure Jason commits the patch, he does know about it.
Comment 6 RHEL Product and Program Management 2007-07-10 17:54:58 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
Comment 9 Jason Baron 2007-07-31 16:25:32 EDT
committed in stream U6 build 55.6. A test kernel with this patch is available
from http://people.redhat.com/~jbaron/rhel4/
Comment 13 errata-xmlrpc 2007-11-15 11:25:10 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.