Bug 114112 - shmat() returns EINVAL with redhat kernel but not with stock 2.4.21 kernel
shmat() returns EINVAL with redhat kernel but not with stock 2.4.21 kernel
Product: Red Hat Enterprise Linux 3
Classification: Red Hat
Component: kernel (Show other bugs)
i686 Linux
medium Severity medium
: ---
: ---
Assigned To: Arjan van de Ven
Brian Brock
Depends On:
  Show dependency treegraph
Reported: 2004-01-22 13:26 EST by Nick Cabatoff
Modified: 2007-11-30 17:07 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2004-01-22 13:46:50 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Nick Cabatoff 2004-01-22 13:26:45 EST
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.5) Gecko/20031007

Description of problem:
I'm trying to use Sybase ASE 11 on Enterprise Linux Release 3.  It
works fine on {E,W,A}S Release 2.1 and Redhat < 9, but not on the
newer versions.  The problem appears to occur when attaching a shared
memory segment.  An strace from the older versions contains lines like

  shmget(1493352882, 260300800, IPC_CREAT|IPC_EXCL|0600) = 32768
  shmat(32768, 0x41000000, 0) = 0x41000000

On WS 3 I see lines like this:

  shmget(16957874, 259203072, IPC_CREAT|IPC_EXCL|0600) = 491520
  shmat(491520, 0xb8000000, 0)         = -1 EINVAL (Invalid argument)

The address is actually valid (it's computed by ASE from the data in
/proc/sys/maps, and I've verified that a block of the size requested
fits in that location.)  According to shmop(2), when shmat fails with
EINVAL it could be:

  Invalid shmid value, unaligned (i.e., not page-aligned and SHM_RND
  was not specified) or invalid shmaddr value, or failing attach at

The shmid is clearly valid having been returned by shmget (and can be
seen using ipcs -a).  The address is page-aligned and valid.  I don't
know what could cause brk to fail other than lack of memory, and as
already stated, that doesn't appear to be the case.  

Here's the contents of /proc/self/maps, also gleaned from the strace

  08048000-0837b000 r-xp 00000000 03:02 1311284   
  0837b000-083f4000 rw-p 00332000 03:02 1311284   
  083f4000-0840a000 rw-p 00000000 00:00 0
  b7485000-b7486000 rw-p 00000000 00:00 0
  b7486000-b75b9000 r-xp 00000000 03:02 49578      /lib/i686/libc-2.2.4.so
  b75b9000-b75be000 rw-p 00132000 03:02 49578      /lib/i686/libc-2.2.4.so
  b75be000-b75c2000 rw-p 00000000 00:00 0
  b75c2000-b75e4000 r-xp 00000000 03:02 49580      /lib/i686/libm-2.2.4.so
  b75e4000-b75e5000 rw-p 00021000 03:02 49580      /lib/i686/libm-2.2.4.so
  b75e5000-b75e8000 r-xp 00000000 03:02 2818441    /lib/libdl-2.2.4.so
  b75e8000-b75e9000 rw-p 00002000 03:02 2818441    /lib/libdl-2.2.4.so 
  b75e9000-b75ea000 rw-p 00000000 00:00 0
  b75ea000-b7600000 r-xp 00000000 03:02 2818428    /lib/ld-2.2.4.so
  b7600000-b7601000 rw-p 00015000 03:02 2818428    /lib/ld-2.2.4.so
  bfff4000-c0000000 rwxp ffff7000 00:00 0

I tried booting a WS 2.1 box with the 2.4.21-9.EL kernel and the
problem then started occuring there.  I've also tried building a
non-Redhat 2.4.21 kernel (i.e. one built from the upstream sources
without any RH patches) and the problem doesn't occur with it.

In summary, the problem occurs with 2.4.21-4.EL and 2.4.21-9.EL, but
not with 2.4.9-e.24 or 2.4.21 [unpatched, non-Redhat].

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
I don't have access to the application's source code and I'm not a
skilled enough C programmer to try to recreate the problem on my own.

Additional info:
Comment 1 Arjan van de Ven 2004-01-22 13:33:13 EST
  shmget(16957874, 259203072, IPC_CREAT|IPC_EXCL|0600) = 491520
  shmat(491520, 0xb8000000, 0)         = -1 EINVAL (Invalid argument)

the application asks for 259203072 bytes at address 0xb8000000.. with
the process address space ending at 0xc0000000 (and the stack being
there) that doesn't look like it will it.....
sounds like a bug in the program to me; it asks for something invalid
and the kernel returns -EINVAL
Comment 2 Nick Cabatoff 2004-01-22 13:46:50 EST
Huh!  I'll be... that makes perfect sense.  If I'd researched the
format of /proc/$$/maps I might've worked that out for myself.  

Thanks very much for the incredibly quick reply.
Comment 3 Need Real Name 2004-09-07 15:23:07 EDT
Hmmm...  Not really sure if this should be classified as "NOTABUG"
because sybase works properly under other non-RHEL3 versions, but OK.

The buzz in the sybase community is that older versions of Sybase do
not work, which will make folks avoid RHEL3 for sybase applications.  

Personally, I think the folks at RedHat tweeked the output of
/proc/self/maps to the point that it breaks some applications.  It
would be nice if RedHat took this seriously to figure out why this is
happening, or provide a workaround.  When sybase boots, it will only
allow a maximum of 128MB of shared memory.  My guess is that whatever
line it is reading in /proc/self/maps incorrectly reports the amount
of free shared memory that is available.

For the benefit of others, here is a workaround that makes older
versions of sybase 11.x, work properly under RHEL3.

To fix sybase, it is necessary to edit the sybase
"dataserver" binary to fool sybase into thinking that
"/proc/self/maps" is invalid.  Older versions of Sybase consulted this
file during initialization to figure out where to map the shared
memory.  If sybase doesn't find this file, it consults the kernel
which aparently works.

Do the following:

vi -b dataserver
search for the **FIRST** occurrence "/proc/self/maps":

/proc/self/maps^@NULL^@key: 0x%lx id: %d file:

Change "/proc" to something different... I used "/Proc".  Change ONLY
the above entry.

Save the binary and sybase will now properly assign the shared memory,
as defined in /proc/sys/kernel/shmmax.


Note You need to log in before you can comment on or make changes to this bug.