Bug 136659 - [PATCH] shared memory can silently overwrite allocated heap in a process from shmat call
[PATCH] shared memory can silently overwrite allocated heap in a process from...
Status: CLOSED WONTFIX
Product: Red Hat Enterprise Linux 2.1
Classification: Red Hat
Component: kernel (Show other bugs)
2.1
All Linux
medium Severity medium
: ---
: ---
Assigned To: Jim Paradis
Brian Brock
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2004-10-21 11:00 EDT by Neil Horman
Modified: 2013-08-05 21:09 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2006-01-12 15:48:54 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
test program from IBM to recreate this issue (2.53 KB, text/plain)
2004-10-21 11:01 EDT, Neil Horman
no flags Details
patch to make shmat return -EINVAL if used addr is passed to shmat (1.25 KB, patch)
2004-10-21 11:03 EDT, Neil Horman
no flags Details | Diff

  None (edit)
Description Neil Horman 2004-10-21 11:00:31 EDT
Description of problem:
IBM reported this to me.  Using their attached test program, it can be
seen that if a call to shmat is made, and an address is passed to it
that is already in use in the process address space, the shmat call
will silently overwrite the previous allocation with the shared memory.

Version-Release number of selected component (if applicable):
all AS2.1 kernels

How reproducible:
always

Steps to Reproduce:
1. Compile the attached program 
2. Make sure to allow sufficient space for shared memory allocations
by bumping up the value of /proc/sys/kernel/shmmax (the test program
will inform as to how much based on the allocation size).
3. Run the compiled program
  
Actual results:
Both malloc and shmat succede.  Looking at the maps file of the
process, you can see the address space of the malloc is overwritten by
the mapping of the shared memory when the call to shmat is made.

Expected results:
shmat call should fail, and return EINVAL.

Additional info:
Technically there appears to be nothing in the SUS that indicates this
should fail, but it certainly seems broken to sliently overwrite
memory mappings like this.  2.6 and the RHEL3 2.4 tree have changed
this behavior to return EINVAL if a already in use memory address is
passed to shmat and the SHM_REMAP flag is not set.
Comment 1 Neil Horman 2004-10-21 11:01:13 EDT
Created attachment 105588 [details]
test program from IBM to recreate this issue

this is the test program for use in the recreation
Comment 2 Neil Horman 2004-10-21 11:03:03 EDT
Created attachment 105589 [details]
patch to make shmat return -EINVAL if used addr is passed to shmat

This is a 2.6 backport of the patch to shmat that causes the system call to
return EINVAL in the event that an in use address is passed in and the
SHM_REMAP flag is not set.
Comment 4 Jim Paradis 2006-01-12 15:48:54 EST
The reported behavior is technically compliant with the POSIX standard, even
though it may be inconsistent with the way other kernels work.  More
importantly, changing the behavior at this time entails unacceptable risk;
returning error codes where we did not do so before can cause a lot of existing
software to break.  Therefore, we are leaving this code as-is.

Note You need to log in before you can comment on or make changes to this bug.