Bug 197395 - mlock near bottom of stack's virtual mem keeps stack from growing
mlock near bottom of stack's virtual mem keeps stack from growing
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: kernel (Show other bugs)
x86_64 Linux
medium Severity medium
: ---
: ---
Assigned To: Peter Martuccelli
Brian Brock
Depends On:
  Show dependency treegraph
Reported: 2006-06-30 16:15 EDT by Mark Allen
Modified: 2012-06-20 12:16 EDT (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2012-06-20 12:16:23 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Mark Allen 2006-06-30 16:15:06 EDT
Description of problem:

On x86_64 and ia64, if I use mlock on a local variable
that lives on the stack, and the locked page is the
current bottom of the virtual memory allocated for the
stack, and then I make more function calls which require
more virtual memory to be added to the stack, the
addition does not occur, and the program segv's when
the bottom of the stack walks off the end of the allocated
virtual memory.

The following test case has a recursive function foo
with its local variable char stuff[1000].  For each
call, foo mlocks stuff[] which lives near the current
bottom of the stack.  So the program is constantly
mlocking right at the tip of the virtual memory allocated
for the stack, and usually after a few iterations the
stack will need to grow and for some reason the mlock
keeps more virtual memory from being added, so it seg

Note I haven't seen been able to reproduce it on ia32.
I've seen the bug on x86_64 and ia64, on RH AS 4.1-4.3
on variants of 2.6.9.  I found an old 2.4 machine that
ran fine, as did a SuSE machine with 2.6.5.  Not sure
beyond that.

Also a workaround is demonstrated in the testcase.  If
I always push the stack out a little bit before doing the
mlock, then the kernel is never forced to grow the virtual
memory directly adjacent to an mlocked page, and the test
runs fine.  The test can also be made to work by making
stuff[] bigger and mlocking on a page boundary into stuff[]
for the same reason.

Here's test code.  Note, nothing about the bug requires
recursion to hit it, that's just an easy way to repeatedly
hit the condition where the bug occurs.

#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <sys/mman.h>

int pregrow = 0;

void grow() {
 *  The volatile and the assignment is just to keep the compiler
 *  from optimizing this away
        volatile char stuff[16384]; /* use your pagesize, whatever that is */
        stuff[0] = 0;
 *  Recursive, so stack grows a bit every call.
void foo(int n) {
        char stuff[1000];
        if (n <= 0) {
                printf("done growing stack, returning\n");
 *  Program passes if you decline to mlock, or if you pre-grow
 *  the stack so the mlocked page right on the bottom of the
 *  stack doesn't keep the stack from growing.
        if (pregrow) { grow(); }
        mlock(stuff, 1000);
int main(int argc, char* argv[]) {
        int i;
        for (i=1; i<argc; ++i) {
                if (strcmp(argv[i], "-grow") == 0) {
                        pregrow = 1;

Version-Release number of selected component (if applicable):

How reproducible:
Very high.

Steps to Reproduce:
Testcase provided.
Actual results:
% ./x
> starting
> Segmentation fault (core dumped)

Expected results:
% ./x
> starting
> done growing stack, returning
> done

Additional info:
Comment 1 Mark Allen 2006-06-30 17:49:54 EDT
Update: a coworker tried it on a plain kernel.org
2.6.17 machine (ia64) and it passed there.
Comment 2 Mark Allen 2006-07-05 14:19:44 EDT
Another update: I forgot to mention, the bug only seems
to happen if you run as non-root.
Comment 4 Jiri Pallich 2012-06-20 12:16:23 EDT
Thank you for submitting this issue for consideration in Red Hat Enterprise Linux. The release for which you requested us to review is now End of Life. 
Please See https://access.redhat.com/support/policy/updates/errata/

If you would like Red Hat to re-consider your feature request for an active release, please re-open the request via appropriate support channels and provide additional supporting details about the importance of this issue.

Note You need to log in before you can comment on or make changes to this bug.