Bug 197395

Description of problem:

On x86_64 and ia64, if I use mlock on a local variable
that lives on the stack, and the locked page is the
current bottom of the virtual memory allocated for the
stack, and then I make more function calls which require
more virtual memory to be added to the stack, the
addition does not occur, and the program segv's when
the bottom of the stack walks off the end of the allocated
virtual memory.

The following test case has a recursive function foo
with its local variable char stuff[1000].  For each
call, foo mlocks stuff[] which lives near the current
bottom of the stack.  So the program is constantly
mlocking right at the tip of the virtual memory allocated
for the stack, and usually after a few iterations the
stack will need to grow and for some reason the mlock
keeps more virtual memory from being added, so it seg
faults.

Note I haven't seen been able to reproduce it on ia32.
I've seen the bug on x86_64 and ia64, on RH AS 4.1-4.3
on variants of 2.6.9.  I found an old 2.4 machine that
ran fine, as did a SuSE machine with 2.6.5.  Not sure
beyond that.

Also a workaround is demonstrated in the testcase.  If
I always push the stack out a little bit before doing the
mlock, then the kernel is never forced to grow the virtual
memory directly adjacent to an mlocked page, and the test
runs fine.  The test can also be made to work by making
stuff[] bigger and mlocking on a page boundary into stuff[]
for the same reason.

Here's test code.  Note, nothing about the bug requires
recursion to hit it, that's just an easy way to repeatedly
hit the condition where the bug occurs.

#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <sys/mman.h>

int pregrow = 0;

void grow() {
/*
 *  The volatile and the assignment is just to keep the compiler
 *  from optimizing this away
 */
        volatile char stuff[16384]; /* use your pagesize, whatever that is */
        stuff[0] = 0;
}
/*
 *  Recursive, so stack grows a bit every call.
 */
void foo(int n) {
        char stuff[1000];
        if (n <= 0) {
                printf("done growing stack, returning\n");
                munlockall();
                return;
        }
/*
 *  Program passes if you decline to mlock, or if you pre-grow
 *  the stack so the mlocked page right on the bottom of the
 *  stack doesn't keep the stack from growing.
 */
        if (pregrow) { grow(); }
        mlock(stuff, 1000);
        foo(n-1);
        return;
}
int main(int argc, char* argv[]) {
        int i;
        for (i=1; i<argc; ++i) {
                if (strcmp(argv[i], "-grow") == 0) {
                        pregrow = 1;
                }
        }
        printf("starting\n");
        foo(4000);
        printf("done\n");
}


Version-Release number of selected component (if applicable):


How reproducible:
Very high.

Steps to Reproduce:
Testcase provided.
  
Actual results:
% ./x
> starting
> Segmentation fault (core dumped)

Expected results:
% ./x
> starting
> done growing stack, returning
> done

Additional info: