Bug 851508

Created attachment 606858 [details]
corrected test case - uncommented comment in first version

Have now identified some additional details, it seems the VmSize,VmmRSS (& ps output) are correct and that the same binary is sometimes showing an
additional 4GB(4194304 kB)  virtual space allocation for each thread,
which makes the total about 2 + 4*5 =22GB in the above example 

The addtional space is being allocated only on some machines and then only
if the job is being executed by the LSF (form platform / IBM) job 
distribution system in which case the job has a process tree like the following

res,12350 -d /tools/lsf/Current/conf -m xl-cbga-03 /home/pmanser/.lsbatch/1345818946.181820
  |-1345818946.1818,12377 /home/pmanser/.lsbatch/1345818946.181820
  |   `-a.out,12379 5
  |       |-{a.out},12380
  |       |-{a.out},12381
  |       |-{a.out},12382
  |       |-{a.out},12383
  |       `-{a.out},12384
  `-{res},12378

if the program is run directly on the machine from a standard linux shell 
I see no issue.

So somehow the executing lsf program is causing the behaviour of the 
threads to be modified.
Inspection of the dynamically linked libraries 

show the same liraries when jobs is executed direcly and via lsf eg.
lc-cam-513 spice_leakage > grep lib64 /proc/12914/maps
3752600000-375261c000 r-xp 00000000 08:01 24674657                       /lib64/ld-2.5.so
375281b000-375281c000 r--p 0001b000 08:01 24674657                       /lib64/ld-2.5.so
375281c000-375281d000 rw-p 0001c000 08:01 24674657                       /lib64/ld-2.5.so
3752e00000-3752f4e000 r-xp 00000000 08:01 24674668                       /lib64/libc-2.5.so
3752f4e000-375314e000 ---p 0014e000 08:01 24674668                       /lib64/libc-2.5.so
375314e000-3753152000 r--p 0014e000 08:01 24674668                       /lib64/libc-2.5.so
3753152000-3753153000 rw-p 00152000 08:01 24674668                       /lib64/libc-2.5.so
3753a00000-3753a16000 r-xp 00000000 08:01 24674670                       /lib64/libpthread-2.5.so
3753a16000-3753c15000 ---p 00016000 08:01 24674670                       /lib64/libpthread-2.5.so
3753c15000-3753c16000 r--p 00015000 08:01 24674670                       /lib64/libpthread-2.5.so
3753c16000-3753c17000 rw-p 00016000 08:01 24674670                       /lib64/libpthread-2.5.so


Thanks for any ideas on likely cause.

Regards, Paul

Hi,
   We believe we identified cause.

If the process limit stacksize is set to a discrete value from executing shell, as opposed to unlimited; then increased virtual memory allocation of stacksize
multiplied by thread count is reserved. If however stacksize limit
is set to unlimited then no additional virtual memory is taken (per thread)

Regards, Paul

I am able to reproduce the issue by increasing the stack size using ulimit,

# ulimit -s 1000240
# ./a.out 5
[..]

# ps -o pid,ppid,pcpu,rss,size,vsize,cputime,lstart -Lp 31913
  PID  PPID %CPU   RSS    SZ    VSZ     TIME                  STARTED
31913 31779  0.0   584 7098644 7104312 00:00:00 Wed Oct  3 11:49:57 2012
31913 31779  0.0   584 7098644 7104312 00:00:00 Wed Oct  3 11:49:57 2012
31913 31779  0.0   584 7098644 7104312 00:00:00 Wed Oct  3 11:49:57 2012
31913 31779  0.0   584 7098644 7104312 00:00:00 Wed Oct  3 11:49:57 2012
31913 31779  0.0   584 7098644 7104312 00:00:00 Wed Oct  3 11:49:57 2012
31913 31779  0.0   584 7098644 7104312 00:00:00 Wed Oct  3 11:49:57 2012


This is an expected behaviour.

From 'man  pthread_create',

On Linux/x86-32, the default stack size for a new thread is 2 megabytes.  Under the NPTL threading implementation, if the RLIMIT_STACK soft resource  limit  at  the  time  the  program  started has any value other than "unlimited", then it determines the default stack size of new threads.  Using pthread_attr_setstacksize(3), the stack size attribute can be explicitly set in the attr argument used to create a thread, in order to obtain a stack size other than the default.

Closing the bugzilla.

This bug/issue is incorrectly diagnosed, I spent many hours with a similar problem before discovering the true cause has nothing to do with setstacksize(), I'm posting this additional info here in case others strike the same problem.

The problem is that malloc() may create multiple arena's for each new thread based on the first threads malloc usage, the result can be that each thread gets an address space similar to the main program, e.g. your main thread is using 100mb, then each thread will get an address space of another 100mb.  

The impact of this on the system is hard to determine, but it does seem to cause performance issues in some situations.  

It appears doing this at the start of your program can stop this from occurring.

    rv = mallopt(-7, 1);  // M_ARENA_TEST
    rv = mallopt(-8, 1);  // M_ARENA_MAX

The real issue is described here by people who know much more than me:

https://sourceware.org/bugzilla/show_bug.cgi?id=11261