Description of problem:

Symptom::
Malloc calls from dlls that are opened with RTLD_DEEPBIND can
not be interposed. As a result, e.g. TotalView Debugger's Heap
Interposition Agent (HIA) is not able keep track of these allocations.


Observations / Comments::

There is a new option to dlopen() called RTLD_DEEPBIND. Here is what the man
page says about it:

------------------------------------------------------------------------
RTLD_DEEPBIND (since glibc 2.3.4)
    Place the lookup scope of the symbols in this library  ahead  of
    the global scope.  This means that a self-contained library will
    use its own symbols in preference to  global  symbols  with  the
    same  name contained in libraries that have already been loaded.
    This flag is not specified in POSIX.1-2001.
------------------------------------------------------------------------

Also see: http://sourceware.org/ml/libc-hacker/2004-09/msg00083.html


We assume that the current situation might be as follows:

          This is 
          preloaded
		          DEEPBIND malloc 
                          call is here

EXE    ->    HIA    ->    USR PROVIDED   ->    LIBC

           malloc(){} 
           defined here
			  no malloc(){} here   
                                              malloc(){} 
                                              defined here


If there is no explicit malloc definition in USR PROVIDED, and the USR
PROVIDED library (with the malloc call) is called with RTLD_DEEPBIND,
the malloc in libc is used, not the malloc defined in Heap
Interposition Agent (HIA). We speculate that the failure could be due to that,
in this scenario, the search path (for malloc that was not defined in USR
PROVIDED under the RTLD_DEEPBIND mechanism) should have started from EXE
downstream, and not from USR PROVIDED downstream.


XXXXXXXXXXXXXXXXXXX
Version-Release number of selected component (if applicable):

glibc-2.5-18

pillbox:/home/seppo > cat /etc/*-rel*
cat: /etc/lsb-release.d: Is a directory
Red Hat Enterprise Linux Server release 5.1 (Tikanga)
pillbox:/home/seppo > 
pillbox:/home/seppo/Bugs/Bug_10786 > rpm -q glibc
glibc-2.5-18
pillbox:/home/seppo/Bugs/Bug_10786 > pillbox:/home/seppo/Bugs/Bug_10786 > uname -a
Linux pillbox.totalviewtech.com 2.6.18-53.el5PAE #1 SMP Wed Oct 10 16:48:18 EDT
2007 i686 i686 i386 GNU/Linux
pillbox:/home/seppo/Bugs/Bug_10786 > 


XXXXXXXXXXXXXXXXXXXXXXXXX
How reproducible:

Reproducer attached.


XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Steps to Reproduce:

> make clean 

> make all
gcc -g ldtest.cpp -Wl,-rpath=`pwd` -lstdc++ -ldl -o ldtest
gcc -g -shared myfunc.cpp -Wl,-rpath=`pwd` -lstdc++ -o libmyfunc.so


Use e.g. TotalView 8.4:

> totalview ./ldtest -a 1

In Process Window: Process > Enable Memory Debugging

Create a pending breakpoint on the function myfun by:

  In Process Window: Action Point > At Location

  In At Location dialog: Type myfun and hit OK

  In Ambig function dialog: select "Make "myfun" a pending breakpoint and hit
ok

In Process Window: Do a Go

In Process Window: Do a Next to step over the call to malloc:

   void* leak_malloc_buf = malloc(90);

You reproduced the problem if you see that the variable leak_malloc_buf is
not annotated with 'allocated'

i.e. on the TotalView Command Line Interface (CLI):

d1.<> p leak_malloc_buf
 leak_malloc_buf = 0x0804a398 -> 0x00000000 (0)

d1.<> dheap -info 0x0804a398
process     1   (13213):

d1.<>


XXXXXXXXXXXXXXXXXXXXXXXXXX
Actual results:

d1.<> p leak_malloc_buf
leak_malloc_buf = 0x0804a398 -> 0x00000000 (0)
d1.<> 


XXXXXXXXXXXXXXXXXXXXXXXXXX
Expected results:

Run the above reproducer, but without any arguments to ldtest:
> totalview ./ldtest 

d1.<> p leak_malloc_buf
 leak_malloc_buf = 0x08684398 (Allocated) -> 0x00000000 (0)
d1.<> 

In this case myfun is called w/o RTLD_DEEPBIND and interposition has
been succesful.


XXXXXXXXXXXXXXXXXXXX
Additional info:

This is present on several platforms with glibc 2.3.4 or higher.


XXXXXXXXXXXXXXXXXXX

Reported by

Seppo Sahrakorpi
Partner Technologies Engineer
TotalView Technologies, 24 Prime Parkway, Natick, MA 01760
http://www.totalviewtech.com
seppo@totalviewtech.com -- 508-652-7781 

XXXXXXXXXXXXXXXXXX