Description of problem: Symptom:: Malloc calls from dlls that are opened with RTLD_DEEPBIND can not be interposed. As a result, e.g. TotalView Debugger's Heap Interposition Agent (HIA) is not able keep track of these allocations. Observations / Comments:: There is a new option to dlopen() called RTLD_DEEPBIND. Here is what the man page says about it: ------------------------------------------------------------------------ RTLD_DEEPBIND (since glibc 2.3.4) Place the lookup scope of the symbols in this library ahead of the global scope. This means that a self-contained library will use its own symbols in preference to global symbols with the same name contained in libraries that have already been loaded. This flag is not specified in POSIX.1-2001. ------------------------------------------------------------------------ Also see: http://sourceware.org/ml/libc-hacker/2004-09/msg00083.html We assume that the current situation might be as follows: This is preloaded DEEPBIND malloc call is here EXE -> HIA -> USR PROVIDED -> LIBC malloc(){} defined here no malloc(){} here malloc(){} defined here If there is no explicit malloc definition in USR PROVIDED, and the USR PROVIDED library (with the malloc call) is called with RTLD_DEEPBIND, the malloc in libc is used, not the malloc defined in Heap Interposition Agent (HIA). We speculate that the failure could be due to that, in this scenario, the search path (for malloc that was not defined in USR PROVIDED under the RTLD_DEEPBIND mechanism) should have started from EXE downstream, and not from USR PROVIDED downstream. XXXXXXXXXXXXXXXXXXX Version-Release number of selected component (if applicable): glibc-2.5-18 pillbox:/home/seppo > cat /etc/*-rel* cat: /etc/lsb-release.d: Is a directory Red Hat Enterprise Linux Server release 5.1 (Tikanga) pillbox:/home/seppo > pillbox:/home/seppo/Bugs/Bug_10786 > rpm -q glibc glibc-2.5-18 pillbox:/home/seppo/Bugs/Bug_10786 > pillbox:/home/seppo/Bugs/Bug_10786 > uname -a Linux pillbox.totalviewtech.com 2.6.18-53.el5PAE #1 SMP Wed Oct 10 16:48:18 EDT 2007 i686 i686 i386 GNU/Linux pillbox:/home/seppo/Bugs/Bug_10786 > XXXXXXXXXXXXXXXXXXXXXXXXX How reproducible: Reproducer attached. XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Steps to Reproduce: > make clean > make all gcc -g ldtest.cpp -Wl,-rpath=`pwd` -lstdc++ -ldl -o ldtest gcc -g -shared myfunc.cpp -Wl,-rpath=`pwd` -lstdc++ -o libmyfunc.so Use e.g. TotalView 8.4: > totalview ./ldtest -a 1 In Process Window: Process > Enable Memory Debugging Create a pending breakpoint on the function myfun by: In Process Window: Action Point > At Location In At Location dialog: Type myfun and hit OK In Ambig function dialog: select "Make "myfun" a pending breakpoint and hit ok In Process Window: Do a Go In Process Window: Do a Next to step over the call to malloc: void* leak_malloc_buf = malloc(90); You reproduced the problem if you see that the variable leak_malloc_buf is not annotated with 'allocated' i.e. on the TotalView Command Line Interface (CLI): d1.<> p leak_malloc_buf leak_malloc_buf = 0x0804a398 -> 0x00000000 (0) d1.<> dheap -info 0x0804a398 process 1 (13213): d1.<> XXXXXXXXXXXXXXXXXXXXXXXXXX Actual results: d1.<> p leak_malloc_buf leak_malloc_buf = 0x0804a398 -> 0x00000000 (0) d1.<> XXXXXXXXXXXXXXXXXXXXXXXXXX Expected results: Run the above reproducer, but without any arguments to ldtest: > totalview ./ldtest d1.<> p leak_malloc_buf leak_malloc_buf = 0x08684398 (Allocated) -> 0x00000000 (0) d1.<> In this case myfun is called w/o RTLD_DEEPBIND and interposition has been succesful. XXXXXXXXXXXXXXXXXXXX Additional info: This is present on several platforms with glibc 2.3.4 or higher. XXXXXXXXXXXXXXXXXXX Reported by Seppo Sahrakorpi Partner Technologies Engineer TotalView Technologies, 24 Prime Parkway, Natick, MA 01760 http://www.totalviewtech.com seppo -- 508-652-7781 XXXXXXXXXXXXXXXXXX
Created attachment 301583 [details] Reproducer package
And your point is? You cannot have the cake and eat it. It's the problem of your proprietary technology and not that of glibc.
This is not limited to our malloc call. Attached please find a general non-malloc reproducer, with the following logic: Write a small ia which interposes, e.g., strdup. The interposer should announc when it's called. Write a small main program which call strdup. Write a small dll which calls strdup from an exported fucntion. call strdup from main. Open the dll from main with RTLD_DEEPBIND. Call the exported function to demonstrate that the call from the dll is not interposed. For details, see the 00readme file in bld-linux.
Created attachment 308722 [details] Non-malloc reproducer
The code works as it is supposed to. There is nothing wrong except your expectation. And I am not going to invent more hooks for proprietary software to work.