Event posted on 2009-09-21 22:04 PDT by woodard

Attached is a distributable document and patch given to us by IBM.
They have found a marked improvement in distributed debugging
using the described enhancements to the runtime linker, and though
the changes were made as part of work on Dawn, our tools team feel
that these changes will also have a big impact on parallel debugging
on our Linux clusters. IBM wants to try to push this patch upstream,
but I don't know if they have begun the process yet.


Here is the antiword output on the Word doc, so you don't have to open
it in openoffice:


Dynamic linker changes to
            Improve performance of debugging dynamic applications

                         Lynn Boger - IBM Rochester


This describes an enhancement that was made to the BGP V1R4 GNU toolchain
to enable improved debugger performance when debugging dynamic applications
that make heavy use of shared libraries.  Totalview 8.7 has been enhanced
to make use of these features.  For detailed information on how a user
would take advantage of this enhancement, refer to the Totalview user
documentation.

Currently the dynamic linker has an interface used by gdb and Totalview
(and possibly other tools) that is used to keep track of the current load
state of the shared libraries used by the debugged program.  The dynamic
linker has defined a function called _dl_debug_state() and will call this
function under various conditions.  A debugger can then set a breakpoint on
this function, stop at various points during a debugging session, and
inspect the state of the shared libraries at that point.  For purposes of
this document, a debug event will refer to the set of operations that take
place during a debugging session,  when the debugger sets a break on this
entry point, the breakpoint is hit, and some processing is done at that
point.

In a parallel programming environment where programs are running on
potentially thousands of nodes, with each node loading a potentially large
set of shared libraries, the amount of time to start up and run a debugging
session with a parallel debugger could be significant.  In the V1R4 BGP
toolchain, an alternate interface has been provided to help reduce the
number of debug events that are generated which will result in improved
startup and debugging time in this situation.  The new implementation will
continue to provide enough detail for a debugger to correctly represent the
current state of the shared libraries that are currently loaded during the
debugging session, so the debugger can keep an accurate representation of
the link_map, which is the set of shared libraries that the dynamic linker
recognizes as being loaded at any given point during execution.

Current implementation in glibc 2.4:

    - A call is made to _dl_debug_state() whenever an attempt is made to
      load a shared library.  This can happen at program load time due to
      dependencies from a binary or other shared library, or during
      execution due to an explicit dlopen call or a statement such as an
      import in Python which results in a dlopen.  When the dynamic linker
      function is called to load the shared library,  the state variable is
      set to ADD and _dl_debug_state() is called.
    - A call is made to _dl_debug_state()  whenever an attempt is made to
      unload or remove a shared library.  This can happen when dlclose is
      called on a shared library, or if another shared library is unloaded
      and it has been determined that one of its dependent libraries is no
      longer needed.  When the dynamic linker function is invoked to unload
      the library, the state variable is set to DELETE and _dl_debug_state()
      is called.
    - Whenever the load or unload of a shared library has completed, a call
      is made to _dl_debug_state().  In this case the state variable is set
      to CONSISTENT.

Note that in all these situations, a call is made to _dl_debug_state() each
time there is a call into the dynamic linker to load or unload a library,
even if it doesn't actually read the shared library into memory.  For
example, if dlopen is called on a shared library that was previously
loaded, the dynamic linker will still make a call to _dl_debug_state() with
the state set to ADD before loading and a call to _dl_debug_state() with
the state set to CONSISTENT after the load attempt even if there was no
need to bring the shared library into memory and no change to the link map.
 The goal of the new implementation is to eliminate the events that don't
result in actually loading or unloading the shared library, and provide
only the events needed to reconstruct the library state at any given point
during a debug session.

New implementation in glibc 2.4:

The following new interfaces have been defined:

    - void _dl_debug_fast_state():  New entry point for debuggers to break
      on.  As in the previous implementation, the dynamic linker will call
      this function at significant points during processing to identify when
      shared libraries have been loaded or unloaded, but under different
      conditions than _dl_debug_state()  There will be fewer calls to
      _dl_debug_fast_state() than to _dl_debug_state().
    - void _dl_debug_disable_events(): Disable all calls to
      _dl_debug_fast_state().  Calls to _dl_debug_state() will still occur.
    - void _dl_debug_enable_events():  Enable calls to
      _dl_debug_fast_state().  It is recommended that the debugger call
      _dl_debug_fast_state() to generate an updated link_map.
    - int _dl_debug_event_state():  Return the status of enable/disable.
      This returns 1 if debug events are currently enabled, 0 if not.

The new implementation works as follows:


    - The previous interface _dl_debug_state() will continue to be supported
      for compatibility reasons.  A call to _dl_debug_state() will be made
      under the same conditions as before with the state variable set
      appropriately.
    - A call to _dl_debug_disable_events() will disable all calls to
      _dl_debug_fast_state() until _dl_debug_enable_events() is later
      called.  Calls to _dl_debug_state() are not affected and will continue
      to be called as in the previous implementation.
    - The new interface _dl_debug_fast_state() will only be called when 1) a
      library operation, such as a load or unload, as been completed 2) if
      that library operation actually resulted in a change to the memory
      mapping, such as the mmap or munmap of the shared library and 3)
      events are not currently in the disabled state due to a previous call
      to the _dl_debug_disable_events() interface.
    - When loading (opening) a shared library, there will be a call to
      _dl_debug_state() with the state set to ADD, but no call to
      _dl_debug_fast_state().
    - When unloading (closing) a shared library, there will be a call to
      _dl_debug_state() with the state set to DELETE but no call to
      _dl_debug_fast_state().
    - When an operation such as an open or close has completed, a call is
      made to _dl_debug_state() with the state set to CONSISTENT, but a call
      will be made to _dl_debug_fast_state() with CONSISTENT state only if
      events are enabled and the previous operation resulted in a change to
      the link map and memory state, as with a mmap or munmap of a shared
      library.  For example, if dlopen was called on a shared library that
      had been previously loaded, no call will be made to
      _dl_debug_fast_state() when the load attempt has completed since the
      library was not read into memory and the link map did not change.
    - A new value has been added to LD_DEBUG so provide debug output which
      describes when debug events will be generated.  In my implementation I
      did not expose this through the LD_DEBUG help text because it is not
      something useful to users but only developers of tools that might use
      this entry point.

Some simple initial experiments resulted in a performance improvement of
over 50%, with the potential for more improvements if the user is able to
enable and disable events appropriately.  In addition, legacy code that
depends on the previous implementation will continue to work as before and
should not be affected.

The original idea for this enhancement came from developers at LLNL
including Dong Ahn among others and the implementation details were the
result of several design discussions by LLNL, John DelSignore of Totalview
Tech and Lynn Boger of IBM.  John DelSignore did the corresponding work in
Totalview 8.7 to make use of the new implementation and did the performance
measurements mentioned above.


© Copyright IBM Corp. 2009  All Rights Reserved.
US Government Users Restricted Rights - Use, duplication or disclosure
restricted by GSA ADP Schedule Contract with IBM Corp.
This event sent from IssueTracker by woodard  [LLNL (HPC)]
 issue 345270

A major change to upstream dynamic linker would not be accepted unless an open source application made use of changes and therefore could be used to test the changes.  GDB or some other open source tool would need to use this feature.

We are open to having discussions on implementing this feature in gdb in future.  There is no capacity to make this happen in time for RHEL 6.0

Can we look at this again and see if we can find another way to skin the cat? How can we enable the desired functionality moving forward? It seems that they have a really legitimate need even if their proposed solution is not acceptable.

As we already said, this needs to be recast as a GDB bug for the manifest performance problem of a particular use case with GDB on an application doing a lot of object loading.  You won't ever move forward by talking about glibc internals as the problem.  Let the GDB team address the real problem, including working out any glibc issues with the glibc maintainers.

Request recast as GDB bug in BZ 698001 [RHEL6] RFE: improve GDB performance on an application performing a lot of object loading.