Description of problem: Some tools run in a 1st party context, they are not part of the application per se but they are linked into the application's address space. There needs to be tool interfaces to the dynamic linker that can be accessed by a tool from inside the process's address space. The needs of this 1st party tools interface is roughly parallel to the 3rd party tools interface currently provided by the dynamic linker to debuggers through interfaces like r_debug but because these tools interfaces are dlmopened or inserted into a process's address space they also need access to symbols in other dlmopen'd namespaces.
One problem that is probably already addressed with dlmopen is one of ABI between one of these 1st party tool libraries. When they are not using dlmopen: When the tool and application both depend on a common library, but different versions of the library. You can get incompatibilities. Often the "common" library is something like libstdc++.so. Suppose, for example, John logs into LLNL and builds a version of HPCToolkit at LLNL using the default g++ 4.8.5 and libstdc++.so.6.0.19. A user builds their application with g++ 4.9.3 and libstdc++.so.6.0.20, and they want to use John's HPCToolkit. If the tool loads first (which tools frequently do), it loads libstdc++.so.6.0.19. The application then crashes with a missing symbol or some other hard-to-interpret error because it expected libstdc++.so.6.0.20 or latter. The poor physicist thinks HPCToolkit is broken. Since the tool and application don't really overlap or share data, we ideally would give them each a copy of their own preferred libstdc++.so. dlmopen allows that. The current solution is for tools to limit themselves to pure C, but that's cutting out a large body of software we'd like to leverage. They want to use C++ and its runtime libraries to write first party tools without worrying about library version conflicts with an application. Not only do they want to use C++ to accelerate our tool development, they also want to use Dyninst, which is written in C++ and uses the boost library (also written in C++), safely inside first-party tools. dlmopen allows them to do this by having the two separate libstdc++'s and likely boost in their own namespace. However, without a 3rd party tool interface to enumerate and access the alternative namespaces, debugging these 1st party tools is fiendishly difficult. See: https://bugzilla.redhat.com/show_bug.cgi?id=1265827 https://bugzilla.redhat.com/show_bug.cgi?id=1292563 Another aspect of this 1st party tools interface is loading the correct one on demand. In theory everything could be written as an audit library and you can bootstrap in all 1st party tools this way but this seems unnecessarily cumbersome. Conventional run time linking has a host of facilities (DT_NEEDED, RPATH, LD_LIBRARY_PATH, LD_PRELOAD...), 1st party tools could also greatly benefit from a parallel set of features that are not as tightly focused as LD_AUDIT but still bring in the 1st party tool library using dlmopen.
LLNL asked for this BZ to remain public, they have an international team of collaborators working on HPC tooling and they want them to have access to this.
t has been on my todo list for quite some time to circle back around to Matt and others and create some RFE's https://bugzilla.redhat.com/show_bug.cgi?id=1419768 Feel free to jump in asking for what you want or if you want me to edit it into the case notes send it to me. I think that I have 4 distinct action item's here: 1) get all my documentation about the runtime linker and its interface into official documentation so that it is readily available to all of you. If you find any errors or omissions please let me know. 2) you need a tools interface for 3rd party debuggers which will enumerate all the namespaces and can find things in their link maps. Along with that you would like debuggers to make use of that new tool interface so that you can debug these 1st party tools that you want to write. 3) you would like more ways to associate 1st party tools with an app above and beyond turning everything into an LD_AUDIT library. Tell me what you need. Do you want DT_MNEEDED in ELF and LD_MPRELOAD, LD_TOOL_LIBRARY_PATH. Do you want TRUNPATH and TRPATH. I think that I kind of understand the need but I don't know what the interface should look like. 4) From inside a 1st party tool, you also would like to have access to some of the things that you rely on when writing 3rd party tools, for example you probably want to be able to find the TLS areas and enumerate threads like you would from a 3rd party tool with libthread_db and you need the link maps and the address maps like you get from r_debug. And what else? I can only make guesses, I haven't written 1st party tools and so am kind of guessing. I think if I were writing a toolkit for 1st party tools, I would also like functions to introspect the ELF and make use of all the DWARF. I am not overlooking your ABI problems but as far as I know, so long as you use dlmopen() to bring in the 1st party tool, its dependencies including boost and libstdc++ should all exist in that same namespace and not collide with application symbols. So to me that problem seems solved until you tell me otherwise.
(In reply to Ben Woodard from comment #1) > When the tool and application both depend on a common library, but different > versions of the library. You can get incompatibilities. > > Often the "common" library is something like libstdc++.so. Suppose, for > example, John logs into LLNL and builds a version of HPCToolkit at LLNL > using the default g++ 4.8.5 and libstdc++.so.6.0.19. A user builds their > application with g++ 4.9.3 and libstdc++.so.6.0.20, and they want to use > John's HPCToolkit. > They want to use C++ and its runtime libraries to write first party tools > without worrying about library version conflicts with an application. Not > only do they want to use C++ to accelerate our tool development, they also > want to use Dyninst, which is written in C++ and uses the boost library > (also written in C++), safely inside first-party tools. > > dlmopen allows them to do this by having the two separate libstdc++'s and > likely boost in their own namespace. However, without a 3rd party tool > interface to enumerate and access the alternative namespaces, debugging > these 1st party tools is fiendishly difficult. See: > https://bugzilla.redhat.com/show_bug.cgi?id=1265827 > https://bugzilla.redhat.com/show_bug.cgi?id=1292563 dlmopen is not the right way to address such a libstdc++ discrepancy. They should use DTS instead, and they need to work on providing a consistent build environment. If dyninst in DTS does not work for them, we need to address that. If the build environments are not properly set up, involving dlmopen will make things only infinitely worse.
(In reply to Florian Weimer from comment #6) > dlmopen is not the right way to address such a libstdc++ discrepancy. They > should use DTS instead, and they need to work on providing a consistent > build environment. If dyninst in DTS does not work for them, we need to > address that. > > If the build environments are not properly set up, involving dlmopen will > make things only infinitely worse. This is a huge endeavor the scale of which you don't seem to conceive the combined number of developers working on application and libraries is approximately double the entire size of RH. The size of the source base far greater than the combined size of every piece of open source software we ship as Red Hat. The range of software spans cutting edge to legacy code decades old. The notion of settling on one compiler is currently inconceivable. We can't perform anywhere as well as Intel's compiler on x86_64 machines nor can we mach LLVM for innovation, or generate as performant of code for PPC as the XL series of compilers. So an approach like having everybody start using GCC from DTS is not practical. We have some teams which are still working on getting their codes working on the GCC in RHEL7. We also have people who are developing using the most cutting edge features in the newest C++ TSs. Into this heterogeneity of applications is a tools team, which is approximately twice the size of RH's tools team, whose focused on making tools to enable these developers to be effective and productive. They want to partner with us to advance the state of the art in tools interfaces. Given the broad range of compilers and compiler versions that they must support when making tools to help characterize performance and debug application code they need to build the tools in such a way that they can interoperate with application code no matter which compiler and compiler version it was compiled with. Hence the ABI issues. Many of these probably could be overcome with dlmopen but there are many unnecessary limitations that come with making use of dlmopen for tool writers. That is what we want are seeking to collaborate on in this case.
This enhancement request is tracked as part of the upstream glibc project: https://sourceware.org/bugzilla/show_bug.cgi?id=15971