Relinking is the process that when you have a package with: shared library A program B depending on A Then on 'make install', B is relinked against the installed copy of A. This is useful on system where objects linked to shared libraries include the full path to the shared library. It is _never_ useful on Linux or other ELF systems, since dependencies are recoreded against the soname, which doesn't include the full path. The is _always_ harmful when installing into a buildroot, either using DESTDIR, or using the make install prefix=/buildroot/usr... hacks. So, we should patch our libtool so that relinking is never done on Linux. (Unfortunately, we can't just disable relinking entirely, since people will want to use our libtool for "make dist")
Could you point me to a package that has a workaround/is affected by this?
The 'pango' package is one of many affected. See the usage of fixed-ltmain.sh in the spec file.
Created attachment 44419 [details] The patch you're effectively applying to ltmain.sh
Ok, I think I more or less understand the patch now. However I'm still confused about how to reproduce the problem. Building pango on my machine with or without the patch gives equivalent rpmbuild output. Is there a simple way to reproduce on my own machine?
If you have the same Pango already installed in /usr/lib it will work fine. Problems occur if you don't have Pango installed or you have a different soname for your installed Pango. (I assume you've modified the Pango spec file to remove the workaround when building and to relibtoolize.)
Of course. Thank you. With pango-0.24-1 installed, I built pango-0.23.90, with and without the modified ltmain.sh. For the record I attach a diff of the rpmbuild outputs.
Created attachment 45641 [details] diff of building pango-0.23.90 without and with fixed ltmain.sh with pango-0.24 installed
Ok, I reduced your patch to just the following! (At least this gives equivalent rpmbuild output to your modified ltmain.) This makes the change more manageable. :) --- ltmain.sh~ Fri Feb 15 16:08:34 2002 +++ ltmain.sh Fri Feb 15 16:08:34 2002 @@ -1716,7 +1716,7 @@ if test "$installed" = no; then notinst_deplibs="$notinst_deplibs $lib" - need_relink=yes +# need_relink=yes fi if test -n "$old_archive_from_expsyms_cmds"; then
Created attachment 45782 [details] the actual simplified patch (not misformatted!)
The patch I attach next to libtool-1.4.2 seems to fix relinking on linux. Any comments before sending it to the libtool list?
Created attachment 45786 [details] patch to libtool-1.4.2 to stop relinking on linux (GNU binutils ld)
It is not correct for the patch to be specific to GNU/Linux. There's nothing specific to GNU/Linux that makes it so different from other ELF platforms in this regard, other than the fact that it uses RPM builds far more often, so the problem of DESTDIR installations becomes more apparent. It appears to me that the best fix would be to verify that all dependency libraries are going to be installed in the default dynamic linker search path. If this is the case, then you don't need relinking. Even then, there are tricky cases when you link with an uninstalled shared library that depends on another uninstalled shared library. I'm not sure doing a test as simple as I suggest above would cover all cases. Basically, what I mean is that ruling relinking out is not right in general. It's there for a reason, and GNU/Linux isn't free from the need of relinking any more than other sane ELF systems. You can be more lenient in doing relinking as long as you stick to putting libraries in directories searched by default, because then you often won't notice the problem because it will be hidden by the fact that the linker ends up finding the library anyway. But you can't just disable it. I strongly recommend that any patch submitted to this effect be tested with some package that has inter-library dependencies (i.e., program A depends on library B depends on library C), each one installed in a different directory, none of each searched by default, installed into a different tree than the one it's going to actually live in (i.e., DESTDIR), and then trying to run the program after the whole package is placed in the correct directory. If this works, the patch is going to be fine. If not, you'll be trading one minor bug (i.e., one that affects a minority of the libtool users) for a major one (i.e., one that would prevent any random user from installing packages properly into his own home directory).
Yes, I know this is applicable to elf architectures in general. Not having a complete list at hand I just put in linux as a starting point (since this is the most urgent). Agreed that verifying that all shared library dependencies are being installed is a sufficient condition for not relinking. Is there any time when all library dependencies wouldn't be installed? Naively it sounds to me like such packages are broken. If truly all shared library dependencies are being installed, then I can't really see how things could go wrong (your paragraph 3). When is relinking useful on elf platforms? When all the shared library dependencies aren't being installed?? Do you know of any (preferably not too big) concrete examples out in the wild of the type you mentioned ("program A depends on library B depends on library C, ...")?
Alexandre, how about Owen's original (the first attachment)? It seems to be closer to what you suggest.
I didn't mean libraries that were not going to be installed. I meant linking with libraries that are not installed *yet*. I.e., think of a package that builds multiple libtool libraries, say , one of which depends on the other. At `make all' time, library C is not installed at the time library B is built. But, at `make install' time, library C will have already been installed, and then, library B has to be relinked with it. For an example of a package that does this, see depdemo in the libtool package. It's part of the libtool testsuite. I kind of like Owen's approach, but I wouldn't go as far as requiring a new command-line argument. I'd rather take the libdir as stored in the .la file (derived from the -rpath specified at library creation time) as the pathname in which the library is to be found at run-time, and the installation pathname as just that. Except that it makes some sense to try to convert libdirs of dependencies such that they become relative to the same installation pathname, for the sake of DESTDIR installs. It probably makes sense to require libdir to be a suffix of the install dir, to make this a possible effort, but this is contrary to the GNU Conding Standards, so we should probably just refrain from attempting the alternate path searching if this condition is not met. That said, I see some problems with Owen's patch. For example, there are unsafe uses of sed, in which a user-supplied argument that may contain `%' is expanded as part of a sed substitution whose delimiters are `%'. This is a no-no, and libtool is insanely careful about not falling prey of this kind of shell-programming trap. Arranging for inst_prefix_dir to be derived from the install pathname and libdir and making sure the caveat above is not an issue would make Owen's patch perfect, IMO.
Just to note - the patch being referred to as "mine", is something that came around when the issue was first discussed (by Alexandre, I believe , maybe from the libtool patch queue?) Anyways, nothing to do with me. Is the issue with simply not relinking that libtool now uses relinking to get rid of the traditional wrapper scripts? Multiple library installation is certainly common in the packages we deal with (GLib, Pango, GTK all do it), but _by itself_ should not cause any need to relink on ELF platforms, since as long as only sonames are being recorded, installation directories don't matter, and rpaths in ELF can refer to directories not yet created.
Oh, I remember the patch now. I hadn't correlated it with that early e-mail exchange. The issue with relinking affects mostly libtool libraries with dependencies on other libtool libraries. The problem is that, when such dependencies exist, libtool creates the library in the build tree such that it looks for its dependencies in the build tree, such that, if you link a program with it, or dlopen it, in the build tree, it will be loaded correctly. The problem of doing it this way is that, if all you want is to install the libraries and programs, never to run them in the build tree, is that you end up having to link the libraries twice. Ideally, we should change libtool such that it links libraries for installation, and creates libraries on demand for execution in the build tree, like we do for executables, but this is close to impossible to do, given that, if you just dlopen a library or run a program that is linked with it, you can't count on libtool being able to intercept the execution/dlopening and linking the library for execution in the build tree. There are other problems in this area, so the decision in libtool was to create libraries prepared for use in the build tree whenever they're found to depend on other uninstalled libraries, and to relink them for installation at install time.
There's is also maddog's relink patch in mdk. I attach it below. How does it compare with the one above?
Created attachment 47080 [details] maddog's relink patch
It's very similar to ltmain-relink.patch. So similar that the same caveats apply.
Is it ok to use "," then instead of "%" for the sed quoting. Also in our patch there is a "exit 1". Is this desirable or not?
Neither `%' nor `,' nor any other character that might appear within the string substituted into the sed command are acceptable. The trick is to quote such appearances in the substituted string, like libtool does in many other locations. Doing otherwise and hoping it won't bite is silly and dangerous. As for the `exit 1', it appears to fix an actual, yet totally unrelated, bug, that has recently shown its ugly head in libjava :-) Still, it's a different problem, so it should be solved with a different patch. However, I seem to remember there's a reason why it's not a fatal error. The idea is that, on systems where proper relinking at DESTDIR install time is impossible (HP-UX), you don't get a hard error at install time, but rather a warning showing what you should do to complete the installation. I'm not sure this is desirable behavior, but I'm sure this is the intent of not failing hardly at that point. Perhaps this behavior should require an option to be given in order to enable it? Anyway, even though I wrote before, I'll do it again: I don't like the idea of passing -inst-prefix-dir explicitly in the command line. It should be inferred from the installation pathname and the -rpath specified at the time the library was built.
I guess the reason for adding the -inst-prefix-dir option was that destdir is determined much later in ltmain than where inst_prefix_dir is being put into deplibs currently.
I have made a patch now which seems to follow to your requirements, except it doesn't pass the depdemo test yet... will continue to work on it tomorrow.
Presumably it's intentional but it seems unfortunate here that libtool uses the variable "libdir" internally, since it hides make's "libdir". So if I'm not missing something, the only shell variable carrying accurate information about the intended install libdir is MAKEFLAGS. But this is probably not portable (GNU make only?). Well prefix and friends may have the right info too, but in principle libdir can be something completely different from them, right? The best solution would seem to be to rename libtool's libdir into its own "namespace" (say "lt_libdir"), so that make's libdir isn't touched by libtool, and can then be used as an accurate indication of the install dir where libraries are being packaged. However when I tried this naively (ie replacing every occurrence of the identifier libdir by lt_libdir in ltmain.in and libtool.m4, this broke "make check" badly. So the substitution seems to be trickier than that. I attach two patches. The first substitutes the buildroot prefix for the ultimate install prefix, which works when the buildroot libdir contains the final prefix. This patch passes "make check" ok and also your depdemo test. The second patch replaces libdir by lt_libdir. The depdemo test seems ok, but lots of "make check" tests fail.
Created attachment 47885 [details] functioning relink patch that uses prefix to determine install libdir
You can't assume libdir is going to be set in the incoming environment. Libtool needs not be run from a Makefile, and libraries don't have to be installed in $(libdir). Libtool knows the expected installation directory of the library since that's specified as the -rpath flag, when the library is created. This directory is stored in the .la script. As for the actual installation pathname (the one that may have DESTDIR in it), it can be taken from the pathname specified as the one in which the library should be installed (say libtool --mode=install install libfoo.la /some/pathname/libfoo.la). That's the only information you can count on.
Created attachment 47886 [details] patch for using lt_libdir and libdir, which seems to break "make check"
You're right, but the problem i saw with that is that the command-line args are being processed in a later part of the file. So I guess I should either do the substitution later (but it should be on add_dir so that doesn't seem feasible), or move the argument processing code up to before that? Or perhaps I can peek at the commands arguments without processing them, just for the install location?
Err... The install location is already available in $dest and $destdir by the time you get to actually process the .la file that you have to install.
True, so a better way then would be to insert some temporary @BUILDROOT_PREFIX@ tag into add_dir and then substitute for it (or remove it) after the commandline args have been processed, but before executation. Does that sound Right? Should/could there be any condition for the tag insertion (like only bother if prefix is set or something) or do we just do it all the time?
I don't understand what you're talking about. Libtool has no notion of prefix whatsoever. What it knows, at library install time, is that $destdir points at the location in which the user requested the library to be installed, and $libdir read from the .la file points at the location the library is supposed to be installed at the end. Look for future_libdirs in ltmain.sh; that's where libtool notices destdir differs from libdir, and prepares to warn the user that additional set up may be necessary before the library is usable. That's where we should add magic to extract DESTDIR by removing libdir from destdir, with code such as: case $destdir in *"$libdir") libdirrx=`$echo "X$libdir" | $Xsed -e 's,.,.,g'` DESTDIR=`$echo "X$destdir" | $Xsed -e "s,$libdirrx$,,"` ;; *) # warn that relinking of libraries may fail because DESTDIR could not be inferred? ;; esac from then on, you can use ${DESTDIR}$libdir wherever you'd use $libdir in -L flags. Hope this helps.
Hmmm, that's all very well, but the problem I still see which I alluded to earlier is that libtool starts a new libtool process when it does the relinking itself, and then DESTDIR and the rest of the env is lost in the new process. It seems to me that from this point of view the original patch introducing the --inst-prefix-dir option seems to make a lot of sense. But please correct me if I'm wrong. [ps I feel I've already spent far too much time on this, wading through and tracing the ltmain script, and still don't feel that much closer to seeing the overall structure. 8-} Seems to me that Alexandre could probably create a working patch satisfying his conditions pretty quickly (half an hour or so?) compared to how long it will likely take me to find it. And that's before even posted it upstream and getting it accepted there...]
You can probably tell that I'm not in a much better position than you by seeing how many times I introduced new elements in the discussion. I've been learning about libtool myself as we discuss this problem. And I haven't been really involved with libtool for at least 9 months now, so I'm also forgetting a lot about it :-( I wish I could be officially assigned to spend time on libtool, but my job is to work in GCC, so libtool is left for my personal spare time, that I haven't had at all lately :-( That said, I was missing the fact that relinking would start a new instance of libtool. I still don't see why it couldn't infer everything just the same as the install process does, but if it can't (for example, because it doesn't get the install pathname), then I see the reason for the new flag, and I'd be all for it.
Ok. Thanks for your continued help. Well my statement is based on the output of a splattering of "echo $LINE; set"'s in ltmain.sh. In "depcomp/", I run % ./configure --prefix=/home/petersen/work/tmp/depdemo/usr ; make clean all and then % make bindir=/var/tmp/depdemo/home/petersen/work/tmp/depdemo/usr/bin libdir=/var/tmp/depdemo/home/petersen/work/tmp/depdemo/usr/lib install Then immediately before the code fragment # Do each of the archive commands. if test -n "$export_symbols" && test -n "$archive_expsym_cmds"; then eval cmds=\"$archive_expsym_cmds\" else eval cmds=\"$archive_cmds\" fi save_ifs="$IFS"; IFS='~' for cmd in $cmds; do IFS="$save_ifs" $show "$cmd" $run eval "$cmd" || exit $? done IFS="$save_ifs" which I found seems to be where the final relinking is done, the list of vars attached below are in scope. As you can see the only vars that contain the buildroot prefix "/var/tmp/depdemo" are MAKEFLAGS and bindir.
Created attachment 48585 [details] list of vars in scope just before relinking
Ok, the need for -inst-prefix-dir makes more sense to me now, but see below. Now I realize the handling of -L flags is wrong. We're modifying them at points in which they're being used for hard-coding library search paths into libraries or executables (on systems in which -L plays a similar role to -R on GNU/Linux). The changes make no difference for GNU/Linux, and will have unwanted effects on systems that actually depend on -L for -R -like functionality. If mangling -L flags or pathnames to libraries is actually necessary, I suggest us to do so in the relink_command line, before we run the relink_command, and hope it just works. This would obviate the need for -inst-prefix-dir. But we'd have to verify that it actually encodes the correct pathnames in programs and shared-library dependencies.
Hmmm. Going back to the original patches, I guess the order of the -L's is significant. This seems to be one difference between our patch and maddog's. In the later add_prefix_dir is added to the end of finalize_deplibs and deplibs, whereas in the former the prefix goes into add_dir which is added at the beginning. The latter seems better wrt to what you're saying about -L/-R behaviour platforms, but less safe wrt to relinking? I mean if a different version of a package is already installed in the final destination ahead of the buildroot install dir in the -L path, it seems to me that relinking should break. But both the depdemo and the pango test seem ok for both patches. However why is the order of the -L's in the final relink command the reverse of the order in deplibs? In the case of the pango test however the biggest difference between the former and the latter patch is that whereas the former seems to stop relinking all together, the latter does not.
Just a note to say that I confirmed that the former patch breaks relinking when there is a different version of the package already installed ahead in the final destination (/usr/lib).
I added aoliva's (old) patch in libtool-1.4.2-8. A relinking patch still needs to be pushed upstream.
The patch, as of libtool-1.4.2-12, appears to list the inst-prefix'ed directory behind its real counterpart. Which is wrong, because an arbitrary version of a dependent library might get linked instead of the bundled one that is installed in the same turn. The immediate order in the code seems right, only it is reverted later. You need to append the paths in reverse order: # Try looking first in the location we're being installed to. add_dir= if test -n "$inst_prefix_dir"; then case "$libdir" in [\\/]*) add_dir="-L$inst_prefix_dir$libdir" ;; esac fi # The order will be reversed later add_dir="-L$libdir $add_dir"
I've filed the bug mentioned in comment 41 above as bug 91110.