Bug 58664 - libtool shouldn't relink on Linux
libtool shouldn't relink on Linux
Status: CLOSED RAWHIDE
Product: Red Hat Linux
Classification: Retired
Component: libtool (Show other bugs)
7.2
i386 Linux
medium Severity medium
: ---
: ---
Assigned To: Jens Petersen
Aaron Brown
:
Depends On:
Blocks: 52246
  Show dependency treegraph
 
Reported: 2002-01-22 10:28 EST by Owen Taylor
Modified: 2007-04-18 12:39 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2002-05-21 04:54:29 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
The patch you're effectively applying to ltmain.sh (3.62 KB, patch)
2002-02-04 01:18 EST, Jens Petersen
no flags Details | Diff
diff of building pango-0.23.90 without and with fixed ltmain.sh with pango-0.24 installed (34.95 KB, patch)
2002-02-14 01:10 EST, Jens Petersen
no flags Details | Diff
the actual simplified patch (not misformatted!) (294 bytes, patch)
2002-02-15 02:30 EST, Jens Petersen
no flags Details | Diff
patch to libtool-1.4.2 to stop relinking on linux (GNU binutils ld) (1.41 KB, patch)
2002-02-15 07:01 EST, Jens Petersen
no flags Details | Diff
maddog's relink patch (3.03 KB, patch)
2002-03-01 04:32 EST, Jens Petersen
no flags Details | Diff
functioning relink patch that uses prefix to determine install libdir (611 bytes, patch)
2002-03-08 04:39 EST, Jens Petersen
no flags Details | Diff
patch for using lt_libdir and libdir, which seems to break "make check" (28.29 KB, patch)
2002-03-08 04:48 EST, Jens Petersen
no flags Details | Diff
list of vars in scope just before relinking (7.43 KB, text/plain)
2002-03-15 03:56 EST, Jens Petersen
no flags Details

  None (edit)
Description Owen Taylor 2002-01-22 10:28:45 EST
Relinking is the process that when you have a package with:
 
 shared library A
 program B depending on A

Then on 'make install', B is relinked against the installed
copy of A.

This is useful on system where objects linked to shared libraries
include the full path to the shared library.

It is _never_ useful on Linux or other ELF systems, since 
dependencies are recoreded against the soname, which doesn't
include the full path.

The is _always_ harmful when installing into a buildroot,
either using DESTDIR, or using the make install prefix=/buildroot/usr...
hacks.

So, we should patch our libtool so that relinking is never done
on Linux. (Unfortunately, we can't just disable relinking entirely,
since people will want to use our libtool for "make dist")
Comment 1 Jens Petersen 2002-01-24 03:28:22 EST
Could you point me to a package that has a workaround/is affected by this?
Comment 2 Owen Taylor 2002-01-24 11:01:48 EST
The 'pango' package is one of many affected. See the usage of 
fixed-ltmain.sh in the spec file.
Comment 3 Jens Petersen 2002-02-04 01:18:35 EST
Created attachment 44419 [details]
The patch you're effectively applying to ltmain.sh
Comment 4 Jens Petersen 2002-02-13 03:41:23 EST
Ok, I think I more or less understand the patch now.
However I'm still confused about how to reproduce the problem.
Building pango on my machine with or without the patch gives equivalent rpmbuild
output.  Is there a simple way to reproduce on my own machine?
Comment 5 Owen Taylor 2002-02-13 11:16:18 EST
If you have the same Pango already installed in /usr/lib it will
work fine. Problems occur if you don't have Pango installed or
you have a different soname for your installed Pango.

(I assume you've modified the Pango spec file to remove the 
workaround when building and to relibtoolize.)
Comment 6 Jens Petersen 2002-02-14 01:02:55 EST
Of course.  Thank you.

With pango-0.24-1 installed, I built pango-0.23.90, with and without the
modified ltmain.sh.  For the record I attach  a diff of the rpmbuild outputs.
Comment 7 Jens Petersen 2002-02-14 01:10:26 EST
Created attachment 45641 [details]
diff of building pango-0.23.90 without and with fixed ltmain.sh with pango-0.24 installed
Comment 8 Jens Petersen 2002-02-15 02:27:56 EST
Ok, I reduced your patch to just the following!  (At least this gives equivalent
rpmbuild output to your modified ltmain.)  This makes the change more manageable. :)

--- ltmain.sh~	Fri Feb 15 16:08:34 2002
+++ ltmain.sh	Fri Feb 15 16:08:34 2002
@@ -1716,7 +1716,7 @@
 
 	  if test "$installed" = no; then
 	    notinst_deplibs="$notinst_deplibs $lib"
-
    need_relink=yes
+# 	    need_relink=yes
 	  fi
 
 	  if test -n "$old_archive_from_expsyms_cmds"; then
Comment 9 Jens Petersen 2002-02-15 02:30:57 EST
Created attachment 45782 [details]
the actual simplified patch (not misformatted!)
Comment 10 Jens Petersen 2002-02-15 06:59:34 EST
The patch I attach next to libtool-1.4.2 seems to fix relinking on linux.  Any
comments before sending it to the libtool list?
Comment 11 Jens Petersen 2002-02-15 07:01:28 EST
Created attachment 45786 [details]
patch to libtool-1.4.2 to stop relinking on linux (GNU binutils ld)
Comment 12 Alexandre Oliva 2002-02-15 10:46:56 EST
It is not correct for the patch to be specific to GNU/Linux.  There's nothing
specific to GNU/Linux that makes it so different from other ELF platforms in
this regard, other than the fact that it uses RPM builds far more often, so the
problem of DESTDIR installations becomes more apparent.

It appears to me that the best fix would be to verify that all dependency
libraries are going to be installed in the default dynamic linker search path. 
If this is the case, then you don't need relinking.

Even then, there are tricky cases when you link with an uninstalled shared
library that depends on another uninstalled shared library.  I'm not sure doing
a test as simple as I suggest above would cover all cases.

Basically, what I mean is that ruling relinking out is not right in general. 
It's there for a reason, and GNU/Linux isn't free from the need of relinking any
more than other sane ELF systems.  You can be more lenient in doing relinking as
long as you stick to putting libraries in directories searched by default,
because then you often won't notice the problem because it will be hidden by the
fact that the linker ends up finding the library anyway.  But you can't just
disable it.

I strongly recommend that any patch submitted to this effect be tested with some
package that has inter-library dependencies (i.e., program A depends on library
B depends on library C), each one installed in a different directory, none of
each searched by default, installed into a different tree than the one it's
going to actually live in (i.e., DESTDIR), and then trying to run the program
after the whole package is placed in the correct directory.  If this works, the
patch is going to be fine.  If not, you'll be trading one minor bug (i.e., one
that affects a minority of the libtool users) for a major one (i.e., one that
would prevent any random user from installing packages properly into his own
home directory).
Comment 13 Jens Petersen 2002-02-18 02:51:00 EST
Yes, I know this is applicable to elf architectures in general. Not having a
complete list at hand I just put in linux as a starting point (since this is the
most urgent).

Agreed that verifying that all shared library dependencies are being installed
is a sufficient condition for not relinking.  Is there any time when all library
dependencies wouldn't be installed?  Naively it sounds to me like such packages are
broken.

If truly all shared library dependencies are being installed, then I can't
really see how things could go wrong (your paragraph 3).

When is relinking useful on elf platforms?  When all the shared library
dependencies aren't being installed??

Do you know of any (preferably not too big) concrete examples out in the wild of
the type you mentioned ("program A depends on library B depends on library C, ...")?
Comment 14 Jens Petersen 2002-02-18 03:59:40 EST
Alexandre, how about Owen's original (the first attachment)?  It seems to be
closer to what you suggest.
Comment 15 Alexandre Oliva 2002-02-18 23:01:25 EST
I didn't mean libraries that were not going to be installed.  I meant linking
with libraries that are not installed *yet*.  I.e., think of a package that
builds multiple libtool libraries, say , one of which depends on the other.  At
`make all' time, library C is not installed at the time library B is built. 
But, at `make install' time, library C will have already been installed, and
then, library B has to be relinked with it.  For an example of a package that
does this, see depdemo in the libtool package.  It's part of the libtool testsuite.

I kind of like Owen's approach, but I wouldn't go as far as requiring a new
command-line argument.  I'd rather take the libdir as stored in the .la file
(derived from the -rpath specified at library creation time) as the pathname in
which the library is to be found at run-time, and the installation pathname as
just that.  Except that it makes some sense to try to convert libdirs of
dependencies such that they become relative to the same installation pathname,
for the sake of DESTDIR installs.  It probably makes sense to require libdir to
be a suffix of the install dir, to make this a possible effort, but this is
contrary to the GNU Conding Standards, so we should probably just refrain from
attempting the alternate path searching if this condition is not met.

That said, I see some problems with Owen's patch.  For example, there are unsafe
uses of sed, in which a user-supplied argument that may contain `%' is expanded
as part of a sed substitution whose delimiters are `%'.  This is a no-no, and
libtool is insanely careful about not falling prey of this kind of
shell-programming trap.

Arranging for inst_prefix_dir to be derived from the install pathname and libdir
and making sure the caveat above is not an issue would make Owen's patch
perfect, IMO.
Comment 16 Owen Taylor 2002-02-19 00:25:43 EST
Just to note - the patch being referred to as "mine", is something that
came around when the issue was first discussed (by Alexandre, I believe
, maybe from the libtool patch queue?) Anyways, nothing to do with me.

Is the issue with simply not relinking that libtool now uses relinking
to get rid of the traditional wrapper scripts? 

Multiple library installation is certainly common in the packages
we deal with (GLib, Pango, GTK all do it), but _by itself_ should
not cause any need to relink on ELF platforms, since as long as
only sonames are being recorded, installation directories don't matter,
and rpaths in ELF can refer to directories not yet created.
Comment 17 Alexandre Oliva 2002-02-19 01:07:36 EST
Oh, I remember the patch now.  I hadn't correlated it with that early e-mail
exchange.

The issue with relinking affects mostly libtool libraries with dependencies on
other libtool libraries.  The problem is that, when such dependencies exist,
libtool creates the library in the build tree such that it looks for its
dependencies in the build tree, such that, if you link a program with it, or
dlopen it, in the build tree, it will be loaded correctly.  The problem of doing
it this way is that, if all you want is to install the libraries and programs,
never to run them in the build tree, is that you end up having to link the
libraries twice.

Ideally, we should change libtool such that it links libraries for installation,
and creates libraries on demand for execution in the build tree, like we do for
executables, but this is close to impossible to do, given that, if you just
dlopen a library or run a program that is linked with it, you can't count on
libtool being able to intercept the execution/dlopening and linking the library
for execution in the build tree.  There are other problems in this area, so the
decision in libtool was to create libraries prepared for use in the build tree
whenever they're found to depend on other uninstalled libraries, and to relink
them for installation at install time.
Comment 18 Jens Petersen 2002-03-01 04:28:52 EST
There's is also maddog's relink patch in mdk. I attach it below.
How does it compare with the one above?
Comment 19 Jens Petersen 2002-03-01 04:32:21 EST
Created attachment 47080 [details]
maddog's relink patch
Comment 20 Alexandre Oliva 2002-03-01 23:49:48 EST
It's very similar to ltmain-relink.patch.  So similar that the same caveats apply.
Comment 21 Jens Petersen 2002-03-04 04:02:06 EST
Is it ok to use "," then instead of "%" for the sed quoting.

Also in our patch there is a "exit 1". Is this desirable or not?
Comment 22 Alexandre Oliva 2002-03-04 15:18:07 EST
Neither `%' nor `,' nor any other character that might appear within the string
substituted into the sed command are acceptable.  The trick is to quote such
appearances in the substituted string, like libtool does in many other
locations. Doing otherwise and hoping it won't bite is silly and dangerous.

As for the `exit 1', it appears to fix an actual, yet totally unrelated, bug,
that has recently shown its ugly head in libjava :-)  Still, it's a different
problem, so it should be solved with a different patch.  However, I seem to
remember there's a reason why it's not a fatal error.  The idea is that, on
systems where proper relinking at DESTDIR install time is impossible (HP-UX),
you don't get a hard error at install time, but rather a warning showing what
you should do to complete the installation.  I'm not sure this is desirable
behavior, but I'm sure this is the intent of not failing hardly at that point. 
Perhaps this behavior should require an option to be given in order to enable it?

Anyway, even though I wrote before, I'll do it again: I don't like the idea of
passing -inst-prefix-dir explicitly in the command line.  It should be inferred
from the installation pathname and the -rpath specified at the time the library
was built.
Comment 23 Jens Petersen 2002-03-05 02:50:44 EST
I guess the reason for adding the -inst-prefix-dir option was that destdir is
determined much later in ltmain than where inst_prefix_dir is being put into
deplibs currently.
Comment 24 Jens Petersen 2002-03-05 04:06:24 EST
I have made a patch now which seems to follow to your requirements, except it
doesn't pass the depdemo test yet...  will continue to work on it tomorrow.

Comment 25 Jens Petersen 2002-03-08 03:54:35 EST
Presumably it's intentional but it seems unfortunate here that libtool uses the
variable "libdir" internally, since it hides make's "libdir". So if I'm not
missing something, the only shell variable carrying accurate information about
the intended install libdir is MAKEFLAGS. But this is probably not portable (GNU
make only?). Well prefix and friends may have the right info too, but in
principle libdir can be something completely different from them, right?

The best solution would seem to be to rename libtool's libdir into its own
"namespace" (say "lt_libdir"), so that make's libdir isn't touched by libtool,
and can then be used as an accurate indication of the install dir where
libraries are being packaged. However when I tried this naively (ie replacing
every occurrence of the identifier libdir by lt_libdir in ltmain.in and
libtool.m4, this broke "make check" badly. So the substitution seems to be
trickier than that.

I attach two patches.  The first substitutes the buildroot prefix for the
ultimate install prefix, which works when the buildroot libdir contains the
final prefix. This patch passes "make check" ok and also your depdemo test.

The second patch replaces libdir by lt_libdir. The depdemo test seems ok, but
lots of "make check" tests fail.
Comment 26 Jens Petersen 2002-03-08 04:39:48 EST
Created attachment 47885 [details]
functioning relink patch that uses prefix to determine install libdir
Comment 27 Alexandre Oliva 2002-03-08 04:41:59 EST
You can't assume libdir is going to be set in the incoming environment.  Libtool
needs not be run from a Makefile, and libraries don't have to be installed in
$(libdir).  Libtool knows the expected installation directory of the library
since that's specified as the -rpath flag, when the library is created.  This
directory is stored in the .la script.

As for the actual installation pathname (the one that may have DESTDIR in it),
it can be taken from the pathname specified as the one in which the library
should be installed (say libtool --mode=install install libfoo.la
/some/pathname/libfoo.la).

That's the only information you can count on.
Comment 28 Jens Petersen 2002-03-08 04:48:14 EST
Created attachment 47886 [details]
patch for using lt_libdir and libdir, which seems to break "make check"
Comment 29 Jens Petersen 2002-03-08 05:00:16 EST
You're right, but the problem i saw with that is that the command-line args are
being processed in a later part of the file.  So I guess I should either do the
substitution later (but it should be on add_dir so that doesn't seem feasible),
or move the argument processing code up to before that?  Or perhaps I can peek
at the commands arguments without processing them, just for the install location?
Comment 30 Alexandre Oliva 2002-03-08 05:14:38 EST
Err...  The install location is already available in $dest and $destdir by the
time you get to actually process the .la file that you have to install.
Comment 31 Jens Petersen 2002-03-08 05:21:19 EST
True, so a better way then would be to insert some temporary @BUILDROOT_PREFIX@
tag into add_dir and then substitute for it (or remove it) after the commandline
args have been processed, but before executation.  Does that sound Right?

Should/could there be any condition for the tag insertion (like only bother if
prefix is set or something) or do we just do it all the time?
Comment 32 Alexandre Oliva 2002-03-08 05:54:09 EST
I don't understand what you're talking about.  Libtool has no notion of prefix
whatsoever.  What it knows, at library install time, is that $destdir points at
the location in which the user requested the library to be installed, and
$libdir read from the .la file points at the location the library is supposed to
be installed at the end.

Look for future_libdirs in ltmain.sh; that's where libtool notices destdir
differs from libdir, and prepares to warn the user that additional set up may be
necessary before the library is usable.  That's where we should add magic to
extract DESTDIR by removing libdir from destdir, with code such as:

case $destdir in
*"$libdir")
  libdirrx=`$echo "X$libdir" | $Xsed -e 's,.,.,g'`
  DESTDIR=`$echo "X$destdir" | $Xsed -e "s,$libdirrx$,,"`
  ;;
*)
  # warn that relinking of libraries may fail because DESTDIR could not be inferred?
  ;;
esac

from then on, you can use ${DESTDIR}$libdir wherever you'd use $libdir in -L flags.

Hope this helps.
Comment 33 Jens Petersen 2002-03-15 02:22:45 EST
Hmmm, that's all very well, but the problem I still see which I alluded to
earlier is that libtool starts a new libtool process when it does the relinking
itself, and then DESTDIR and the rest of the env is lost in the new process. It
seems to me that from this point of view the original patch introducing the
--inst-prefix-dir option seems to make a lot of sense. But please correct me if
I'm wrong.

[ps I feel I've already spent far too much time on this, wading through and
tracing the ltmain script, and still don't feel that much closer to seeing the 
overall structure. 8-} Seems to me that Alexandre could probably create a
working patch satisfying his conditions pretty quickly (half an hour or so?)
compared to how long it will likely take me to find it.  And that's before even
posted it upstream and getting it accepted there...]
Comment 34 Alexandre Oliva 2002-03-15 02:31:57 EST
You can probably tell that I'm not in a much better position than you by seeing
how many times I introduced new elements in the discussion.  I've been learning
about libtool myself as we discuss this problem.  And I haven't been really
involved with libtool for at least 9 months now, so I'm also forgetting a lot
about it :-(  I wish I could be officially assigned to spend time on libtool,
but my job is to work in GCC, so libtool is left for my personal spare time,
that I haven't had at all lately :-(

That said, I was missing the fact that relinking would start a new instance of
libtool.  I still don't see why it couldn't infer everything just the same as
the install process does, but if it can't (for example, because it doesn't get
the install pathname), then I see the reason for the new flag, and I'd be all
for it.
Comment 35 Jens Petersen 2002-03-15 03:54:33 EST
Ok.  Thanks for your continued help.

Well my statement is based on the output of a
splattering of "echo $LINE; set"'s in ltmain.sh.

In "depcomp/", I run
% ./configure --prefix=/home/petersen/work/tmp/depdemo/usr ; make clean all
and then
% make bindir=/var/tmp/depdemo/home/petersen/work/tmp/depdemo/usr/bin
libdir=/var/tmp/depdemo/home/petersen/work/tmp/depdemo/usr/lib install

Then immediately before the code fragment

	# Do each of the archive commands.
	if test -n "$export_symbols" && test -n "$archive_expsym_cmds"; then
	  eval cmds=\"$archive_expsym_cmds\"
	else
	  eval cmds=\"$archive_cmds\"
	fi
	save_ifs="$IFS"; IFS='~'
	for cmd in $cmds; do
	  IFS="$save_ifs"
	  $show "$cmd"
	  $run eval "$cmd" || exit $?
	done
	IFS="$save_ifs"

which I found seems to be where the final relinking is done,
the list of vars attached below are in scope.  As you can see
the only vars that contain the buildroot prefix
"/var/tmp/depdemo" are MAKEFLAGS and bindir.
Comment 36 Jens Petersen 2002-03-15 03:56:56 EST
Created attachment 48585 [details]
list of vars in scope just before relinking
Comment 37 Alexandre Oliva 2002-03-15 04:35:43 EST
Ok, the need for -inst-prefix-dir makes more sense to me now, but see below.

Now I realize the handling of -L flags is wrong.  We're modifying them at points
in which they're being used for hard-coding library search paths into libraries
or executables (on systems in which -L plays a similar role to -R on GNU/Linux).
 The changes make no difference for GNU/Linux, and will have unwanted effects on
systems that actually depend on -L for -R -like functionality.

If mangling -L flags or pathnames to libraries is actually necessary, I suggest
us to do so in the relink_command line, before we run the relink_command, and
hope it just works.  This would obviate the need for -inst-prefix-dir.  But we'd
have to verify that it actually encodes the correct pathnames in programs and
shared-library dependencies.
Comment 38 Jens Petersen 2002-03-18 00:48:44 EST
Hmmm.  Going back to the original patches, I guess the order of the
-L's is significant.  This seems to be one difference between our patch
and maddog's.  In the later add_prefix_dir is added to the end of
finalize_deplibs and deplibs, whereas in the former the prefix goes into
add_dir  which is added at the beginning. The latter seems better wrt to what
you're saying about -L/-R behaviour platforms, but less safe wrt to relinking?
I mean if a different version of a package is already installed in the final
destination ahead of the buildroot install dir in the -L path, it seems to me
that relinking should break. But both the depdemo and the pango test seem
ok for both patches. However why is the order of the -L's in the final relink
command the reverse of the order in deplibs?

In the case of the pango test however the biggest difference between the former
and the latter patch is that whereas the former seems to stop relinking all
together, the latter does not.
Comment 39 Jens Petersen 2002-03-18 01:34:35 EST
Just a note to say that I confirmed that the former patch breaks relinking when
there is a different version of the package already installed ahead in the final
destination (/usr/lib).
Comment 40 Jens Petersen 2002-05-12 22:21:30 EDT
I added aoliva's (old) patch in libtool-1.4.2-8.

A relinking patch still needs to be pushed upstream.
Comment 41 Mikhail Zabaluev 2002-09-22 16:06:00 EDT
The patch, as of libtool-1.4.2-12, appears to list the inst-prefix'ed directory
behind its real counterpart. Which is wrong, because an arbitrary version of a
dependent library might get linked instead of the bundled one that is installed
in the same turn. The immediate order in the code seems right, only it is
reverted later. You need to append the paths in reverse order:

	      # Try looking first in the location we're being installed to.
	      add_dir=
	      if test -n "$inst_prefix_dir"; then
		case "$libdir" in
		[\\/]*)
		  add_dir="-L$inst_prefix_dir$libdir"
		  ;;
		esac
	      fi
	      # The order will be reversed later
	      add_dir="-L$libdir $add_dir"
Comment 42 Joe Orton 2003-05-18 17:31:02 EDT
I've filed the bug mentioned in comment 41 above as bug 91110.

Note You need to log in before you can comment on or make changes to this bug.