Bug 1467409 - LTO broken support
LTO broken support
Status: CLOSED WONTFIX
Product: Fedora
Classification: Fedora
Component: binutils (Show other bugs)
27
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Nick Clifton
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-03 13:05 EDT by Tomasz Kłoczko
Modified: 2017-08-15 07:38 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-08-15 07:38:57 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Tomasz Kłoczko 2017-07-03 13:05:48 EDT
I've been trying to play with -flto option abd seems gcc has no problems with this option however ar, ranlib are not ready to LTO or LTO support is broken because those commands are printing "plugin needed to handle lto object" warning messages.
Comment 1 Nick Clifton 2017-07-17 06:20:40 EDT
Hi Tomasz,

  Do you have a specific example that I could try out ?

  The ar tool is built with plugin support enabled, so it should work.
  Ranlib does not support plugins, but that should not be an issue, since
  ar can do everything that ranlib does, and more.

Cheers
  Nick
Comment 2 Tomasz Kłoczko 2017-07-18 05:13:44 EDT
You can take any package and alter CFLAGS adding -flto.
If package uses autoconf you can change in spec file:

%configure
     <autoconf-params>

to:

%configure
     CFALGS="%{optflags} -flto" \
     <autoconf-params>

and try to rebuild such package.
Comment 3 Nick Clifton 2017-07-18 06:34:19 EDT
Hi Tomasz,

  OK, well I tried rebuilding the binutils package with -flto enabled and sure enough there was an warning from ar:

  libtool: link: ar rc .libs/libopcodes.a  dis-buf.o disassemble.o dis-init.o i386-dis.o i386-opc.o
  ar: dis-buf.o: plugin needed to handle lto object

But, when I looked at the actual command being invoked, I saw that the --plugin option to ar was missing.  When I ran the command by hand, adding in the missing --plugin option, everything worked. 

  ar rc --plugin /usr/libexec/gcc/x86_64-redhat-linux/7/liblto_plugin.so.0.0.0 .libs/libopcodes.a dis-buf.o disassemble.o dis-init.o i386-dis.o i386-opc.o

So to me this looks like  a libtool problem, in that ar is being invoked incorrectly.  Unfortunately I am not a libtool expert, so I am not sure how to proceed from here.

Cheers
  Nick

PS.  Oh, and I was wrong about the ranlib program.  It does accept the --plugin command line option.

PPS. There are other errors later on in the -lfto enabled build of the binutils, this time about undefined symbols.  I am assuming that these are due to ar and ranlib not being able to create libraries properly.
Comment 4 Tomasz Kłoczko 2017-07-19 06:42:55 EDT
It would be really bad if switching to LTO would require two changes:
- add -flto to compile time options
- add --plugin <lto_pluging_DSO_PATH> param to all binutils programs.

If it was original plan IMO it is a bit to complicated.
As object is already generated with in LTO format/metadata it should be only ld switch to use LTO to link executable/DSO. All other tools generating static libraries like ar, ranlib should be transparent and if they will find that on the list are objects LTO metadata they should automatically load LTO plugin.

Do you know maybe where is original LTO design documentation?
Comment 5 Jakub Jelinek 2017-07-19 06:50:52 EDT
If you use gcc-ar instead of ar, gcc-nm instead of nm and gcc-ranlib instead of ranlib, then those programs take care of adding the needed --plugin arguments.
See https://gcc.gnu.org/wiki/LinkTimeOptimizationFAQ
There are some words about newer binutils automatically being able to find the plugin, but it isn't clear to me where exactly is it searching for those and how would it work in presence of multiple compilers.  Nick?
Comment 6 Tomasz Kłoczko 2017-07-19 08:05:45 EDT
I'm experimenting with building zabbix using LTO and seems I just been able to build my first LTO optimized binaries.
As this project is not using libtool it was necessary to teak a bit configure.ac to add 

+AM_PROG_LIBTOOL

and remove:

-RANLIB="ranlib"
-AC_SUBST(RANLIB)

With above change combined with:

%configure \
        CFLAGS="%{optflags} -flto" \
        AR="gcc-ar" RANLIB="gcc-ranlib" NM="gcc-nm" \
        LDFLAGS="-Wl,--as-needed" \
        --disable-static \
        <other-ac_options>

I was able to produce my first LTO optimized binaries.
Looks like adapting to LTO any AM/AC/LT project will be relatively easy however I'm not going to recommend to use LTO that way as IMO it would probably better to replace ar, ranlib, nm by versions of those binaries from gcc to make LTO IMO way easier to use.
Other way is to change ar, ranlib, nm binutils to fully transparent and stop provide gcc-{ar,ranlib,nm} by gcc.

Nevertheless I have now possibility to test LTO :)
Comment 7 Nick Clifton 2017-07-19 12:01:51 EDT
(In reply to Jakub Jelinek from comment #5)

> See https://gcc.gnu.org/wiki/LinkTimeOptimizationFAQ
> There are some words about newer binutils automatically being able to find
> the plugin, but it isn't clear to me where exactly is it searching for those
> and how would it work in presence of multiple compilers.  Nick?

Hmm - this is probably a reference to:

  https://sourceware.org/ml/binutils/2014-01/msg00213.html

Although for that to work the lto plugin has to be copied/linked into a non-standard location: /usr/lib/bfd-plugins.  This does solve the problem, but I am not sure if it is an acceptable solution.  I could change the binutils sources so that they look in a different directory, but what is the standard directory for plugins and, since I expect that it depends upon the compiler(s) installed, how can the BFD library find out the name/location of this directory ?

It seems to me however that another way we might solve this problem is to add an enhancement to the redhat-rpm-config package so that the necessary AR="gcc-ar", RANLIB="gcc-ranlib" and NM="gcc-nm" defines are automatically added to CFLAGS.  (And CXXFLAGS of course).  It should not matter if the plugin is used when -flto is not used, although I suppose that is could slow down the build process a little bit.  But it would then mean that package maintainers could add the -flto flag to their builds as and when they see fit and it should "just work"(tm).

Cheers
  Nick
Comment 8 Jakub Jelinek 2017-07-19 12:08:12 EDT
(In reply to Nick Clifton from comment #7)
> Although for that to work the lto plugin has to be copied/linked into a
> non-standard location: /usr/lib/bfd-plugins.  This does solve the problem,

Ok, I'll talk to Honza what is intended.

> It seems to me however that another way we might solve this problem is to
> add an enhancement to the redhat-rpm-config package so that the necessary
> AR="gcc-ar", RANLIB="gcc-ranlib" and NM="gcc-nm" defines are automatically
> added to CFLAGS.  (And CXXFLAGS of course).  It should not matter if the
> plugin is used when -flto is not used, although I suppose that is could slow
> down the build process a little bit.  But it would then mean that package
> maintainers could add the -flto flag to their builds as and when they see
> fit and it should "just work"(tm).

First of all, I think LTO isn't in the state I'd want to actively encourage it for packages, the debug info story is still extremely bad, lots of code is practically undebuggable with that.  And debugging GCC bugs with LTO is also painfully hard.  And I would prefer not to impose gcc-{ar,nm,ranlib} into everyone just because a package or three use LTO.
Comment 9 Jan Kurik 2017-08-15 03:53:37 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 27 development cycle.
Changing version to '27'.
Comment 10 Tomasz Kłoczko 2017-08-15 07:21:34 EDT
I think that this ticket can be closed or it should be kept in opened mode if you want in future solve LTO linking without gcc-{ar,nm,ranlib} wrappers.
Comment 11 Nick Clifton 2017-08-15 07:38:57 EDT
Lets close this bug for now.  I am sure that once LTO is ready for mainstream use the issue will be looked at again.

Note You need to log in before you can comment on or make changes to this bug.