Bug 1600035 - FreeIPA's ipa-kdb compile error "multiple definition of `_end'"
Summary: FreeIPA's ipa-kdb compile error "multiple definition of `_end'"
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: samba
Version: 29
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Alexander Bokovoy
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: 1599441
TreeView+ depends on / blocked
 
Reported: 2018-07-11 09:20 UTC by Christian Heimes
Modified: 2019-11-27 19:28 UTC (History)
15 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2019-11-27 19:28:44 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1599441 0 unspecified CLOSED Failed to link testing program with error: multiple definition of `_end' 2021-02-22 00:41:40 UTC
Red Hat Bugzilla 1599521 0 unspecified CLOSED linker errors since binutils-2.30.90-1.fc29 : _end: invalid version 21 (max 0), error adding symbols: bad value 2021-02-22 00:41:40 UTC

Internal Links: 1599441 1599521

Description Christian Heimes 2018-07-11 09:20:22 UTC
Description of problem:
FreeIPA's ipa-kdb plugin from git master fails to compile on rawhide. /usr/lib64/libndr-standard.so and /usr/lib64/libsamba-util.so export duplicate symbols "_end"

Version-Release number of selected component (if applicable):
samba-devel-4.8.3-2.fc29.1.x86_64


How reproducible:
always

Steps to Reproduce:
1. https://github.com/freeipa/freeipa.git
2. cd freeipa
3. dnf builddep -b -D "with_python2 0" -D "with_wheels 1" -D "with_lint 1" --spec freeipa.spec.in --best --allowerasing
4. ./autogen.sh
5. make

Actual results:
make[3]: Entering directory '/root/freeipa/daemons/ipa-kdb'
/bin/sh ../../libtool  --tag=CC   --mode=link gcc  -Wall -Wshadow -Wstrict-prototypes -Wpointer-arith -Wcast-align -Werror-implicit-function-declaration  -g -O2 -Werror=implicit-function-declaration -avoid-version -module -Wl,--version-script,./ipa_kdb.exports  -o ipadb.la -rpath /usr/local/lib/krb5/plugins/kdb ipa_kdb.lo ipa_kdb_common.lo ipa_kdb_mkey.lo ipa_kdb_passwords.lo ipa_kdb_principals.lo ipa_kdb_pwdpolicy.lo ipa_kdb_mspac.lo ipa_kdb_delegation.lo ipa_kdb_audit_as.lo  ipa_kdb_certauth.lo -lkrb5 -lk5crypto -lcom_err  -lldap_r -llber -lndr-krb5pac -lndr-standard -lndr -lsamba-util -ltevent -ltalloc  -lunistring -lssl3 -lsmime3 -lnss3 -lnssutil3 -lplds4 -lplc4 -lnspr4 -lpthread -ldl  -lsss_certmap  ../../util/libutil.la  
libtool: link: gcc -shared  -fPIC -DPIC  .libs/ipa_kdb.o .libs/ipa_kdb_common.o .libs/ipa_kdb_mkey.o .libs/ipa_kdb_passwords.o .libs/ipa_kdb_principals.o .libs/ipa_kdb_pwdpolicy.o .libs/ipa_kdb_mspac.o .libs/ipa_kdb_delegation.o .libs/ipa_kdb_audit_as.o .libs/ipa_kdb_certauth.o  -Wl,--whole-archive ../../util/.libs/libutil.a -Wl,--no-whole-archive  -lndr-krb5pac -lndr-standard -lndr -lsamba-util -ltevent -ltalloc -lunistring -lsss_certmap -lcrypto -lkrb5 -lk5crypto -lcom_err -lldap_r -llber -lssl3 -lsmime3 -lnss3 -lnssutil3 -lplds4 -lplc4 -lnspr4 -lpthread -ldl  -g -O2 -Wl,--version-script -Wl,./ipa_kdb.exports   -Wl,-soname -Wl,ipadb.so -o .libs/ipadb.so
/usr/bin/ld: /usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/libndr-standard.so:(*IND*+0x0): multiple definition of `_end'
/usr/bin/ld: /usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/libsamba-util.so:(*IND*+0x0): multiple definition of `_end'


Expected results:
no error

Additional info:
# objdump -TC /usr/lib64/libndr-standard.so | grep " _end"
0000000000339001 g    D  .data.rel.ro.local     0000000000000000  NDR_STANDARD_0.0.1 _end
0000000000339001 g    D  .data.rel.ro.local     0000000000000000 (NDR_STANDARD_0.0.1) _end
# objdump -TC /usr/lib64/libsamba-util.so | grep " _end"
000000000007a2f0 g    D  .data.rel.ro.local     0000000000000000  SAMBA_UTIL_0.0.1 _end
000000000007a2f0 g    D  .data.rel.ro.local     0000000000000000 (SAMBA_UTIL_0.0.1) _end

Comment 1 Christian Heimes 2018-07-11 09:29:02 UTC
This may be a duplicate of BZ #1599441.

# rpm -qa binutils samba gcc
samba-4.8.3-2.fc29.1.x86_64
gcc-8.1.1-4.fc29.1.x86_64
binutils-2.30.90-1.fc29.x86_64

Comment 2 Christian Heimes 2018-07-11 09:33:17 UTC
FreeIPA's ipa-sam plugin also fails to compile:

make[3]: Entering directory '/root/freeipa/daemons/ipa-sam'
/bin/sh ../../libtool  --tag=CC   --mode=link gcc  -Wall -Wshadow -Wstrict-prototypes -Wpointer-arith -Wcast-align -Werror-implicit-function-declaration  -g -O2 -Werror=implicit-function-declaration -avoid-version -module   -o ipasam.la -rpath /usr/local/lib/samba/pdb ipa_sam.lo  -lcrypto  -lldap_r -llber -lkrb5 -lk5crypto -lcom_err  -ltalloc  -lsamba-util -ltevent -ltalloc  -lndr -lsamba-util -ltevent -ltalloc  -L/usr/lib64/samba -Wl,-rpath=/usr/lib64/samba -lsmbldap -lpdb -lsmbconf  -lsss_idmap  ../../asn1/libipaasn1.la ../../util/libutil.la  
libtool: link: gcc -shared  -fPIC -DPIC  .libs/ipa_sam.o  -Wl,--whole-archive ../../asn1/.libs/libipaasn1.a ../../util/.libs/libutil.a -Wl,--no-whole-archive  -lndr -lsamba-util -ltevent -ltalloc -L/usr/lib64/samba -lsmbldap -lpdb -lsmbconf -lsss_idmap -lcrypto -lkrb5 -lk5crypto -lcom_err -lldap_r -llber -lssl3 -lsmime3 -lnss3 -lnssutil3 -lplds4 -lplc4 -lnspr4 -lpthread -ldl  -g -O2 -Wl,-rpath=/usr/lib64/samba   -Wl,-soname -Wl,ipasam.so -o .libs/ipasam.so
/usr/bin/ld: cannot find -lpdb
/usr/bin/ld: /usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/libsmbconf.so:(*IND*+0x0): multiple definition of `_edata'
/usr/bin/ld: /usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/libsmbconf.so:(*IND*+0x0): multiple definition of `__bss_start'
/usr/bin/ld: /usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/libsmbconf.so:(*IND*+0x0): multiple definition of `_end'

Comment 3 Alexander Bokovoy 2018-07-11 09:42:08 UTC
Nick, could you please explain is this change in binutils behavior related to https://sourceware.org/ml/binutils/2018-05/msg00127.html and how we should fix it?

Comment 4 Christian Heimes 2018-07-11 09:44:50 UTC
I guess Samba packages have be rebuild after the binutil bug has been fixed.

Comment 5 Alexander Bokovoy 2018-07-11 09:56:11 UTC
I pushed a rebuild https://koji.fedoraproject.org/koji/taskinfo?taskID=28126741

Comment 6 Christian Heimes 2018-07-11 10:59:55 UTC
Alexander's rebuild didn't the issue:

# koji download-task 28126841
...
# rm samba-4.8.3-3.fc29.1.src.rpm
# dnf install -y ./*.rpm
...
# rpm -qa samba-devel
samba-devel-4.8.3-3.fc29.1.x86_64
# make
...
libtool: link: gcc -shared  -fPIC -DPIC  .libs/ipa_kdb.o .libs/ipa_kdb_common.o .libs/ipa_kdb_mkey.o .libs/ipa_kdb_passwords.o .libs/ipa_kdb_principals.o .libs/ipa_kdb_pwdpolicy.o .libs/ipa_kdb_mspac.o .libs/ipa_kdb_delegation.o .libs/ipa_kdb_audit_as.o .libs/ipa_kdb_certauth.o  -Wl,--whole-archive ../../util/.libs/libutil.a -Wl,--no-whole-archive  -lndr-krb5pac -lndr-standard -lndr -lsamba-util -ltevent -ltalloc -lunistring -lsss_certmap -lcrypto -lkrb5 -lk5crypto -lcom_err -lldap_r -llber -lssl3 -lsmime3 -lnss3 -lnssutil3 -lplds4 -lplc4 -lnspr4 -lpthread -ldl  -g -O2 -Wl,--version-script -Wl,./ipa_kdb.exports   -Wl,-soname -Wl,ipadb.so -o .libs/ipadb.so
/usr/bin/ld: /usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/libndr-standard.so:(*IND*+0x0): multiple definition of `_end'
/usr/bin/ld: /usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/libsamba-util.so:(*IND*+0x0): multiple definition of `_end'

Comment 7 Alexander Bokovoy 2018-07-11 12:54:50 UTC
Yes, rebuilding against binutils 2.30.90-2.fc29 does not fix the problem. We need a guidance from binutils guys.

Comment 8 Nick Clifton 2018-07-11 16:15:21 UTC
(In reply to Alexander Bokovoy from comment #3)
> Nick, could you please explain is this change in binutils behavior related
> to https://sourceware.org/ml/binutils/2018-05/msg00127.html and how we
> should fix it?

Right - H.J's patch, and several others that followed it, are all in the 
about-to-be-released 2.31 FSF binutils.  They change how shared libraries
are linked, such that they always provide their own local definitions of
the _end, _edata and _bss_start symbols.  This would all be fine, except
for shared libraries that export all symbols be default.  (Rather than
just exporting those symbols that form part of their API).

Obviously there are libraries in the Fedora ecosystem that now trigger this
problem, and the samba libraries are some of them.  When the bug/mis-feature
was reported on Monday I added a patch to the linker to revert part of the
fix for PR 23162.  I had hoped that this would be enough.  It appears however
that I may need to revert some more.

The best way to fix the problem, in my opinion, would be to fix the build
machinery for the samba libraries so that a version script is used when they
are linked, and this script explicitly defines which symbols are exported
by the library.  (And also explicitly ensures that all other symbols are kept
local).  This would be a lot of hassle for the samba developers however, and
I can appreciate that it is not something that they might want to do.

So instead I need to investigate why these multiple definitions are occurring
and see if I can find another way to prevent them.  The other alternative is
to just abandon the binutils rebase to 2.31 and go back to the good old 2.30
release.  I would prefer not to do that if possible however.

Cheers
  Nick

Comment 9 Nick Clifton 2018-07-11 16:18:01 UTC
A couple of quick questions:

> /usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/libndr-standard.so:
> (*IND*+0x0): multiple definition of `_end'

Is this error x86_64 specific, or does it happen with other architectures as well ?

Is libndr-standard part of the samba libraries, and was it rebuilt with Alexander's rebuild ?

Thanks.

Nick

Comment 10 Nick Clifton 2018-07-11 16:20:53 UTC
(In reply to Christian Heimes from comment #6)

Third question:

> /usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/libndr-standard.so:
> (*IND*+0x0): multiple definition of `_end'

Have the multiple definitions in the libsmbconf.so library (as reported in
comment #2) gone away ?

Comment 11 Alexander Bokovoy 2018-07-12 07:29:01 UTC
libndr-standard is part of Samba, yes.

I did a patch upstream that is going to hind these three symbols by default in all Samba libraries. Here is a build pipeline: https://gitlab.com/samba-team/devel/samba/pipelines/25612925, once it succeeds, I'll submit that upstream.

Comment 12 Alexander Bokovoy 2018-07-12 11:22:39 UTC
A new pipeline after I fixed abi tests at Samba side is this: https://gitlab.com/samba-team/devel/samba/pipelines/25615509. I submitted patch upstream.

I also have rebuilt samba in Rawhide using the patch: https://koji.fedoraproject.org/koji/taskinfo?taskID=28154430

A check for libraries shows that now these symbols aren't public anymore.

Comment 13 Lukas Slebodnik 2018-07-12 11:59:41 UTC
I still can see linking failures with latest update in koji

 samba-client-libs        x86_64   2:4.8.3-4.fc29               rawhide   4.9 M
 samba-common             noarch   2:4.8.3-4.fc29               rawhide   139 k
 samba-common-libs        x86_64   2:4.8.3-4.fc29               rawhide   101 k
 samba-common-tools       x86_64   2:4.8.3-4.fc29               rawhide   388 k
 samba-dc-libs            x86_64   2:4.8.3-4.fc29               rawhide   522 k
 samba-devel              x86_64   2:4.8.3-4.fc29               rawhide   238 k
 samba-libs               x86_64   2:4.8.3-4.fc29               rawhide   216 k
 samba-winbind            x86_64   2:4.8.3-4.fc29               rawhide   485 k
 samba-winbind-modules    x86_64   2:4.8.3-4.fc29               rawhide    53 k


/bin/dash ./libtool  --tag=CC   --mode=link gcc  -Wall -Wshadow -Wstrict-prototypes -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings -Wundef -Werror-implicit-function-declaration 
-Winit-self -Wmissing-include-dirs -fno-strict-aliasing -std=gnu99   -g3 -O2 -Werror -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -avoid-version -module  -o libsss_krb
5.la -rpath /usr/local/lib/sssd src/providers/krb5/libsss_krb5_la-krb5_init.lo -ltalloc  -ldhash  -lkrb5 -lk5crypto -lcom_err -lpcre  libsss_util.la libsss_crypt.la libsss_debug.la libsss
_child.la  libsss_krb5_common.la 
/usr/bin/ld: /usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/libndr-standard.so:(*IND*+0x0): multiple definition of `__bss_start'
collect2: error: ld returned 1 exit status
libtool: link: ( cd ".libs" && rm -f "_py2sss.la" && ln -s "../_py2sss.la" "_py2sss.la" )
make[2]: *** [Makefile:16678: sssd_pac] Error 1



sh$ rpm -qf /usr/lib64/libndr-standard.so
samba-devel-4.8.3-4.fc29.x86_64
samba-client-libs-4.8.3-4.fc29.x86_64

Comment 14 Alexander Bokovoy 2018-07-12 12:20:22 UTC
Yes, that's with a typo _bss_start (one, not two underscores at the beginning). I'm going to ship a new build.

Comment 15 Christian Heimes 2018-07-12 13:05:57 UTC
Samba build samba-4.8.3-4.1.fc29 fixed the problem for me. FreeIPA compiles again on F29 rawhide. Good work, Alexander!

Comment 16 Alan Modra 2018-07-12 14:33:44 UTC
I'd like to know how the bad samba libraries were built.  If you can manage to have both _end.1 and _end@@SAMBA_UTIL_0.0.1 defined then we can probably get into the same mess with other symbols.  Every trick I tried to construct a testcase failed..

Comment 17 Alexander Bokovoy 2018-07-12 15:12:39 UTC
When samba builds, it generates a vscript for each library it has. The code to generate it is defined in buildtools/wafsamba/samba_abi.py. It is tested in buildtools/wafsamba/tests/test_abi.py

You can run that test suite with the following sequence inside samba source code:

$ PYTHONPATH=third_party/waf/wafadmin python2 -mpytest buildtools/wafsamba/tests/test_abi.py
==================================================================================== test session starts =====================================================================================
platform linux2 -- Python 2.7.15, pytest-3.4.2, py-1.5.3, pluggy-0.6.0
rootdir: /home/abokovoy/src/samba, inifile:
plugins: sourceorder-0.5, multihost-3.0
collected 9 items                                                                                                                                                                            

buildtools/wafsamba/tests/test_abi.py .........                                                                                                                                        [100%]

================================================================================== 9 passed in 0.03 seconds ==================================================================================

Comment 18 Alexander Bokovoy 2018-07-12 15:24:50 UTC
If you want to see all generated vscripts, you need to run a samba build, then do

$ find bin -name '*vscript'

in the source tree. $srcdir/bin is the default target.

Comment 19 Alan Modra 2018-07-13 02:36:28 UTC
So I found a reproducer for this gold bug, which appears to be present in at least 2.29, 2.30 and 2.31.
https://sourceware.org/bugzilla/show_bug.cgi?id=23409

Comment 20 Jan Kurik 2018-08-14 11:21:23 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 29 development cycle.
Changing version to '29'.

Comment 21 Ben Cotton 2019-10-31 20:41:48 UTC
This message is a reminder that Fedora 29 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 29 on 2019-11-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '29'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 29 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 22 Ben Cotton 2019-11-27 19:28:44 UTC
Fedora 29 changed to end-of-life (EOL) status on 2019-11-26. Fedora 29 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.