Bug 1570246 - glibc: When built with file 5.33, valgrind stops working due to RPM ELF processing [Fedora]
Summary: glibc: When built with file 5.33, valgrind stops working due to RPM ELF proce...
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: glibc
Version: 28
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Florian Weimer
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Keywords:
: 1582189 (view as bug list)
Depends On: 1581224
Blocks: 1580957
TreeView+ depends on / blocked
 
Reported: 2018-04-21 06:44 UTC by Lukas Slebodnik
Modified: 2018-05-27 19:31 UTC (History)
14 users (show)

(edit)
Clone Of:
(edit)
Last Closed: 2018-05-27 19:31:59 UTC


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Bugzilla 1581343 None CLOSED file-5.33 identification of gstreamer1 plugins changed 2019-07-11 14:50 UTC

Internal Trackers: 1581343

Description Lukas Slebodnik 2018-04-21 06:44:38 UTC
Description of problem:
The latest auto-sync update with master caused issues with valgrind.
It cannot run as previously without installed debug symbols for glibc.

valgrind:  Fatal error at startup: a function redirection
valgrind:  which is mandatory for this platform-tool combination
valgrind:  cannot be set up.  Details of the redirection are:
valgrind:  
valgrind:  A must-be-redirected function
valgrind:  whose name matches the pattern:      strlen
valgrind:  in an object with soname matching:   ld-linux-x86-64.so.2
valgrind:  was not found whilst processing
valgrind:  symbols from the object with soname: ld-linux-x86-64.so.2
valgrind:  
valgrind:  Possible fixes: (1, short term): install glibc's debuginfo
valgrind:  package on this machine.  (2, longer term): ask the packagers
valgrind:  for your Linux distribution to please in future ship a non-
valgrind:  stripped ld.so (or whatever the dynamic linker .so is called)
valgrind:  that exports the above-named function using the standard
valgrind:  calling conventions for this platform.  The package you need
valgrind:  to install for fix (1) is called


Version-Release number of selected component (if applicable):
sh$ rpm -qa gcc valgrind glibc\*
gcc-8.0.1-0.21.fc29.x86_64
glibc-2.27.9000-15.fc29.x86_64
valgrind-3.13.0-18.fc29.x86_64
glibc-minimal-langpack-2.27.9000-15.fc29.x86_64
glibc-headers-2.27.9000-15.fc29.x86_64
glibc-langpack-en-2.27.9000-15.fc29.x86_64
glibc-common-2.27.9000-15.fc29.x86_64
glibc-devel-2.27.9000-15.fc29.x86_64

How reproducible:
Deterministic

Steps to Reproduce:
1. dnf install -y gcc glibc-devel valgrind
2. cat >text.c <<EOF
#include <stdio.h>

int main(void) {   
   printf("Hello world!\n");
   return 0;
}
EOF
3. gcc test.c -o test
4. valgrind ./test

Actual results:
[build@cb5c29a48d0f ~]$ rpm -qa gcc valgrind glibc\*
gcc-8.0.1-0.21.fc29.x86_64
glibc-2.27.9000-15.fc29.x86_64
valgrind-3.13.0-18.fc29.x86_64
glibc-minimal-langpack-2.27.9000-15.fc29.x86_64
glibc-headers-2.27.9000-15.fc29.x86_64
glibc-langpack-en-2.27.9000-15.fc29.x86_64
glibc-common-2.27.9000-15.fc29.x86_64
glibc-devel-2.27.9000-15.fc29.x86_64
[build@cb5c29a48d0f ~]$ valgrind ./test
==196== Memcheck, a memory error detector
==196== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==196== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==196== Command: ./test
==196== 

valgrind:  Fatal error at startup: a function redirection
valgrind:  which is mandatory for this platform-tool combination
valgrind:  cannot be set up.  Details of the redirection are:
valgrind:  
valgrind:  A must-be-redirected function
valgrind:  whose name matches the pattern:      strlen
valgrind:  in an object with soname matching:   ld-linux-x86-64.so.2
valgrind:  was not found whilst processing
valgrind:  symbols from the object with soname: ld-linux-x86-64.so.2
valgrind:  
valgrind:  Possible fixes: (1, short term): install glibc's debuginfo
valgrind:  package on this machine.  (2, longer term): ask the packagers
valgrind:  for your Linux distribution to please in future ship a non-
valgrind:  stripped ld.so (or whatever the dynamic linker .so is called)
valgrind:  that exports the above-named function using the standard
valgrind:  calling conventions for this platform.  The package you need
valgrind:  to install for fix (1) is called
valgrind:  
valgrind:    On Debian, Ubuntu:                 libc6-dbg
valgrind:    On SuSE, openSuSE, Fedora, RHEL:   glibc-debuginfo
valgrind:  
valgrind:  Note that if you are debugging a 32 bit process on a
valgrind:  64 bit system, you will need a corresponding 32 bit debuginfo
valgrind:  package (e.g. libc6-dbg:i386).
valgrind:  
valgrind:  Cannot continue -- exiting now.  Sorry.

[build@cb5c29a48d0f ~]$ echo $?
1

Expected results:
//valgrind will not fail with error code 1; the same as with 2.27.9000-14
[build@cb5c29a48d0f ~]$ valgrind ./test
==171== Memcheck, a memory error detector
==171== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==171== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==171== Command: ./test
==171== 
Hello world!
==171== 
==171== HEAP SUMMARY:
==171==     in use at exit: 0 bytes in 0 blocks
==171==   total heap usage: 1 allocs, 1 frees, 1,024 bytes allocated
==171== 
==171== All heap blocks were freed -- no leaks are possible
==171== 
==171== For counts of detected and suppressed errors, rerun with: -v
==171== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
[build@cb5c29a48d0f ~]$ echo $?
0


Additional info:
Changelog mentions following upstream commits
* Thu Apr 19 2018 Florian Weimer <fweimer@redhat.com> - 2.27.9000-15 
- Auto-sync with upstream branch master, commit 0085be1415a38b40a5a1a12e49368498f1687380.
* Mon Apr 09 2018 Florian Weimer <fweimer@redhat.com> - 2.27.9000-14 
- Auto-sync with upstream branch master, commit 583a27d525ae189bdfaa6784021b92a9a1dae12e.

Comment 1 Lukas Slebodnik 2018-04-21 06:47:15 UTC
Short term fix recommended by valgrind (to install debug symbols) is a functional workaround:

[build@cb5c29a48d0f ~]$ rpm -qa gcc valgrind glibc\*
gcc-8.0.1-0.21.fc29.x86_64
glibc-2.27.9000-15.fc29.x86_64
glibc-debuginfo-common-2.27.9000-15.fc29.x86_64
valgrind-3.13.0-18.fc29.x86_64
glibc-minimal-langpack-2.27.9000-15.fc29.x86_64
glibc-headers-2.27.9000-15.fc29.x86_64
glibc-langpack-en-2.27.9000-15.fc29.x86_64
glibc-debuginfo-2.27.9000-15.fc29.x86_64
glibc-common-2.27.9000-15.fc29.x86_64
glibc-devel-2.27.9000-15.fc29.x86_64
[build@cb5c29a48d0f ~]$ valgrind ./test
==214== Memcheck, a memory error detector
==214== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==214== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==214== Command: ./test
==214== 
Hello world!
==214== 
==214== HEAP SUMMARY:
==214==     in use at exit: 0 bytes in 0 blocks
==214==   total heap usage: 1 allocs, 1 frees, 1,024 bytes allocated
==214== 
==214== All heap blocks were freed -- no leaks are possible
==214== 
==214== For counts of detected and suppressed errors, rerun with: -v
==214== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Comment 2 Florian Weimer 2018-04-21 19:22:23 UTC
.symtab is gone from /lib64/ld-linux-x86-64.so.2.  I'm not aware of anything which would have caused this.  Could this be a change in the way we separate debugging information?

Comment 3 Mark Wielaard 2018-04-21 21:51:42 UTC
(In reply to Florian Weimer from comment #2)
> .symtab is gone from /lib64/ld-linux-x86-64.so.2.  I'm not aware of anything
> which would have caused this.  Could this be a change in the way we separate
> debugging information?

The issue is indeed that .symtab is somehow gone. It was there in glibc-2.27.9000-14.fc29. But I don't immediate see something that would cause this in the changes between -14 and -15. You pass -g as find_debuginfo_args which should do the right thing.

Comment 4 Mark Wielaard 2018-04-22 21:44:39 UTC
This is weird, I can replicate it when doing a fedpkg mockbuild on master (the produced ld-2.27.9000.so in the glibc-2.27.9000-15.fc29.x86_64.rpm doesn't contain a SYMTAB). But when I then do a fedpkg mockbuild --no-clean-all to investigate what happened the resulting ld-2.27.9000.so looks fine (contains a SYMTAB)... I don't understand what is going on (or why the --no-clean-all build would produce different results).

Comment 5 Florian Weimer 2018-04-24 15:34:57 UTC
(In reply to Mark Wielaard from comment #4)
> This is weird, I can replicate it when doing a fedpkg mockbuild on master
> (the produced ld-2.27.9000.so in the glibc-2.27.9000-15.fc29.x86_64.rpm
> doesn't contain a SYMTAB). But when I then do a fedpkg mockbuild
> --no-clean-all to investigate what happened the resulting ld-2.27.9000.so
> looks fine (contains a SYMTAB)... I don't understand what is going on (or
> why the --no-clean-all build would produce different results).

Did you update to the “local” (buildroot) package versions?  Often, when issues fail to reproduce in this way, it's because package version skew between whatever is installed locally (from the rawhide compose), and what's in the “local” (buildroot) repositories.

I'll try to free up some time to debug this further, but I'm pretty swamped during the coming days, so I'd appreciate any help.

Comment 6 Lukas Slebodnik 2018-05-04 19:14:14 UTC
Just a note:
It is reproducible even with glibc-2.27.9000-16.fc29.x86_64.rpm.

Maybe, it would worth to try rebuilding glibc-2.27.9000-14.fc29.x86_64.rpm with latest rawhide. It might be related to toolchain and not to glibc.

Comment 7 Florian Weimer 2018-05-16 13:51:00 UTC
(In reply to Florian Weimer from comment #5)
> Did you update to the “local” (buildroot) package versions?  Often, when
> issues fail to reproduce in this way, it's because package version skew
> between whatever is installed locally (from the rawhide compose), and what's
> in the “local” (buildroot) repositories.

To replicate more closely what is used by the builders, you need to use something like this:

mock -r fedora-rawhide-x86_64 --init
mock -r fedora-rawhide-x86_64 --enablerepo=local --update
mock -r fedora-rawhide-x86_64 --enablerepo=local --noclean $SRPM

I didn't pick the “local” name, but it's what's in the DNF configuration supplied by mock.  The “local” repositories are directly supplied by Koji, they are not composes, and are not backed by the Fedora mirror network.  Repositories like these:

  https://kojipkgs.fedoraproject.org/repos/rawhide/latest/

Comment 8 Lukas Slebodnik 2018-05-22 09:18:40 UTC
It is reproducible even with f28

The oldest non-working version is:
[build@804cc6388406 ~]$ rpm -q glibc
glibc-2.27-12.fc28.x86_64

I was not able to find any older version (2.27-9 .. 2.27-11) in koji

[build@804cc6388406 ~]$ rpm -q --changelog glibc | head -n 14
* Fri May 11 2018 Florian Weimer <fweimer@redhat.com> - 2.27-12
- Unconditionally build downstream with -mstackrealign for now

* Fri May 11 2018 Florian Weimer <fweimer@redhat.com> - 2.27-11
- Inherit compiler flags in the original order

* Fri May 11 2018 Florian Weimer <fweimer@redhat.com> - 2.27-10
- Inherit the -mstackrealign flag if it is set

* Fri May 11 2018 Florian Weimer <fweimer@redhat.com> - 2.27-9
- Use /usr/bin/python3 for benchmarks scripts (#1577223)

* Thu Mar 29 2018 Florian Weimer <fweimer@redhat.com> - 2.27-8
- Auto-sync with upstream branch release/2.27/master,

Comment 9 Florian Weimer 2018-05-22 09:40:09 UTC
(In reply to Lukas Slebodnik from comment #8)
> It is reproducible even with f28
> 
> The oldest non-working version is:
> [build@804cc6388406 ~]$ rpm -q glibc
> glibc-2.27-12.fc28.x86_64

Hmm.  glibc-2.27-8.fc28.x86_64 appears to be okay.  glibc-2.27-14.fc28.x86_64 is not.

I did not consciously change anything on the glibc side since then.

Comment 10 Florian Weimer 2018-05-22 11:10:40 UTC
I rebuild with the current Fedora 28 buildroot and downgraded binutils to binutils-2.29.1-20.fc28.x86_64.  The bug is still there.

Comment 11 Florian Weimer 2018-05-22 11:32:13 UTC
On a hunch, I downgraded file and file-libs from file-5.33-2.fc28 to  file-5.32-3.fc28.  This seems to have fixed it.

5.33 prints: ELF 64-bit LSB pie executable x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=56aca38d02690c73020768752ad47d642694d4ba, not stripped
5.32 prints: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=56aca38d02690c73020768752ad47d642694d4ba, not stripped

This seems to cause scripts/brp-strip-shared and scripts/brp-strip not do the right thing for glibc anymore.

Comment 12 Kamil Dudka 2018-05-22 12:53:38 UTC
(In reply to Florian Weimer from comment #11)
> 5.33 prints: ELF 64-bit LSB pie executable x86-64, version 1 (SYSV),
> dynamically linked, BuildID[sha1]=56aca38d02690c73020768752ad47d642694d4ba,
> not stripped

Is the above output technically incorrect?

Comment 13 Florian Weimer 2018-05-24 13:19:13 UTC
*** Bug 1582189 has been marked as a duplicate of this bug. ***

Comment 14 Florian Weimer 2018-05-24 14:08:41 UTC
We are definitely blocked on the rpm update (bug 1581224).  The file update (bug 1581343) is not sufficient.

Comment 15 Kamil Dudka 2018-05-24 14:25:54 UTC
(In reply to Florian Weimer from comment #14)
> We are definitely blocked on the rpm update (bug 1581224).  The file update
> (bug 1581343) is not sufficient.

Indeed.  I should have looked at the actual RPM fix.  We need to revert also the following upstream commit to fix the MIME output of file:

https://github.com/file/file/commits/FILE5_32-81-g7dbecfe4

Comment 16 Kamil Dudka 2018-05-24 14:27:55 UTC
Oops, typo.  I meant this commit:

https://github.com/file/file/commit/FILE5_32-81-g7dbecfe4

Comment 17 Fedora Update System 2018-05-24 17:42:34 UTC
glibc-2.27-15.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-916dfe0d86

Comment 18 Fedora Update System 2018-05-25 18:45:17 UTC
glibc-2.27-15.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-916dfe0d86

Comment 19 Fedora Update System 2018-05-27 19:31:59 UTC
glibc-2.27-15.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.