Bug 1421272 - rpm: debuginfo stripping is not completely reproducible
Summary: rpm: debuginfo stripping is not completely reproducible
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: rpm
Version: 8.0
Hardware: Unspecified
OS: Unspecified
medium
low
Target Milestone: rc
: 8.1
Assignee: Pavlina Moravcova Varekova
QA Contact: Eva Mrakova
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-02-10 19:42 UTC by Florian Weimer
Modified: 2019-11-05 22:22 UTC (History)
8 users (show)

Fixed In Version: rpm-4.14.2-16.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-11-05 22:22:32 UTC
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)
rpm-hardlink-order.spec (803 bytes, text/plain)
2019-04-04 12:11 UTC, Florian Weimer
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:3584 None None None 2019-11-05 22:22:59 UTC

Description Florian Weimer 2017-02-10 19:42:03 UTC
I built glibc twice in a row, the only difference in patches which are not expected to affect generated files.  However, I ended up with many spurious differences in binaries.  The difference looks like this under “eu-objcopy -s”:

 Contents of section .gnu_debuglink:
- 0000 69636f6e 76636f6e 6669672e 64656275  iconvconfig.debu
- 0010 67000000 70dd83e1                    g...p...
+ 0000 69636f6e 76636f6e 6669672e 69363836  iconvconfig.i686
+ 0010 2e646562 75670000 70dd83e1           .debug..p... 

I believe what happens is this: iconvconfig and iconvconfig.i686 are hard-linked.  Depending on the directory iteration order (which is unpredictable), one of these files is processed first and stripped.  The contents of the .gnu_debuglink section depends on the file name of the binary, so whatever name comes first determines the name, and the choice of ordering affects the contents of the generated executable file.

It would be nice to fix this because it makes it harder to figure out if a change has no impact on (some) compiled binaries.

Comment 1 Florian Festi 2017-07-13 09:40:56 UTC
Piping the list of processed files through "sort" might at least make the result reproducible.

Comment 2 Panu Matilainen 2018-10-17 10:20:59 UTC
At least one such fix went in just recently, related to dwz: https://github.com/rpm-software-management/rpm/commit/801ee2e8eb2e5b2566fdb8b8bed8f7a64235b2e1 

fweimer, can you test whether that fixes what you're seeing?

Comment 3 Florian Weimer 2018-10-17 10:24:50 UTC
Is this change already in rawhide?  Otherwise, it is going to be very difficult for me to test.

Comment 5 Mark Wielaard 2018-10-17 10:34:45 UTC
(In reply to Florian Weimer from comment #3)
> Is this change already in rawhide?  Otherwise, it is going to be very
> difficult for me to test.

Yes, it already is in Fedora 28. You can check with:
$ grep readarray /usr/lib/rpm/find-debuginfo.sh
  readarray dwz_files < <(cd "${RPM_BUILD_ROOT}/usr/lib/debug"; find -type f -name \*.debug | LC_ALL=C sort)

Comment 6 Florian Weimer 2018-10-17 12:09:37 UTC
There appears to be another issue.  /usr/lib/debug/usr/libexec/getconf contains:

954635 -r--r--r--. 6 fweimer fweimer 26K Oct 17 14:03 POSIX_V6_ILP32_OFF32.debug
954635 -r--r--r--. 6 fweimer fweimer 26K Oct 17 14:03 POSIX_V6_ILP32_OFFBIG.debug
954635 -r--r--r--. 6 fweimer fweimer 26K Oct 17 14:03 POSIX_V7_ILP32_OFF32.debug
954635 -r--r--r--. 6 fweimer fweimer 26K Oct 17 14:03 POSIX_V7_ILP32_OFFBIG.debug
954635 -r--r--r--. 6 fweimer fweimer 26K Oct 17 14:03 XBS5_ILP32_OFF32.debug
954635 -r--r--r--. 6 fweimer fweimer 26K Oct 17 14:03 XBS5_ILP32_OFFBIG.debug

But the generation of the symbolic links in /usr/lib/debug/.build-id/a3 is not reproducible.  I see the following targets:

../../../../../usr/lib/debug/usr/libexec/getconf/XBS5_ILP32_OFFBIG.debug
../../../../../usr/lib/debug/usr/libexec/getconf/XBS5_ILP32_OFF32.debug
../../../../../usr/lib/debug/usr/libexec/getconf/POSIX_V7_ILP32_OFF32.debug

I don't know if this is a separate bug, but it looks like debuginfo splitting is still not reproducible.

Comment 7 Florian Weimer 2018-10-17 12:10:20 UTC
(In reply to Florian Weimer from comment #6)

> But the generation of the symbolic links in /usr/lib/debug/.build-id/a3 is
> not reproducible.  I see the following targets:

Sorry, I meant the target of this symbolic link:

/usr/lib/debug/.build-id/a3/e85f4fea4dace43a364e95af5d01b8430c8f1c.1.debug

Comment 9 Mark Wielaard 2018-10-17 12:28:02 UTC
I assume those files are hard linked together.

A list of hard linked files is generated by the followow snippet in find-debuginfo.sh:

# Build a list of unstripped ELF files and their hardlinks
touch "$temp/primary"
find "$RPM_BUILD_ROOT" ! -path "${debugdir}/*.debug" -type f \
                     \( -perm -0100 -or -perm -0010 -or -perm -0001 \) \
                     -print |
file -N -f - | sed -n -e 's/^\(.*\):[   ]*.*ELF.*, not stripped.*/\1/p' |
xargs --no-run-if-empty stat -c '%h %D_%i %n' |
while read nlinks inum f; do
  if [ $nlinks -gt 1 ]; then
    var=seen_$inum
    if test -n "${!var}"; then
      echo "$inum $f" >>"$temp/linked"
      continue
    else
      read "$var" < <(echo 1)
    fi
  fi
  echo "$nlinks $inum $f" >>"$temp/primary"
done

That list probably also needs to be sorted.

Comment 10 Florian Weimer 2018-10-17 12:31:06 UTC
(In reply to Mark Wielaard from comment #9)
> I assume those files are hard linked together.

Yes, they are, the inode number is 954635 (in comment #6).

Comment 19 Mark Wielaard 2019-04-03 11:13:23 UTC
This issue depends on the directory traversal order that find uses. So to reproduce it, it might matter on which file system the build was done. And how much file activity there was between builds.

Comment 20 Florian Weimer 2019-04-04 12:11:01 UTC
Created attachment 1551848 [details]
rpm-hardlink-order.spec

Sorry, the glibc build has too many moving parts.  We do not yet use --g-libs (bug 1689810), among other things, which may interfere with reproduction.

This RPM spec file should reproduce the issue with high likelihood.  On the first run, I get:

# objdump -j .gnu_debuglink -s usr/bin/example-1

usr/bin/example-1:     file format elf64-x86-64

Contents of section .gnu_debuglink:
 0000 6578616d 706c652d 31362d31 2d312e65  example-16-1-1.e
 0010 6c382e78 38365f36 342e6465 62756700  l8.x86_64.debug.
 0020 1636d37a                             .6.z

On the second run, its:

$ objdump -j .gnu_debuglink -s usr/bin/example-1

usr/bin/example-1:     file format elf64-x86-64

Contents of section .gnu_debuglink:
 0000 6578616d 706c652d 31352d31 2d312e65  example-15-1-1.e
 0010 6c382e78 38365f36 342e6465 62756700  l8.x86_64.debug.
 0020 1636d37a                             .6.z

Comment 21 Florian Weimer 2019-04-04 12:12:13 UTC
I should have said that the reproducer is still file-system-specific, but I think it should work with XFS and ext4.  I tested it with XFS.

Comment 29 errata-xmlrpc 2019-11-05 22:22:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3584


Note You need to log in before you can comment on or make changes to this bug.