Piping the list of processed files through "sort" might at least make the result reproducible.

At least one such fix went in just recently, related to dwz: https://github.com/rpm-software-management/rpm/commit/801ee2e8eb2e5b2566fdb8b8bed8f7a64235b2e1 

fweimer, can you test whether that fixes what you're seeing?

Is this change already in rawhide?  Otherwise, it is going to be very difficult for me to test.

(In reply to Florian Weimer from comment #3)
> Is this change already in rawhide?  Otherwise, it is going to be very
> difficult for me to test.

Yes, it already is in Fedora 28. You can check with:
$ grep readarray /usr/lib/rpm/find-debuginfo.sh
  readarray dwz_files < <(cd "${RPM_BUILD_ROOT}/usr/lib/debug"; find -type f -name \*.debug | LC_ALL=C sort)

There appears to be another issue.  /usr/lib/debug/usr/libexec/getconf contains:

954635 -r--r--r--. 6 fweimer fweimer 26K Oct 17 14:03 POSIX_V6_ILP32_OFF32.debug
954635 -r--r--r--. 6 fweimer fweimer 26K Oct 17 14:03 POSIX_V6_ILP32_OFFBIG.debug
954635 -r--r--r--. 6 fweimer fweimer 26K Oct 17 14:03 POSIX_V7_ILP32_OFF32.debug
954635 -r--r--r--. 6 fweimer fweimer 26K Oct 17 14:03 POSIX_V7_ILP32_OFFBIG.debug
954635 -r--r--r--. 6 fweimer fweimer 26K Oct 17 14:03 XBS5_ILP32_OFF32.debug
954635 -r--r--r--. 6 fweimer fweimer 26K Oct 17 14:03 XBS5_ILP32_OFFBIG.debug

But the generation of the symbolic links in /usr/lib/debug/.build-id/a3 is not reproducible.  I see the following targets:

../../../../../usr/lib/debug/usr/libexec/getconf/XBS5_ILP32_OFFBIG.debug
../../../../../usr/lib/debug/usr/libexec/getconf/XBS5_ILP32_OFF32.debug
../../../../../usr/lib/debug/usr/libexec/getconf/POSIX_V7_ILP32_OFF32.debug

I don't know if this is a separate bug, but it looks like debuginfo splitting is still not reproducible.

(In reply to Florian Weimer from comment #6)

> But the generation of the symbolic links in /usr/lib/debug/.build-id/a3 is
> not reproducible.  I see the following targets:

Sorry, I meant the target of this symbolic link:

/usr/lib/debug/.build-id/a3/e85f4fea4dace43a364e95af5d01b8430c8f1c.1.debug

I assume those files are hard linked together.

A list of hard linked files is generated by the followow snippet in find-debuginfo.sh:

# Build a list of unstripped ELF files and their hardlinks
touch "$temp/primary"
find "$RPM_BUILD_ROOT" ! -path "${debugdir}/*.debug" -type f \
                     \( -perm -0100 -or -perm -0010 -or -perm -0001 \) \
                     -print |
file -N -f - | sed -n -e 's/^\(.*\):[   ]*.*ELF.*, not stripped.*/\1/p' |
xargs --no-run-if-empty stat -c '%h %D_%i %n' |
while read nlinks inum f; do
  if [ $nlinks -gt 1 ]; then
    var=seen_$inum
    if test -n "${!var}"; then
      echo "$inum $f" >>"$temp/linked"
      continue
    else
      read "$var" < <(echo 1)
    fi
  fi
  echo "$nlinks $inum $f" >>"$temp/primary"
done

That list probably also needs to be sorted.

(In reply to Mark Wielaard from comment #9)
> I assume those files are hard linked together.

Yes, they are, the inode number is 954635 (in comment #6).

This issue depends on the directory traversal order that find uses. So to reproduce it, it might matter on which file system the build was done. And how much file activity there was between builds.

Created attachment 1551848 [details]
rpm-hardlink-order.spec

Sorry, the glibc build has too many moving parts.  We do not yet use --g-libs (bug 1689810), among other things, which may interfere with reproduction.

This RPM spec file should reproduce the issue with high likelihood.  On the first run, I get:

# objdump -j .gnu_debuglink -s usr/bin/example-1

usr/bin/example-1:     file format elf64-x86-64

Contents of section .gnu_debuglink:
 0000 6578616d 706c652d 31362d31 2d312e65  example-16-1-1.e
 0010 6c382e78 38365f36 342e6465 62756700  l8.x86_64.debug.
 0020 1636d37a                             .6.z

On the second run, its:

$ objdump -j .gnu_debuglink -s usr/bin/example-1

usr/bin/example-1:     file format elf64-x86-64

Contents of section .gnu_debuglink:
 0000 6578616d 706c652d 31352d31 2d312e65  example-15-1-1.e
 0010 6c382e78 38365f36 342e6465 62756700  l8.x86_64.debug.
 0020 1636d37a                             .6.z

I should have said that the reproducer is still file-system-specific, but I think it should work with XFS and ext4.  I tested it with XFS.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3584