Bug 2232086

Summary: .gdb_index sections are not reproducible
Product: [Fedora] Fedora Reporter: Zbigniew Jędrzejewski-Szmek <zbyszek>
Component: gdbAssignee: Kevin Buettner <kevinb>
Status: NEW --- QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 38CC: ahajkova, blarsen, fweimer, jan, keiths, kevinb, mcermak, mjw, mkolar
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Zbigniew Jędrzejewski-Szmek 2023-08-15 08:44:10 UTC 
I'm trying to reproducibly rebuild rpms (see [1] for the background), and repeat builds are identical except for the .gnu_debuglink and .gdb_index sections [2].

.gnu_debuglink has the same filename, but the CRC hash differs. IIUC, this is because the .gdb_index section in the .debug file is different. If the latter is fixed, both issues will go away.

.gdb_index seems to have the same content, but just in different order (see [2]).
The invocations are the same, except for cpu count:
+ /usr/bin/find-debuginfo -j6 --strict-build-id -i --build-id-seed 3.21.0-8.fc39 --unique-debug-suffix -3.21.0-8.fc39.x86_64 --unique-debug-src-base valgrind-3.21.0-8.fc39.x86_64 --run-dwz --dwz-low-mem-die-limit 10000000 --dwz-max-die-limit 110000000 -S debugsourcefiles.list /builddir/build/BUILD/valgrind-3.21.0
+ /usr/bin/find-debuginfo -j8 --strict-build-id -i --build-id-seed 3.21.0-8.fc39 --unique-debug-suffix -3.21.0-8.fc39.x86_64 --unique-debug-src-base valgrind-3.21.0-8.fc39.x86_64 --run-dwz --dwz-low-mem-die-limit 10000000 --dwz-max-die-limit 110000000 -S debugsourcefiles.list /builddir/build/BUILD/valgrind-3.21.0

Please make the output stable so that repeat builds are identical.

I don't know anything about this process, so please redirect me to the right place if it's not debuginfo that is responsible for the contents of that section.

[1] https://discussion.fedoraproject.org/t/report-from-the-reproducible-builds-hackfest-during-flock-2023/87469
[2] https://pagure.io/fedora-reproducible-builds/project/issue/5
Comment 1 Zbigniew Jędrzejewski-Szmek 2023-08-15 09:07:13 UTC 
The issue only occurs in some cases. E.g. for valgrind, it's only the files under /usr/libexec/valgrind/,
and for some other packages I rebuilt, the issue does not occur at all.
Setting RPM_BUILD_NCPUS= doesn't seem to have an effect.
Comment 2 Zbigniew Jędrzejewski-Szmek 2023-08-15 15:38:50 UTC 
After looking at this some more, find-debuginfo calls gdb-add-index, which calls gdb.minimal:

$GDB --batch -nx -iex 'set auto-load no' \
    -iex 'set debuginfod enabled off' \
    -ex "file $file" -ex "save gdb-index $dwarf5 $dir"
$OBJCOPY --add-section $section="$index" \
	 --set-section-flags $section=readonly \
		 $(if $debugstradd; then \
	       echo --add-section .debug_str="$debugstrmerge"; \
	       echo --set-section-flags .debug_str=readonly; \
	   fi; \
	   if $debugstrupdate; then \
	       echo --update-section .debug_str="$debugstrmerge"; \
	   fi) \
	 "$fpath" "$fpath"

+ /usr/bin/gdb.minimal --batch -nx -iex 'set auto-load no' -iex 'set debuginfod enabled off' -ex 'file /builddir/build/BUILDROOT/valgrind-3.21.0-8.fc40.x86_64_5/usr/libexec/valgrind/massif-amd64-linux' -ex 'save gdb-index  /builddir/build/BUILDROOT/valgrind-3.21.0-8.fc40.x86_64_5/usr/libexec/valgrind'
+ objcopy --add-section .gdb_index=/builddir/build/BUILDROOT/valgrind-3.21.0-8.fc40.x86_64_5/usr/libexec/valgrind/massif-amd64-linux.gdb-index --set-section-flags .gdb_index=readonly /builddir/build/BUILDROOT/valgrind-3.21.0-8.fc40.x86_64_5/usr/libexec/valgrind/massif-amd64-linux /builddir/build/BUILDROOT/valgrind-3.21.0-8.fc40.x86_64_5/usr/libexec/valgrind/massif-amd64-linux

So it seems the contents of that section are generated by gdb.minimal and then
just copied over. So maybe this needs to be reassigned to gdb?
Comment 3 Mark Wielaard 2023-08-17 15:21:13 UTC 
Yes, gdb-add-index is part of gdb.

I don't immediately see anything that could cause non-deterministic output in the gdb-add-index script itself.
So it must be something in gdb itself that generates or saves the gdb-index data non-deterministically.

Do you happen to have an example invocation of gdb-add-index that produces different results on different runs?
Is it the contents of .gdb_index section or something else in the ELF file that is different?