Bug 2232086

Summary: .gdb_index sections are not reproducible
Product: [Fedora] Fedora Reporter: Zbigniew Jędrzejewski-Szmek <zbyszek>
Component: gdbAssignee: Alexandra Petlanová Hájková <ahajkova>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 38CC: aburgess, ahajkova, fweimer, guinevere, jan, keiths, kevinb, mcermak, mjw, mkolar
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: gdb-13.2-11.fc39 gdb-13.2-7.fc38 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-12-01 01:23:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
homectl-254.1-2.fc40.x86_64.debug from koji build
none
homectl-254.1-2.fc40.x86_64.debug from local mock build
none
build.log from local rebuild of blosc2-2.10.5-2.fc40 none

Description Zbigniew Jędrzejewski-Szmek 2023-08-15 08:44:10 UTC 
I'm trying to reproducibly rebuild rpms (see [1] for the background), and repeat builds are identical except for the .gnu_debuglink and .gdb_index sections [2].

.gnu_debuglink has the same filename, but the CRC hash differs. IIUC, this is because the .gdb_index section in the .debug file is different. If the latter is fixed, both issues will go away.

.gdb_index seems to have the same content, but just in different order (see [2]).
The invocations are the same, except for cpu count:
+ /usr/bin/find-debuginfo -j6 --strict-build-id -i --build-id-seed 3.21.0-8.fc39 --unique-debug-suffix -3.21.0-8.fc39.x86_64 --unique-debug-src-base valgrind-3.21.0-8.fc39.x86_64 --run-dwz --dwz-low-mem-die-limit 10000000 --dwz-max-die-limit 110000000 -S debugsourcefiles.list /builddir/build/BUILD/valgrind-3.21.0
+ /usr/bin/find-debuginfo -j8 --strict-build-id -i --build-id-seed 3.21.0-8.fc39 --unique-debug-suffix -3.21.0-8.fc39.x86_64 --unique-debug-src-base valgrind-3.21.0-8.fc39.x86_64 --run-dwz --dwz-low-mem-die-limit 10000000 --dwz-max-die-limit 110000000 -S debugsourcefiles.list /builddir/build/BUILD/valgrind-3.21.0

Please make the output stable so that repeat builds are identical.

I don't know anything about this process, so please redirect me to the right place if it's not debuginfo that is responsible for the contents of that section.

[1] https://discussion.fedoraproject.org/t/report-from-the-reproducible-builds-hackfest-during-flock-2023/87469
[2] https://pagure.io/fedora-reproducible-builds/project/issue/5
Comment 1 Zbigniew Jędrzejewski-Szmek 2023-08-15 09:07:13 UTC 
The issue only occurs in some cases. E.g. for valgrind, it's only the files under /usr/libexec/valgrind/,
and for some other packages I rebuilt, the issue does not occur at all.
Setting RPM_BUILD_NCPUS= doesn't seem to have an effect.
Comment 2 Zbigniew Jędrzejewski-Szmek 2023-08-15 15:38:50 UTC 
After looking at this some more, find-debuginfo calls gdb-add-index, which calls gdb.minimal:

$GDB --batch -nx -iex 'set auto-load no' \
    -iex 'set debuginfod enabled off' \
    -ex "file $file" -ex "save gdb-index $dwarf5 $dir"
$OBJCOPY --add-section $section="$index" \
	 --set-section-flags $section=readonly \
		 $(if $debugstradd; then \
	       echo --add-section .debug_str="$debugstrmerge"; \
	       echo --set-section-flags .debug_str=readonly; \
	   fi; \
	   if $debugstrupdate; then \
	       echo --update-section .debug_str="$debugstrmerge"; \
	   fi) \
	 "$fpath" "$fpath"

+ /usr/bin/gdb.minimal --batch -nx -iex 'set auto-load no' -iex 'set debuginfod enabled off' -ex 'file /builddir/build/BUILDROOT/valgrind-3.21.0-8.fc40.x86_64_5/usr/libexec/valgrind/massif-amd64-linux' -ex 'save gdb-index  /builddir/build/BUILDROOT/valgrind-3.21.0-8.fc40.x86_64_5/usr/libexec/valgrind'
+ objcopy --add-section .gdb_index=/builddir/build/BUILDROOT/valgrind-3.21.0-8.fc40.x86_64_5/usr/libexec/valgrind/massif-amd64-linux.gdb-index --set-section-flags .gdb_index=readonly /builddir/build/BUILDROOT/valgrind-3.21.0-8.fc40.x86_64_5/usr/libexec/valgrind/massif-amd64-linux /builddir/build/BUILDROOT/valgrind-3.21.0-8.fc40.x86_64_5/usr/libexec/valgrind/massif-amd64-linux

So it seems the contents of that section are generated by gdb.minimal and then
just copied over. So maybe this needs to be reassigned to gdb?
Comment 3 Mark Wielaard 2023-08-17 15:21:13 UTC 
Yes, gdb-add-index is part of gdb.

I don't immediately see anything that could cause non-deterministic output in the gdb-add-index script itself.
So it must be something in gdb itself that generates or saves the gdb-index data non-deterministically.

Do you happen to have an example invocation of gdb-add-index that produces different results on different runs?
Is it the contents of .gdb_index section or something else in the ELF file that is different?
Comment 4 Zbigniew Jędrzejewski-Szmek 2023-08-19 16:39:28 UTC 
> I don't immediately see anything that could cause non-deterministic output in the gdb-add-index script itself.
> So it must be something in gdb itself that generates or saves the gdb-index data non-deterministically.

Yes, that was my conclusion too.

> Do you happen to have an example invocation of gdb-add-index that produces different results on different runs?

No. I tried a few times and on the same machine, I get the same contents.
But when trying to rebuild packages from koji, the .gdb_index is (in the examples I have done so far)
the only thing that differs. Nevertheless, it only differs in some files, and not in all packages.
It looks like either there's some input that I'm not controlling (most likely some detail about
the machine). It'd be helpful to know what gdb uses for initialization of seeds and such.

(FWIW, I suspected locale differences, but it doesn't seems to be the case. For example,
C.UTF-8 and C sort capital letters differently, but the a) calling gdb-add-index with in a different
locale doesn't seem to change the result, and b) the differences I linked above are not consistent
with being caused by locale sorting.)
Comment 5 Zbigniew Jędrzejewski-Szmek 2023-08-20 12:57:53 UTC 
Another example: systemd-254.1-2.fc40 build.
The original package is at https://kojipkgs.fedoraproject.org//packages/systemd/254.1/2.fc40/x86_64/systemd-debugsource-254.1-2.fc40.x86_64.rpm.
I rebuild the srpm and get a second systemd-debugsource-254.1-2.fc40.x86_64.rpm.

$ (mkcd /var/tmp/extr1 && rpm2cpio orig/systemd-debuginfo-254.1-2.fc40.x86_64.rpm | cpio -i --make-dir) && (mkcd /var/tmp/extr2 && rpm2cpio rebuild/systemd-debuginfo-254.1-2.fc40.x86_64.rpm | cpio -i --make-dir) 
$ sha256sum /var/tmp/extr*/usr/lib/debug/usr/bin/homectl-254.1-2.fc40.x86_64.debug    
b00149e3158dc585e4d03af775f6ffe5799369ccae2fe1894f3127a4ea2e213f  /var/tmp/extr1/usr/lib/debug/usr/bin/homectl-254.1-2.fc40.x86_64.debug
cf59594bda406f21dcebd0b230c13f7938bc21f0f4f57e708cafebb661eac1aa  /var/tmp/extr2/usr/lib/debug/usr/bin/homectl-254.1-2.fc40.x86_64.debug

diffoscope says that those files only differ in the .gdb_index section.
The easiest way I could think of to confirm that it is correct:
$ cp /var/tmp/extr1/usr/lib/debug/usr/bin/homectl-254.1-2.fc40.x86_64.debug{,.mod}
$ objcopy --remove-section=.gdb_index /var/tmp/extr1/usr/lib/debug/usr/bin/homectl-254.1-2.fc40.x86_64.debug.mod
$ cp /var/tmp/extr2/usr/lib/debug/usr/bin/homectl-254.1-2.fc40.x86_64.debug{,.mod}
$ objcopy --remove-section=.gdb_index /var/tmp/extr2/usr/lib/debug/usr/bin/homectl-254.1-2.fc40.x86_64.debug.mod
$ sha256sum /var/tmp/extr*/usr/lib/debug/usr/bin/homectl-254.1-2.fc40.x86_64.debug.mod
d5920a0531cf92a24d1bc67a1af07e7803afbffe49bdff366b4ba27d34fd64ce  /var/tmp/extr1/usr/lib/debug/usr/bin/homectl-254.1-2.fc40.x86_64.debug.mod
d5920a0531cf92a24d1bc67a1af07e7803afbffe49bdff366b4ba27d34fd64ce  /var/tmp/extr2/usr/lib/debug/usr/bin/homectl-254.1-2.fc40.x86_64.debug.mod

And the sections themselves:
$ readelf --wide --debug-dump=gdb_index /var/tmp/extr1/usr/lib/debug/usr/bin/homectl-254.1-2.fc40.x86_64.debug >/var/tmp/dump1
$ readelf --wide --debug-dump=gdb_index /var/tmp/extr2/usr/lib/debug/usr/bin/homectl-254.1-2.fc40.x86_64.debug >/var/tmp/dump2

$ diff -U0 /var/tmp/dump*
--- /var/tmp/dump1	2023-08-20 14:42:06.514343768 +0200
+++ /var/tmp/dump2	2023-08-20 14:42:14.831380346 +0200
@@ -1 +1 @@
-Contents of the .gdb_index section (loaded from /var/tmp/extr1/usr/lib/debug/usr/bin/homectl-254.1-2.fc40.x86_64.debug):
+Contents of the .gdb_index section (loaded from /var/tmp/extr2/usr/lib/debug/usr/bin/homectl-254.1-2.fc40.x86_64.debug):
@@ -93 +93 @@
-[114] SPECIAL_GLYPH_ARROW_DOWN: 32 [static, variable]
+[114] signed char: 29 [static, type]
@@ -125 +125 @@
-[238] __rlimit_resource: 33 [static, type]
+[238] SpecialGlyph: 32 [static, type]
@@ -159 +159 @@
-[361] json_variant_strv: 27 [global, function]
+[361] __fsid_t: 33 [static, type]
@@ -231 +231 @@
-[561] __start_SYSTEMD_STATIC_DESTRUCT: 33 [global, variable]
+[561] arg_transport: 27 [static, variable]
@@ -259,3 +259 @@
-[637] array_cleanup:
-	28 [static, function]
-	33 [static, function]
+[637] SPECIAL_GLYPH_BLACK_CIRCLE: 32 [static, variable]
@@ -278 +276 @@
-[687] _TABLE_DATA_TYPE_INVALID: 33 [static, variable]
+[687] extract_first_word: 27 [global, function]
@@ -294 +291,0 @@
-[739] all_capabilities: 33 [static, function]
@@ -321 +318 @@
-[823] stdout: 32 [global, variable]
+[823] AUTO_RESIZE_OFF: 29 [static, variable]
@@ -338 +335 @@
-[859] ASK_PASSWORD_CONSOLE_COLOR: 33 [static, variable]
+[859] _AUTO_RESIZE_MODE_MAX: 29 [static, variable]
@@ -455 +452 @@
-[1178] JSON_FORMAT_COLOR: 29 [static, variable]
+[1178] __unique_prefix_static_destructor_wrapper25: 33 [static, function]
@@ -479 +476 @@
-[1237] FORK_NEW_MOUNTNS: 33 [static, variable]
+[1237] SPECIAL_GLYPH_ELLIPSIS: 32 [static, variable]
@@ -512 +509 @@
-[1373] SpecialGlyph: 32 [static, type]
+[1373] JSON_PARSE_SENSITIVE: 33 [static, variable]
@@ -515 +512 @@
-[1385] JSON_VARIANT_STRING: 30 [static, variable]
+[1385] json_variant_string: 27 [global, function]
@@ -545 +542 @@
-[1513] StaticDestructorType: 33 [static, type]
+[1513] USER_DIRECTORY: 29 [static, variable]
@@ -612,3 +609 @@
-[1738] json_variant_unref:
-	27 [global, function]
-	28 [global, function]
+[1738] SPECIAL_GLYPH_DARK_SHADE: 32 [static, variable]
@@ -643,3 +638 @@
-[1853] update_last_change:
-	27 [static, function]
-	33 [static, function]
+[1853] EXTRACT_RETAIN_SEPARATORS: 33 [static, variable]
@@ -653 +646 @@
-[1897] _AUTO_RESIZE_MODE_MAX: 29 [static, variable]
+[1897] ASK_PASSWORD_CONSOLE_COLOR: 33 [static, variable]
@@ -657 +650,3 @@
-[1917] SPECIAL_GLYPH_DARK_SHADE: 32 [static, variable]
+[1917] json_variant_unref:
+	27 [global, function]
+	28 [global, function]
@@ -686 +681,3 @@
-[1980] SPECIAL_GLYPH_EXTERNAL_LINK: 32 [static, variable]
+[1980] update_home:
+	27 [static, function]
+	33 [static, function]
@@ -689,0 +687 @@
+[1988] JSON_VARIANT_UNSIGNED: 30 [static, variable]
@@ -692,4 +689,0 @@
-[2013] errno_or_else:
-	30 [static, function]
-	31 [static, function]
-	32 [static, function]
@@ -708 +702 @@
-[2047] JSON_VARIANT_UNSIGNED: 30 [static, variable]
+[2047] DNS_LABEL_LEAVE_TRAILING_DOT: 29 [static, variable]
@@ -783 +777 @@
-[2341] __unique_prefix_static_destructor_wrapper25: 33 [static, function]
+[2341] JSON_FORMAT_COLOR: 29 [static, variable]
@@ -796 +790 @@
-[2388] json_variant_string: 27 [global, function]
+[2388] JSON_VARIANT_STRING: 30 [static, variable]
@@ -804 +798 @@
-[2402] __fsid_t: 33 [static, type]
+[2402] json_variant_strv: 27 [global, function]
@@ -813 +807 @@
-[2429] USER_CIFS: 29 [static, variable]
+[2429] __RLIMIT_NPROC: 33 [static, variable]
@@ -856 +850 @@
-[2526] AUTO_RESIZE_OFF: 29 [static, variable]
+[2526] stdout: 32 [global, variable]
@@ -878 +872,4 @@
-[2638] __RLIMIT_NPROC: 33 [static, variable]
+[2636] update_last_change:
+	27 [static, function]
+	33 [static, function]
+[2638] USER_CIFS: 29 [static, variable]
@@ -899 +896 @@
-[2737] signed char: 29 [static, type]
+[2737] SPECIAL_GLYPH_ARROW_DOWN: 32 [static, variable]
@@ -937,3 +934 @@
-[2873] update_home:
-	27 [static, function]
-	33 [static, function]
+[2873] SPECIAL_GLYPH_EXTERNAL_LINK: 32 [static, variable]
@@ -1005 +1000 @@
-[3081] CGROUP_CONTROLLER_DEVICES: 33 [static, variable]
+[3081] print_qrcode: 27 [global, function]
@@ -1013,0 +1009 @@
+[3123] _JSON_VARIANT_TYPE_MAX: 30 [static, variable]
@@ -1031 +1027 @@
-[3199] JSON_PARSE_SENSITIVE: 33 [static, variable]
+[3199] __rlimit_resource: 33 [static, type]
@@ -1041 +1037 @@
-[3249] _JSON_VARIANT_TYPE_MAX: 30 [static, variable]
+[3249] all_capabilities: 33 [static, function]
@@ -1050 +1046,3 @@
-[3274] SPECIAL_GLYPH_BLACK_CIRCLE: 32 [static, variable]
+[3274] array_cleanup:
+	28 [static, function]
+	33 [static, function]
@@ -1087 +1084,0 @@
-[3415] EXTRACT_RETAIN_SEPARATORS: 33 [static, variable]
@@ -1108 +1105 @@
-[3482] SPECIAL_GLYPH_ELLIPSIS: 32 [static, variable]
+[3482] FORK_NEW_MOUNTNS: 33 [static, variable]
@@ -1140 +1137 @@
-[3565] extract_first_word: 27 [global, function]
+[3565] _TABLE_DATA_TYPE_INVALID: 33 [static, variable]
@@ -1215 +1212 @@
-[3803] USER_DIRECTORY: 29 [static, variable]
+[3803] StaticDestructorType: 33 [static, type]
@@ -1218 +1215 @@
-[3812] print_qrcode: 27 [global, function]
+[3812] CGROUP_CONTROLLER_DEVICES: 33 [static, variable]
@@ -1253 +1250 @@
-[3995] arg_transport: 27 [static, variable]
+[3995] __start_SYSTEMD_STATIC_DESTRUCT: 33 [global, variable]
@@ -1276 +1273,4 @@
-[4078] DNS_LABEL_LEAVE_TRAILING_DOT: 29 [static, variable]
+[4078] errno_or_else:
+	30 [static, function]
+	31 [static, function]
+	32 [static, function]
@@ -1280 +1280 @@
-Contents of the .gdb_index section (loaded from /var/tmp/extr1/usr/lib/debug/usr/bin/../../.dwz/systemd-254.1-2.fc40.x86_64):
+Contents of the .gdb_index section (loaded from /var/tmp/extr2/usr/lib/debug/usr/bin/../../.dwz/systemd-254.1-2.fc40.x86_64):

I'll attach the two .debug files.
Comment 6 Zbigniew Jędrzejewski-Szmek 2023-08-20 12:59:18 UTC 
Created attachment 1984170 [details]
homectl-254.1-2.fc40.x86_64.debug from koji build
Comment 7 Zbigniew Jędrzejewski-Szmek 2023-08-20 13:02:06 UTC 
Created attachment 1984171 [details]
homectl-254.1-2.fc40.x86_64.debug from local mock build
Comment 8 Zbigniew Jędrzejewski-Szmek 2023-08-20 13:03:37 UTC 
Oh, and as mentioned, most of the .debug files are identical. 

$ md5sum /var/tmp/extr*/usr/lib/debug/usr/bin/*debug|sed 's|/var/tmp/extr.||' | sort | uniq -c
      2 020e7d87e1a174a10d48f3e680200180  /usr/lib/debug/usr/bin/busctl-254.1-2.fc40.x86_64.debug
      2 0b9e28db383c7aa562fdd4cc2094b2b4  /usr/lib/debug/usr/bin/systemd-stdio-bridge-254.1-2.fc40.x86_64.debug
      2 0e1fe0dddedb2075e7266bd8483d1e43  /usr/lib/debug/usr/bin/oomctl-254.1-2.fc40.x86_64.debug
      2 0f6ba1d7c1297443a6fcf5dfacef6b1d  /usr/lib/debug/usr/bin/systemd-detect-virt-254.1-2.fc40.x86_64.debug
      1 1e14bbd084995b0e13c019cd13d0d5ee  /usr/lib/debug/usr/bin/systemctl-254.1-2.fc40.x86_64.debug
      2 322f2a378264464ba941d599c12ab177  /usr/lib/debug/usr/bin/systemd-ask-password-254.1-2.fc40.x86_64.debug
      2 35b76789633bdd1f05aea5e9b7777ee2  /usr/lib/debug/usr/bin/systemd-notify-254.1-2.fc40.x86_64.debug
      2 36692e687fdb1fb429f1ca5506b595fe  /usr/lib/debug/usr/bin/systemd-inhibit-254.1-2.fc40.x86_64.debug
      2 37ccf65d5cb9ef0cc37bc44711364fd8  /usr/lib/debug/usr/bin/systemd-sysext-254.1-2.fc40.x86_64.debug
      2 3b60212a1a2f6f9546efadea5b98b20a  /usr/lib/debug/usr/bin/systemd-sysusers-254.1-2.fc40.x86_64.debug
      2 3b66ed8cfbd8777368b37b7f7d8a891b  /usr/lib/debug/usr/bin/hostnamectl-254.1-2.fc40.x86_64.debug
      2 3cdd5072b9ec8de9319edbdd17b300b4  /usr/lib/debug/usr/bin/systemd-cgtop-254.1-2.fc40.x86_64.debug
      2 4da9b47f2f734001877515d287c2e9f7  /usr/lib/debug/usr/bin/systemd-path-254.1-2.fc40.x86_64.debug
      1 5b708ff29116ccf113f7094726e79a3a  /usr/lib/debug/usr/bin/homectl-254.1-2.fc40.x86_64.debug
      2 6328ceb4e6c0749bd3d09c851b1f5193  /usr/lib/debug/usr/bin/systemd-escape-254.1-2.fc40.x86_64.debug
      2 647c2ba0812fe09286eb5f21501be8ff  /usr/lib/debug/usr/bin/portablectl-254.1-2.fc40.x86_64.debug
      2 76bcb66ffcb98ce7cc1d65186425a40c  /usr/lib/debug/usr/bin/journalctl-254.1-2.fc40.x86_64.debug
      2 809f733018fc3e87a1340845e57db4e6  /usr/lib/debug/usr/bin/systemd-ac-power-254.1-2.fc40.x86_64.debug
      2 889e281f61166502d2f5816d4f52dad2  /usr/lib/debug/usr/bin/localectl-254.1-2.fc40.x86_64.debug
      1 94a635c08b72b6473d0eab2217018efa  /usr/lib/debug/usr/bin/systemd-analyze-254.1-2.fc40.x86_64.debug
      2 a55f954e76bcd3f5169fa886e93188a9  /usr/lib/debug/usr/bin/systemd-socket-activate-254.1-2.fc40.x86_64.debug
      2 afd88151596d3011b32d4987d1715627  /usr/lib/debug/usr/bin/systemd-firstboot-254.1-2.fc40.x86_64.debug
      2 b0da1d36fbaffa618898d191b23ea7fc  /usr/lib/debug/usr/bin/loginctl-254.1-2.fc40.x86_64.debug
      2 b237a8d875895302a0450dedecc81af3  /usr/lib/debug/usr/bin/systemd-machine-id-setup-254.1-2.fc40.x86_64.debug
      1 b3cb416ff1ce169db254e22a31d57d0b  /usr/lib/debug/usr/bin/homectl-254.1-2.fc40.x86_64.debug
      2 b8e2e89a9c0f373589cee9771c9279c8  /usr/lib/debug/usr/bin/systemd-dissect-254.1-2.fc40.x86_64.debug
      1 bf5fc5594cf4b7d965f98dca78a09548  /usr/lib/debug/usr/bin/resolvectl-254.1-2.fc40.x86_64.debug
      1 c0b6c605ccb01cae8e965b322e88c9c1  /usr/lib/debug/usr/bin/systemd-analyze-254.1-2.fc40.x86_64.debug
      2 c3110cdded6182838e38fbd368c4c3bd  /usr/lib/debug/usr/bin/userdbctl-254.1-2.fc40.x86_64.debug
      2 c7f34b4fdc732435fcff706d776998a0  /usr/lib/debug/usr/bin/systemd-creds-254.1-2.fc40.x86_64.debug
      2 c8402412e1348da96282d9e59a55b256  /usr/lib/debug/usr/bin/systemd-delta-254.1-2.fc40.x86_64.debug
      1 cf03f0fde2858da39f61a53cd0cd3bce  /usr/lib/debug/usr/bin/resolvectl-254.1-2.fc40.x86_64.debug
      2 d773008c10d6db42250786769b53414b  /usr/lib/debug/usr/bin/systemd-cat-254.1-2.fc40.x86_64.debug
      2 dc85c2e10f5aacf345b4292b8537ca7c  /usr/lib/debug/usr/bin/systemd-run-254.1-2.fc40.x86_64.debug
      2 dd94a14657017453d5aad3990ee60311  /usr/lib/debug/usr/bin/timedatectl-254.1-2.fc40.x86_64.debug
      2 e79766eade4cca1fb0f32664e5ab807a  /usr/lib/debug/usr/bin/systemd-cgls-254.1-2.fc40.x86_64.debug
      2 f04958191a5b70c3ddc0dd6964022bd5  /usr/lib/debug/usr/bin/systemd-id128-254.1-2.fc40.x86_64.debug
      2 f427d3aad806dc24e8935756e9b15417  /usr/lib/debug/usr/bin/systemd-mount-254.1-2.fc40.x86_64.debug
      2 f58d586905011c81a6cc44024be6a5e0  /usr/lib/debug/usr/bin/systemd-tmpfiles-254.1-2.fc40.x86_64.debug
      1 f5b66824a126685e8e537b3e85039fc0  /usr/lib/debug/usr/bin/systemctl-254.1-2.fc40.x86_64.debug
      2 f88cf65b8c66cd348d16e6de0b87d81b  /usr/lib/debug/usr/bin/systemd-tty-ask-password-agent-254.1-2.fc40.x86_64.debug
Comment 9 Mark Wielaard 2023-10-25 08:51:48 UTC 
Could you attach/point to a koji build.log and a local mockbuild build.log for these builds?
Does the issue trigger only for a local build vs a remote koji build? Or can you trigger it for two local builds (or two remote builds) too?
Comment 10 Mark Wielaard 2023-10-25 09:00:31 UTC 
Also, what is the smallest package you are seeing this issue for?
Having a package with just one or two binaries would really help replicate/investigate this issue.
Comment 11 Mark Wielaard 2023-10-25 09:06:19 UTC 
(In reply to Mark Wielaard from comment #9)
> Does the issue trigger only for a local build vs a remote koji build? Or can
> you trigger it for two local builds (or two remote builds) too?

Sorry, you answered that in comment #4 already:

> No. I tried a few times and on the same machine, I get the same contents.

So it is some property of the build machine, maybe size of memory?
Comment 12 Zbigniew Jędrzejewski-Szmek 2023-10-25 16:38:20 UTC 
(In reply to Mark Wielaard from comment #9)
> Could you attach/point to a koji build.log and a local mockbuild build.log
> for these builds?

I think I lost the build logs for the two examples I quoted above.
But I was looking for a simpler package, and blosc2 is fairly simple and also shows the issue:

blosc2-2.10.5-2.fc40:
koji build: https://koji.fedoraproject.org/koji/buildinfo?buildID=2311591
build log: https://kojipkgs.fedoraproject.org//packages/blosc2/2.10.5/2.fc40/data/logs/x86_64/build.log

I'll attach a local build log in a moment.

The diff:

├── content
│ ├── ./usr/lib/debug/usr/lib64/libblosc2.so.2.10.5-2.10.5-2.fc40.x86_64.debug
│ │┄ File has been modified after NT_GNU_BUILD_ID has been applied.
│ │ ├── readelf --wide --debug-dump=gdb_index {}
│ │ │ @@ -301,15 +301,17 @@
│ │ │   198 [global, function]
│ │ │   199 [global, function]
│ │ │  [142] BLOSC2_IO_FILESYSTEM: 247 [static, variable]
│ │ │  [144] zfp_field_is_contiguous: 239 [global, function]
│ │ │  [155] blosc2_frame_get_offsets:
│ │ │   198 [global, function]
│ │ │   251 [global, function]
│ │ │ -[166] zfp_decode_partial_block_strided_int32_4.isra.0: 201 [global, function]
│ │ │ +[166] compress_strided_int32_1:
│ │ │ + 196 [static, function]
│ │ │ + 239 [static, function]
│ │ │  [186] _mm_unpacklo_epi64:
│ │ │   263 [global, function]
│ │ │   264 [global, function]
│ │ │  [191] is_little_endian:
│ │ │   247 [static, function]
│ │ │   250 [static, function]
│ │ │   251 [static, function]
...

> Does the issue trigger only for a local build vs a remote koji build? Or can
> you trigger it for two local builds (or two remote builds) too?

So far only local-vs-remote. I tried "local" builds in a VM and the result was the same as on the host.
For remote builds, koji generally doesn't allow the same nevra to be built twice in koji as official builds, so we get different $SOURCE_EPOCH_DATE, and hence different results.
I started a scratch build of blosc2 now, I'll report when it's done.

> Also, what is the smallest package you are seeing this issue for?

I tried with some "hello world" package, and it didn't reproduce the issue. But as I said earlier, only some files exhibibit the issue. E.g. in the systemd case, it was maybe ~10% of files, so I guess that it's possible that simple packages are also affected, but it's just harder to hit the problem when there is few files in the package.
Comment 13 Zbigniew Jędrzejewski-Szmek 2023-10-25 16:41:24 UTC 
Created attachment 1995436 [details]
build.log from local rebuild of blosc2-2.10.5-2.fc40
Comment 14 Zbigniew Jędrzejewski-Szmek 2023-10-25 21:33:17 UTC 
scratch build of blosc2-2.10.5-2.fc40:
https://koji.fedoraproject.org/koji/taskinfo?taskID=108094630
The result is different from both of the two previous builds (official koji and local rebuild).

── content
│ ├── ./usr/lib/debug/usr/lib64/libblosc2.so.2.10.5-2.10.5-2.fc40.x86_64.debug
│ │┄ File has been modified after NT_GNU_BUILD_ID has been applied.
│ │ ├── readelf --wide --debug-dump=gdb_index {}
│ │ │ @@ -376,15 +376,15 @@
│ │ │  [601] ZSTD_CCtx: 247 [static, type]
│ │ │  [609] uint8: 239 [static, type]
│ │ │  [610] zfp_stream_set_bit_stream: 239 [global, function]
│ │ │  [611] __pthread_cond_s: 205 [static, type]
│ │ │  [625] blosc2_set_delta:
│ │ │   197 [global, function]
│ │ │   247 [global, function]
│ │ │ -[627] rev_decode_block_float_3.isra.0: 201 [static, function]
│ │ │ +[627] zfp_stream_set_accuracy.isra.0: 200 [global, function]
...

I now also rebuilt the package locally 5 times and I always get the same hash.
Comment 15 Alexandra Petlanová Hájková 2023-10-30 21:26:34 UTC 
What do the debuginfo package look like for the packages that differ in .gdb_index section? Do they also differ?
Comment 16 Zbigniew Jędrzejewski-Szmek 2023-11-02 13:40:53 UTC 
The .gdb_index section is in the debuginfo package. The packages are identical except for this section.
Comment 17 Andrew Burgess 2023-11-28 10:32:52 UTC 
There is now a fix for this in upstream GDB.  We likely need to backport these commits:

* 3644f41dc80 gdb: generate dwarf-5 index identically as worker-thread count changes
* aff250145af gdb: generate gdb-index identically regardless of work thread count
* acc117b57f7 gdb: C++-ify mapped_symtab from dwarf2/index-write.c
* aa19bc1d259 gdb: reduce size of generated gdb-index file
* 1f0fab7ff86 gdb/testsuite: small refactor in selftest-support.exp

I'll look into doing this shortly.
Comment 18 Andrew Burgess 2023-11-28 16:48:14 UTC 
I created pull requests https://src.fedoraproject.org/rpms/gdb/pull-request/107 , https://src.fedoraproject.org/rpms/gdb/pull-request/108, and https://src.fedoraproject.org/rpms/gdb/pull-request/109 to merge the required patches into f38, f39, and rawhide.
Comment 19 Fedora Update System 2023-11-28 20:27:33 UTC 
FEDORA-2023-a4c8643cc8 has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-a4c8643cc8
Comment 20 Fedora Update System 2023-11-28 20:31:14 UTC 
FEDORA-2023-b1bd0758d0 has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2023-b1bd0758d0
Comment 21 Fedora Update System 2023-11-29 02:42:27 UTC 
FEDORA-2023-a4c8643cc8 has been pushed to the Fedora 38 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-a4c8643cc8`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-a4c8643cc8

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Comment 22 Fedora Update System 2023-11-29 03:02:50 UTC 
FEDORA-2023-b1bd0758d0 has been pushed to the Fedora 39 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-b1bd0758d0`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-b1bd0758d0

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Comment 23 Fedora Update System 2023-12-01 01:23:05 UTC 
FEDORA-2023-b1bd0758d0 has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.
Comment 24 Fedora Update System 2023-12-14 01:52:03 UTC 
FEDORA-2023-a4c8643cc8 has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.
Comment 25 Zbigniew Jędrzejewski-Szmek 2024-02-23 14:13:33 UTC 
I'm now testing some rebuilds and they all seem to reproduce exactly.
Thank you for the fix!