Bug 2390105 - Massively increased debuginfo generation time with debugedit 5.2
Summary: Massively increased debuginfo generation time with debugedit 5.2
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: debugedit
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Mark Wielaard
QA Contact:
URL:
Whiteboard: AcceptedFreezeException
Depends On:
Blocks: 2383272 F43BetaFreezeException
TreeView+ depends on / blocked
 
Reported: 2025-08-21 15:09 UTC by Orion Poplawski
Modified: 2025-09-08 14:34 UTC (History)
8 users (show)

Fixed In Version: debugedit-5.2-3.fc43
Clone Of:
Environment:
Last Closed: 2025-09-07 12:29:18 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Orion Poplawski 2025-08-21 15:09:52 UTC
Description of problem:

koji build times for gdal went from about 1 hour to about 24 hours with the change of debugedit to 5.2 (among other things - so I'm not 100% sure it's this, but it's a good place to start):

https://koschei.fedoraproject.org/package/gdal?last_seen_ts=1752888173&collection=f44

the build spends most of its time here:

find-debuginfo: starting
Extracting debug info from 43 files

Version-Release number of selected component (if applicable):
debugedit 5.2-1

Comment 1 Frank Ch. Eigler 2025-08-22 00:38:44 UTC
NB: this is not a bug, it is the cost of a new capability.  It can already be turned off with a command line option to find-debuginfo.sh (--no-ar-files), or via a spec directive in the rpm spec file.  However, see https://sourceware.org/PR33305 for hypothetical future improvements.

Comment 2 Orion Poplawski 2025-08-22 02:38:19 UTC
I'm not sure that:

%undefine _preserve_static_debuginfo

is working.  I've submitted a new gdal build with that and it is still stuck in Extacting debug info: https://koji.fedoraproject.org/koji/taskinfo?taskID=136315181

I guess I'll try:

%define _find_debuginfo_opts --no-ar-files

Comment 3 Orion Poplawski 2025-08-22 03:10:56 UTC
It works a lot better with that.  Not particularly surprising since find-debuginfo runs before brp-strip-static-archive:

find-debuginfo: done
+ /usr/lib/rpm/check-buildroot
+ /usr/lib/jvm/java-25-openjdk/bin/java -cp /usr/share/java/xmvn-generator.jar org.fedoraproject.xmvn.generator.callback.CallbackEntry /tmp/xmvngen-1880179414245525861/socket
+ /usr/lib/rpm/redhat/brp-ldconfig
+ /usr/lib/rpm/brp-compress
+ /usr/lib/rpm/redhat/brp-strip-lto /usr/bin/strip
+ /usr/lib/rpm/brp-strip-static-archive /usr/bin/strip

Is the %undefine expected to work?  That is all the advice I saw on the mailing list thread.  Should one have to do both items?

Comment 4 Jens Petersen 2025-08-22 17:12:20 UTC
Sorry but calling this "notabug" seems unacceptable to me - this is a very serious problem and regression in behavior.

I think ghc build times have doubled probably because of this new feature :-(

Comment 5 Jens Petersen 2025-08-22 17:20:30 UTC
I wanted to add that I already had to kill find-debuginfo on my laptop recently a couple of times to keep it under control.

Comment 6 Mark Wielaard 2025-08-22 22:27:35 UTC
(In reply to Jens Petersen from comment #4)
> Sorry but calling this "notabug" seems unacceptable to me - this is a very
> serious problem and regression in behavior.

There is a workaround, so I think unacceptable is a little strong.
But I agree we should keep this bug open till we got debugedit/find-debuginfo do the right thing by default in fedora.
We have some ideas how to detect "problematic" ar automatically and just skip them.

> I think ghc build times have doubled probably because of this new feature :-(

Do you happen to have before/after koji builds that show the impact?

Comment 7 Mark Wielaard 2025-08-22 23:48:22 UTC
(In reply to Orion Poplawski from comment #3)
> It works a lot better with that.  Not particularly surprising since
> find-debuginfo runs before brp-strip-static-archive:
> 
> find-debuginfo: done
> + /usr/lib/rpm/check-buildroot
> + /usr/lib/jvm/java-25-openjdk/bin/java -cp
> /usr/share/java/xmvn-generator.jar
> org.fedoraproject.xmvn.generator.callback.CallbackEntry
> /tmp/xmvngen-1880179414245525861/socket
> + /usr/lib/rpm/redhat/brp-ldconfig
> + /usr/lib/rpm/brp-compress
> + /usr/lib/rpm/redhat/brp-strip-lto /usr/bin/strip
> + /usr/lib/rpm/brp-strip-static-archive /usr/bin/strip
> 
> Is the %undefine expected to work?  That is all the advice I saw on the
> mailing list thread.  Should one have to do both items?

Yes, sorry, you should do both:

%undefine _preserve_static_debuginfo
%define _find_debuginfo_opts --no-ar-files

The first makes sure all debuginfo will be removed from ar archive members.
The second makes find-debuginfo not process ar files (so it doesn't unpack, rewrite and repack the ar member objects)

Comment 8 Jens Petersen 2025-08-23 05:49:40 UTC
(In reply to Mark Wielaard from comment #6)
> (In reply to Jens Petersen from comment #4)
> But I agree we should keep this bug open till we got
> debugedit/find-debuginfo do the right thing by default in fedora.
> We have some ideas how to detect "problematic" ar automatically and just
> skip them.

Thank you for reopening, Mark, and I am glad you are looking into the problem.

> > I think ghc build times have doubled probably because of this new feature :-(
> 
> Do you happen to have before/after koji builds that show the impact?

GHC takes several hours to build in koji so testing this is non-trivial,
but here are a few samples using koji-tool output.
To be fair, I tried to exclude the mass rebuilds since their times tend to be more skewed anyway.

# Example 1

$ koji-tool builds -p ghc9.8*fc4* -d -s complete -u petersen -l3

ghc9.8-9.8.4-16.fc43 BuildComplete (petersen)
https://koji.fedoraproject.org/koji/buildinfo?buildID=2789225
https://koji.fedoraproject.org/koji/taskinfo?taskID=135931706
start:      Mon Aug 11 17:57:08 +08 2025
completion: Mon Aug 11 22:19:23 +08 2025
duration: 4 hours 22 min
https://kojipkgs.fedoraproject.org/packages/ghc9.8/9.8.4/16.fc43

ghc9.8-9.8.4-14.fc42 BuildComplete (petersen)
https://koji.fedoraproject.org/koji/buildinfo?buildID=2633719
https://koji.fedoraproject.org/koji/taskinfo?taskID=128090259
start:      Sun Jan 19 02:17:20 +08 2025
completion: Sun Jan 19 04:41:58 +08 2025
duration: 2 hours 24 min
https://kojipkgs.fedoraproject.org/packages/ghc9.8/9.8.4/14.fc42

# Example 2

$ koji-tool builds -p ghc9.10*fc4* -d -s complete -l3

ghc9.10-9.10.2-10.fc43 BuildComplete (releng)
https://koji.fedoraproject.org/koji/buildinfo?buildID=2756513
https://koji.fedoraproject.org/koji/taskinfo?taskID=135171252
start:      Thu Jul 24 12:58:12 +08 2025
completion: Thu Jul 24 20:10:19 +08 2025
duration: 7 hours 12 min
https://kojipkgs.fedoraproject.org/packages/ghc9.10/9.10.2/10.fc43
:
ghc9.10-9.10.2-9.fc43 BuildComplete (petersen)
https://koji.fedoraproject.org/koji/buildinfo?buildID=2740364
https://koji.fedoraproject.org/koji/taskinfo?taskID=134548824
start:      Sun Jun 29 20:49:46 +08 2025
completion: Sun Jun 29 23:07:39 +08 2025
duration: 2 hours 17 min
https://kojipkgs.fedoraproject.org/packages/ghc9.10/9.10.2/9.fc43

# Example 3

$ koji-tool builds -p ghc9.12*fc4* -d -s complete -u petersen -l3

ghc9.12-9.12.2-8.fc43 BuildComplete (petersen)
https://koji.fedoraproject.org/koji/buildinfo?buildID=2756515
https://koji.fedoraproject.org/koji/taskinfo?taskID=135786391
start:      Thu Aug  7 10:57:30 +08 2025
completion: Thu Aug  7 20:21:48 +08 2025
duration: 9 hours 24 min
https://kojipkgs.fedoraproject.org/packages/ghc9.12/9.12.2/8.fc43
:
ghc9.12-9.12.2-7.fc43 BuildComplete (petersen)
https://koji.fedoraproject.org/koji/buildinfo?buildID=2716687
https://koji.fedoraproject.org/koji/taskinfo?taskID=132968756
start:      Mon May 19 21:34:48 +08 2025
completion: Tue May 20 04:02:02 +08 2025
duration: 6 hours 27 min
https://kojipkgs.fedoraproject.org/packages/ghc9.12/9.12.2/7.fc43

# Example 4

$ koji-tool builds -p ghc9.4*fc4* -d -s complete -u petersen -l3

ghc9.4-9.4.8-36.fc43 BuildComplete (petersen)
https://koji.fedoraproject.org/koji/buildinfo?buildID=2803913
https://koji.fedoraproject.org/koji/taskinfo?taskID=136259713
start:      Wed Aug 20 14:57:06 +08 2025
completion: Wed Aug 20 19:03:23 +08 2025
duration: 4 hours 6 min
https://kojipkgs.fedoraproject.org/packages/ghc9.4/9.4.8/36.fc43

ghc9.4-9.4.8-36.fc44 BuildComplete (petersen)
https://koji.fedoraproject.org/koji/buildinfo?buildID=2803670
https://koji.fedoraproject.org/koji/taskinfo?taskID=136240701
start:      Wed Aug 20 00:55:44 +08 2025
completion: Wed Aug 20 06:15:11 +08 2025
duration: 5 hours 19 min
https://kojipkgs.fedoraproject.org/packages/ghc9.4/9.4.8/36.fc44

ghc9.4-9.4.8-33.fc42 BuildComplete (petersen)
https://koji.fedoraproject.org/koji/buildinfo?buildID=2632270
https://koji.fedoraproject.org/koji/taskinfo?taskID=128213723
start:      Mon Jan 20 23:39:24 +08 2025
completion: Tue Jan 21 02:33:09 +08 2025
duration: 2 hours 53 min
https://kojipkgs.fedoraproject.org/packages/ghc9.4/9.4.8/33.fc42

# Example 5

$ koji-tool builds -p ghc9.2*fc4* -d -s complete -u petersen -l3

ghc9.2-9.2.8-30.fc43 BuildComplete (petersen)
https://koji.fedoraproject.org/koji/buildinfo?buildID=2804150
https://koji.fedoraproject.org/koji/taskinfo?taskID=136275561
start:      Thu Aug 21 02:24:49 +08 2025
completion: Thu Aug 21 06:08:28 +08 2025
duration: 3 hours 43 min
https://kojipkgs.fedoraproject.org/packages/ghc9.2/9.2.8/30.fc43

ghc9.2-9.2.8-30.fc44 BuildComplete (petersen)
https://koji.fedoraproject.org/koji/buildinfo?buildID=2804084
https://koji.fedoraproject.org/koji/taskinfo?taskID=136269867
start:      Wed Aug 20 21:48:53 +08 2025
completion: Thu Aug 21 01:49:13 +08 2025
duration: 4 hours 0 min
https://kojipkgs.fedoraproject.org/packages/ghc9.2/9.2.8/30.fc44

ghc9.2-9.2.8-28.fc42 BuildComplete (petersen)
https://koji.fedoraproject.org/koji/buildinfo?buildID=2642095
https://koji.fedoraproject.org/koji/taskinfo?taskID=128240927
start:      Tue Jan 21 16:51:24 +08 2025
completion: Tue Jan 21 19:09:35 +08 2025
duration: 2 hours 18 min
https://kojipkgs.fedoraproject.org/packages/ghc9.2/9.2.8/28.fc42

----

So it is probably more accurate to say that it adds around 1-3 hours to ghc builds.
I would need to dig more into the individual archs to get more detailed stats.
But watching the build.log's one can see they are stuck in the find-debuginfo for long periods.

Comment 9 Jens Petersen 2025-08-23 09:56:41 UTC
I tried to test a bit more locally with ghc9.10, using "rpmbuild -bi --short-circuit".
(I am actually building with ghc-9.10.3 RC4, but I think the results should be same with the current ghc9.10-9.10.2 in Fedora.)

It seems that the workaround
  %define _find_debuginfo_opts --no-ar-files
is sufficient to avoid the problem for ghc builds.
(A normal full build takes about 37min locally for me using 4 of my cores.)

(The reason I had to kill find-debuginfo manually is that I build via another toolbox container.
Also although I have set "%_smp_ncpus_max 4" nevertheless find-debuginfo gets -j16
(ie dd (& ar) processes take most of my cores) unless I also set "%_smp_build_ncpus 4".
Anyway I was planning to open a separate bug about that.)

It looks like this towards the end, when there are only 2 dd's left:

 597365 ?        Ssl+   0:00      \_ fbrnch local -s
 597962 ?        S+     0:00          \_ sh -c rpmbuild --define '_sourcedir /var/home/petersen/fedora/haskell/ghc9.10' --define 'dist %{?distprefix}.fc44' -bi --short-circuit ghc9.10.spec |& tee .build-9.10.2.20250817-10.3.fc44.log && exit ${PIPESTATUS[0]}
 597963 ?        S+     0:00              \_ /bin/sh /var/home/petersen/bin/rpmbuild --define _sourcedir /var/home/petersen/fedora/haskell/ghc9.10 --define dist %{?distprefix}.fc44 -bi --short-circuit ghc9.10.spec
 597965 ?        S+     0:00              |   \_ /usr/bin/rpmbuild --define _sourcedir /var/home/petersen/fedora/haskell/ghc9.10 --define dist %{?distprefix}.fc44 -bi --short-circuit ghc9.10.spec
 598158 ?        S+     0:00              |       \_ /bin/sh -e /var/tmp/rpm-tmp.HArGVk
 651253 ?        S+     0:00              |           \_ /usr/bin/bash /usr/bin/find-debuginfo -j16 --strict-build-id -m -i --build-id-seed 9.10.2.20250817-10.3.fc44 --unique-debug-suffix -9.10.2.20250817-10.3.fc44.x86_64 --unique-debug-src-base ghc9.10-9.10.2.20250817-10.3.fc44.x86_64 --run-dwz --dwz-low-mem-die-limit 10000000 --dwz-max-die-limit 110000000 -S debugsourcefiles.list /var/home/petersen/fedora/haskell/ghc9.10/BUILD/ghc9.10-9.10.2.20250817-build/ghc-9.10.2.20250817
 651278 ?        S+     0:00              |               \_ /usr/bin/bash /usr/bin/find-debuginfo -j16 --strict-build-id -m -i --build-id-seed 9.10.2.20250817-10.3.fc44 --unique-debug-suffix -9.10.2.20250817-10.3.fc44.x86_64 --unique-debug-src-base ghc9.10-9.10.2.20250817-10.3.fc44.x86_64 --run-dwz --dwz-low-mem-die-limit 10000000 --dwz-max-die-limit 110000000 -S debugsourcefiles.list /var/home/petersen/fedora/haskell/ghc9.10/BUILD/ghc9.10-9.10.2.20250817-build/ghc-9.10.2.20250817
 651279 ?        S+     0:00              |                   \_ /usr/bin/bash /usr/bin/find-debuginfo -j16 --strict-build-id -m -i --build-id-seed 9.10.2.20250817-10.3.fc44 --unique-debug-suffix -9.10.2.20250817-10.3.fc44.x86_64 --unique-debug-src-base ghc9.10-9.10.2.20250817-10.3.fc44.x86_64 --run-dwz --dwz-low-mem-die-limit 10000000 --dwz-max-die-limit 110000000 -S debugsourcefiles.list /var/home/petersen/fedora/haskell/ghc9.10/BUILD/ghc9.10-9.10.2.20250817-build/ghc-9.10.2.20250817
 654722 ?        S+     0:00              |                   |   \_ /usr/bin/bash /usr/bin/find-debuginfo -j16 --strict-build-id -m -i --build-id-seed 9.10.2.20250817-10.3.fc44 --unique-debug-suffix -9.10.2.20250817-10.3.fc44.x86_64 --unique-debug-src-base ghc9.10-9.10.2.20250817-10.3.fc44.x86_64 --run-dwz --dwz-low-mem-die-limit 10000000 --dwz-max-die-limit 110000000 -S debugsourcefiles.list /var/home/petersen/fedora/haskell/ghc9.10/BUILD/ghc9.10-9.10.2.20250817-build/ghc-9.10.2.20250817
 700955 ?        R+     0:02              |                   |       \_ dd status=none if=/var/home/petersen/fedora/haskell/ghc9.10/BUILD/ghc9.10-9.10.2.20250817-build/BUILDROOT/usr/lib64/ghc-9.10.2.20250817/lib/x86_64-linux-ghc-9.10.2.20250817/ghc-9.10.2.20250817-188a/libHSghc-9.10.2.20250817-188a_p.a of=Make.p_o bs=1 skip=154362956 count=3632520
 651292 ?        S+     0:00              |                   \_ /usr/bin/bash /usr/bin/find-debuginfo -j16 --strict-build-id -m -i --build-id-seed 9.10.2.20250817-10.3.fc44 --unique-debug-suffix -9.10.2.20250817-10.3.fc44.x86_64 --unique-debug-src-base ghc9.10-9.10.2.20250817-10.3.fc44.x86_64 --run-dwz --dwz-low-mem-die-limit 10000000 --dwz-max-die-limit 110000000 -S debugsourcefiles.list /var/home/petersen/fedora/haskell/ghc9.10/BUILD/ghc9.10-9.10.2.20250817-build/ghc-9.10.2.20250817
 654031 ?        S+     0:00              |                       \_ /usr/bin/bash /usr/bin/find-debuginfo -j16 --strict-build-id -m -i --build-id-seed 9.10.2.20250817-10.3.fc44 --unique-debug-suffix -9.10.2.20250817-10.3.fc44.x86_64 --unique-debug-src-base ghc9.10-9.10.2.20250817-10.3.fc44.x86_64 --run-dwz --dwz-low-mem-die-limit 10000000 --dwz-max-die-limit 110000000 -S debugsourcefiles.list /var/home/petersen/fedora/haskell/ghc9.10/BUILD/ghc9.10-9.10.2.20250817-build/ghc-9.10.2.20250817
 700976 ?        R+     0:00              |                           \_ dd status=none if=/var/home/petersen/fedora/haskell/ghc9.10/BUILD/ghc9.10-9.10.2.20250817-build/BUILDROOT/usr/lib64/ghc-9.10.2.20250817/lib/x86_64-linux-ghc-9.10.2.20250817/ghc-9.10.2.20250817-188a/libHSghc-9.10.2.20250817-188a.a of=Usage.o bs=1 skip=156197708 count=131664
 597964 ?        S+     0:00              \_ tee .build-9.10.2.20250817-10.3.fc44.log

I think those are indeed the two largest libraries in ghc.


On my local machine (Thinkpad P16v gen1 AMD 16 cores + 64GB), the warmed-up rpmbuild --short-circuit with "_smp_build_ncpus 4" takes:

- With the workarounds: 2m19s  (mostly rpath and subpkg files checking)
- Without workarounds: 41m30s

So the find-debuginfo took roughly 40min of time (and this isn't exactly a sluggish laptop),
which is longer than the actual package build itself (see above).

Comment 10 Nikita Popov 2025-08-27 07:49:52 UTC
Why does redhat-rpm-config not automatically set --no-ar-files if _preserve_static_debuginfo is not defined?

Comment 11 Nikita Popov 2025-08-27 10:33:13 UTC
I filed https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/344 to automatically set the option.

Comment 12 Nikita Popov 2025-08-27 12:24:59 UTC
For people with cross-distro spec files, you need to use:

%if 0%{?fedora} >= 43
%define _find_debuginfo_opts --no-ar-files
%endif

As the option is not supported on older Fedora versions or RHEL and will result in a build error.

Comment 13 Mark Wielaard 2025-08-27 14:29:45 UTC
Two observations that could reduce the work that is done a lot:

- For gdal (and other packages that contain some mingw builds) there is a large *dll.a that contains thousands of members, but none of them is an ELF object, so we don't have to process them at all.

- For ghc there are lots of .a archives which do contain ELF objects, but only a handful debug.a archives contain objects with debug sections. We don't have to process an archive that contains only .o files without .debug sections.

I created patches for elfutils eu-elfclassify to detect these archives:
https://code.wildebeest.org/git/user/mjw/elfutils/log/?h=classify-ar

It introduces two options, one --ar-member to classify "inside" an ar file, and one --has-debug-sections that is like --unstripped but doesn't consider .symtab to be a debug section.

Comment 14 Jens Petersen 2025-08-28 11:21:53 UTC
Thanks

I believe llvm is also quite affected by this: the main package just opted out so far...

Comment 15 Jens Petersen 2025-08-28 11:23:01 UTC
(In reply to Nikita Popov from comment #12)
> %if 0%{?fedora} >= 43
> %define _find_debuginfo_opts --no-ar-files
> %endif

+1 I noticed this too with ghc9.14

Comment 16 Jens Petersen 2025-08-28 11:24:10 UTC
Also just want to point out that we are already in F43 Beta freeze now.

Comment 17 Mark Wielaard 2025-08-29 15:37:00 UTC
Just added a quick and dirty debugedit-classify-ar helper to debugedit based on the ideas from elfutils eu-elfclassify
  https://inbox.sourceware.org/debugedit/20250829151551.1511031-1-mark@klomp.org/

This is now in debugedit-5.2-3.fc44
  https://bodhi.fedoraproject.org/updates/FEDORA-2025-f12d187e89

This seems to resolve things at least for openblas (detects the archives are just too big 20.000+ members), ghc (detects that most archives can be skipped because they don't contain any debuginfo) and gdal (detects that the dll.a archives don't contain any ELF members with debuginfo).

Comment 18 Mark Wielaard 2025-08-29 15:39:29 UTC
(In reply to Jens Petersen from comment #16)
> Also just want to point out that we are already in F43 Beta freeze now.

If debugedit-5.2-3.fc44 works out I want to push the fix to f43 too.
Do you believe this is a beta blocker? And if so, how to request debugedit gets updated before the actual beta?

Comment 19 Mark Wielaard 2025-08-29 15:41:30 UTC
(In reply to Nikita Popov from comment #11)
> I filed
> https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/344 to
> automatically set the option.

Thanks. Should this maybe be its own separate bug?
I am not really clear on the interaction between the macros defined by rpm itself and redhat-rpm-config.

Comment 20 Mark Wielaard 2025-08-29 15:48:33 UTC
(In reply to Jens Petersen from comment #9)
> (The reason I had to kill find-debuginfo manually is that I build via
> another toolbox container.
> Also although I have set "%_smp_ncpus_max 4" nevertheless find-debuginfo
> gets -j16
> (ie dd (& ar) processes take most of my cores) unless I also set
> "%_smp_build_ncpus 4".
> Anyway I was planning to open a separate bug about that.)

Yeah, while testing I also got the rpmbuild to use more cpus than should be available in the mock.
I think this is an rpm bug, not debugedit though. It just uses the -j value find-debuginfo gets.
Could you CC me on any new bug around this? Thanks.
 
> I think those are indeed the two largest libraries in ghc.
> 
> On my local machine (Thinkpad P16v gen1 AMD 16 cores + 64GB), the warmed-up
> rpmbuild --short-circuit with "_smp_build_ncpus 4" takes:
> 
> - With the workarounds: 2m19s  (mostly rpath and subpkg files checking)
> - Without workarounds: 41m30s
> 
> So the find-debuginfo took roughly 40min of time (and this isn't exactly a
> sluggish laptop),
> which is longer than the actual package build itself (see above).

Please test out debugedit-5.2-3.fc44 if you can. It should skip most of those archive since they don't actually contain debuginfo, it should only handle the ...debug.a files.

Comment 21 Jens Petersen 2025-09-02 12:48:41 UTC
Thanks, Mark

I tested debugedit-5.2-3.fc44.x86_64 locally with ghc9.14
for --short-circuit and with profiling libraries disabled:

- with --no-ar-files: duration 1m45s
- without           : duration 14m28s
(with debugedit-5.2-2.fc43, this takes: 1h9m3s - not entirely sure why longer than above!?)

So that is certainly an improvement, though still not great I suppose.


Should I try some other options (or they are just internal)?
Well anyway I can live with "--no-ar-files" in ghc*.spec for now
(as far as I can tell so far only GHC is affected not Haskell packages thankfully:
mostly because they disable debuginfo anyway).
Eventually debuginfo should get enabled in our ghc too -
I did try it once but seem to remember running into something: need to try again.

Comment 22 Fedora Update System 2025-09-02 15:30:54 UTC
FEDORA-2025-50a0376e48 (debugedit-5.2-3.fc43) has been submitted as an update to Fedora 43.
https://bodhi.fedoraproject.org/updates/FEDORA-2025-50a0376e48

Comment 23 Mark Wielaard 2025-09-02 15:58:28 UTC
Hi Jens,

Thanks for testing. I now also did an f43 build. It will likely not be a blocker for the beta, but land as soon as the beta is unfrozen.

(In reply to Jens Petersen from comment #21)
> I tested debugedit-5.2-3.fc44.x86_64 locally with ghc9.14
> for --short-circuit and with profiling libraries disabled:
> 
> - with --no-ar-files: duration 1m45s
> - without           : duration 14m28s
> (with debugedit-5.2-2.fc43, this takes: 1h9m3s - not entirely sure why
> longer than above!?)

So with 5.2.2 find-debuginfo was doing a lot of unnecessary work.
Opening up, extracting and repackaging all archives in the package.
With 5.2.3 it basically skips all static archives that don't contain any ELF object with debuginfo.
It still processes the handful of archives that do contain ELF objects with debuginfo.
Which is still a bit slow. But much faster than doing all the unnecessary work.

> So that is certainly an improvement, though still not great I suppose.

Whether that is great or not depends on your viewpoint I guess.
If you see it as saving 1 hour of package build time it is great!
If you see it as adding 12 minutes to your build it isn't great.

> Should I try some other options (or they are just internal)?

No, this is as good as it gets for now.

There are some ideas how to make the actual processing faster by using a hacked up eu-ar:
https://sourceware.org/bugzilla/show_bug.cgi?id=33329
But that is really a future followup fix.

> Well anyway I can live with "--no-ar-files" in ghc*.spec for now
> (as far as I can tell so far only GHC is affected not Haskell packages
> thankfully:
> mostly because they disable debuginfo anyway).
> Eventually debuginfo should get enabled in our ghc too -
> I did try it once but seem to remember running into something: need to try
> again.

I hope we can speed things up a little more in the future.
For now I am glad we at least don't do an hour of unnecessary work.

Comment 24 Fedora Update System 2025-09-03 01:14:20 UTC
FEDORA-2025-50a0376e48 has been pushed to the Fedora 43 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2025-50a0376e48`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2025-50a0376e48

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 25 Jens Petersen 2025-09-03 11:29:59 UTC
I also tested a llvm20 rawhide x86_64 scratch build (the full llvm package is huge):

- a scratch build with --no-ar-files took 1h 54m 38s (https://koji.fedoraproject.org/koji/taskinfo?taskID=136689230 )
- without --no-ar-files took 4h 10m 10s (https://koji.fedoraproject.org/koji/taskinfo?taskID=136711183)

Comment 26 Kamil Páral 2025-09-04 12:08:35 UTC
Jens, you nominated this for a freeze exception, but provided no justification. Please give us a brief summary of the bug and reasoning for breaking the freeze, either here or even better in our voting ticket:
https://pagure.io/fedora-qa/blocker-review/issue/1900
Thank you.

Comment 27 Adam Williamson 2025-09-05 18:21:54 UTC
+4 in https://pagure.io/fedora-qa/blocker-review/issue/1900 , marking accepted FE.

Comment 28 Fedora Update System 2025-09-07 12:29:18 UTC
FEDORA-2025-50a0376e48 (debugedit-5.2-3.fc43) has been pushed to the Fedora 43 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 29 Frank Ch. Eigler 2025-09-08 14:34:08 UTC
Thanks for bearing with us during this transition, and thanks to Mark for building a small workable solution!


Note You need to log in before you can comment on or make changes to this bug.