The error messages show the binaries containing DW_UT_TYPE units (Unit type 2), which aren't supported by debugedit (and also not by dwz).

Try removing the -fdebug-types-section compile flag.

Thanks, will try, seems the flag is inserted by WebKit's build system ...

Currently the webkitgtk build disables LTO on ppc64le, because it OOMs on the builders. Could it be related?

(In reply to Dan Horák from comment #4)
> Currently the webkitgtk build disables LTO on ppc64le, because it OOMs on
> the builders. Could it be related?

I don't really know how -fdebug-types-section and LTO interact for g++ (CCed jakub who might).
But if you are only seeing this on ppc64le and not on other arches then I suspect it might.
Do your other arch builds also use -fdebug-types-section and are you seeing any issue there?

For the LTO vs. -fdebug-types-section interaction, I've recently filed and fixed https://gcc.gnu.org/PR116614
(but so far on trunk only and it isn't in any Fedora rpm yet).
I think dwz doesn't handle -fdebug-types-section and gives up with that though (could be wrong on that).

With -fdebug-types-section dropped the build passes. The "Unable to find program interpreter name" remains there.

...
+ /usr/bin/find-debuginfo -j160 --strict-build-id -m -i --build-id-seed 2.45.92-3.fc42 --unique-debug-suffix -2.45.92-3.fc42.ppc64le --unique-debug-src-base webkitgtk-2.45.92-3.fc42.ppc64le --run-dwz --dwz-low-mem-die-limit 10000000 --dwz-max-die-limit 50000000 -j30 -S debugsourcefiles.list /builddir/build/BUILD/webkitgtk-2.45.92-build/webkitgtk-2.45.92
find-debuginfo: starting
Extracting debug info from 15 files
readelf: Error: Unable to find program interpreter name
readelf: Error: Unable to find program interpreter name
readelf: Error: Unable to find program interpreter name
readelf: Error: Unable to find program interpreter name
DWARF-compressing 15 files
dwz: ./usr/lib64/libwebkit2gtk-4.1.so.0.16.1-2.45.92-3.fc42.ppc64le.debug: Too many DIEs, not optimizing
dwz: ./usr/lib64/libwebkitgtk-6.0.so.4.10.1-2.45.92-3.fc42.ppc64le.debug: Too many DIEs, not optimizing
sepdebugcrcfix: Updated 15 CRC32s, 0 CRC32s did match.
Creating .debug symlinks for symlinks to ELF files
...

(In reply to Mark Wielaard from comment #5)
> Do your other arch builds also use -fdebug-types-section and are you seeing
> any issue there?

It's used on every architecture, but this build failure only occurs for ppc64le.

I can remove -fdebug-types-section, but I think we should understand why the error is specific to ppc64le. LTO is also disabled on i686, but this error isn't happening there.

Also "-fdebug-types-section" is a new option passed to the build in webkitgtk-2.45.92, it wasn't there in webkitgtk-2.45.91 if I see right.

Correct, the flag was added recently to "avoid linker relocation errors on Debug/RelWithDebInfo builds" https://bugs.webkit.org/show_bug.cgi?id=278861

"Currently RelWithDebInfo/Debug builds fail to link on aarch64 - relocation errors appear: relocation R_AARCH64_ABS32 out of range: 4312197985 is not in [-2147483648, 4294967295]. Adding -fdebug-types-section as compiler flag helps to migitate the problem."

which doesn't look good either. I wonder what that error message means.

I thought I actually *did* test this change quite recently with a scratch build on all Fedora architectures specifically because I was nervous about any changes that affect debuginfo, but perhaps I messed up or am misremembering.

That means you have too large debug info that some debug section doesn't fit in 32-bit DWARF anymore.
You could always try -gdwarf-64 and use 64-bit DWARF (which admittedly is significantly less tested and larger).
That said, if dwz gives up on the compression anyway (see the
dwz: ./usr/lib64/libwebkit2gtk-4.1.so.0.16.1-2.45.92-3.fc42.ppc64le.debug: Too many DIEs, not optimizing
dwz: ./usr/lib64/libwebkitgtk-6.0.so.4.10.1-2.45.92-3.fc42.ppc64le.debug: Too many DIEs, not optimizing
; dwz has some limits so that it doesn't consume too much memory; those can be obviously increased also on a per package basis))
perhaps using -fdebug-types-section offers at least some possibility to reduce the debug info size.

We already have bug #2017171 for the DIE limit problem. The settings in webkitgtk.spec are:

# Increase the DIE limit so our debuginfo packages can be size-optimized.
# This previously decreased the size for x86_64 from ~5G to ~1.1G, but as of
# 2022 it's more like 850 MB -> 675 MB. This requires lots of RAM on the
# builders, so only do this for x86_64 and aarch64 to avoid overwhelming
# builders with less RAM.
# https://bugzilla.redhat.com/show_bug.cgi?id=1456261
%global _dwz_max_die_limit_x86_64 250000000
%global _dwz_max_die_limit_aarch64 250000000

I guess we could try raising the limit on all 64-bit architectures now that https://pagure.io/fedora-infrastructure/issue/11839 is fixed. (Could also try reenabling LTO.)

Wait, sorry, the DIE limit is already raised for aarch64, and the build there is not hitting the DIE limit.

I am going to open another bug for failed LTO

...
lto-wrapper: fatal error: Too many copied sections: Operation not supported
compilation terminated.
/usr/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.

lto-wrapper: fatal error: Too many copied sections: Operation not supported
is most likely that https://gcc.gnu.org/PR116614 I was talking about.
Running into the DWZ limits is fromk that
dwz: ./usr/lib64/libwebkit2gtk-4.1.so.0.16.1-2.45.92-3.fc42.ppc64le.debug: Too many DIEs, not optimizing
dwz: ./usr/lib64/libwebkitgtk-6.0.so.4.10.1-2.45.92-3.fc42.ppc64le.debug: Too many DIEs, not optimizing
in #c7, so on ppc64le where the limit isn't increased.
The aarch64 error presumably happens during linking, so dwz can't help with that, one needs to successfully link and only after that it can be dwz optimized.
dwz can't e.g. optimize relocatable objects, so you can't relocatably link say 4 parts of the library, optimize them separately and then link together.
And right now I believe dwz can't deal with -fdebug-types-section, so you can choose that or dwz but not both; if both work, you want to pick the
one which gives you smaller debug info.
One way to decrease size of debug info without -fdebug-types-section might be try to link fewer larger objects instead of many small ones.
Kind of what e.g. Qt/KDE does, creating *.all.cpp which includes all or most source files in each directory in a single larger one.  Then
debug info for stuff included in large C++ headers is emitted just once rather than 50 times or how many sources one has in a directory (provided
you include usually same or very similar set of headers).

(In reply to Jakub Jelinek from comment #15)
> And right now I believe dwz can't deal with -fdebug-types-section, so you
> can choose that or dwz but not both; if both work, you want to pick the
> one which gives you smaller debug info.

What does this mean for Fedora and other distro builds? Should the package not use -fdebug-types-section?

> One way to decrease size of debug info without -fdebug-types-section might
> be try to link fewer larger objects instead of many small ones.
> Kind of what e.g. Qt/KDE does, creating *.all.cpp which includes all or most
> source files in each directory in a single larger one.  Then
> debug info for stuff included in large C++ headers is emitted just once
> rather than 50 times or how many sources one has in a directory (provided
> you include usually same or very similar set of headers).

We generally #include approximately 8 C++ source files together into each "source bundle" which is what actually gets submitted to the compiler, although the number varies. E.g. the first example file I checked, UnifiedSource-0e543b66-1.cpp, only has 5:

#include "interpreter/CLoopStack.cpp"
#include "interpreter/CallFrame.cpp"
#include "interpreter/Interpreter.cpp"
#include "interpreter/ShadowChicken.cpp"
#include "interpreter/StackVisitor.cpp"

Increasing the number is possible, but awkward. We use the same bundles for all platforms and architectures, and the number is chosen to optimize build performance for macOS, not for Linux. Also, the higher the number of #included source files per source bundle, the greater the RAM required to build WebKit.

(In reply to Jakub Jelinek from comment #15)
> And right now I believe dwz can't deal with -fdebug-types-section, so you
> can choose that or dwz but not both; if both work, you want to pick the
> one which gives you smaller debug info.

I'm confused. It looks like we unconditionally use both -fdebug-types-section and also dwz, yes? And this works everywhere except ppc64le?

(In reply to Jakub Jelinek from comment #15)
> lto-wrapper: fatal error: Too many copied sections: Operation not supported
> is most likely that https://gcc.gnu.org/PR116614 I was talking about.

Are you sure? We are building with Clang. Won't lto-wrapper come from Clang instead of GCC?

Oops, sorry. I forgot about this part of the spec file:

# Clang is preferred: https://skia.org/docs/user/build/#supported-and-preferred-compilers
# However, it crashes on ppc64le.
%ifnarch ppc64le
%global toolchain clang
%endif

So ppc64le is special once again.

(In reply to Michael Catanzaro from comment #16)
> What does this mean for Fedora and other distro builds? Should the package
> not use -fdebug-types-section?

-fdebug-types-section is simply one way to decrease redundancies in DWARF (one way
in DWARF 4 where it used special .debug_types section, hence the switch name, in DWARF 5
it uses DW_UT_type units of .debug_info instead; and with various problems around the
DWARF design of that feature), dwz is another one.  The reason dwz doesn't have
-fdebug-types-section support is mostly that normally -fdebug-types-section only brings
pain and generally can't do what dwz does, so it wasn't something we'd be recommending.
Of course, when you trigger link limits without -fdebug-types-section, that is a different
thing.
dwz isn't done during linking because it needs a lot of RAM and sharing that RAM
together with the linker would make it worse, and because it has a mode in which it
optimizes debug info from multiple shared libraries/binaries in the same package.
Adding dwz support for -fdebug-types-section is theoretically possible, ideally it would
just undo the mess it added (e.g. the huge 16-byte references it uses).

> We generally #include approximately 8 C++ source files together into each
> "source bundle" which is what actually gets submitted to the compiler,
> although the number varies. E.g. the first example file I checked,
> UnifiedSource-0e543b66-1.cpp, only has 5:
> 
> #include "interpreter/CLoopStack.cpp"
> #include "interpreter/CallFrame.cpp"
> #include "interpreter/Interpreter.cpp"
> #include "interpreter/ShadowChicken.cpp"
> #include "interpreter/StackVisitor.cpp"
> 
> Increasing the number is possible, but awkward. We use the same bundles for
> all platforms and architectures, and the number is chosen to optimize build
> performance for macOS, not for Linux. Also, the higher the number of
> #included source files per source bundle, the greater the RAM required to
> build WebKit.

I'm just surprised that even when you do it you run into clearly largest debug info from any Fedora packages.
How many *.o files are linked together?

(In reply to Michael Catanzaro from comment #17)
> (In reply to Jakub Jelinek from comment #15)
> > And right now I believe dwz can't deal with -fdebug-types-section, so you
> > can choose that or dwz but not both; if both work, you want to pick the
> > one which gives you smaller debug info.
> 
> I'm confused. It looks like we unconditionally use both
> -fdebug-types-section and also dwz, yes? And this works everywhere except
> ppc64le?

dwz is used always, but if it sees something it doesn't handle (like DW_UT_type),
it just bails out and doesn't optimize anything.

As for the aarch64 linking errors, might be interesting to try using gcc instead of clang if you still run into it.

(In reply to Jakub Jelinek from comment #21)
> As for the aarch64 linking errors, might be interesting to try using gcc
> instead of clang if you still run into it.

I don't know what compiler was used in that case. https://bugs.webkit.org/show_bug.cgi?id=278861 is not an error I've encountered myself.

(In reply to Jakub Jelinek from comment #20)
> I'm just surprised that even when you do it you run into clearly largest
> debug info from any Fedora packages.
> How many *.o files are linked together?

I copied the link command for libwebkitgtk-6.0.so.4.11.0 from my personal development build (not identical to the Fedora build configuration) and did a search for ".o" and there are 2221 results. That should be approximately correct; it looks like it's not matching anything else.

(It looks like there are an additional ~800 .o files linked into lib/libjavascriptcoregtk-6.0.so.1.4.0.)

So I still don't know what to do here. For now I've patched the Fedora package to just not use this flag anymore, but I also can't confidently land that change upstream. My best explanation at this point is "the flag is incompatible with dwz and debugedit," but these tools are surely used on all architectures, and it's only a problem for ppc64le.

File an upstream bug for debugedit to support DW_UT_type. https://sourceware.org/bugzilla/show_bug.cgi?id=32157

for the record, webkit builds OK on aarch64 in Rawhide with gcc and without the -fdebug-types-section option

I assume the aarch64 build failure probably only affects WPE WebKit, though I'm not certain. WPE has only one shared library, instead of the two shared libraries installed by WebKitGTK, so the largest shared library is going to be bigger.

(In reply to Michael Catanzaro from comment #24)
> So I still don't know what to do here. For now I've patched the Fedora
> package to just not use this flag anymore, but I also can't confidently land
> that change upstream. My best explanation at this point is "the flag is
> incompatible with dwz and debugedit," but these tools are surely used on all
> architectures, and it's only a problem for ppc64le.

Does anybody have any ideas why the -fdebug-types failure is specific to ppc64le only?

(In reply to Michael Catanzaro from comment #28)
> Does anybody have any ideas why the -fdebug-types failure is specific to
> ppc64le only?

If https://bugzilla.redhat.com/show_bug.cgi?id=2310828#c19 is in, then ppc64le would be built by a different compiler, which emits different debug info, for which LTO means something significantly different, etc.

If it is the
lto-wrapper: fatal error: Too many copied sections: Operation not supported
error, I can have a fix within 2-3 days, if it is something different (like debugedit not handling it), Mark needs to answer that.
Does clang default to DWARF 5 (i.e. does it use DW_UT_type for -fdebug-types-section) or DWARF 4 (so .debug_types sections)?

(In reply to Jakub Jelinek from comment #29)
> (In reply to Michael Catanzaro from comment #28)
> > Does anybody have any ideas why the -fdebug-types failure is specific to
> > ppc64le only?
> 
> If https://bugzilla.redhat.com/show_bug.cgi?id=2310828#c19 is in, then
> ppc64le would be built by a different compiler, which emits different debug
> info, for which LTO means something significantly different, etc.

Right. I suspect that ppc64le is special because it uses g++ and -fdebug-types-section
I don't know what clang++ and -fdebug-types-section produces.

> If it is the
> lto-wrapper: fatal error: Too many copied sections: Operation not supported
> error, I can have a fix within 2-3 days, if it is something different (like
> debugedit not handling it), Mark needs to answer that.

I filed an upstream bug for debugedit to support DW_UT_type https://sourceware.org/bugzilla/show_bug.cgi?id=32157 but haven't worked on it yet assuming the removal of -fdebug-types from the compile flags did the trick already.

for the record, the clang issue has been reported as https://github.com/llvm/llvm-project/issues/108014, internal clang/llvm team is aware

(In reply to Jakub Jelinek from comment #29)
> If https://bugzilla.redhat.com/show_bug.cgi?id=2310828#c19 is in, then
> ppc64le would be built by a different compiler, which emits different debug
> info, for which LTO means something significantly different, etc.

Good point. You're right. That explains it.
 
> If it is the
> lto-wrapper: fatal error: Too many copied sections: Operation not supported
> error, I can have a fix within 2-3 days,

Thank you! But the issue with LTO is a different bug.

> if it is something different (like
> debugedit not handling it), Mark needs to answer that.
> Does clang default to DWARF 5 (i.e. does it use DW_UT_type for
> -fdebug-types-section) or DWARF 4 (so .debug_types sections)?

Found https://www.phoronix.com/news/LLVM-Clang-DWARFv5-Default so this particular mystery remains.

for the record, aarch64 is hitting the same issue as ppc64le when building with gcc and -fdebug-types-section (and disabled LTO due "too many sections"), so this problem isn't ppc64le specific

I have opened https://src.fedoraproject.org/rpms/webkitgtk/pull-request/12 to enable clang builds for ppc64le.

Lets keep this bug for debugedit not handling DW_UT_type.
There is a patch for that now:
https://inbox.sourceware.org/debugedit/20240928230445.390836-1-mark@klomp.org/

Can we open new bugs for any of the other issues?

I don't think we have any other remaining issues here? The LLVM musttail issue is tracked as https://github.com/llvm/llvm-project/issues/98859.

ack, the gcc/lto fix should be in gcc-14.2.1-3, needs rechecking ...

And rechecked, neither aarch64 or ppc64le fail due LTO issues, so https://gcc.gnu.org/PR116614 should be fixed now. This is with -fdebug-types-section enabled (the default in webkitgtk).

FEDORA-2024-d0c875e1ea (debugedit-5.0-18.fc41) has been submitted as an update to Fedora 41.
https://bodhi.fedoraproject.org/updates/FEDORA-2024-d0c875e1ea

FEDORA-2024-520747442e (debugedit-5.0-18.fc40) has been submitted as an update to Fedora 40.
https://bodhi.fedoraproject.org/updates/FEDORA-2024-520747442e

FEDORA-2024-d0c875e1ea has been pushed to the Fedora 41 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-d0c875e1ea`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-d0c875e1ea

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2024-520747442e has been pushed to the Fedora 40 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-520747442e`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-520747442e

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

I have checked webkit build with gcc and -fdebug-types-section and updated debugedit and is builds OK.

FEDORA-2024-520747442e (debugedit-5.0-18.fc40) has been pushed to the Fedora 40 stable repository.
If problem still persists, please make note of it in this bug report.

FEDORA-2024-d0c875e1ea (debugedit-5.0-18.fc41) has been pushed to the Fedora 41 stable repository.
If problem still persists, please make note of it in this bug report.