rpm currently discards the debugging information in static libraries. As a result, statically linked programs lack debugging information for all system libraries. Except for select packages which copy .a files around, there is no way to recover the missing debugging information, leading to poor developer experience. I suggest we simply stop stripping static libraries. Relevant thread on the devel list: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/OFNQT4QPQB4TRHAD4FQTOHXBMORXJWQV/ I'll write up a proposal for a System-Wide Change.
I must say I strongly don't like this proposal. Unstripped static libraries are huge, we are in GCC intentionally not optimizing e.g. debug info for size of debug info with relocation, only for size of debug info after linking. And generally the debug info is not really very useful without the corresponding *-debuginfo packages installed. So I think it is better to have the non-stripped variants of static libraries in the *-debuginfo packages. After all, static linking is generally not recommended in Fedora, so for the rare case when one needs it it is possible to just tweak gcc command line to point it to the *-debuginfo variants.
For cases like Rust, static linking is the only sane choice, as there's no stable ABI to use for shared libraries. But since rpm strips all of Rust's rlibs, which are just ar archives, you miss a lot of debuginfo for standard library code. (The exception is generic code, which is only kept as metadata in the rlibs, thus that debuginfo is generated later when it's actually monomorphized.) However, I've heard rumors that Rust might start writing MIR in its rlibs instead of actual object code, in which case it will no longer be relevant to rpm here.
(In reply to Jakub Jelinek from comment #1) > I must say I strongly don't like this proposal. Unstripped static libraries > are huge, we are in GCC intentionally not optimizing e.g. debug info for > size of debug info with relocation, only for size of debug info after > linking. And generally the debug info is not really very useful without the > corresponding *-debuginfo packages installed. > So I think it is better to have the non-stripped variants of static > libraries in the *-debuginfo packages. I'll check with a mirror administrator, but I suspect -debuginfo packages are far more often installed than -static packages or even -devel packages. So please clarify what do you want to minimize: -devel packages, -static packages (if they exist), or -debuginfo packages? Or mirror size? > After all, static linking is generally not recommended in Fedora, so for the > rare case when one needs it it is possible to just tweak gcc command line to > point it to the *-debuginfo variants. Then we really should put the unstripped static archives into a directory which is not full of corrupted ELF files, otherwise we will just confuse users and have to deal with bug reports related to ld choking on separated debuginfo files. But I really don't think this is necessary, but I'll wait for the mirror statistics to provide some additional insight.
Jakub was right: It benefits most users if we move the unstripped static libraries to -debuginfo packages. For example, glibc, gmp, and llvm all have -static and -debuginfo packages, and on the mirror I had checked, -static had very few downloads (llvm-static just one), but at least there were some. For -debuginfo, there were none. It's not an artifact of mirror selection (but it could be related to how dnf handles -debuginfo package updates). This phenomenon is also visible in the public mirror statistics: https://admin.fedoraproject.org/mirrormanager/statistics/2016-11/repositories The -debug repositories are only consulted very rarely. This does not match my own usage at all, which is why I'm so surprised. Unfortunately, there is another problem with code in -debuginfo packages: Those packages cannot depend on the -devel headers, so you can easily install a static library which is incompatible with the headers you compile against, leading to difficult-to-diagnose bugs. How can we solve this? We probably need a separate subpackage to address this.
FESCo approved the Change today, but it sounds like you still have some open questions in comment #4. Do you know how you're going to approach this?
Based on the download statistics mentioned in comment #4, maybe we should do this: · Put the unstripped .a files into /usr/lib/static-debug and /usr/lib64/static-debug. (I think we should not reuse /usr/lib/debug paths because the directory would contain corrupt .so files as well, possibly confusing the linker.) · Install those files into the -debuginfo packages. · Rewrite the archives to reference the usual source paths. Then developers can install the -debuginfo packages and link with -L/usr/lib64/static-debug to access the unstripped libraries. (If we had -debuginfo-static packages, there should be an exact versioned dependency on the corresponding -devel package for the headers, but we can't put this into the -debuginfo package because it would interfere with parallel installation of multiple -debuginfo packages.) We probably need to move this to Fedora 27.
This bug appears to have been reported against 'rawhide' during the Fedora 26 development cycle. Changing version to '26'.
I don't really know what the best solution is. Maybe we should look at a solution that uses GNU Debug Fission or DWARF5 split DWARF where most of of the non-relocationable DWARF structures are put in a separate .dwo file. Then you could keep the skeleton units in the (unstripped) static library. And package up the .dwo units into a .dwp file that goes into the debuginfo. Those skeleton units contain an ID of the .dwo units. We would then need to create some mapping that ties together the debuginfo package containing the skeleton units to the debuginfo package containing the .dwp file?
This doesn't seem to be going anywhere anytime soon, closing as deferred. Feel free to reopen if it these plans become reactivated.
Out of curiosity, did a round of mass builds of Fedora 41 RPMs that export -static library subrpms, but this time disabling the static-archive stripping thusly in .rpmmacros: %__brp_strip_static_archive /bin/true jistone informs us that several rust packages are doing this sort of thing today. This "works" in that the .a files are indeed packaged unstripped, and debuginfo makes it into the client application instead of totally disappearing. But: - they are about 4-10x the size they were before, due to the retained debuginfo - the .a files are not subjected to debugedit processing, therefore retain build-tree pathnames (rather than canonicalized under /usr/src/debug), and don't get source files listed into a -debugsource subrpms. This makes the source files unavailable, even via debuginfod, even if debuginfo were kept. The former appears to be basically an aesthetic issue that may or may not be a problem at the distribution level. The latter is a tooling shortcoming that is IMO a showstopper. If corrected, then the libfoo.a source files would also be packaged (into the libfoo-debugsource subrpm, not that of the client application that links libfoo.a), and accessible via debuginfod. We should consider extending the debugedit processing toolchain to work on .a/.o files, so that some forward progress can be made on this. Packages can then opt into it by disabling %__brp_strip_static_archive and get full debuggability.
(In reply to Frank Ch. Eigler from comment #10) > Out of curiosity, did a round of mass builds of Fedora 41 RPMs that export > -static library subrpms, but this time disabling the static-archive > stripping thusly in .rpmmacros: > > %__brp_strip_static_archive /bin/true > > jistone informs us that several rust packages are doing this sort of thing > today. > This "works" in that the .a files are indeed packaged unstripped, and > debuginfo > makes it into the client application instead of totally disappearing. Should the Status of this bug be changed from CLOSED? Or is it still Deferred? Or should we file new bug reports to enhance the various tools? > But: > > - they are about 4-10x the size they were before, due to the retained > debuginfo Do you happen to have some samples of such .a (ar archives) available for inspection? I am mainly interested where the size comes from. If it is relocations then we could look whether split-dwarf improves the situation. With split-dwarf various relocations are traded for indirection tables. > - the .a files are not subjected to debugedit processing, therefore retain > build-tree pathnames (rather than canonicalized under /usr/src/debug), and > don't get source files listed into a -debugsource subrpms. This makes the > source files unavailable, even via debuginfod, even if debuginfo were kept. > > The former appears to be basically an aesthetic issue that may or may not > be a problem at the distribution level. > > The latter is a tooling shortcoming that is IMO a showstopper. In theory debugedit could be made to work on ar files. Those are just a collection of .o (ET_REL) files. And debugedit already handles kernel modules (.ko ET_REL files). Normally debugedit does work "in-place", in this case debugedit should probable parse the whole ar file and create a newly rewritten one. > If corrected, > then the libfoo.a source files would also be packaged (into the > libfoo-debugsource > subrpm, not that of the client application that links libfoo.a), and > accessible via debuginfod. How would debuginfod find the correct libfoo-debugsource package? It cannot associate a build-id with it because the .o objects in the .a archive don't have any. And I don't know if you can retrieve the archive name they came from after linking.
debuginfod would find sources for .a by the canonicalized uniqueish path names debugedit would assign to the foo.o file/directory table, no matter which foo-debugsource.rpm it is actually in. This should Just Work (tm). Resetting to redhat-rpm-config, as this is more of a distro packaging policy consideration.
find-debuginfo draft patch for static-archive processing: https://inbox.sourceware.org/debugedit/20250306195336.GA11809@redhat.com/T/#u
Upstream find-debuginfo (debugedit) now contains the static-archive debuginfo processing code above. It is active by default. That, plus %__brp_strip_static_archive /bin/true in .rpmmacros, results in unstripped, and fully usable debuginfo in -static libraries and automatically accessible source code in -debugsource rpms.
(In reply to Frank Ch. Eigler from comment #14) > Upstream find-debuginfo (debugedit) now contains the static-archive > debuginfo processing code above. It is active by default. That, plus > > %__brp_strip_static_archive /bin/true > > in .rpmmacros, results in unstripped, and fully usable debuginfo in -static > libraries and automatically accessible source code in -debugsource rpms. Could you submit a F43 Change Proposal for this? https://docs.fedoraproject.org/en-US/program_management/changes_policy/ The Proposal submission deadline (System Wide Changes) is July 1st. https://fedorapeople.org/groups/schedule/f-43/f-43-all-tasks.html
How about pushing Florian's 8-year-old one into "ready for wrangler" state? https://fedoraproject.org/wiki/Changes/StaticLibraryDebuginfo
(In reply to Frank Ch. Eigler from comment #16) > How about pushing Florian's 8-year-old one into "ready for wrangler" state? > > https://fedoraproject.org/wiki/Changes/StaticLibraryDebuginfo Does this mean we plan to ship debugging symbols in *crt*.o and libgcc.a? It will alter gcc command behavior: default builds now always contain some debugging information. Previously, there were objections to this.
We'd have to hash that out during fedora-devel discussions. But it seems a default is fairly easily opted-out: People who are concerned about "gcc foo.c" producing an a.out that has no debuginfo from foo.o but does have some from libgcc could add a "strip" to their pipeline. People who don't want to ship debuginfo in their static-libs rpms could reactivate %__brp_strip_static_archive for themselves.
Please add me under "Other developers" when you propose this. It seems the proposal is simply two things: - Backport the ar processing patch to debugedit (or do a new release with it) - change %__brp_strip_static_archive in rpmmacros (redhat-rpm-config)? What about the ideas in comment #6? Do we really want a separate debuginfo-static like package and separate install location? (My preference would be no.)
The Proposal submission deadline (Changes requiring mass rebuild) is in one week, Tue 2025-07-01 Check contingency plans (mass rebuild) is Tue 2025-07-22 Mass Rebuild is Wed 2025-07-23 I am on vacation from Fri 2025-07-18 till Mon 2025-07-28 (inclusive) So I won't be a great help when it matters most, sorry.
https://fedoraproject.org/wiki/Changes/StaticLibraryPreserveDebuginfo
This package has changed maintainer in Fedora. Reassigning to the new maintainer of this component.