Bug 1970578 - please create a separate vdso subpackage to accelerate fedora debuginfod service
Summary: please create a separate vdso subpackage to accelerate fedora debuginfod service
Keywords:
Status: NEW
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-06-10 18:00 UTC by Frank Ch. Eigler
Modified: 2022-12-20 17:39 UTC (History)
23 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2021-06-11 18:44:30 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Frank Ch. Eigler 2021-06-10 18:00:00 UTC
With Fedora 35 serving debuginfo to gdb etc. via the debuginfod service by default, we see regular queries looking for debuginfo of the userspace-relevant vdso* libraries.  This happens even if the user is running gdb on some ordinary program like /bin/ls, because vdso is mapped into userspace, has a buildid, thus gdb tries to fetch its dwarf data.  With debuginfod, this request can now be serviced.

However, this particular request is a challenge to the server, because it involves decompressing the appropriate kernel-debuginfo file on the fly, and saving the vdso*.so.debug file, then relaying it to to the remote client.  The problem is that these files are practically at the END of the -debuginfo compressed-cpio archive, after all the module .ko.debug's (many hundreds), just before the final vmlinux (just the one! :-).  That ends up taking something like 60 cpu seconds (!).

If it were possible to sort the kernel's main debuginfo*.list file, to put the vdso* files at the front of the file list, these requests could be serviced much earlier, without decompressing the bulk of the file.  I believe this code in the rawhide/master branch kernel.spec file is responsible for the sequencing:

   1065 %description %{?1:%{1}-}debuginfo\
   1066 This package provides debug information for package %{name}%{?1:-%{1}}.\
   1067 This is required to use SystemTap with %{name}%{?1:-%{1}}-%{KVERREL}.\
   1068 %{expand:%%global _find_debuginfo_opts %{?_find_debuginfo_opts} -p '.*\/usr\/src\/kernels/.*|XXX' -o ignored-debuginfo.lis   1068 t -p '/.*/%%{KVERREL_RE}%{?1:[+]%{1}}/.*|/.*%%{KVERREL_RE}%{?1:\+%{1}}(\.debug)?' -o debuginfo%{?1}.list}\
   1069 %{nil}

Would you consider adding a brief postprocessor (two greps?) to sort the vdso* file names earlier?  (Another alternative may be to put these into a whole separate subrpm, and other options may exist too.)

Comment 1 Frank Ch. Eigler 2021-06-11 18:44:30 UTC
Never mind.
I spent some time experimenting with overriding

   %global __os_install_post \
      # sort debugfiles*list differently \
      %__os_install_post

and this hooking process worked.

However, rpm's build/files.c genCpioListAndHeader() sorts rpm manifest lists ANYWAY, so there appears to be nothing a .spec file can do.

Comment 2 Frank Ch. Eigler 2022-08-12 11:29:08 UTC
(The kernel spec file could still create a separate subrpm for vdso files, and that would completely fix this particular problem.)

Comment 3 Frank Ch. Eigler 2022-08-23 19:23:14 UTC
Our friends at suse have switched to a separate vdso subpackage, to solve precisely this problem.  It makes fedora debugging much faster to start up (60s in the worst case down to fractions of a second).  Please consider doing the same.  Search for separate_vdso here:

https://build.opensuse.org/package/view_file/Kernel:stable/kernel-default/kernel-default.spec?expand=1

Comment 4 Jonathan Wakely 2022-12-20 17:16:16 UTC
(In reply to Frank Ch. Eigler from comment #3)
> (60s in the worst case down to fractions of a second).

Currently about 80s, and due to a server-side misconfiguration it wasn't being cached on the server, so it was quite painful if you're impatient like me.

A separate debuginfo package would be nice.

Comment 5 Aaron Merey 2022-12-20 17:30:18 UTC
Caching helps with this issue but each time a vdso debuginfo is added to the server's cache, there will have to be a client that needs to wait for the decompression to occur. It can be annoying to wait over a minute for this, especially if this occurs during an interactive session (ex. GDB).

A separate debuginfo package for the vdso will eliminate all of this extra download time.

Comment 6 Mark Wielaard 2022-12-20 17:39:50 UTC
Yeah, if you are unlucky this takes ~80 seconds just for fetching the vdso debug file. Having a separate user space only debuginfo package would be helpful also in case someone just wants to install the debuginfo and not all the kernel space debuginfo they might not use.


Note You need to log in before you can comment on or make changes to this bug.