Bug 1970578

Summary: please create a separate vdso subpackage to accelerate fedora debuginfod service
Product: [Fedora] Fedora Reporter: Frank Ch. Eigler <fche>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: NEW --- QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: acaringi, adscvr, airlied, alciregi, amerey, bskeggs, hdegoede, jarodwilson, jeremy, jglisse, jonathan, josef, jwakely, kernel-maint, lgoncalv, linville, masami256, mchehab, mjw, mjw, mliska, ptalbert, steved
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-06-11 18:44:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Frank Ch. Eigler 2021-06-10 18:00:00 UTC
With Fedora 35 serving debuginfo to gdb etc. via the debuginfod service by default, we see regular queries looking for debuginfo of the userspace-relevant vdso* libraries.  This happens even if the user is running gdb on some ordinary program like /bin/ls, because vdso is mapped into userspace, has a buildid, thus gdb tries to fetch its dwarf data.  With debuginfod, this request can now be serviced.

However, this particular request is a challenge to the server, because it involves decompressing the appropriate kernel-debuginfo file on the fly, and saving the vdso*.so.debug file, then relaying it to to the remote client.  The problem is that these files are practically at the END of the -debuginfo compressed-cpio archive, after all the module .ko.debug's (many hundreds), just before the final vmlinux (just the one! :-).  That ends up taking something like 60 cpu seconds (!).

If it were possible to sort the kernel's main debuginfo*.list file, to put the vdso* files at the front of the file list, these requests could be serviced much earlier, without decompressing the bulk of the file.  I believe this code in the rawhide/master branch kernel.spec file is responsible for the sequencing:

   1065 %description %{?1:%{1}-}debuginfo\
   1066 This package provides debug information for package %{name}%{?1:-%{1}}.\
   1067 This is required to use SystemTap with %{name}%{?1:-%{1}}-%{KVERREL}.\
   1068 %{expand:%%global _find_debuginfo_opts %{?_find_debuginfo_opts} -p '.*\/usr\/src\/kernels/.*|XXX' -o ignored-debuginfo.lis   1068 t -p '/.*/%%{KVERREL_RE}%{?1:[+]%{1}}/.*|/.*%%{KVERREL_RE}%{?1:\+%{1}}(\.debug)?' -o debuginfo%{?1}.list}\
   1069 %{nil}

Would you consider adding a brief postprocessor (two greps?) to sort the vdso* file names earlier?  (Another alternative may be to put these into a whole separate subrpm, and other options may exist too.)

Comment 1 Frank Ch. Eigler 2021-06-11 18:44:30 UTC
Never mind.
I spent some time experimenting with overriding

   %global __os_install_post \
      # sort debugfiles*list differently \
      %__os_install_post

and this hooking process worked.

However, rpm's build/files.c genCpioListAndHeader() sorts rpm manifest lists ANYWAY, so there appears to be nothing a .spec file can do.

Comment 2 Frank Ch. Eigler 2022-08-12 11:29:08 UTC
(The kernel spec file could still create a separate subrpm for vdso files, and that would completely fix this particular problem.)

Comment 3 Frank Ch. Eigler 2022-08-23 19:23:14 UTC
Our friends at suse have switched to a separate vdso subpackage, to solve precisely this problem.  It makes fedora debugging much faster to start up (60s in the worst case down to fractions of a second).  Please consider doing the same.  Search for separate_vdso here:

https://build.opensuse.org/package/view_file/Kernel:stable/kernel-default/kernel-default.spec?expand=1

Comment 4 Jonathan Wakely 2022-12-20 17:16:16 UTC
(In reply to Frank Ch. Eigler from comment #3)
> (60s in the worst case down to fractions of a second).

Currently about 80s, and due to a server-side misconfiguration it wasn't being cached on the server, so it was quite painful if you're impatient like me.

A separate debuginfo package would be nice.

Comment 5 Aaron Merey 2022-12-20 17:30:18 UTC
Caching helps with this issue but each time a vdso debuginfo is added to the server's cache, there will have to be a client that needs to wait for the decompression to occur. It can be annoying to wait over a minute for this, especially if this occurs during an interactive session (ex. GDB).

A separate debuginfo package for the vdso will eliminate all of this extra download time.

Comment 6 Mark Wielaard 2022-12-20 17:39:50 UTC
Yeah, if you are unlucky this takes ~80 seconds just for fetching the vdso debug file. Having a separate user space only debuginfo package would be helpful also in case someone just wants to install the debuginfo and not all the kernel space debuginfo they might not use.