Bug 751195

Summary: make it possible to install multiple kernel-debuginfos
Product: [Fedora] Fedora Reporter: Frank Ch. Eigler <fche>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: acme, aquini, fche, gansalmon, itamar, jeder, joe, jonathan, kernel-maint, labbott, madhu.chinakonda, mjw, smakarov
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 770177 (view as bug list) Environment:
Last Closed: 2019-07-17 19:32:10 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1340819    
Bug Blocks: 770177    

Description Frank Ch. Eigler 2011-11-03 20:03:21 UTC
Due to various packaging policies and accidents, rpm generally refuses
to install more than one kernel-*-debuginfo variant on a machine at a time,
even though more than one kernel may be installed.  One reason appears to be
a conflict over the /usr/src/debug/kernel-V.A directory's contents.  This 
directory is not versioned in the sense that the -R (rpm release) bits are not 
embedded.  That in turn means that two closely related kernels, different only in 
patchlevel, will generate immediate rpm -i conflicts:

  file /usr/src/debug/kernel-3.0.6/linux-3.0.6.x86_64/arch/x86/kernel/hpet.c conflicts between attempted installs of kernel-rt-debuginfo-common-x86_64-3.0.6-rt17.34.el6rt.x86_64 and kernel-rt-debuginfo-common-x86_64-3.0.6-rt18.35.el6rt.x86_64
  file /usr/src/debug/kernel-3.0.6/linux-3.0.6.x86_64/fs/ioprio.c conflicts between attempted installs of kernel-rt-debuginfo-common-x86_64-3.0.6-rt17.34.el6rt.x86_64 and kernel-rt-debuginfo-common-x86_64-3.0.6-rt18.35.el6rt.x86_64
  file /usr/src/debug/kernel-3.0.6/linux-3.0.6.x86_64/include/linux/hardirq.h conflicts between attempted installs of kernel-rt-debuginfo-common-x86_64-3.0.6-rt17.34.el6rt.x86_64 and kernel-rt-debuginfo-common-x86_64-3.0.6-rt18.35.el6rt.x86_64

etc.

Please investigate whether in kernel.spec, the debugedit /usr/src/debug 
invocation can be smartened up to be similar to the /usr/src/kernels
/%{KVERREL}... bit, which does include the -R (from N-V-R) substring.
Maybe it's as simple as

export AFTER_LINK=\
'sh -xc "/usr/lib/rpm/debugedit -b $$RPM_BUILD_DIR -d /usr/src/debug/kernel-%{KVERREL} \
                                -i $@ > $@.id"'

Comment 1 Josh Boyer 2012-02-22 16:34:35 UTC
It's not really trivial to do that it seems.

debugedit can't deal with the destdir being a longer string length than the basedir (probably because it edits the binaries and doesn't want to go mucking with section sizes and such).  So just changing destdir to /usr/src/debug/kernel-%{KVERREL} fails.

Trying to change the build directory so that it winds up including release in the basedir isn't exactly trivial either.  The kernel spec file has a bunch of stuff it does to try and minimize the impact of creating exploded kernel trees during the %prep section.  This results in $RPM_BUILD_DIR/kernel-%{kversion}/ containing the trees under linux-%{kversion}.%{_target_cpu}.  That tree is what is really causing the conflict.

I might try having it explode the tree to linux-%{KVERREL}.%{_target_cpu} instead, which would solve the file conflict.  We'll see how horrible that looks.

Comment 3 Frank Ch. Eigler 2012-02-22 18:04:11 UTC
Josh, are you sure /usr/src/debug/kernel-%{KVERREL} is necessarily longer than
/usr/src/debug/kernel-V/linux-V-R-A?  It would eliminate the kernel/linux and V substring duplication.

Comment 4 Josh Boyer 2012-02-22 18:10:42 UTC
(In reply to comment #3)
> Josh, are you sure /usr/src/debug/kernel-%{KVERREL} is necessarily longer than
> /usr/src/debug/kernel-V/linux-V-R-A?  It would eliminate the kernel/linux and V
> substring duplication.

The conflict comes from the -b and -d options to debugedit.  -b is $RPMBUILD, which translates to -b /builddir/build/BUILD.  -d is /usr/src/debug right now, which is definitely shorter.  The current situation is tenuous at best, but it works and making -d anything with a path longer than /builddir/build/BUILD will break it.

Anyway, the solution I mentioned at the bottom of my comment seems to be working just fine.  That should allow multiple debuginfo to be installed.

Comment 5 Josh Boyer 2012-02-22 22:01:06 UTC
Poking at this more, it seems this is getting less and less possible.  The patch I have to adjust /usr/src/debug/kernel-<version>.<dist>/linux-<version>.<arch> to /usr/src/debug/kernel-<version>.<dist>/linux-KVERREL works just fine to avoid duplicate source files being installed.  There are no conflicts from kernel-debuginfo-common due to that.

However, the main kernel-debuginfo packages contain the build-id hash directories and files.  There _are_ conflicts among these, and that makes sense.  The build-id for two binaries should be basically identical or build-id wouldn't really be working.  We can't move those files out of /usr/lib/debug/.build-id/ as far as I'm aware, so really this bug isn't going to get solved fully.

I'm willing to commit the patch I have anyway, since it will align us better with RHEL but it isn't going to allow kernel-debuginfo packages of similar kernels to be installed at the same time.

See also bug 606620

Comment 7 Josh Boyer 2012-02-22 23:57:34 UTC
I've committed the change to switch to linux-%{KVERREL} for the build directory, which fixes the conflicts in kernel-debuginfo-common.

As noted above, this won't actually solve the issue of muliple kernel debuginfo installs, which doesn't seem to have a fix while build-id is in use.  Because of that, I'm going to close this bug as CANTFIX.

Comment 8 Frank Ch. Eigler 2012-02-23 16:16:21 UTC
(FWIW, conflicting build-ids are probably not a big problem.
The binaries should be effectively identical if the build-id file names 
are the same.)

Comment 9 Frank Ch. Eigler 2013-08-27 23:11:29 UTC
Maybe there is another solution, if those conflicts cannot be fixed
at the RPM level.  I haven't heard whether those .build-id/FOO files
are actually identical (so RPM should be able to suppress the conflict).

In any case, GNU LD can generate build-ids in several ways.  The 
"normative-content" SHA1 based one is only the default.  For the
kernel, we could try -Wl,--buildid=uuid, where the resulting
build-id file names should be different for every compiler invocation.

Comment 10 Frank Ch. Eigler 2013-08-28 00:07:59 UTC
By the way, all it takes in the kernel build system is this:

diff --git a/Makefile b/Makefile
index a35f72a..a661550 100644
--- a/Makefile
+++ b/Makefile
@@ -674,7 +674,7 @@ KBUILD_CFLAGS += $(KCFLAGS)
 
 # Use --build-id when available.
 LDFLAGS_BUILD_ID = $(patsubst -Wl$(comma)%,%,\
-                             $(call cc-ldoption, -Wl$(comma)--build-id,))
+                             $(call cc-ldoption, -Wl$(comma)--build-id=uuid,))
 KBUILD_LDFLAGS_MODULE += $(LDFLAGS_BUILD_ID)
 LDFLAGS_vmlinux += $(LDFLAGS_BUILD_ID)

Comment 12 Joseph D. Wagner 2017-05-21 22:13:43 UTC
I would love to see some traction on this six year old request.  It would really help kernel debugging in rawhide.

Comment 13 Laura Abbott 2017-05-22 15:29:31 UTC
There have been significant changes to debuginfo and build-ids and everything over the past n years. Do the specific complaints still apply?

Comment 14 Mark Wielaard 2017-05-22 16:45:45 UTC
(In reply to Laura Abbott from comment #13)
> There have been significant changes to debuginfo and build-ids and
> everything over the past n years. Do the specific complaints still apply?

Yes, they still do apply even though rpmbuild could now help with it.
But for the kernel we are still using %undefine _unique_build_ids.
It takes a bit more work to get that set to 1 for the current kernel.spec.

I know I own you a review for your latest kernel.spec changes (sorry, I do have it in my INBOX marked urgent...). And I think we could get there for the kernel eventually, but it will take a bit more work since the kernel build uses build-ids both before and after rpm debugedit makes them unique.

Comment 15 Frank Ch. Eigler 2017-05-22 19:06:48 UTC
Laura, I gather from Mark that you are well on the way to being able to do this: install N kernel/kernel-devel/kernel-debuginfo sets concurrently.  How about making that the predicate for closing this BZ?

Comment 16 Joseph D. Wagner 2018-05-31 09:11:08 UTC
This has been mostly working until now.

file /usr/lib/debug/.build-id/d5/8a63e566b3cb9218d225a1e7df3947e314f413.1 from install of kernel-debuginfo-4.17.0-0.rc7.git0.1.fc29.x86_64 conflicts with file from package kernel-debuginfo-4.17.0-0.rc6.git1.1.fc29.x86_64

Comment 17 Mark Wielaard 2018-05-31 09:43:00 UTC
(In reply to Joseph D. Wagner from comment #16)
> This has been mostly working until now.
> 
> file /usr/lib/debug/.build-id/d5/8a63e566b3cb9218d225a1e7df3947e314f413.1
> from install of kernel-debuginfo-4.17.0-0.rc7.git0.1.fc29.x86_64 conflicts
> with file from package kernel-debuginfo-4.17.0-0.rc6.git1.1.fc29.x86_64

If you have it already installed, could you ls -lah /usr/lib/debug/.build-id/d5/8a63e566b3cb9218d225a1e7df3947e314f413.1 to see where it is pointing to?

BTW. The .1 postfix says it already was a duplicate in the package itself, which means it is probably a hardlinked file with debuginfo, which is legit, but not very common. So there is probably already something odd going on.

Comment 18 Joseph D. Wagner 2018-06-01 23:31:22 UTC
lrwxrwxrwx. 1 root root 78 May 29 09:31 /usr/lib/debug/.build-id/d5/8a63e566b3cb9218d225a1e7df3947e314f413.1 -> ../../../../../usr/src/kernels/4.17.0-0.rc7.git0.1.fc29.x86_64/scripts/unifdef

It's blinking red.

Comment 19 Laura Abbott 2018-06-04 17:50:41 UTC
This is one of the userspace binaries that must be
packaged with kernel-devel. I'm guessing this was a duplicate with the debug kernel because of how we do packaging:

warning: Duplicate build-ids /builddir/build/BUILDROOT/kernel-4.17.0-0.rc7.git0.1.fc29.x86_64/usr/src/kernels/4.17.0-0.rc7.git0.1.fc29.x86_64+debug/scripts/unifdef and /builddir/build/BUILDROOT/kernel-4.17.0-0.rc7.git0.1.fc29.x86_64/usr/src/kernels/4.17.0-0.rc7.git0.1.fc29.x86_64/scripts/unifdef

The userspace binaries don't have the unique build-id setting because we need that feature turned off for the kernel itself. Honestly I'd rather just not include the debuginfo for any of the userspace binaries but I haven't gotten around to thinking about this too much.

Comment 20 Frank Ch. Eigler 2018-06-04 18:03:10 UTC
(mjw/et al., any progress on getting buildid salting from n-v-r strings?)

Comment 21 Laura Abbott 2018-07-12 23:36:38 UTC
The patches to complete this work have been accepted upstream so I applied them to the rawhide branch. This means that parallel debuginfo should work for future kernel rawhide builds. Please test it out and let me know if I missed anything.