Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2229804

Summary: Backports needed to install debug kernel
Product: Red Hat Enterprise Linux 8 Reporter: Joseph Marrero <jmarrero>
Component: rpm-ostreeAssignee: Joseph Marrero <jmarrero>
Status: CLOSED ERRATA QA Contact: Aashish Radhakrishnan <aaradhak>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: CentOS StreamCC: aaradhak, bstinson, hhei, jwboyer
Target Milestone: rcKeywords: Triaged
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: rpm-ostree-2022.10.117.g52714b51-2.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-11-14 15:27:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Joseph Marrero 2023-08-07 18:04:58 UTC 
Backport:

https://github.com/coreos/rpm-ostree/commit/fb97c48f3cd070c1ad559f3f43f86ad6548f6b02
https://github.com/coreos/rpm-ostree/commit/eae7e1d8d692b5ce6d3d6eef29abbd7512ae4682

for: https://issues.redhat.com/browse/OCPBUGS-17383
Comment 1 HuijingHei 2023-08-23 03:19:18 UTC 
Do pre-testing on 412.86.202308221642-0 with rpm-ostree-2022.10.117.g52714b51-2.el8.x86_64, can install kernel-debug successfully.

1) Test with unfixed rpm-ostree, install kernel-debug failed

[core@cosa-devsh ~]$ sudo -i
[root@cosa-devsh ~]# rpm -q rpm-ostree
rpm-ostree-2022.10.117.g52714b51-1.el8.x86_64

[root@cosa-devsh ~]# rpm-ostree override remove kernel kernel-{core,modules,modules-extra} \
> --install kernel-debug-4.18.0-512.el8.x86_64.rpm \
> --install kernel-debug-core-4.18.0-512.el8.x86_64.rpm \
> --install kernel-debug-modules-4.18.0-512.el8.x86_64.rpm \
> --install kernel-debug-modules-extra-4.18.0-512.el8.x86_64.rpm
Checking out tree 3821eab... done
No enabled rpm-md repositories.
Importing rpm-md... done
Resolving dependencies... done
Applying 4 overrides and 4 overlays
Processing packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
error: Running %posttrans for kernel-debug-modules: bwrap(/bin/sh): Child process killed by signal 1; run `journalctl -t 'rpm-ostree(kernel-debug-modules.posttrans)'` for more information


2) Upgrade rpm-ostree to 2022.10.117.g52714b51-2.el8, install kernel-debug successfully.

[root@cosa-devsh ~]# rpm-ostree override replace rpm-ostree-2022.10.117.g52714b51-2.el8.x86_64.rpm rpm-ostree-libs-2022.10.117.g52714b51-2.el8.x86_64.rpm
[root@cosa-devsh ~]# reboot

[core@cosa-devsh ~]$ rpm -q rpm-ostree
rpm-ostree-2022.10.117.g52714b51-2.el8.x86_64

[root@cosa-devsh ~]# rpm-ostree override remove kernel kernel-{core,modules,modules-extra} \
>  --install kernel-debug-4.18.0-512.el8.x86_64.rpm \
>  --install kernel-debug-core-4.18.0-512.el8.x86_64.rpm \
>  --install kernel-debug-modules-4.18.0-512.el8.x86_64.rpm \
>  --install kernel-debug-modules-extra-4.18.0-512.el8.x86_64.rpm
Checking out tree 3821eab... done
No enabled rpm-md repositories.
Importing rpm-md... done
Resolving dependencies... done
Applying 6 overrides and 4 overlays
Processing packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Generating initramfs... done
Writing OSTree commit... done
Staging deployment... done
Removed:
  kernel-4.18.0-372.69.1.el8_6.x86_64
  kernel-core-4.18.0-372.69.1.el8_6.x86_64
  kernel-modules-4.18.0-372.69.1.el8_6.x86_64
  kernel-modules-extra-4.18.0-372.69.1.el8_6.x86_64
Added:
  kernel-debug-4.18.0-512.el8.x86_64
  kernel-debug-core-4.18.0-512.el8.x86_64
  kernel-debug-modules-4.18.0-512.el8.x86_64
  kernel-debug-modules-extra-4.18.0-512.el8.x86_64
Use "rpm-ostree override reset" to undo overrides
Run "systemctl reboot" to start a reboot

[root@cosa-devsh ~]# reboot

[core@cosa-devsh ~]$ uname -r
4.18.0-512.el8.x86_64+debug

[core@cosa-devsh ~]$ rpm -qa | grep kernel-debug
kernel-debug-core-4.18.0-512.el8.x86_64
kernel-debug-4.18.0-512.el8.x86_64
kernel-debug-modules-4.18.0-512.el8.x86_64
kernel-debug-modules-extra-4.18.0-512.el8.x86_64
Comment 2 Joseph Marrero 2023-08-24 05:51:42 UTC 
need to create the errata to get this to move to ON_QA, but I am waiting to land v2023.6 (https://bugzilla.redhat.com/show_bug.cgi?id=2234352) on centos stream 9 before pushing this forward to avoid getting this fix on RHEL8 before RHEL9.
Comment 3 Joseph Marrero 2023-08-24 15:47:19 UTC 
pushing to errata as build for stream 9 will include this soon.
Comment 7 Aashish Radhakrishnan 2023-08-24 20:44:26 UTC 
Verification done based on comment 1

aaradhak@toolbox ~/rhcos/rhcos-4.12 $ cosa run --qemu-image rhcos-412.86.202308221642-0-qemu.x86_64.qcow2 

Last login: Thu Aug 24 16:57:54 2023
[core@cosa-devsh ~]$ sudo -i
[root@cosa-devsh ~]# rpm -q rpm-ostree
rpm-ostree-2022.10.117.g52714b51-1.el8.x86_64


[root@cosa-devsh ~]# curl - http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-debug-modules-extra-4.18.0-511.el8.x86_64.rpm
curl: option -: is unknown
curl: try 'curl --help' or 'curl --manual' for more information
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# curl -kO http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-debug-modules-extra-4.18.0-511.el8.x86_64.rpm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11.3M  100 11.3M    0     0  1246k      0  0:00:09  0:00:09 --:--:-- 1967k


[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# curl -kO http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-debug-modules-4.18.0-511.el8.x86_64.rpm      
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 64.6M  100 64.6M    0     0  13.1M      0  0:00:04  0:00:04 --:--:-- 15.1M
[root@cosa-devsh ~]# curl -kO http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-debug-core-4.18.0-511.el8.x86_64.rpm   
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 71.3M  100 71.3M    0     0  11.9M      0  0:00:05  0:00:05 --:--:-- 13.1M
[root@cosa-devsh ~]# curl -kO http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/Packages/kernel-debug-4.18.0-511.el8.x86_64.rpm     
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  9.9M  100  9.9M    0     0   9.8M      0  0:00:01  0:00:01 --:--:--  9.8M


[root@cosa-devsh ~]# ls
--install                               kernel-debug-core-4.18.0-511.el8.x86_64.rpm     kernel-debug-modules-extra-4.18.0-511.el8.x86_64.rpm
kernel-debug-4.18.0-511.el8.x86_64.rpm  kernel-debug-modules-4.18.0-511.el8.x86_64.rpm

[root@cosa-devsh ~]# rpm-ostree override remove kernel kernel-{core,modules,modules-extra} --install kernel-debug-4.18.0-511.el8.x86_64.rpm --install kernel-debug-core-4.18.0-511.el8.x86_64.rpm --install kernel-debug-modules-4.18.0-511.el8.x86_64.rpm --install kernel-debug-modules-extra-4.18.0-511.el8.x86_64.rpm  
Checking out tree 3821eab... done
No enabled rpm-md repositories.
Importing rpm-md... done
Resolving dependencies... done
Applying 4 overrides and 4 overlays
Processing packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
error: Running %posttrans for kernel-debug-modules: bwrap(/bin/sh): Child process killed by signal 1; run `journalctl -t 'rpm-ostree(kernel-debug-modules.posttrans)'` for more information


[root@cosa-devsh ~]# curl -kO https://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/rpm-ostree/2022.10.117.g52714b51/2.el8/x86_64/rpm-ostree-2022.10.117.g52714b51-2.el8.x86_64.rpm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 3435k  100 3435k    0     0  1257k      0  0:00:02  0:00:02 --:--:-- 1257k
[root@cosa-devsh ~]# curl -kO https://download.eng.bos.redhat.com/brewroot/vol/rhel-8/packages/rpm-ostree/2022.10.117.g52714b51/2.el8/x86_64/rpm-ostree-libs-2022.10.117.g52714b51-2.el8.x86_64.rpm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 54700  100 54700    0     0   160k      0 --:--:-- --:--:-- --:--:--  160k

-bash: Ls: command not found
[root@cosa-devsh ~]# ls
--install                                    kernel-debug-modules-4.18.0-511.el8.x86_64.rpm        rpm-ostree-libs-2022.10.117.g52714b51-2.el8.x86_64.rpm
kernel-debug-4.18.0-511.el8.x86_64.rpm       kernel-debug-modules-extra-4.18.0-511.el8.x86_64.rpm
kernel-debug-core-4.18.0-511.el8.x86_64.rpm  rpm-ostree-2022.10.117.g52714b51-2.el8.x86_64.rpm
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]#  rpm-ostree override replace rpm-ostree-2022.10.117.g52714b51-2.el8.x86_64.rpm rpm-ostree-libs-2022.10.117.g52714b51-2.el8.x86_64.rpm
Checking out tree 3821eab... done
No enabled rpm-md repositories.
Importing rpm-md... done
Resolving dependencies... done
Applying 2 overrides
Processing packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
Freed: 170.4 MB (pkgcache branches: 4)
Upgraded:
  rpm-ostree 2022.10.117.g52714b51-1.el8 -> 2022.10.117.g52714b51-2.el8
  rpm-ostree-libs 2022.10.117.g52714b51-1.el8 -> 2022.10.117.g52714b51-2.el8
Use "rpm-ostree override reset" to undo overrides
Run "systemctl reboot" to start a reboot
[root@cosa-devsh ~]# reboot

[core@cosa-devsh ~]$ sudo -i

[root@cosa-devsh ~]# rpm-ostree override remove kernel kernel-{core,modules,modules-extra}   --install kernel-debug-4.18.0-511.el8.x86_64.rpm   --install kernel-debug-core-4.18.0-511.el8.x86_64.rpm   --install kernel-debug-modules-4.18.0-511.el8.x86_64.rpm   --install kernel-debug-modules-extra-4.18.0-511.el8.x86_64.rpm

Checking out tree 3821eab... done
No enabled rpm-md repositories.
Importing rpm-md... done
Resolving dependencies... done
Applying 6 overrides and 4 overlays
Processing packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Generating initramfs... done
Writing OSTree commit... done
Staging deployment... done
Removed:
  kernel-4.18.0-372.69.1.el8_6.x86_64
  kernel-core-4.18.0-372.69.1.el8_6.x86_64
  kernel-modules-4.18.0-372.69.1.el8_6.x86_64
  kernel-modules-extra-4.18.0-372.69.1.el8_6.x86_64
Added:
  kernel-debug-4.18.0-511.el8.x86_64
  kernel-debug-core-4.18.0-511.el8.x86_64
  kernel-debug-modules-4.18.0-511.el8.x86_64
  kernel-debug-modules-extra-4.18.0-511.el8.x86_64
Use "rpm-ostree override reset" to undo overrides
Run "systemctl reboot" to start a reboot
[root@cosa-devsh ~]# 
[root@cosa-devsh ~]# reboot
Red Hat Enterprise Linux CoreOS 412.86.202308221642-0
  Part of OpenShift 4.12, RHCOS is a Kubernetes native operating system
  managed by the Machine Config Operator (`clusteroperator/machine-config`).

WARNING: Direct SSH access to machines is not recommended; instead,
make configuration changes via `machineconfig` objects:
  https://docs.openshift.com/container-platform/4.12/architecture/architecture-rhcos.html

---
Last login: Thu Aug 24 20:39:42 2023
[core@cosa-devsh ~]$ 
[core@cosa-devsh ~]$ 
[core@cosa-devsh ~]$ uname -r
4.18.0-511.el8.x86_64+debug
[core@cosa-devsh ~]$ 
[core@cosa-devsh ~]$ rpm -qa | grep kernel-debug
kernel-debug-core-4.18.0-511.el8.x86_64
kernel-debug-4.18.0-511.el8.x86_64
kernel-debug-modules-4.18.0-511.el8.x86_64
kernel-debug-modules-extra-4.18.0-511.el8.x86_64
Comment 9 errata-xmlrpc 2023-11-14 15:27:08 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (rpm-ostree bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:6937