Bug 1367792

Summary: After kernel update, zipl on s390x defaults to booting "_with_debugging" kernel
Product: Red Hat Enterprise Linux 7 Reporter: Andrew Price <anprice>
Component: grubbyAssignee: Peter Jones <pjones>
Status: CLOSED WONTFIX QA Contact: Release Test Team <release-test-team-automation>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.3CC: anprice, dhorak, herrold, jonsulman, jstodola, mueller, radoslaw.piliszek, sbueno, schoudha
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-12-15 07:44:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Andrew Price 2016-08-17 13:28:13 UTC
Description of problem:

Some time between RHEL7.2 and current RHEL7.3 candidate, updating the kernel package on an s390x system, that boots with zipl, started switching the default from booting the normal boot menu entry to booting the _with_debugging menu entry.

Version-Release number of selected component (if applicable):

grubby-8.28-18.el7.s390x

How reproducible:

100%

Steps to Reproduce:
1. Install RHEL7.3 development on s390x virt
2. Make sure the default boot entry in /etc/zipl.conf is not a debugging entry
3. Update the kernel package
4. Observe that the default boot entry has become a _with_debugging entry

Actual results:

# zipl
Using config file '/etc/zipl.conf'
Building bootmap in '/boot'
Building menu 'zipl-automatic-menu'
Adding #1: IPL section '3.10.0-489.el7.s390x'
Adding #2: IPL section '3.10.0-489.el7.s390x_with_debugging' (default)
Adding #3: IPL section '3.10.0-481.el7.s390x'
Adding #4: IPL section '3.10.0-481.el7.s390x_with_debugging'
Adding #5: IPL section '3.10.0-462.el7.s390x'
Preparing boot device: dasda (0100).
Done.

Expected results:

# zipl
Using config file '/etc/zipl.conf'
Building bootmap in '/boot'
Building menu 'zipl-automatic-menu'
Adding #1: IPL section '3.10.0-489.el7.s390x' (default)
Adding #2: IPL section '3.10.0-489.el7.s390x_with_debugging'
Adding #3: IPL section '3.10.0-481.el7.s390x'
Adding #4: IPL section '3.10.0-481.el7.s390x_with_debugging'
Adding #5: IPL section '3.10.0-462.el7.s390x'
Preparing boot device: dasda (0100).
Done.


Additional info:

How my zipl.conf looks after the kernel update:

[defaultboot]
defaultauto
prompt=1
timeout=5
default=3.10.0-489.el7.s390x_with_debugging
target=/boot
[3.10.0-489.el7.s390x]
        image=/boot/vmlinuz-3.10.0-489.el7.s390x
        parameters="crashkernel=auto rd.lvm.lv=rhel_qe-c02-m02/root rd.dasd=0.0.0100 undefined root=/dev/mapper/rhel_qe--c02--m02-root rd.lvm.lv=rhel_qe-c02-m02/swap rd.dasd=0.0.0101 LANG=en_US.UTF-8"
        ramdisk=/boot/initramfs-3.10.0-489.el7.s390x.img
[3.10.0-489.el7.s390x_with_debugging]
        image=/boot/vmlinuz-3.10.0-489.el7.s390x
        parameters="crashkernel=auto rd.lvm.lv=rhel_qe-c02-m02/root rd.dasd=0.0.0100 undefined root=/dev/mapper/rhel_qe--c02--m02-root rd.lvm.lv=rhel_qe-c02-m02/swap rd.dasd=0.0.0101 LANG=en_US.UTF-8 systemd.log_level=debug systemd.log_target=kmsg"
        ramdisk=/boot/initramfs-3.10.0-489.el7.s390x.img
[3.10.0-481.el7.s390x]
        image=/boot/vmlinuz-3.10.0-481.el7.s390x
        parameters="crashkernel=auto rd.lvm.lv=rhel_qe-c02-m02/root rd.dasd=0.0.0100 undefined root=/dev/mapper/rhel_qe--c02--m02-root rd.lvm.lv=rhel_qe-c02-m02/swap rd.dasd=0.0.0101 LANG=en_US.UTF-8"
        ramdisk=/boot/initramfs-3.10.0-481.el7.s390x.img
[3.10.0-481.el7.s390x_with_debugging]
        image=/boot/vmlinuz-3.10.0-481.el7.s390x
        parameters="crashkernel=auto rd.lvm.lv=rhel_qe-c02-m02/root rd.dasd=0.0.0100 undefined root=/dev/mapper/rhel_qe--c02--m02-root rd.lvm.lv=rhel_qe-c02-m02/swap rd.dasd=0.0.0101 LANG=en_US.UTF-8 systemd.log_level=debug systemd.log_target=kmsg"
        ramdisk=/boot/initramfs-3.10.0-481.el7.s390x.img
[3.10.0-462.el7.s390x]
        image=/boot/vmlinuz-3.10.0-462.el7.s390x
        parameters="crashkernel=auto rd.lvm.lv=rhel_qe-c02-m02/root rd.dasd=0.0.0100 undefined root=/dev/mapper/rhel_qe--c02--m02-root rd.lvm.lv=rhel_qe-c02-m02/swap rd.dasd=0.0.0101 LANG=en_US.UTF-8"
        ramdisk=/boot/initramfs-3.10.0-462.el7.s390x.img

Comment 1 Peter Jones 2016-09-08 17:16:50 UTC
Are you sure this was done with grubby-8.28-18.el7.s390x , and that wasn't merely the version installed after the rpm update was finished?  To me this looks like something I'd expect -17.el7 to do, but -18.el7 should have fixed.

That is, if you upgrade grubby in its own transaction before updating anything else, does the error still occur?

Comment 2 Jan Stodola 2016-09-09 13:36:07 UTC
Reproduced with grubby-8.28-18.el7.s390x (compose RHEL-7.3-20160901.1):

[root@rtt7 ~]# zipl
Using config file '/etc/zipl.conf'
Building bootmap in '/boot'
Building menu 'zipl-automatic-menu'
Adding #1: IPL section 'linux' (default)
Adding #2: IPL section 'linux-0-rescue-28b817321aa14230b612ddd82169abe9'
Preparing boot device: dasda (3627).
Done.
[root@rtt7 ~]# echo "MAKEDEBUG=yes" >> /etc/sysconfig/kernel
[root@rtt7 ~]# yum install ./kernel-3.10.0-501.el7.s390x.rpm
[root@rtt7 ~]# zipl
Using config file '/etc/zipl.conf'
Building bootmap in '/boot'
Building menu 'zipl-automatic-menu'
Adding #1: IPL section '3.10.0-501.el7.s390x'
Adding #2: IPL section '3.10.0-501.el7.s390x_with_debugging' (default)
Adding #3: IPL section 'linux'
Adding #4: IPL section 'linux-0-rescue-28b817321aa14230b612ddd82169abe9'
Preparing boot device: dasda (3627).
Done.
[root@rtt7 ~]# rpm -q grubby
grubby-8.28-18.el7.s390x
[root@rtt7 ~]#

Comment 3 Jan Stodola 2016-09-09 13:43:33 UTC
A note - to reproduce it, it is necessary to add MAKEDEBUG=yes to /etc/sysconfig/kernel. This option is not there by default, so without this change this bug problem won't appear.

Comment 4 Samantha N. Bueno 2016-09-09 21:04:09 UTC
(In reply to Jan Stodola from comment #3)
> A note - to reproduce it, it is necessary to add MAKEDEBUG=yes to
> /etc/sysconfig/kernel. This option is not there by default, so without this
> change this bug problem won't appear.

Based on this information -- which indicates to me that default behavior works fine -- this is not a blocker, so I'm deferring this to 7.4 planning.

Clearing pjones' needinfo request as well since jstodola tested it with -18 version.

Comment 5 Jan Stodola 2016-09-16 09:05:01 UTC
BTW, it's not only on s390x with zipl, but also on x86_64, see bug 1285601#c5.

Comment 6 Thomas Mueller 2016-11-09 15:45:26 UTC
I have this issue with x86_64 and grub2 since upgrade to RHEL 7.3.

The /boot/grub2/grub.cfg entries are now in correct order (first entry: normal boot, second entry: with debugging, RHEL 7.2 had this in reverse) but grub2-editenv list show saved_entry with `... with debugging`.

Comment 7 Thomas Mueller 2016-11-09 15:51:48 UTC
(In reply to Thomas Mueller from comment #6)
> I have this issue with x86_64 and grub2 since upgrade to RHEL 7.3.
> 
> The /boot/grub2/grub.cfg entries are now in correct order (first entry:
> normal boot, second entry: with debugging, RHEL 7.2 had this in reverse) but
> grub2-editenv list show saved_entry with `... with debugging`.

/etc/syconfig/kernel has MAKEDEBUG=yes altough i never added this. interestingly 

rm /etc/sysconfig/kernel
rpm --erase --nodeps grubby
yum install grubby

does not restore the file. It seems it is added somewhere while install?

Comment 8 Radosław Piliszek 2017-02-16 08:56:53 UTC
The same happens on x86_64 with GRUB.

New kernels default to debug when MAKEDEBUG=yes which in turn got installed with some update but did not get reverted.

The workaround is to remove MAKEDEBUG=yes from /etc/sysconfig/kernel and set correct default manually. Next kernel update will not break it unless MAKEDEBUG=yes appears magically again.

Please fix new-kernel-pkg not to default to debug.

Comment 9 Radosław Piliszek 2017-02-16 09:12:59 UTC
More info:
Documentation states:
'MAKEDEBUG specifies if new-kernel-pkg should create non-default "debug" entries for new kernels.'
So they should be non-default by design, not default. This is clearly a nasty bug.

Comment 12 RHEL Program Management 2020-12-15 07:44:39 UTC
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.