So, this was literally the behavior requested in https://bugzilla.redhat.com/show_bug.cgi?id=957681 , but I guess the debug->systemd.debug change that happens elsewhere never happened here.  Nevertheless, I'm going to just use this bz to change it to the expanded version.

I.e. "systemd.log_level=debug systemd.log_target=kmsg"

It seems MAKEDEBUG=yes in /etc/sysconfig/kernel by default. Is it intentional?

Changing to 'no' and 'sudo grub2-mkconfig -o /boot/grub2/grub.cfg' removes 'systemd.debug' from grub and messages from logs.

(In reply to Eugene Lobko from comment #3)

> It seems MAKEDEBUG=yes in /etc/sysconfig/kernel by default. Is it
> intentional?

I guess it is, as this option seems to be checked in `new-kernel-pkg'. There it seems to control whether the debug entry (which includes `systemd.debug' option) is created at all in the boot loader config.

> Changing to 'no' and 'sudo grub2-mkconfig -o /boot/grub2/grub.cfg' removes
> 'systemd.debug' from grub and messages from logs.

Indeed it does, but disabling `MAKEDEBUG' should merely disable the debug entry creation. So this is a side effect -- the non-debug entry be created regardless and should never contain `systemd.debug'.

I have also noticed other options being copied from other kernel's entries to the new kernel's entry. Which looks like the effect of `--copy-default' option being passed to `grubby' and this might be the root cause of this bug.

(In reply to Peter Jones from comment #2)

> So, this was literally the behavior requested in
> https://bugzilla.redhat.com/show_bug.cgi?id=957681
> {...}

What is bug 957681 about? Access to it is restricted.

*** Bug 1226343 has been marked as a duplicate of this bug. ***

"systemd.log_level=debug systemd.log_target=kmsg" is added to the kernel command line for menu entries "with debugging". Non-debugging entries don't have debugging enabled:

menuentry 'Red Hat Enterprise Linux Server (3.10.0-313.el7.x86_64) 7.1 (Maipo)' --class fedora --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gn
ulinux-3.10.0-229.el7.x86_64-advanced-500512a3-dae4-4572-8b7b-21ee9268a2fd' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_msdos
        insmod xfs
        set root='hd0,msdos1'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint='hd0,msdos1'  c6fb26fc-e637-461f-86f7-aa8581085000
        else
          search --no-floppy --fs-uuid --set=root c6fb26fc-e637-461f-86f7-aa8581085000
        fi
        linux16 /vmlinuz-3.10.0-313.el7.x86_64 root=/dev/mapper/rhel-root ro rd.lvm.lv=rhel/swap crashkernel=auto rd.lvm.lv=rhel/root rhgb quiet LANG=en_US.UTF-8
        initrd16 /initramfs-3.10.0-313.el7.x86_64.img
}
menuentry 'Red Hat Enterprise Linux Server (3.10.0-313.el7.x86_64) 7.1 (Maipo) with debugging' --class fedora --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-229.el7.x86_64-advanced-500512a3-dae4-4572-8b7b-21ee9268a2fd' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_msdos
        insmod xfs
        set root='hd0,msdos1'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint='hd0,msdos1'  c6fb26fc-e637-461f-86f7-aa8581085000
        else
          search --no-floppy --fs-uuid --set=root c6fb26fc-e637-461f-86f7-aa8581085000
        fi
        linux16 /vmlinuz-3.10.0-313.el7.x86_64 root=/dev/mapper/rhel-root ro rd.lvm.lv=rhel/swap crashkernel=auto rd.lvm.lv=rhel/root rhgb quiet LANG=en_US.UTF-8 systemd.log_level=debug systemd.log_target=kmsg
        initrd16 /initramfs-3.10.0-313.el7.x86_64.img

Tested with grubby-8.28-14.el7.

Moving to VERIFIED.

Jan, I was unable to match your test:

I tested the latest version of this package against a RHEL 7.1 system per a request from mgmt.

RHEL 7.1 system running kernel-3.10.0-229.11.1.el7.x86_64 -->

Ran a yum update to obtain new grub2 packages from this RHBA: https://rhn.redhat.com/errata/RHBA-2015-1784.html
This was to verify that it was unrelated to menu-ordering in grub2.
Installed kernel-3.10.0-229.14.1.el7.x86_64, same issue with menu listings.
Removed kernel-3.10.0-229.14.1.el7.x86_64, ensured menu was back to normal with original kernel

Installed grubby-8.28-16.el7.x86_64.rpm obtained from brew.

Re-installed kernel-3.10.0-229.14.1.el7.x86_64 and ended up with same results. kernel boot lines contained default line and debug line (due to MAKEDEBUG=yes), but BOTH lines contain the debug options. Although they are no longer 'systemd.debug', they are: 'systemd.log_level=debug systemd.log_target=kmsg'

grub2.cfg as below for these autogenerated entries upon kernel installation:

### BEGIN /etc/grub.d/10_linux ###
menuentry 'Red Hat Enterprise Linux Server (3.10.0-229.14.1.el7.x86_64) 7.1 (Maipo)' --class fedora --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-229.11.1.el7.x86_64-advanced-84b4e488-7936-4485-ad08-b97664ddb26e' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_msdos
        insmod xfs
        set root='hd0,msdos1'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint='hd0,msdos1'  8cd9c0a8-1c86-43b1-94da-840f4b845db5
        else
          search --no-floppy --fs-uuid --set=root 8cd9c0a8-1c86-43b1-94da-840f4b845db5
        fi
        linux16 /vmlinuz-3.10.0-229.14.1.el7.x86_64 root=/dev/mapper/rhel_unused-root ro rd.lvm.lv=rhel_unused/swap rd.lvm.lv=rhel_unused/root rhgb quiet systemd.log_level=debug systemd.log_target=kmsg LANG=en_US.UTF-8
        initrd16 /initramfs-3.10.0-229.14.1.el7.x86_64.img
}
menuentry 'Red Hat Enterprise Linux Server (3.10.0-229.14.1.el7.x86_64) 7.1 (Maipo) with debugging' --class fedora --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-229.11.1.el7.x86_64-advanced-84b4e488-7936-4485-ad08-b97664ddb26e' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_msdos
        insmod xfs
        set root='hd0,msdos1'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint='hd0,msdos1'  8cd9c0a8-1c86-43b1-94da-840f4b845db5
        else
          search --no-floppy --fs-uuid --set=root 8cd9c0a8-1c86-43b1-94da-840f4b845db5
        fi
        linux16 /vmlinuz-3.10.0-229.14.1.el7.x86_64 root=/dev/mapper/rhel_unused-root ro rd.lvm.lv=rhel_unused/swap rd.lvm.lv=rhel_unused/root rhgb quiet systemd.log_level=debug systemd.log_target=kmsg LANG=en_US.UTF-8
        initrd16 /initramfs-3.10.0-229.14.1.el7.x86_64.img
}

If you need further information, please let me know.

Thanks Craig for catching this, this issue is not completely fixed, I can reproduce it as well. Moving back to ASSIGNED based on comment 9.

(In reply to Craig Donnelly from comment #9)
> Re-installed kernel-3.10.0-229.14.1.el7.x86_64 and ended up with same
> results. kernel boot lines contained default line and debug line (due to
> MAKEDEBUG=yes), but BOTH lines contain the debug options. Although they are
> no longer 'systemd.debug', they are: 'systemd.log_level=debug
> systemd.log_target=kmsg'

When you say you re-installed the kernel, what specifically did you do?  Can you remove it and add it again, and verify that the config file is reasonable in between?

(In reply to Peter Jones from comment #11)
> When you say you re-installed the kernel, what specifically did you do?  Can
> you remove it and add it again, and verify that the config file is
> reasonable in between?

I have several RHEL 7.1 test systems available, this one in particular I cleaned up for the test, by rebuilding the grub2.cfg.

Basically, i ran grub2-mkconfig to regenerate the file, rebooted to verify I got rid of the messed up debug lines, installed the new kernel, rebooted to verify i had messed up debug lines, then removed that kernel: This reset my grub2.cfg back to the normal lines from before the kernel was installed. Everything was normal.

I downloaded and installed grubby-8.28-16.el7.x86_64.rpm locally, rebooted.
I then installed the new kernel again, and checked the list. Still messed up.

Is there a particular way you would like me to test? This seems pretty solid to me.

TLDR; when removing the kernel the first time, the grub2.cfg was solid. Was correct.

Can you actually attach the intermediate grub2.cfg , from between the remove and the install?  If the debug entry isn't there, I don't see where this problem is coming from.

Peter,

I am attaching the grub2.cfg from the time before the new kernel is installed, and after it is installed. This is all with the current grubby pkg present.

Let me know if you would like to see anything else.

Created attachment 1081475 [details]
Prior to new kernel install with latest grubby from brew.

Ran a 'yum remove kernel-3.10.0-229.14.1.el7.x86_64.rpm' to eliminate the broken kernel entry. No need for a grub2-mkconfig.

Created attachment 1081476 [details]
After installing new kernel, with latest grubby pkg.

After running 'yum install kernel-3.10.0-229.14.1.el7.x86_64.rpm'. Entry automatically generated. No grub2 cmds.

Created attachment 1082051 [details]
new-kernel-pkg that may behave better?

I'm thinking what's happening is that somehow we're picking up the arguments from the debug entry instead of the older kernel's entry when we're creating the new kernel's non debug entry.

Attached is a version of new-kernel-pkg that does them in the opposite order instead.  Can you sub it in and see if that helps?

Created attachment 1082298 [details]
grub.cfg

Peter, the new-kernel-pkg works fine for me. There is one change - the debug boot item is above the non-debug boot item in the bootloader. The non-debug is selected as the default boot item as expected.

Peter, using that updated script, I received the same result as Jan. The system installed the new kernel, and grub2.cfg generated the debug line and default line separately, debug options were only attached to the debugging entry in grub2, they were just in the wrong order as Jan reported. It did however default to the correct entry, the non-debugging one (i.e. Line 2).

grub2.cfg attached.

Created attachment 1082981 [details]
grub2.cfg w/new-kernel-pkg update

Hi Peter,

If this is a known issue for 7.2 that should be documented in the Release Notes, then please switch the Doc Type to Known Issue (David switched it already and Bugzilla turned it back...) and provide information about what the problem is and what the user can do to fix it in the Doc Text field.

Thanks!

Two scenarios were tested:
1) only regular kernel installed
2) both kernel and kernel-debug packages installed

In both cases, RHEL-7.1 was installed, then the grubby package was updated to grubby-8.28-17.el7. After that, the kernel (and kernel-debug) packages from the latest compose were installed.

Scenario 1) passed the test and only the "with debugging" boot item contained additional systemd debug options.

Scenario 2) failed, since all new boot items contained additional systemd debug options.

Since scenario 2) failed, moving to ASSIGNED. grub.cfg before and after will be attached.

Created attachment 1087612 [details]
grub.cfg before

Created attachment 1087613 [details]
grub.cfg after

with grubby-8.28-17.el7

Hey,

Having the same issue, I looked into new-kernel-pkg source. See bug 1289174 for details (sadly created it before finding this one).

Cheers, Phil

*** Bug 1288591 has been marked as a duplicate of this bug. ***

Created attachment 1162253 [details]
RHEL 7.2 Steps - Cannot Reproduce

I tried to reproduce this in RHEL 7.2 - it seems to work just fine.

Here is a high level outline of what I did:

-- Install RHEL7.2 from local repo release.
-- Subscribe to RHN
-- Started logging at this point
-- check grubby version - 8.28.17
-- add 'set -x' to /usr/sbin/new-kernel-pkg
-- add MAKEDEBUG=yes to /etc/sysconfig/kernel
-- yum install -y kernel-debug
-- yum update -y kernel

The systemd.debug statements are only where I expected.

The compressed file I have attached contains the details of the work done to try to reproduce this on RHEL 7.2 including the /etc/os-release to verify the version I was using.

anaconda-ks.cfg - kickstart file generated by minimal install
final_packages.txt - final list of installed packages
grubby.log.debugkernel - /var/log/grubby after installing kernel-debug
grubby.log.newkernel - /var/log/grubby after updating kernel
grub.cfg.debugkernel - /boot/grub2/grub.cfg after installing kernel-debug
grub.cfg.install - /boot/grub2/grub.cfg after initial installation
grub.cfg.newkernel - /boot/grub2/grub.cfg after updating kernel
grubenv.debugkernel - /boot/grub2/grubenv after installing kernel-debug
grubenv.install - /boot/grub2/grubenv after initial installation
grubenv.newkernel - /boot/grub2/grubenv after updating kernel
initial_packages.txt - the installed packages on the clean system
keystrokes.log - log of my work except for where I put in my password to subscribe to RHN
os-release - the /etc/os-release file from the virtual machine

Robert,
I wasn't able to reproduce it using RHEL-7.2 with grubby-8.28-17.el7, so I suppose this is not an issue any more.
I've asked Phil to provide some details in bug 1289174 to see if we can safely close this bug or not.

Hi,

(In reply to Jan Stodola from comment #30)
> Robert,
> I wasn't able to reproduce it using RHEL-7.2 with grubby-8.28-17.el7, so I
> suppose this is not an issue any more.
> I've asked Phil to provide some details in bug 1289174 to see if we can
> safely close this bug or not.

As pointed out in [1], I can't reproduce it anymore with build 17 as well.

Thanks, Phil

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1289174

*** Bug 1289174 has been marked as a duplicate of this bug. ***

Given the confirmations in comment #30 and comment #31 I am closing this issue as resolved in the current release.