Bug 1261569 - old kernel boots by default after upgrade
old kernel boots by default after upgrade
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: anaconda (Show other bugs)
23
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Anaconda Maintenance Team
Fedora Extras Quality Assurance
AcceptedBlocker https://fedoraproject...
: CommonBugs
Depends On:
Blocks: F23FinalBlocker
  Show dependency treegraph
 
Reported: 2015-09-09 13:22 EDT by Chris Murphy
Modified: 2016-01-08 16:10 EST (History)
18 users (show)

See Also:
Fixed In Version: anaconda-23.19.10-1.fc23
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-10-26 17:01:40 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
screenshot showing wrong kernel selected by default (40.99 KB, image/png)
2015-09-09 13:22 EDT, Chris Murphy
no flags Details
grub.cfg after installation (4.09 KB, text/plain)
2015-09-09 13:54 EDT, Chris Murphy
no flags Details
grub.cfg after kernel upgrade (4.91 KB, text/plain)
2015-09-09 13:55 EDT, Chris Murphy
no flags Details

  None (edit)
Description Chris Murphy 2015-09-09 13:22:37 EDT
Created attachment 1071857 [details]
screenshot showing wrong kernel selected by default

Description of problem: After dnf upgrade, the old kernel still boots by default.


Version-Release number of selected component (if applicable):
grubby-8.40-2.fc23.x86_64

How reproducible:
Always


Steps to Reproduce:
1. Install Fedora-Live-Workstation-x86_64-23_Beta-TC4.iso
2. reboot
3. dnf upgrade
4. reboot

Actual results:

In GRUB menu, old kernel menu entry is the default.

Expected results:

In GRUB menu, new kernel entry should be default.


Additional info:

Anaconda program log shows:
10:33:14,450 INFO program: Running... grub2-install --no-floppy /dev/sda
10:33:15,396 INFO program: Running... grub2-set-default 0
10:33:15,500 INFO program: Running... grub2-mkconfig -o /boot/grub2/grub.cfg


And yet after the first boot of the installed system:

# grub2-editenv list
saved_entry=Fedora (4.2.0-1.fc23.x86_64) 23 (Workstation Edition)

Grubby appears to have changed the grubenv to that kernel version at the time of installation, I'm guessing because grubenv was 0.

If all I do is rerun grub2-mkconfig -o /boot/grub2/grub.cfg to wipe out the grubby modified grub.cfg, the new kernel is now default.
Comment 1 Chris Murphy 2015-09-09 13:54:52 EDT
Created attachment 1071862 [details]
grub.cfg after installation
Comment 2 Chris Murphy 2015-09-09 13:55:05 EDT
Created attachment 1071863 [details]
grub.cfg after kernel upgrade
Comment 3 Chris Murphy 2015-09-09 13:58:51 EDT
Reproduces on i686 baremetal and x86_64 VM. I'm not sure why it's happening now, because it's the same version of grubby since April. The suspicious part is the grubenv contents:

# grub2-editenv list
saved_entry=Fedora (4.2.0-1.fc23.x86_64) 23 (Workstation Edition)
Comment 4 Alexander Todorov 2015-09-10 04:10:15 EDT
Could it be that GRUB_DEFAULT=saved in /etc/default/grub ? Btw I'm seeing the same issue on Rawhide.
Comment 5 Juan Orti 2015-10-03 10:16:25 EDT
I'm experiencing the same bug in F23. The new installed kernels aren't set as default. But in my case, grub2-mkconfig doesn't make a difference.

This is a clean F23-Beta Workstation install with these packages:
grub2-tools-2.02-0.23.fc23.x86_64
grub2-2.02-0.23.fc23.x86_64
grubby-8.40-2.fc23.x86_64
grub2-efi-2.02-0.23.fc23.x86_64
Comment 6 Chris Murphy 2015-10-13 22:15:47 EDT
This is still a bug, with a clean net install of Workstation I got kernel 4.2.1 and on a subsequent Software update 4.2.3 was installed. But GRUB menu defaults to 4.2.1, the newer kernel is not booted by default.
Comment 7 Chris Murphy 2015-10-13 22:23:42 EDT
There is no specific single release criteria that applies, but it's been successfully argued in the past that a new kernel needs to actually boot by default once installed because it could contain security updates and the user may not be aware that the new kernel isn't booting.

Beta criteria for updates:
"The installed system must be able to download and install updates with the default graphical package manager in all release-blocking desktops." So here I'll argue it's not enough that they're installed, they need to actually be used (which for any other package would be true but the kernel is unique in that multiple versions can exist at the same time).

Final criterial for security.
"The release must contain no known security bugs of 'important' or higher impact according to the Red Hat severity classification scale which cannot be satisfactorily resolved by a package update (e.g. issues during installation)." It's uncertain whether this bug can be fixed with an update.
Comment 8 Adam Williamson 2015-10-14 15:03:09 EDT
I can't reproduce this. Fresh install of Final TC1 Workstation live (to get a kernel older than the current stable), 'dnf update kernel', reboot, and it boots the newly-installed kernel by default, not the old one.

The old one *is* listed in /boot/grub2/grubenv , but nevertheless, grub boots the new one.
Comment 9 Adam Williamson 2015-10-14 17:41:14 EDT
Can't reproduce going from Beta to current, either.
Comment 10 Chris Murphy 2015-10-14 17:44:02 EDT
(In reply to awilliam@redhat.com from comment #8)
Can you attach or post contents of /etc/default/grub ? I wonder if live vs net have different settings that causes this; clearly in your case grubenv is ignored while mine is honored. I'd check this myself but I have 150KB/s Internet in BFE mountains for another couple days.
Comment 11 Adam Williamson 2015-10-14 18:40:44 EDT
Hmm, but I can reproduce it with a UEFI install. Not sure what the difference would be there, though. Indeed, install Final TC1 Workstation on UEFI system, 'dnf update kernel', reboot, and you get the TC1 kernel, not the updated kernel.
Comment 12 Adam Williamson 2015-10-14 18:48:46 EDT
I don't think /etc/default/grub *does* anything unless you run grub2-mkconfig , does it? grubby doesn't care about it at all. Still, I checked: /etc/default/grub is exactly the same in both BIOS and UEFI installs.
Comment 13 Adam Williamson 2015-10-14 19:35:31 EDT
In an F22 install, the saved entry in grubenv is updated when I update the kernel:

[root@localhost test]# grub2-editenv /boot/efi/EFI/fedora/grubenv list
saved_entry=Fedora (4.0.4-301.fc22.x86_64) 22 (Twenty Two)
[root@localhost test]# uname -r
4.0.4-301.fc22.x86_64
[root@localhost test]# dnf update kernel
...
  Installing  : kernel-core-4.1.10-200.fc22.x86_64
...
Complete!
[root@localhost test]# grub2-editenv /boot/efi/EFI/fedora/grubenv list
saved_entry=Fedora (4.1.10-200.fc22.x86_64) 22 (Twenty Two)

so, something's definitely broken there, F23 grubby is not updating the saved entry when it updates the kernel...
Comment 14 Adam Williamson 2015-10-14 22:01:23 EDT
So here's a thing:

[root@localhost test]# rpm -q --scripts kernel-core-4.2.3-300.fc23.x86_64
...
posttrans scriptlet (using /bin/sh):
/bin/kernel-install add 4.2.3-300.fc23.x86_64 /lib/modules/4.2.3-300.fc23.x86_64/vmlinuz || exit $?

[root@localhost test]# sh -x /bin/kernel-install add 4.2.3-300.fc23.x86_64 /lib/modules/4.2.3-300.fc23.x86_64/vmlinuz
/bin/kernel-install: line 35: syntax error near unexpected token `<'
/bin/kernel-install: line 35: `    readarray -t files < <('

seems significant.
Comment 15 Adam Williamson 2015-10-14 22:07:21 EDT
oh, never mind that - it fails in 'sh -x' because 'readarray' is a bashism. But still, there's a mystery here:

[root@localhost test]# grub2-editenv list
saved_entry=Fedora (4.2.0-300.fc23.x86_64) 23 (Workstation Edition)
[root@localhost test]# dnf update kernel
...
  Installing  : kernel-core-4.2.3-300.fc23.x86_64                           1/3 
[root@localhost test]# grub2-editenv list
saved_entry=Fedora (4.2.0-300.fc23.x86_64) 23 (Workstation Edition)
...
[root@localhost test]# /bin/kernel-install add 4.2.3-300.fc23.x86_64 /lib/modules/4.2.3-300.fc23.x86_64/vmlinuz
[root@localhost test]# grub2-editenv list
saved_entry=Fedora (4.2.3-300.fc23.x86_64) 23 (Workstation Edition)

so somehow when the kernel is first installed, the kernel-install (which calls new-kernel-pkg and ultimately grubby) doesn't updated the saved entry, but running it manually afterwards does...
Comment 16 Adam Williamson 2015-10-14 23:09:43 EDT
Aha! I think we may have us a Python 3 issue. /etc/sysconfig/kernel on F23:

# UPDATEDEFAULT specifies if new-kernel-pkg should make
# new kernels the default
UPDATEDEFAULT=yes

# DEFAULTKERNEL specifies the default kernel package type
DEFAULTKERNEL=b'kernel-core'

/etc/sysconfig/kernel on F22:

# UPDATEDEFAULT specifies if new-kernel-pkg should make
# new kernels the default
UPDATEDEFAULT=yes

# DEFAULTKERNEL specifies the default kernel package type
DEFAULTKERNEL=kernel-core

spot the difference. Looks like good ol' py2 vs. py3 string types.
Comment 17 Adam Williamson 2015-10-15 01:10:43 EDT
https://github.com/rhinstaller/anaconda/pull/409
Comment 18 Adam Williamson 2015-10-15 01:15:24 EDT
I tested that; it works for me. In both BIOS and UEFI cases it causes grubenv to be correctly updated, and it does indeed make the new kernel the default boot option in the UEFI case, for me.

https://www.happyassassin.net/updates/1261569.0.img is an updates.img built against anaconda-23.19.6-1.fc23 , which is the version in Final TC6 and TC9, for others to test.
Comment 19 Adam Williamson 2015-10-15 16:32:17 EDT
Discussed at 2015-10-15 blocker review meeting: https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2015-10-15/f23-blocker-review.2015-10-15-20.11.log.txt . Accepted as a blocker per criterion "The installed system must be able to download and install updates with the default graphical package manager in all release-blocking desktops." As we have in the past for a similar issue, we held that kernel updates aren't properly 'installed' if the updated kernel is not the default boot choice, as intended: the practical impact is bad on remote-managed servers where the admin doesn't usually see the boot screen, they expect to be able to simply apply updates and reboot the system to update to the newest kernel.
Comment 20 Fedora Update System 2015-10-20 14:25:01 EDT
anaconda-23.19.8-1.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-cad91ea1ae
Comment 21 Fedora Update System 2015-10-20 17:56:34 EDT
anaconda-23.19.8-1.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update anaconda'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-cad91ea1ae
Comment 22 Fedora Update System 2015-10-21 14:32:36 EDT
python-blivet-1.12.8-1.fc23 anaconda-23.19.9-1.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-da7ada8825
Comment 23 Kamil Páral 2015-10-22 07:28:09 EDT
Fixed for me with RC2.
Comment 24 Christian Stadelmann 2015-10-22 17:52:04 EDT
Looks like this issue won't be fixed for already installed F23 installations. Is that correct?
How do I manually fix this issue for an existing F23 installation?
Comment 25 Andrew Dunn 2015-10-22 18:44:00 EDT
Would like to know the same as comment 24.
Comment 26 Adam Williamson 2015-10-22 19:50:41 EDT
"Looks like this issue won't be fixed for already installed F23 installations. Is that correct?"

Yup.

"How do I manually fix this issue for an existing F23 installation?"

Edit /etc/sysconfig/kernel and fix the line, as per #c16. That *should* be all that's needed, though I haven't checked. Kernels installed after that point should become the default.
Comment 27 Fedora Update System 2015-10-24 08:09:03 EDT
anaconda-23.19.10-1.fc23, python-blivet-1.12.8-1.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update anaconda python-blivet'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-da7ada8825
Comment 28 Kamil Páral 2015-10-26 08:34:16 EDT
Verified with RC3 (on UEFI).
Comment 29 Fedora Update System 2015-10-26 17:00:55 EDT
anaconda-23.19.10-1.fc23, python-blivet-1.12.8-1.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.
Comment 30 teppot 2015-11-05 03:49:05 EST
(In reply to awilliam@redhat.com from comment #26)
> Edit /etc/sysconfig/kernel and fix the line, as per #c16. That *should* be
> all that's needed, though I haven't checked. Kernels installed after that
> point should become the default.

I can't figure out what I need to do here. Also, are all F23 pre-release testers expected to find this bug report on their own?
Comment 31 Kamil Páral 2015-11-05 08:21:32 EST
(In reply to teppot from comment #30)
> I can't figure out what I need to do here. 

I believe you need to change
DEFAULTKERNEL=b'kernel-core'
to
DEFAULTKERNEL=kernel-core

> Also, are all F23 pre-release
> testers expected to find this bug report on their own?

We could add this issue to https://fedoraproject.org/wiki/Common_F23_bugs since it requires manually intervention. I don't have better ideas.
Comment 32 Adam Williamson 2015-11-05 12:06:59 EST
There isn't really anything better we can do, I don't think it's a good idea to ship an update that goes around sed'ing people's config files. As a pre-release tester you are expected to be able to figure stuff like this out, generally speaking - there are enough references to the bug that it should be reasonably simple to google.
Comment 33 c72578 2015-11-20 06:06:22 EST
(In reply to awilliam@redhat.com from comment #26)
> "Looks like this issue won't be fixed for already installed F23
> installations. Is that correct?"
> 
> Yup.
> 
> "How do I manually fix this issue for an existing F23 installation?"
> 
> Edit /etc/sysconfig/kernel and fix the line, as per #c16. That *should* be
> all that's needed, though I haven't checked. Kernels installed after that
> point should become the default.

In addition to the modification of /etc/sysconfig/kernel, the default entry should be set in GRUB, e.g. according to:

https://fedoraproject.org/wiki/GRUB_2#Setting_default_entry

Doing this, the default entry is set immediately and security is ensured.
Otherwise it would take until there is the next update to the kernel ...
Comment 34 Chris Murphy 2015-11-20 13:59:28 EST
There is still something screwy going on with grubenv, where "grub2-editenv list" after a recent kernel update shows the previous kernel version. I had set this to 0 previously, but something is changing it, and changing it incorrectly.
Comment 35 Kaan Cappon 2016-01-08 16:10:26 EST
I had the same error.

"cat /etc/system-release fedora release 23 but uname -r fc22"

I have two disk on my laptop and others Linux distributions. The boot sequence of the two discs was generated by other Linux distributions.
I started on the distribution that generated the file, and I restarted the laptop. Everything is in order.
Hope this can help.
Best regards.
Kaan.

Note You need to log in before you can comment on or make changes to this bug.