Description of problem: After using rpm-ostree upgrade, the resulting system doesn't reboot either tree because the grub.cfg contains the wrong root=UUID. Instead of containing the UUID for the actual root fs, it contains the UUID for the OS it was originally installed to. Version-Release number of selected component (if applicable): Fedora-Cloud_Atomic-x86_64-23.iso rpm-ostree-2015.9-2.fc23.x86_64 ostree-2015.9-1.fc23.x86_64 How reproducible: Once, only one new tree to test right now Steps to Reproduce: 1. rpm-ostree upgrade 2. reboot either 23 or 23.29 trees Actual results: Boot fails because root=UUID=24e16099-88fc-4a84-96d8-eabea2cbef82 specified in the grub.cfg doesn't exist. Since the original grub.cfg is wiped out in the course of the ugprade, both trees have the wrong UUID, so rollback doesn't work. Expected results: Should boot. The grub.cfg should have root=UUID=8b0c4840-4fc7-4782-a4c0-25fec8a40dd4. Additional info: Normally I'd use 'bash -x grub2-mkconfig' to figure out why/how it's pulling in wrong information like this. But that command, and even "grub2-mkconfig" as root in a normal shell produces a menu entry free configuration - nothing is listed under the 15_ostree header at all. Is ostree/rpm-ostree running grub2-mkconfig is some kind of chroot or other environment that'd cause it to produce different results than directly calling that script from the command line?
Created attachment 1104217 [details] grub.cfg post upgrade grub.cfg created via 'rpm-ostree upgrade' (presumably calling grub2-mkconfig) This file has the wrong root=UUID= value as boot param.
Created attachment 1104218 [details] bash -x grub2-mkconfig This is the output from 'bash -x grub2-mkconfig' which shows it's aware of the correct UUID for root. But it doesn't contain the menu entries for some reason, as if there's a difference in a regular shell environment calling grub2-mkconfig, and rpm-ostree calling it. (?) Weird. Makes this harder to troubleshoot.
OK so I'm finding the wrong UUID is in /boot/loader.1/entries/*.conf which are then pulled into the grub.cfg by grub2-mkconfig. Now the question is what creates those conf files? And does is just copy root=UUID= from an earlier conf file, or does it determine it some other way? Normally grub2-mkconfig determines this via grub2-probe, but ostree/rpm-ostree seems to inhibit that in favor of it's own determination of the root fs UUID.
This is an Anaconda install? I'll take a look.
Oh, this is for your clone-installed-system-to-btrfs thing? Yeah, so OSTree indeed owns the bootloader config and propagates the whole kernel command line forward from the loader config files. If you change your clone script to edit the UUID in the loader files, that should fix it.
The clone script was just btrfs send/receive, there isn't that much to clean up. Manually I modified ostree-fedora-atomic-0.conf and ostree-fedora-atomic-1.conf with the correct volume UUID and the problem is fixed. It's a really minor regression that grub isn't permitted to insert the correct root=UUID= as it's really unlikely the UUID changes from installation. The cloning is a work around for bug 1289752.
This message is a reminder that Fedora 23 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 23. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '23'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 23 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 23 changed to end-of-life (EOL) status on 2016-12-20. Fedora 23 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.