Red Hat Bugzilla – Bug 872088
fedup giving grubby backtrace
Last modified: 2013-01-03 02:26:24 EST
Description of problem:
I tried testing fedup on a fresh F18 x86_64 minimal + @base install.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. As root I ran:
# fedup-cli --network 18 --repo fedup=http://tflink.fedorapeople.org/fedup/f18-upgrade/x86_64 --instrepo=fedup
1. After downloading all the f18 packages to be upgraded I get:
setting up system for upgrade
Traceback (most recent call last):
File "/bin/fedup-cli", line 267, in <module>
File "/bin/fedup-cli", line 244, in main
prep_boot(kernel, initrd, bootloader=args.bootloader)
File "/usr/lib/python2.7/site-packages/fedup/download.py", line 330, in prep_boot
File "/usr/lib/python2.7/site-packages/fedup/download.py", line 300, in modify_bootloader
default = bootloader.default_entry()
File "/usr/lib/python2.7/site-packages/fedup/grubby.py", line 74, in default_entry
File "/usr/lib/python2.7/site-packages/fedup/grubby.py", line 50, in get_entry out = self._grubby("--info", index)
File "/usr/lib/python2.7/site-packages/fedup/grubby.py", line 45, in _grubby
return check_output(cmd + [str(a) for a in args], stderr=PIPE)
File "/usr/lib64/python2.7/subprocess.py", line 544, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['grubby', '--info', '-3']' returned non-zero exit status 1
No backtrace and process to proceed to reboot and upgrade. :)
# rpm -q kernel
Created attachment 636479 [details]
(In reply to comment #0)
> subprocess.CalledProcessError: Command '['grubby', '--info', '-3']' returned
> non-zero exit status 1
So what happens when you run "grubby --info -3" manually? Can you attach your grub2.conf?
I get the same error.
grubby --info -3 says: kernel not found
Okay, so '-3' is the index grubby returns when grub2.cfg has:
(for grub1 it'd be '-2').
But even though grubby tells us the index is -3, it doesn't accept --info -3, so we crash.
BUT: we don't need to call bootloader.default_entry() at all - it was just being used for a line in the debugging logs. I'll just drop that code and this will go away.
Should be fixed by https://github.com/wgwoods/fedup/commit/7f2f2f1
Thanks - looks good now.
I guess better to leave this open until fix is pushed to Bodhi.
BTW a workaround seems to be to install another kernel in f17.
eg yum install --enablerepo=updates-testing kernel
fedup-0.7.1-1.fc18 has been submitted as an update for Fedora 18.
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing fedup-0.7.1-1.fc18'
as soon as you are able to.
Please go to the following url:
then log in and leave karma (feedback).
fedup-0.7.1-1.fc17 has been submitted as an update for Fedora 17.
*** Bug 879089 has been marked as a duplicate of this bug. ***
I'm pretty sure this was an issue with the fix not having made it out to the mirrors when Adam tested and filed #879089. We talked about it over IRC it sounds like he was using fedup-0.7-1.fc17 which does not have the needed fix.
I haven't seen this recently and AFAIK, it's fixed. Since there haven't been any other reports, I'm moving this to VERIFIED.
My test with 0.7.1-1 certainly worked. agree with VERIFIED.
fedup-0.7.1-1.fc18 has been pushed to the Fedora 18 stable repository. If problems still persist, please make note of it in this bug report.
fedup-0.7.1-1.fc17 has been pushed to the Fedora 17 stable repository. If problems still persist, please make note of it in this bug report.
I just hit this with fedup-0.7.1-1.fc17 on an F17 x86_64 UEFI install from livecd. I'll attach the debug logs but when I run the grubby command manually, I get:
sudo grubby --add-kernel /boot/upgrade/vmlinuz --title 'System Upgrade' --initrd /boot/upgrade/upgrade.img --args 'systemd.unit=system-upgrade.target' --copy-default
error opening /boot/grub/grub.cfg for read: No such file or directory
The grub.cfg is in /boot/efi/EFI/redhat/ for this machine - not /boot/grub/grub.cfg
Created attachment 653204 [details]
debug log from EFI run
This seems somewhat related to Bug 768106. I really don't know whether Fedora uses saved_entry in grub env or "hardcode" numbers in grub.cfg.
(In reply to comment #17)
> when I run the grubby command
> manually, I get:
> sudo grubby --add-kernel /boot/upgrade/vmlinuz --title 'System Upgrade'
> --initrd /boot/upgrade/upgrade.img --args
> 'systemd.unit=system-upgrade.target' --copy-default
> error opening /boot/grub/grub.cfg for read: No such file or directory
grybby-the-binary defaults to this, upstreams Single True Location of grub.cfg. In fedora we still use the grub2 name transformation and grubby-the-binary thus needs -c to point at the right config file. new-kernel-pkg will do that.
> The grub.cfg is in /boot/efi/EFI/redhat/ for this machine - not
Except for the fact that both paths are slightly different for Fedora:
Yes, it is sad and confusing that we have two different grub.cfg locations: /boot/grub2/grub.cfg and /boot/efi/EFI/fedora/grub.cfg. It is thus no longer possible to give simple and clear instructions on how grub2-mkconfig should be invoked.
It would be much better if /boot/efi/EFI/fedora/grub.cfg only contained a 'search' command and a 'configfile' command that pointed at /boot/grub2/grub.cfg.
Discussed at 2012-12-03 blocker review meeting: http://meetbot.fedoraproject.org/fedora-qa/2012-12-03/f18final-blocker-review-1.2.2012-12-03-17.25.log.txt . We agreed to delay a decision on this until there is a clear action plan rather than a vague 'it needs to be more secure!' feeling, at least.
Sorry, disregard above comment, it was attached to the wrong bug. Correct comment follows:
Discussed at 2012-12-03 blocker review meeting: http://meetbot.fedoraproject.org/fedora-qa/2012-12-03/f18final-blocker-review-1.2.2012-12-03-17.25.log.txt . We agreed to delay a decision on this until we have a clearer understanding of what can cause this bug to occur in 0.7.1 - it seems clear that it is still happening, but we have not clearly determined the cause.
This is a different problem than the original bug, but let's ignore that for now..
(In reply to comment #19)
> (In reply to comment #17)
> > when I run the grubby command
> > manually, I get:
> > sudo grubby --add-kernel /boot/upgrade/vmlinuz --title 'System Upgrade'
> > --initrd /boot/upgrade/upgrade.img --args
> > 'systemd.unit=system-upgrade.target' --copy-default
> > error opening /boot/grub/grub.cfg for read: No such file or directory
Right - fedup doesn't directly touch *any* boot config files; all configuration is done through grubby. So if we're invoking it correctly, and it's not finding the bootloader config, this is a grubby problem.
> grybby-the-binary defaults to this, upstreams Single True Location of
> grub.cfg. In fedora we still use the grub2 name transformation and
> grubby-the-binary thus needs -c to point at the right config file.
> new-kernel-pkg will do that.
Ah, so maybe we *are* invoking it incorrectly. How does new-kernel-pkg get the correct path to the config file?
(In reply to comment #22)
> Ah, so maybe we *are* invoking it incorrectly. How does new-kernel-pkg get
> the correct path to the config file?
It follows/reads /etc/grub2.cfg and /etc/grub2-efi.cfg (and /etc/grub.conf and ...) and patches all the files it can find.
Yeah, so it runs grubby with a bunch of special arguments that differ for each available bootloader it finds. Fun!
I guess this means we're going to need to run new-kernel-pkg instead of grubby, since there's a bunch of special magic locked up in new-kernel-pkg that isn't available otherwise.
If I understand correctly, this means we need to download the images to:
new-kernel-pkg --initrdfile /boot/initramfs-fedup.img \
--banner "System Upgrade" \
--kernel-args "systemd.unit=system-upgrade.target" \
And to remove it,
new-kernel-pkg --remove fedup
Does that seem correct?
FWIW, I don't spot any obvious errors there.
But you should verify that the entry for the new f18 kernel doesn't use the fedup entry for cloning and inherits the extra systemd.unit kernel arg.
Grubby is smart enough to prefer the *default* boot item to the *first* boot item, so as long as the new boot item isn't the default this works fine. We've tested that pretty extensively.
We'll probably also remove the upgrade boot entry once the upgrade has started in the near future. So that shouldn't really be a problem.
Discussed at 2012-12-05 blocker review meeting - http://meetbot.fedoraproject.org/fedora-bugzappers/2012-12-05/f18final-blocker-review-2.2012-12-05-17.01.log.txt . We agreed that we're worried about people still hitting grubby backtraces, but the precise cause(s) haven't yet been identified and we'd like to know how unusual they are before taking any bugs as blockers. We note it might be best to split off at least one separate report here, since the cases we have now aren't the original case.
I tried upgrading with fedup-0.7.2-0.fc17 from koji and ended up with a similar error from 'grubby --default-kernel'.
As will mentioned in c#22, this is a different issue so I filed it as https://bugzilla.redhat.com/show_bug.cgi?id=884696
Any objections to closing this bug, as the case it initially reported seems clearly fixed?
Maybe move it to F18-accepted? If there are still some edge/corner-cases
left then better either to file new bugs for those or leave this open I think.
(In reply to comment #29)
> Any objections to closing this bug, as the case it initially reported seems
> clearly fixed?
No objections from me. I filed a new bug for the exact issue that I was hitting: https://bugzilla.redhat.com/show_bug.cgi?id=884696
If you choose not to close it now, this bug will be closed by bodhi when the fedup-0.7.2 update gets pushed.
Putting it in ON_DEV until the 0.7.2 update is ready.
Let's go ahead and close it out, please file post-0.7 issues separately.
fedup-0.7.2-1.fc17 has been submitted as an update for Fedora 17.
fedup-0.7.2-1.fc17 has been pushed to the Fedora 17 stable repository. If problems still persist, please make note of it in this bug report.