Bug 872088

Summary: fedup giving grubby backtrace
Product: [Fedora] Fedora Reporter: Jens Petersen <petersen>
Component: fedupAssignee: Will Woods <wwoods>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 17CC: awilliam, collura, mads, mclasen, mishu, robatino, tflink, wwoods
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-12-12 05:45:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 752661    
Attachments:
Description Flags
debuglog
none
debug log from EFI run none

Description Jens Petersen 2012-11-01 08:00:09 UTC
Description of problem:
I tried testing fedup on a fresh F18 x86_64 minimal + @base install.

Version-Release number of selected component (if applicable):
0.7
git1b6992b

How reproducible:
100%

Steps to Reproduce:
1. As root I ran:

# fedup-cli --network 18 --repo fedup=http://tflink.fedorapeople.org/fedup/f18-upgrade/x86_64 --instrepo=fedup
  
Actual results:
1. After downloading all the f18 packages to be upgraded I get:

:
setting up system for upgrade
Traceback (most recent call last):
  File "/bin/fedup-cli", line 267, in <module>
    main(args)
  File "/bin/fedup-cli", line 244, in main
    prep_boot(kernel, initrd, bootloader=args.bootloader)
  File "/usr/lib/python2.7/site-packages/fedup/download.py", line 330, in prep_boot
    modify_bootloader(kernel, initrd)
  File "/usr/lib/python2.7/site-packages/fedup/download.py", line 300, in modify_bootloader
    default = bootloader.default_entry()
  File "/usr/lib/python2.7/site-packages/fedup/grubby.py", line 74, in default_entry
    return self.get_entry(self.default_index())
  File "/usr/lib/python2.7/site-packages/fedup/grubby.py", line 50, in get_entry    out = self._grubby("--info", index)
  File "/usr/lib/python2.7/site-packages/fedup/grubby.py", line 45, in _grubby
    return check_output(cmd + [str(a) for a in args], stderr=PIPE)
  File "/usr/lib64/python2.7/subprocess.py", line 544, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['grubby', '--info', '-3']' returned non-zero exit status 1

Expected results:
No backtrace and process to proceed to reboot and upgrade. :)

Comment 1 Jens Petersen 2012-11-01 08:06:32 UTC
# rpm -q kernel
kernel-3.6.3-1.fc17
#

Comment 2 Jens Petersen 2012-11-01 08:11:33 UTC
Created attachment 636479 [details]
debuglog

Comment 3 Will Woods 2012-11-01 15:00:53 UTC
(In reply to comment #0)
> subprocess.CalledProcessError: Command '['grubby', '--info', '-3']' returned 
> non-zero exit status 1

So what happens when you run "grubby --info -3" manually? Can you attach your grub2.conf?

Comment 4 Matthias Clasen 2012-11-02 15:12:51 UTC
I get the same error.

grubby --info -3 says: kernel not found

Comment 5 Will Woods 2012-11-02 15:48:27 UTC
Okay, so '-3' is the index grubby returns when grub2.cfg has:

  set default="${saved_entry}"

(for grub1 it'd be '-2'). 

But even though grubby tells us the index is -3, it doesn't accept --info -3, so we crash.

BUT: we don't need to call bootloader.default_entry() at all - it was just being used for a line in the debugging logs. I'll just drop that code and this will go away.

Comment 6 Will Woods 2012-11-02 15:55:56 UTC
Should be fixed by https://github.com/wgwoods/fedup/commit/7f2f2f1

Comment 7 Jens Petersen 2012-11-05 02:26:57 UTC
Thanks - looks good now.

I guess better to leave this open until fix is pushed to Bodhi.

Comment 8 Jens Petersen 2012-11-05 08:39:53 UTC
BTW a workaround seems to be to install another kernel in f17.

eg yum install --enablerepo=updates-testing kernel

Comment 9 Fedora Update System 2012-11-20 04:15:03 UTC
fedup-0.7.1-1.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/fedup-0.7.1-1.fc18

Comment 10 Fedora Update System 2012-11-20 07:04:55 UTC
Package fedup-0.7.1-1.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing fedup-0.7.1-1.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-18606/fedup-0.7.1-1.fc18
then log in and leave karma (feedback).

Comment 11 Fedora Update System 2012-11-21 22:00:45 UTC
fedup-0.7.1-1.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/fedup-0.7.1-1.fc17

Comment 12 Tim Flink 2012-11-22 03:47:10 UTC
*** Bug 879089 has been marked as a duplicate of this bug. ***

Comment 13 Tim Flink 2012-11-22 18:24:24 UTC
I'm pretty sure this was an issue with the fix not having made it out to the mirrors when Adam tested and filed #879089. We talked about it over IRC it sounds like he was using fedup-0.7-1.fc17 which does not have the needed fix.

I haven't seen this recently and AFAIK, it's fixed. Since there haven't been any other reports, I'm moving this to VERIFIED.

Comment 14 Adam Williamson 2012-11-23 04:35:46 UTC
My test with 0.7.1-1 certainly worked. agree with VERIFIED.

Comment 15 Fedora Update System 2012-11-23 07:49:55 UTC
fedup-0.7.1-1.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 16 Fedora Update System 2012-11-24 03:27:58 UTC
fedup-0.7.1-1.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 17 Tim Flink 2012-11-27 23:52:38 UTC
I just hit this with fedup-0.7.1-1.fc17 on an F17 x86_64 UEFI install from livecd. I'll attach the debug logs but when I run the grubby command manually, I get:

sudo grubby --add-kernel /boot/upgrade/vmlinuz --title 'System Upgrade' --initrd /boot/upgrade/upgrade.img --args 'systemd.unit=system-upgrade.target' --copy-default
error opening /boot/grub/grub.cfg for read: No such file or directory

The grub.cfg is in /boot/efi/EFI/redhat/ for this machine - not /boot/grub/grub.cfg

Comment 18 Tim Flink 2012-11-27 23:53:17 UTC
Created attachment 653204 [details]
debug log from EFI run

Comment 19 Mads Kiilerich 2012-11-28 00:29:46 UTC
This seems somewhat related to Bug 768106. I really don't know whether Fedora uses saved_entry in grub env or "hardcode" numbers in grub.cfg.

Also:

(In reply to comment #17)
> when I run the grubby command
> manually, I get:
> 
> sudo grubby --add-kernel /boot/upgrade/vmlinuz --title 'System Upgrade'
> --initrd /boot/upgrade/upgrade.img --args
> 'systemd.unit=system-upgrade.target' --copy-default
> error opening /boot/grub/grub.cfg for read: No such file or directory

grybby-the-binary defaults to this, upstreams Single True Location of grub.cfg. In fedora we still use the grub2 name transformation and grubby-the-binary thus needs -c to point at the right config file. new-kernel-pkg will do that.

And:

> The grub.cfg is in /boot/efi/EFI/redhat/ for this machine - not
> /boot/grub/grub.cfg

Except for the fact that both paths are slightly different for Fedora:

Yes, it is sad and confusing that we have two different grub.cfg locations: /boot/grub2/grub.cfg and /boot/efi/EFI/fedora/grub.cfg. It is thus no longer possible to give simple and clear instructions on how grub2-mkconfig should be invoked.

It would be much better if /boot/efi/EFI/fedora/grub.cfg only contained a 'search' command and a 'configfile' command that pointed at /boot/grub2/grub.cfg.

Comment 20 Adam Williamson 2012-12-03 20:05:56 UTC
Discussed at 2012-12-03 blocker review meeting: http://meetbot.fedoraproject.org/fedora-qa/2012-12-03/f18final-blocker-review-1.2.2012-12-03-17.25.log.txt . We agreed to delay a decision on this until there is a clear action plan rather than a vague 'it needs to be more secure!' feeling, at least.

Comment 21 Adam Williamson 2012-12-03 20:15:08 UTC
Sorry, disregard above comment, it was attached to the wrong bug. Correct comment follows:

Discussed at 2012-12-03 blocker review meeting: http://meetbot.fedoraproject.org/fedora-qa/2012-12-03/f18final-blocker-review-1.2.2012-12-03-17.25.log.txt . We agreed to delay a decision on this until we have a clearer understanding of what can cause this bug to occur in 0.7.1 - it seems clear that it is still happening, but we have not clearly determined the cause.

Comment 22 Will Woods 2012-12-03 21:25:49 UTC
This is a different problem than the original bug, but let's ignore that for now..

(In reply to comment #19)
> (In reply to comment #17)
> > when I run the grubby command
> > manually, I get:
> > 
> > sudo grubby --add-kernel /boot/upgrade/vmlinuz --title 'System Upgrade'
> > --initrd /boot/upgrade/upgrade.img --args
> > 'systemd.unit=system-upgrade.target' --copy-default
> > error opening /boot/grub/grub.cfg for read: No such file or directory

Right - fedup doesn't directly touch *any* boot config files; all configuration is done through grubby. So if we're invoking it correctly, and it's not finding the bootloader config, this is a grubby problem.

> grybby-the-binary defaults to this, upstreams Single True Location of
> grub.cfg. In fedora we still use the grub2 name transformation and
> grubby-the-binary thus needs -c to point at the right config file.
> new-kernel-pkg will do that.

Ah, so maybe we *are* invoking it incorrectly. How does new-kernel-pkg get the correct path to the config file?

Comment 23 Mads Kiilerich 2012-12-03 21:36:03 UTC
(In reply to comment #22)
> Ah, so maybe we *are* invoking it incorrectly. How does new-kernel-pkg get
> the correct path to the config file?

It follows/reads /etc/grub2.cfg and /etc/grub2-efi.cfg (and /etc/grub.conf and ...) and patches all the files it can find.

Comment 24 Will Woods 2012-12-03 23:01:26 UTC
Yeah, so it runs grubby with a bunch of special arguments that differ for each available bootloader it finds. Fun!

I guess this means we're going to need to run new-kernel-pkg instead of grubby, since there's a bunch of special magic locked up in new-kernel-pkg that isn't available otherwise.

If I understand correctly, this means we need to download the images to:

  /boot/vmlinuz-fedup
  /boot/initramfs-fedup.img

And run:

  new-kernel-pkg --initrdfile /boot/initramfs-fedup.img \
                 --banner "System Upgrade" \
                 --kernel-args "systemd.unit=system-upgrade.target" \
                 --install fedup

And to remove it,

  new-kernel-pkg --remove fedup

Does that seem correct?

Comment 25 Mads Kiilerich 2012-12-03 23:28:38 UTC
FWIW, I don't spot any obvious errors there.

But you should verify that the entry for the new f18 kernel doesn't use the fedup entry for cloning and inherits the extra systemd.unit kernel arg.

Comment 26 Will Woods 2012-12-03 23:50:31 UTC
Grubby is smart enough to prefer the *default* boot item to the *first* boot item, so as long as the new boot item isn't the default this works fine. We've tested that pretty extensively.

We'll probably also remove the upgrade boot entry once the upgrade has started in the near future. So that shouldn't really be a problem.

Comment 27 Adam Williamson 2012-12-05 18:39:18 UTC
Discussed at 2012-12-05 blocker review meeting - http://meetbot.fedoraproject.org/fedora-bugzappers/2012-12-05/f18final-blocker-review-2.2012-12-05-17.01.log.txt . We agreed that we're worried about people still hitting grubby backtraces, but the precise cause(s) haven't yet been identified and we'd like to know how unusual they are before taking any bugs as blockers. We note it might be best to split off at least one separate report here, since the cases we have now aren't the original case.

Comment 28 Tim Flink 2012-12-06 15:52:36 UTC
I tried upgrading with fedup-0.7.2-0.fc17 from koji and ended up with a similar error from 'grubby --default-kernel'.

As will mentioned in c#22, this is a different issue so I filed it as https://bugzilla.redhat.com/show_bug.cgi?id=884696

Comment 29 Adam Williamson 2012-12-11 00:21:06 UTC
Any objections to closing this bug, as the case it initially reported seems clearly fixed?

Comment 30 Jens Petersen 2012-12-11 01:13:19 UTC
Maybe move it to F18-accepted?  If there are still some edge/corner-cases
left then better either to file new bugs for those or leave this open I think.

Comment 31 Tim Flink 2012-12-11 14:46:09 UTC
(In reply to comment #29)
> Any objections to closing this bug, as the case it initially reported seems
> clearly fixed?

No objections from me. I filed a new bug for the exact issue that I was hitting: https://bugzilla.redhat.com/show_bug.cgi?id=884696

Comment 32 Will Woods 2012-12-11 16:01:13 UTC
If you choose not to close it now, this bug will be closed by bodhi when the fedup-0.7.2 update gets pushed.

Putting it in ON_DEV until the 0.7.2 update is ready.

Comment 33 Adam Williamson 2012-12-12 05:45:22 UTC
Let's go ahead and close it out, please file post-0.7 issues separately.

Comment 34 Fedora Update System 2012-12-21 04:49:14 UTC
fedup-0.7.2-1.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/fedup-0.7.2-1.fc17

Comment 35 Fedora Update System 2013-01-03 07:26:24 UTC
fedup-0.7.2-1.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.