This service will be undergoing maintenance at 03:30 UTC, 2016-05-27. It is expected to last about 2 hours
Bug 872088 - fedup giving grubby backtrace
fedup giving grubby backtrace
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: fedup (Show other bugs)
17
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Will Woods
Fedora Extras Quality Assurance
: Reopened
: 879089 (view as bug list)
Depends On:
Blocks: F18Blocker/F18FinalBlocker
  Show dependency treegraph
 
Reported: 2012-11-01 04:00 EDT by Jens Petersen
Modified: 2013-01-03 02:26 EST (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-12-12 00:45:22 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
debuglog (32.21 KB, application/x-gzip)
2012-11-01 04:11 EDT, Jens Petersen
no flags Details
debug log from EFI run (492.19 KB, text/plain)
2012-11-27 18:53 EST, Tim Flink
no flags Details

  None (edit)
Description Jens Petersen 2012-11-01 04:00:09 EDT
Description of problem:
I tried testing fedup on a fresh F18 x86_64 minimal + @base install.

Version-Release number of selected component (if applicable):
0.7
git1b6992b

How reproducible:
100%

Steps to Reproduce:
1. As root I ran:

# fedup-cli --network 18 --repo fedup=http://tflink.fedorapeople.org/fedup/f18-upgrade/x86_64 --instrepo=fedup
  
Actual results:
1. After downloading all the f18 packages to be upgraded I get:

:
setting up system for upgrade
Traceback (most recent call last):
  File "/bin/fedup-cli", line 267, in <module>
    main(args)
  File "/bin/fedup-cli", line 244, in main
    prep_boot(kernel, initrd, bootloader=args.bootloader)
  File "/usr/lib/python2.7/site-packages/fedup/download.py", line 330, in prep_boot
    modify_bootloader(kernel, initrd)
  File "/usr/lib/python2.7/site-packages/fedup/download.py", line 300, in modify_bootloader
    default = bootloader.default_entry()
  File "/usr/lib/python2.7/site-packages/fedup/grubby.py", line 74, in default_entry
    return self.get_entry(self.default_index())
  File "/usr/lib/python2.7/site-packages/fedup/grubby.py", line 50, in get_entry    out = self._grubby("--info", index)
  File "/usr/lib/python2.7/site-packages/fedup/grubby.py", line 45, in _grubby
    return check_output(cmd + [str(a) for a in args], stderr=PIPE)
  File "/usr/lib64/python2.7/subprocess.py", line 544, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['grubby', '--info', '-3']' returned non-zero exit status 1

Expected results:
No backtrace and process to proceed to reboot and upgrade. :)
Comment 1 Jens Petersen 2012-11-01 04:06:32 EDT
# rpm -q kernel
kernel-3.6.3-1.fc17
#
Comment 2 Jens Petersen 2012-11-01 04:11:33 EDT
Created attachment 636479 [details]
debuglog
Comment 3 Will Woods 2012-11-01 11:00:53 EDT
(In reply to comment #0)
> subprocess.CalledProcessError: Command '['grubby', '--info', '-3']' returned 
> non-zero exit status 1

So what happens when you run "grubby --info -3" manually? Can you attach your grub2.conf?
Comment 4 Matthias Clasen 2012-11-02 11:12:51 EDT
I get the same error.

grubby --info -3 says: kernel not found
Comment 5 Will Woods 2012-11-02 11:48:27 EDT
Okay, so '-3' is the index grubby returns when grub2.cfg has:

  set default="${saved_entry}"

(for grub1 it'd be '-2'). 

But even though grubby tells us the index is -3, it doesn't accept --info -3, so we crash.

BUT: we don't need to call bootloader.default_entry() at all - it was just being used for a line in the debugging logs. I'll just drop that code and this will go away.
Comment 6 Will Woods 2012-11-02 11:55:56 EDT
Should be fixed by https://github.com/wgwoods/fedup/commit/7f2f2f1
Comment 7 Jens Petersen 2012-11-04 21:26:57 EST
Thanks - looks good now.

I guess better to leave this open until fix is pushed to Bodhi.
Comment 8 Jens Petersen 2012-11-05 03:39:53 EST
BTW a workaround seems to be to install another kernel in f17.

eg yum install --enablerepo=updates-testing kernel
Comment 9 Fedora Update System 2012-11-19 23:15:03 EST
fedup-0.7.1-1.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/fedup-0.7.1-1.fc18
Comment 10 Fedora Update System 2012-11-20 02:04:55 EST
Package fedup-0.7.1-1.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing fedup-0.7.1-1.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-18606/fedup-0.7.1-1.fc18
then log in and leave karma (feedback).
Comment 11 Fedora Update System 2012-11-21 17:00:45 EST
fedup-0.7.1-1.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/fedup-0.7.1-1.fc17
Comment 12 Tim Flink 2012-11-21 22:47:10 EST
*** Bug 879089 has been marked as a duplicate of this bug. ***
Comment 13 Tim Flink 2012-11-22 13:24:24 EST
I'm pretty sure this was an issue with the fix not having made it out to the mirrors when Adam tested and filed #879089. We talked about it over IRC it sounds like he was using fedup-0.7-1.fc17 which does not have the needed fix.

I haven't seen this recently and AFAIK, it's fixed. Since there haven't been any other reports, I'm moving this to VERIFIED.
Comment 14 Adam Williamson 2012-11-22 23:35:46 EST
My test with 0.7.1-1 certainly worked. agree with VERIFIED.
Comment 15 Fedora Update System 2012-11-23 02:49:55 EST
fedup-0.7.1-1.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 16 Fedora Update System 2012-11-23 22:27:58 EST
fedup-0.7.1-1.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 17 Tim Flink 2012-11-27 18:52:38 EST
I just hit this with fedup-0.7.1-1.fc17 on an F17 x86_64 UEFI install from livecd. I'll attach the debug logs but when I run the grubby command manually, I get:

sudo grubby --add-kernel /boot/upgrade/vmlinuz --title 'System Upgrade' --initrd /boot/upgrade/upgrade.img --args 'systemd.unit=system-upgrade.target' --copy-default
error opening /boot/grub/grub.cfg for read: No such file or directory

The grub.cfg is in /boot/efi/EFI/redhat/ for this machine - not /boot/grub/grub.cfg
Comment 18 Tim Flink 2012-11-27 18:53:17 EST
Created attachment 653204 [details]
debug log from EFI run
Comment 19 Mads Kiilerich 2012-11-27 19:29:46 EST
This seems somewhat related to Bug 768106. I really don't know whether Fedora uses saved_entry in grub env or "hardcode" numbers in grub.cfg.

Also:

(In reply to comment #17)
> when I run the grubby command
> manually, I get:
> 
> sudo grubby --add-kernel /boot/upgrade/vmlinuz --title 'System Upgrade'
> --initrd /boot/upgrade/upgrade.img --args
> 'systemd.unit=system-upgrade.target' --copy-default
> error opening /boot/grub/grub.cfg for read: No such file or directory

grybby-the-binary defaults to this, upstreams Single True Location of grub.cfg. In fedora we still use the grub2 name transformation and grubby-the-binary thus needs -c to point at the right config file. new-kernel-pkg will do that.

And:

> The grub.cfg is in /boot/efi/EFI/redhat/ for this machine - not
> /boot/grub/grub.cfg

Except for the fact that both paths are slightly different for Fedora:

Yes, it is sad and confusing that we have two different grub.cfg locations: /boot/grub2/grub.cfg and /boot/efi/EFI/fedora/grub.cfg. It is thus no longer possible to give simple and clear instructions on how grub2-mkconfig should be invoked.

It would be much better if /boot/efi/EFI/fedora/grub.cfg only contained a 'search' command and a 'configfile' command that pointed at /boot/grub2/grub.cfg.
Comment 20 Adam Williamson 2012-12-03 15:05:56 EST
Discussed at 2012-12-03 blocker review meeting: http://meetbot.fedoraproject.org/fedora-qa/2012-12-03/f18final-blocker-review-1.2.2012-12-03-17.25.log.txt . We agreed to delay a decision on this until there is a clear action plan rather than a vague 'it needs to be more secure!' feeling, at least.
Comment 21 Adam Williamson 2012-12-03 15:15:08 EST
Sorry, disregard above comment, it was attached to the wrong bug. Correct comment follows:

Discussed at 2012-12-03 blocker review meeting: http://meetbot.fedoraproject.org/fedora-qa/2012-12-03/f18final-blocker-review-1.2.2012-12-03-17.25.log.txt . We agreed to delay a decision on this until we have a clearer understanding of what can cause this bug to occur in 0.7.1 - it seems clear that it is still happening, but we have not clearly determined the cause.
Comment 22 Will Woods 2012-12-03 16:25:49 EST
This is a different problem than the original bug, but let's ignore that for now..

(In reply to comment #19)
> (In reply to comment #17)
> > when I run the grubby command
> > manually, I get:
> > 
> > sudo grubby --add-kernel /boot/upgrade/vmlinuz --title 'System Upgrade'
> > --initrd /boot/upgrade/upgrade.img --args
> > 'systemd.unit=system-upgrade.target' --copy-default
> > error opening /boot/grub/grub.cfg for read: No such file or directory

Right - fedup doesn't directly touch *any* boot config files; all configuration is done through grubby. So if we're invoking it correctly, and it's not finding the bootloader config, this is a grubby problem.

> grybby-the-binary defaults to this, upstreams Single True Location of
> grub.cfg. In fedora we still use the grub2 name transformation and
> grubby-the-binary thus needs -c to point at the right config file.
> new-kernel-pkg will do that.

Ah, so maybe we *are* invoking it incorrectly. How does new-kernel-pkg get the correct path to the config file?
Comment 23 Mads Kiilerich 2012-12-03 16:36:03 EST
(In reply to comment #22)
> Ah, so maybe we *are* invoking it incorrectly. How does new-kernel-pkg get
> the correct path to the config file?

It follows/reads /etc/grub2.cfg and /etc/grub2-efi.cfg (and /etc/grub.conf and ...) and patches all the files it can find.
Comment 24 Will Woods 2012-12-03 18:01:26 EST
Yeah, so it runs grubby with a bunch of special arguments that differ for each available bootloader it finds. Fun!

I guess this means we're going to need to run new-kernel-pkg instead of grubby, since there's a bunch of special magic locked up in new-kernel-pkg that isn't available otherwise.

If I understand correctly, this means we need to download the images to:

  /boot/vmlinuz-fedup
  /boot/initramfs-fedup.img

And run:

  new-kernel-pkg --initrdfile /boot/initramfs-fedup.img \
                 --banner "System Upgrade" \
                 --kernel-args "systemd.unit=system-upgrade.target" \
                 --install fedup

And to remove it,

  new-kernel-pkg --remove fedup

Does that seem correct?
Comment 25 Mads Kiilerich 2012-12-03 18:28:38 EST
FWIW, I don't spot any obvious errors there.

But you should verify that the entry for the new f18 kernel doesn't use the fedup entry for cloning and inherits the extra systemd.unit kernel arg.
Comment 26 Will Woods 2012-12-03 18:50:31 EST
Grubby is smart enough to prefer the *default* boot item to the *first* boot item, so as long as the new boot item isn't the default this works fine. We've tested that pretty extensively.

We'll probably also remove the upgrade boot entry once the upgrade has started in the near future. So that shouldn't really be a problem.
Comment 27 Adam Williamson 2012-12-05 13:39:18 EST
Discussed at 2012-12-05 blocker review meeting - http://meetbot.fedoraproject.org/fedora-bugzappers/2012-12-05/f18final-blocker-review-2.2012-12-05-17.01.log.txt . We agreed that we're worried about people still hitting grubby backtraces, but the precise cause(s) haven't yet been identified and we'd like to know how unusual they are before taking any bugs as blockers. We note it might be best to split off at least one separate report here, since the cases we have now aren't the original case.
Comment 28 Tim Flink 2012-12-06 10:52:36 EST
I tried upgrading with fedup-0.7.2-0.fc17 from koji and ended up with a similar error from 'grubby --default-kernel'.

As will mentioned in c#22, this is a different issue so I filed it as https://bugzilla.redhat.com/show_bug.cgi?id=884696
Comment 29 Adam Williamson 2012-12-10 19:21:06 EST
Any objections to closing this bug, as the case it initially reported seems clearly fixed?
Comment 30 Jens Petersen 2012-12-10 20:13:19 EST
Maybe move it to F18-accepted?  If there are still some edge/corner-cases
left then better either to file new bugs for those or leave this open I think.
Comment 31 Tim Flink 2012-12-11 09:46:09 EST
(In reply to comment #29)
> Any objections to closing this bug, as the case it initially reported seems
> clearly fixed?

No objections from me. I filed a new bug for the exact issue that I was hitting: https://bugzilla.redhat.com/show_bug.cgi?id=884696
Comment 32 Will Woods 2012-12-11 11:01:13 EST
If you choose not to close it now, this bug will be closed by bodhi when the fedup-0.7.2 update gets pushed.

Putting it in ON_DEV until the 0.7.2 update is ready.
Comment 33 Adam Williamson 2012-12-12 00:45:22 EST
Let's go ahead and close it out, please file post-0.7 issues separately.
Comment 34 Fedora Update System 2012-12-20 23:49:14 EST
fedup-0.7.2-1.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/fedup-0.7.2-1.fc17
Comment 35 Fedora Update System 2013-01-03 02:26:24 EST
fedup-0.7.2-1.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.