Bug 826537
Summary: | Pre-upgraded system never install fc17 kernel in grub2 | ||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Kamil Páral <kparal> | ||||||||||||||||||||||
Component: | grubby | Assignee: | Peter Jones <pjones> | ||||||||||||||||||||||
Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||||||||||||||||||
Severity: | urgent | Docs Contact: | |||||||||||||||||||||||
Priority: | urgent | ||||||||||||||||||||||||
Version: | 17 | CC: | akvasu, awilliam, bcl, bloch, bugzilla.redhat, chris, collura, dan, emailjonathananderson-fedora, garrett.mitchener, germano.massullo, g.kaviyarasu, htl10, hughsient, jonathan, mads, mlatelcom, mt, nsoranzo, pjones, sergio, solaris.smoke, thomas, vanmeeuwen+fedora | ||||||||||||||||||||||
Target Milestone: | --- | ||||||||||||||||||||||||
Target Release: | --- | ||||||||||||||||||||||||
Hardware: | Unspecified | ||||||||||||||||||||||||
OS: | Unspecified | ||||||||||||||||||||||||
Whiteboard: | |||||||||||||||||||||||||
Fixed In Version: | grubby-8.12-1.fc17 | Doc Type: | Bug Fix | ||||||||||||||||||||||
Doc Text: | Story Points: | --- | |||||||||||||||||||||||
Clone Of: | Environment: | ||||||||||||||||||||||||
Last Closed: | 2012-06-04 18:37:31 UTC | Type: | Bug | ||||||||||||||||||||||
Regression: | --- | Mount Type: | --- | ||||||||||||||||||||||
Documentation: | --- | CRM: | |||||||||||||||||||||||
Verified Versions: | Category: | --- | |||||||||||||||||||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||||||||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||||||||||||
Embargoed: | |||||||||||||||||||||||||
Attachments: |
|
Description
Kamil Páral
2012-05-30 12:59:39 UTC
Created attachment 587722 [details]
grub.cfg in the upgraded system
Created attachment 587723 [details]
anaconda.log
Created attachment 587724 [details]
program.log
Created attachment 587725 [details]
storage.log
Created attachment 587726 [details]
syslog
Created attachment 587727 [details]
yum.log
To illustrate the main problem - it is not easy to fix it. "yum update" doesn't help until a newer kernel is released. You have to edit grub.cfg manually or run grub2-mkconfig or meddle with kernels (reinstall fc17 kernel). Layman users might have problems doing that. Also most people won't know what to do, because they don't read CommonBugs. That grub.cfg has not been generated by the f17 grub2. I don't see any invocations of grub2-install / grub2-mkconfig in the anaconda logs. Is the boot loader installation step always skipped with preupgrade or was that an explicit choice here? In this case the problem must be caused by the grubby template error message - it was for once really fatal. Is the f16 kernel fully installed and with both vmlinuz and initramfs in /boot? Can you confirm that grub2-mkconfig will generate a working configuration? (In reply to comment #8) > That grub.cfg has not been generated by the f17 grub2. I don't see any > invocations of grub2-install / grub2-mkconfig in the anaconda logs. Is the > boot loader installation step always skipped with preupgrade or was that an > explicit choice here? In my case (comment #21 of bug 820351) I made no explicit choices for preupgrade; the result came from the default preupgrade process once F17 was made generally available yesterday. > In this case the problem must be caused by the grubby template error message > - it was for once really fatal. > > Is the f16 kernel fully installed and with both vmlinuz and initramfs in > /boot? In my case, the F16 kernel was fully installed and had both vmlinux and initramfs. So did the F17 kernel. > > Can you confirm that grub2-mkconfig will generate a working configuration? Running grub2-mkconfig > /etc/grub2.cfg && grub2-install /dev/sda generates a working configuration (the results are much different than what had been living in /etc/grub2.cfg - now I understand what people are talking about with respect to the grub video modes), so for this naive user (heh) it seems like a decent workaround / manual fix for the bug. This problem does not happen when using netinst.iso to upgrade - a single fc17 kernel is installed and the old fc16 kernel is removed. So this issue is probably really caused by preupgrade in some way. Created attachment 587752 [details]
preupgrade errors
Preugprade-cli finishes with some weird errors.
Created attachment 587760 [details]
kickstart generated by preupgrade
Created attachment 587761 [details]
original grub.cfg before running preupgrade
Created attachment 587762 [details]
modified grub.cfg after preupgrade finished but before reboot
This is probably a duplicate of bug 820340, but there are more logs here. I have tested preupgrade-1.1.11-1. It fixes the errors printed out at the end of preupgrade process (attachment 587752 [details]). But it does not fix this issue. fc17 kernel is still installed but not present in grub.cfg.
Could you please tell my how to get these logs? I have other computers that are running upgrades so I can reproduce the problem (In reply to comment #11) > Created attachment 587752 [details] > preupgrade errors > > Preugprade-cli finishes with some weird errors. That is bug 815473 and bug 826473 and bug 811964 - preupgrade isn't selected once in the bootloader as it should. I have a computer on which the anaconda (you mean preupgrade but is anaconda) is not avaible at all in grub *** Bug 826529 has been marked as a duplicate of this bug. *** I concur, this bug occurred for me after preupgrade fc16/fc17beta and again at fc16/fc17gold. Also concur grub2-mkconfig > /etc/grub2.cfg && grub2-install /dev/sda repairs this issue and creates proper grub.cfg loading fc17 kernel (In reply to comment #21) (Please do the grub2 stuff correctly - and stop spreading the wrong way. First install stuff mkconfig can pick up: grub2-install /dev/sda Then create the config securely: grub2-mkconfig -o /boot/grub2/grub.cfg ) https://admin.fedoraproject.org/updates/grubby-8.12-1.fc17 is expected to fix this , I believe. If people can re-test it'd be good. Testing before the update hits stable is a bit tricky, though. I saw this too on two different machines and I think this is probably the correct bug to post a comment. On one machine, the upgrade seemed to go well, but then the machine rebooted back into the F16 kernel as described here. I had to manually run grub2-install /dev/sda followed by grub2-mkconfig -o /boot/grub2/grub.cfg. I wasn't paying too much attention, but I believe all the files in /boot/grub2 before this were all from the F16 version of grub2. On the other machine (upgraded the same way) I'm quite sure that the files in /boot/grub2 were from F16 after reboot. To make matters worse, once I got to the command line, grub2-install was somehow not able to identify that the "target" should be i386-pc for this machine, and I had to dig through a lot of documentation to figure out that the command to fix it was grub2-install --target=i386-pc /dev/sda In short, I don't think grubby is necessarily the (only) culprit here. I think grub2 and/or anaconda and/or preupgrade are also at fault for not installing the new bootloader files. I have a third F16 machine that I'll be upgrading, probably in about a week. I'd be willing to test a proposed fix on it then if you can tell me how. Seriously: We've got to do better on upgrades, especially for something as critical and technical as the bootloader. There's a new Fedora every six months or so. Every time I've upgraded, no matter what method, something goes wrong with the bootloader, I get no error message, just a malfunctioning system. And the fix is always to get to a command line and manually run the grub installation commands that ought to be done automatically. That doesn't sound like the same bug. Do you have the kickstart file from the preupgrade? (In reply to comment #25) > That doesn't sound like the same bug. Okay, on reading this page over again, maybe my bug is different, but I think it's related. Part of the original bug may have been fixed before I ran my upgrade. I used the preupgrade graphical interface, didn't see any error messages then, and afterward grub2.cfg did have an entry for the upgrade, which ran more or less correctly, but it generated this error in /root/upgrade.log: 09:00:36 Upgrading grubby-8.11-1.fc17.i686 ... 09:47:23 Upgrading kernel-PAE-3.3.7-1.fc17.i686 grubby fatal error: unable to find a suitable template ... 10:45:19 Upgrading grub2-2.0-0.25.beta4.fc17.i686 warning: /etc/default/grub created as /etc/default/grub.rpmnew ... What I'm thinking here is that grub2 isn't getting installed properly somehow (maybe too late in the process), and grub2.cfg isn't getting rebuilt, so grubby fails during the install of the kernel package. In my case, all of that went wrong during the actual upgrade. Did all the malfunction in the original bug report happen while running preupgrade but before booting into the upgrade itself? Comment #8 makes me think something also went wrong during the upgrade. > Do you have the kickstart file from the preupgrade? Where would I find it? Ooh. I just figured something out that I didn't realize before. We're doing something different in preupgrade to what we're doing in DVD/netinst upgrades. DVD/netinst default bootloader handling on upgrade is still 'install new bootloader' (which has been default since F16); i.e. anaconda will entirely reinstall the bootloader on upgrade. That's not actually the case for preupgrade from F16 to F17, though. The code we put in preupgrade during F16 cycle to ensure it did a new bootloader install when upgrading from F15 to F16 was quite specific: # Fedora 16 switched to grub2 if float(self.source_version) < 16.0 and float(self.target_version) >= 16.0: ks[3] += ' --location=mbr' else: ks[3] += ' --upgrade --location=none' So - when you're upgrading from 16 to 17, you get the good old pre-F16 default behaviour back. You get "bootloader --upgrade --location=none" . So in preupgrade, we're just telling anaconda to update the existing bootloader configuration, not to reinstall the bootloader. That's obviously significant here. So the problems we're having are when the attempt to update the bootloader configuration fails. I think Kamil and Garrett are actually both seeing a similar case where the bootloader config update is just failing, and I was wrong to say they're different bugs, but I'm not sure how to be sure... (In reply to comment #27) > ks[3] += ' --upgrade --location=none' > > So - when you're upgrading from 16 to 17, you get the good old pre-F16 > default behaviour back. You get "bootloader --upgrade --location=none" . So > in preupgrade, we're just telling anaconda to update the existing bootloader > configuration, not to reinstall the bootloader. Great job. That's exactly what I see in my generated /boot/upgrade/ks.cfg. So we need to fix: 1. kickstart generation in preupgrade 2. grub2.cfg generation by grubby What about merging these 2 bugreports https://bugzilla.redhat.com/show_bug.cgi?id=820340 https://bugzilla.redhat.com/show_bug.cgi?id=826537 into https://bugzilla.redhat.com/show_bug.cgi?id=820351 ? (In reply to comment #28) > 2. grub2.cfg generation by grubby Just a clarification: grubby will never generate grub.cfg, neither directly nor by calling grub2-mkconfig. It is only anaconda that will generate grub.cfg by invoking grub2-mkconfig. (The role of grubby in the brave new world with the over-engineered grub2 configuration scheme is a bit unclear. Perhaps grubby should do something different.) Patching of grub.conf with new kernels while preupgrading should however work with https://admin.fedoraproject.org/updates/grubby-8.12-1.fc17 . *** Bug 820340 has been marked as a duplicate of this bug. *** Caterpillar: we'd really prefer not to, because they're different problems. IIUC we need a new preupgrade package that fixes kickstart generation, as stated in comment 27. And soon, before all our users decide to upgrade (and hit the issue). Richard, can you provide one? Thanks. Kamil: I don't necessarily agree. It's at least not urgent. There's nothing intrinsically wrong with doing a config update instead of a reinstall (though obviously it's a bit odd for preupgrade and DVD to take different paths here). With grubby 8.12, it should actually work okay. It just explains why grubby is important and why we're hitting issues on preupgrade we're not hitting on DVD. If grubby 8.12 does fix the bugs with actually correctly updating the grub config, then I'd say there's no urgent problem any more. We might decide that we always want to reinstall grub2 on upgrade until its code and config format start stabilizing a bit more as a matter of policy, but that's not 'OMG needs to happen yesterday'. So I tested, and 8.12 seems to do the trick. If anyone else wants to try, the trick is to run the F16 part of preupgrade, then edit /boot/upgrade/ks.cfg (or /boot/update/ks.cfg, i forget which it is) and add a line: repo --name=side --baseurl=http://www.happyassassin.net/extras/repo2/x86_64 then go ahead and reboot and do the preupgrade. That repo is on my personal server, it has grubby 8.12 in it. (I don't have an i686 version up, sorry). After the preupgrade completes, it should properly add the F17 kernel. I compared by doing it with two identical cloned VMs, making the edit in one case but not in the other. It worked with the updated grubby, it didn't work without. The update needs one more karma point. (In reply to comment #35) > So I tested, and 8.12 seems to do the trick. If anyone else wants to try, > the trick is to run the F16 part of preupgrade, then edit > /boot/upgrade/ks.cfg (or /boot/update/ks.cfg, i forget which it is) and add > a line: > > repo --name=side --baseurl=http://www.happyassassin.net/extras/repo2/x86_64 > > then go ahead and reboot and do the preupgrade. That repo is on my personal > server, it has grubby 8.12 in it. (I don't have an i686 version up, sorry). > After the preupgrade completes, it should properly add the F17 kernel. I > compared by doing it with two identical cloned VMs, making the edit in one > case but not in the other. It worked with the updated grubby, it didn't work > without. > > The update needs one more karma point. My two virtualmachines are actually in trouble after a normal Fedora 16 update (lol, yes). As soon as possible I will fix them and test your patch. (In reply to comment #35) > So I tested, and 8.12 seems to do the trick. If anyone else wants to try, > the trick is to run the F16 part of preupgrade, then edit > /boot/upgrade/ks.cfg (or /boot/update/ks.cfg, i forget which it is) and add > a line: > > repo --name=side --baseurl=http://www.happyassassin.net/extras/repo2/x86_64 Have I to add it at the bottom of file? (In reply to comment #35) > So I tested, and 8.12 seems to do the trick. If anyone else wants to try, > the trick is to run the F16 part of preupgrade, then edit > /boot/upgrade/ks.cfg (or /boot/update/ks.cfg, i forget which it is) and add > a line: > > repo --name=side --baseurl=http://www.happyassassin.net/extras/repo2/x86_64 > > then go ahead and reboot and do the preupgrade. That repo is on my personal > server, it has grubby 8.12 in it. (I don't have an i686 version up, sorry). > After the preupgrade completes, it should properly add the F17 kernel. I > compared by doing it with two identical cloned VMs, making the edit in one > case but not in the other. It worked with the updated grubby, it didn't work > without. > > The update needs one more karma point. On my Fedora 16 64bit LXDE VirtualBOX virtual machine I did: - yum update - reboot - preupgrade - putted on bottom of file /boot/upgrade/ks.cfg repo --name=side --baseurl=http://www.happyassassin.net/extras/repo2/x86_64 - preupgrade again - reboot and let start anaconda to do upgrades - rebooted The final result is that I still have Grub 2 with old Fedora 16 kernels. Maybe something went wrong, dunno, I cannot vote -1 since I don't know if I correctly followed the procedure (In reply to comment #38) > - preupgrade > - putted on bottom of file /boot/upgrade/ks.cfg repo --name=side > --baseurl=http://www.happyassassin.net/extras/repo2/x86_64 > - preupgrade again I guess that step created a new ks without your changes. Running preupgrade once should be enough. I completely erased old virtual machine and installed a new Fedora 16 64bit LXDE virtual machine, then I did: - yum update - reboot - preupgrade - putted on bottom of file /boot/upgrade/ks.cfg repo --name=side --baseurl=http://www.happyassassin.net/extras/repo2/x86_64 - reboot and let start anaconda to do upgrades - rebooted It works. I voted on https://admin.fedoraproject.org/updates/grubby-8.12-1.fc17 to increase Karma *** Bug 817289 has been marked as a duplicate of this bug. *** Hi I've done the yum update, and actually already did preupgrade 2 times - Initally I had to manually edit the Grub command to enter the temporary upgrade kernel path. After I did that I got some dracut error saying no image found. I checked this and then used grub2 options to reboot the machine and it completed the preupgrade perfectly as per the upgrade logs. However I don't find any kernet installed - [root@localhost ~]# cd /boot/ [root@localhost boot]# ls config-3.3.2-1.fc16.x86_64 initrd-plymouth.img config-3.3.6-3.fc16.x86_64 lost+found config-3.3.7-1.fc16.x86_64 memtest86+-4.20 efi System.map-3.3.2-1.fc16.x86_64 elf-memtest86+-4.20 System.map-3.3.6-3.fc16.x86_64 grub System.map-3.3.7-1.fc16.x86_64 grub2 upgrade initramfs-3.3.2-1.fc16.x86_64.img vmlinuz-3.3.2-1.fc16.x86_64 initramfs-3.3.6-3.fc16.x86_64.img vmlinuz-3.3.6-3.fc16.x86_64 initramfs-3.3.7-1.fc16.x86_64.img vmlinuz-3.3.7-1.fc16.x86_64 [root@localhost boot]# Is there a different path for Kernel Install? I couldn't find any 17 kernel installed. I am going to try that option provided above, but in your cases I think the kernel did get installed? Thanks Anup (In reply to comment #42) 'rpm -q kernel' only shows f16 kernels, and 'yum update' will not install kernel-3.3.7-1.fc17.x86_64? That sounds like a different issue. Better file a new issue with a full description and attach your logs there - and paste a link here. Hi I had to boot into the 3.3.7-1.x86_64 kernel and yum update didnt find any updates. yum distro-sync actually pointed out the packeges that were at higher versions and tried to remove them. [root@localhost boot]# rpm -q kernel kernel-3.3.2-1.fc16.x86_64 kernel-3.3.6-3.fc16.x86_64 kernel-3.3.7-1.fc16.x86_64 [root@localhost boot]# There are no other kernels, the 500 mb /boot is only at 20% usage. What other info can I collect? I will fine a new bug. I do have a solid state HDD - I think in F15/F16 there had been issues with the solid state devices. Thanks Anup (In reply to comment #44) Please attach the anaconda logs there - and show the yum commands and their output. akvasu: do you have the 17 updates repo enabled? cat upgrade.log | grep -i grub -C2 21:20:04 Upgrading python-batchhttp-1.1-4.fc17.noarch 21:20:05 Upgrading python-transaction-1.1.1-4.fc17.noarch 21:20:06 Upgrading grubby-8.11-1.fc17.x86_64 21:20:06 Upgrading libmount-2.21.2-1.fc17.x86_64 21:20:09 Upgrading xfsprogs-3.1.8-1.fc17.x86_64 -- 22:41:45 Upgrading wireless-tools-devel-29-7.1.fc17.x86_64 22:41:46 Upgrading hpijs-3.12.4-2.fc17.x86_64 22:41:48 Upgrading grub2-2.0-0.25.beta4.fc17.x86_64 warning: /etc/default/grub created as /etc/default/grub.rpmnew 22:41:52 Upgrading lighttpd-fastcgi-1.4.30-1.fc17.x86_64 22:41:53 Upgrading avahi-ui-tools-0.6.30-7.fc17.x86_64 -- 23:45:15 Upgrading libgcc-4.7.0-5.fc17.i686 23:45:16 Upgrading kernel-3.3.7-1.fc17.x86_64 grubby fatal error: unable to find a suitable template 23:45:24 Upgrading libstdc++-4.7.0-5.fc17.i686 23:45:29 Upgrading zlib-1.2.5-6.fc17.i686 (In reply to comment #46) > akvasu: do you have the 17 updates repo enabled? He apparently filed Bug 827876. (In reply to comment #47) > cat upgrade.log | grep -i grub -C2 Sergio, why are pasting that and what do you want to say? (In reply to comment #48) > (In reply to comment #46) > > akvasu: do you have the 17 updates repo enabled? > > He apparently filed Bug 827876. > > (In reply to comment #47) > > cat upgrade.log | grep -i grub -C2 > > Sergio, why are pasting that and what do you want to say? Anaconda logs , isn't upgrade.log and upgrade.log.syslog on /root ? I find out on upgrade.log why preupgrade never install F17 kernel , because gives the error "grubby fatal error: unable to find a suitable template" and using grubby-8.11-1.fc17.x86_64 (In reply to comment #49) > I find out on upgrade.log why preupgrade never install F17 kernel , because > gives the error "grubby fatal error: unable to find a suitable template" That message is usually not relevant. It usually come from grubby failing when trying to update an irrelevant and unused bootloader. It might however also be a real problem that is fixed by https://admin.fedoraproject.org/updates/grubby-8.12-1.fc17 . (In reply to comment #50) > (In reply to comment #49) > > I find out on upgrade.log why preupgrade never install F17 kernel , because > > gives the error "grubby fatal error: unable to find a suitable template" > > That message is usually not relevant. It usually come from grubby failing > when trying to update an irrelevant and unused bootloader. It might however > also be a real problem that is fixed by > https://admin.fedoraproject.org/updates/grubby-8.12-1.fc17 . Yes, that message has been there for months - I think it was a left-over from the grub 1 to grub 2 switch in f15->f16 - i.e. if one is too lazy to remove grub1 bits from /boot - I am just not motivated to touch /boot (even for "cleaning up") unless it is a clear problem. Hi I don't have the f17 repo installed, however I am on a F16 boot. The thing is I did preupgrade and I expected it to install these repos, booting back into f16 i think it shouldn't indicate the F17 repos. I've done the manual update via yum a couple of times in earlier releases (f14-f15) - thats when I used to manually add the repo. I can try adding the F17 repo and then retrying the preupgrade, considering that it is has been able to install a few of the f17 apps, I thought it was able to get the repo info correctly. Looks like I am stuck on F16 for now.... the DVD download will take time and i'll have to hunt for a DVD to try the DVD upgrade route The grubby update fixed the issue (tested i686 minimal install and preupgrade-cli). Hi Will this be added to mainstream? Or should I manually update grubby and then try preupgrade ? Thanks Anup Anup, fixed grubby is now in Fedora 17 stable updates, no manual steps should be necessary. Everything should work out-of-the-box. hi, no update for F16 ? and preupgrade from F15 to F16 ? I will not hit the same problem ? Hi I still dont get the new Kernel installed.... Looks like I am still hitting #820351 Thanks Anup akvasu: I meant did the updated system, after preupgrade had completed, have the f17 repos enabled. 8.12 is pushed stable now, so let's close this. sergio: things work differently in an f15->f16 preupgrade, this bug would not happen. (In reply to comment #27) > So - when you're upgrading from 16 to 17, you get the good old pre-F16 > default behaviour back. You get "bootloader --upgrade --location=none" . So > in preupgrade, we're just telling anaconda to update the existing bootloader > configuration, not to reinstall the bootloader. That is still an issue. Leaving the system in a working state where a grub2-mkconfig cause boot time error messages is not a good idea. That require that we in all future should say "whenever you run grub2-mkconfig you have to run grub2-install first". That issue in f16 preupgrade is now tracked in Bug 827987. I ran preupgrade-cli and all packages where downloaded. after rebooting I got a grub entry to start the installation process but once it starts booting suddenly drops me to the dracut shell with the no /dev/root msg. Booted into another OS on the same machine(another HDD)to check the files and all the installation files are under /var/cache/yum/ directory but there is nothing in /boot directory.Some directories are empty too; so it looks like the installation process haven't start yet. If you get a 'no /dev/root' error at that point, then you're hitting some other problem from what's discussed in this bug report. That error message in itself isn't very useful - it just means that anaconda wasn't able to initialize for some reason, the reason could be one of many, and always produces the same ultimate /dev/root error. One known outstanding bug which can trigger that error is https://bugzilla.redhat.com/show_bug.cgi?id=813973 , if you have a small /boot partition. You could check and see if you're hitting that. If not, file a new report. thanks! |