Bug 750469

Summary: anaconda considers the USB stick it's installing from a valid bootloader target
Product: [Fedora] Fedora Reporter: Sandro Mathys <sandro>
Component: anacondaAssignee: Brian Lane <bcl>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 16CC: anaconda-maint-list, awilliam, cwickert, dlehman, jarsmith, j.clarke3, jonathan, k.georgiou, kparal, mishu, rbergero, robatino, samuel-rhbugs, satellitgo, tflink, vanmeeuwen+fedora
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: anaconda-16.25-1.fc16 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-11-18 11:58:14 EST Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Attachments:
Description Flags
upgrade.log from F15->F16 RC3 DVD upgrade with Grub 2 option
none
upgrade.log.syslog from F15->F16 RC3 DVD upgrade with Grub 2 option
none
anaconda.program.log from F15->F16 RC3 DVD upgrade with Grub Legacy option
none
anaconda.storage.log from F15->F16 RC3 DVD upgrade with Grub Legacy option
none
storage.log from RC5 none

Description Sandro Mathys 2011-11-01 05:05:32 EDT
Description of problem:
After upgrading F15 to F16 with the Final RC3 DVD, I see two issues:
1) Even though I chose to use Grub 2, the system will run Grub Legacy
2) No Fedora boot options are shown in Grub Legacy (only "Other" as there's a Win 7 installed)

I guess those bugs are closely related, therefore I chose to only open one bug report.

Version-Release number of selected component (if applicable):
F16 Final RC3 DVD

How reproducible:
Only tried once

Steps to Reproduce:
1. Install from F15 Live KDE
2. Upgrade to F16 with Final RC3 DVD and choose to use Grub 2
  
Actual results:
Grub Legacy with no Fedora boot options.

Expected results:
Grub 2 with Fedora boot option(s).

Additional info:
Comment 1 Sandro Mathys 2011-11-01 05:10:42 EDT
Proposing F16Blocker.

I have no idea whether this fits any specific test case, though. Also, by choosing to stick with Grub Legacy during upgrade there's probably a valid workaround available (untested yet).

Still, the option to change to Grub 2 during upgrade seems to be an (important) anaconda feature and the issue obviously can't be fixed post-release.
Comment 2 Sandro Mathys 2011-11-01 05:36:09 EDT
Created attachment 531089 [details]
upgrade.log from F15->F16 RC3 DVD upgrade with Grub 2 option
Comment 3 Sandro Mathys 2011-11-01 05:36:31 EDT
Created attachment 531090 [details]
upgrade.log.syslog from F15->F16 RC3 DVD upgrade with Grub 2 option
Comment 4 Sandro Mathys 2011-11-01 07:00:29 EDT
(In reply to comment #1)
> Proposing F16Blocker.
> 
> I have no idea whether this fits any specific test case, though. Also, by
> choosing to stick with Grub Legacy during upgrade there's probably a valid
> workaround available (untested yet).

Correction, there's no such workaround. Choosing to stick with the Grub Legacy results in the same issue.

So right now, DVD upgrade is not usable for me. To reproduce, it *might* (purely my guess based on heavy speculation) be necessary to have a "Other"-type of OS installed initially.
Comment 5 David Lehman 2011-11-01 11:14:36 EDT
Please attach /var/log/anaconda/anaconda.storage.log and /var/log/anaconda/anaconda.program.log from the installed system.
Comment 6 Sandro Mathys 2011-11-01 12:46:18 EDT
Damn, sorry I missed them earlier. Unfortunately your request came just late by ~5 minutes and I went home (produced this in the office)...and the system seems to be down now, can't get through to it remotely.

Since time is pressing: this should be fairly easy to reproduce if you have the necessary pieces (which I currently don't). I guess if you don't have some "Other"-OS available, a manually added fake boot entry (at top? as default?) should do the job (maybe it's even reproducible without non-Fedora entries).
Comment 7 Sandro Mathys 2011-11-01 13:47:20 EDT
Sorry, forgot to add one thing: during F15 installation I dropped lvm and /boot and only made a single / instead. I know, that's not standard and bad to block test cases...but I initially wanted to reproduce a very different bug than this one ;)
Comment 8 Sandro Mathys 2011-11-01 15:08:17 EDT
I changed the summary to reflect the actual issue as it's currently understood. If someone can make this shorter and more precise, be my guest :)
Comment 9 Adam Williamson 2011-11-01 15:43:14 EDT
Discussed at the 2011-11-01 emergency blocker review meeting. We agreed that we really need more information in order to assess this one; we trust Sandro that he didn't do anything really dumb, but on the other hand, we do have multiple other tests of upgrading that did work. We could really do with seeing Sandro's logs to know what went wrong.

I'm currently trying to reproduce this in a VM, by doing an F15 KDE live install alongside Windows and then upgrading to F16.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 10 Adam Williamson 2011-11-01 16:59:08 EDT
Couldn't reproduce this. I installed F15 from KDE live alongside Windows - actually used a similar layout to Sandro as I'm low on space in that VM, a 300MB /boot, 500MB swap, and ~5GB ext4 / . Updated F15 to latest. Booted from F16 DVD and upgraded the installed system. Upgrade completed successfully and boots to grub2 with entries for F16 and Windows, which all work.

Would still like to see Sandro's logs, though.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 11 Sandro Mathys 2011-11-01 18:43:06 EDT
Adam did an additional test without /boot and swap, i.e. just with a Windows and a / partition, exactly like my tests were performed. He still couldn't reproduce the issue like that.

So my next and probably last (unqualified) guess would be that the issue's root cause has something to do with installation/upgrade being performed from an usb stick. We've recently seen/fixed one or two bugs like that IIRC so it's not totally unlikable.

Still, can't do any tests myself yet but will attach the logs of my last test and do more tests at around 7 AM UTC. Will also be back in IRC by then, afk in the meantime ;)
Comment 12 Adam Williamson 2011-11-01 19:06:01 EDT
it could be related to the USB stick, yeah. Did you check and see if the bootloader got written to the USB stick?



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 13 Adam Williamson 2011-11-01 19:06:48 EDT
Voting -1 blocker on this as, whatever's causing it, it doesn't seem terribly easy to hit, and our upgrade criterion is intentionally modest: we never expect all upgrades to work, we mostly just expect the most simple case to definitely work clean.
Comment 14 Tim Flink 2011-11-01 23:32:42 EDT
Also voting -1 blocker on this since we can't seem to reproduce it easily in reasonable scenarios.
Comment 15 Adam Williamson 2011-11-02 00:24:02 EDT
rejecting as blocker based on these votes and the discussion at today's meeting.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 16 Sandro Mathys 2011-11-02 03:20:17 EDT
Created attachment 531267 [details]
anaconda.program.log from F15->F16 RC3 DVD upgrade with Grub Legacy option
Comment 17 Sandro Mathys 2011-11-02 03:20:40 EDT
Created attachment 531268 [details]
anaconda.storage.log from F15->F16 RC3 DVD upgrade with Grub Legacy option
Comment 18 Sandro Mathys 2011-11-02 03:23:24 EDT
There you go, the log files you wanted are now waiting for your inspection.

But let me add new important intel, too: during upgrade, anaconda _by default_ chooses the usb stick (Corsair Voyager) to install the bootloader to. It says the usb stick is the "First BIOS drive", the SATA-disk is the second only.
Comment 19 Adam Williamson 2011-11-02 04:14:33 EDT
*** Bug 737430 has been marked as a duplicate of this bug. ***
Comment 20 Adam Williamson 2011-11-02 04:18:05 EDT
Adjusting summary, assigning.

So it looks like the work we did to filter out the USB stick anaconda is running from as a valid bootloader target didn't apply to upgrades :(. Impact is that if you try to upgrade to F16 using the installer written to a USB stick, the bootloader winds up on the USB stick.

Not sure if this is workaroundable while still using the stick. The dupe - 737430 - reports that you used to be able to 're-order' the drives to get around this, but that option isn't there any more. The obvious workaround would be 'write it to a disc instead', I guess. dlehman, if you can think of another workaround, that'd be good...

Re-proposing as a blocker so we can kick it around at the go/no-go tomorrow.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 21 Adam Williamson 2011-11-02 04:57:04 EDT
ccs for blocker info



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 22 Kamil Páral 2011-11-02 09:07:36 EDT
If the bootloader location defaults to an USB stick when doing upgrade, I'm +1 to blocker.
Comment 23 David Lehman 2011-11-02 11:22:20 EDT
Someone has changed the way to initramfs stuff works and in the process removed /run/initramfs/tmp/root.info, which is the only known way of identifying the actual block device that backs the live device.

If you booted from the USB your system will tell anaconda that that is the preferred boot device, so yeah -- given that we have no way of knowing that the USB device is the live backing device and the BIOS tells us it's the boot disk of course we default to that as the bootloader target. People need to give us something to work with if they expect pleasant results.
Comment 24 Adam Williamson 2011-11-02 12:03:27 EDT
/run/initramfs/tmp/root.info exists on both my booted F16 systems, and there's a stanza in /usr/share/dracut/modules.d/99base/init which tries to generate it. So I'm not sure why it's missing in the anaconda environment.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 25 Adam Williamson 2011-11-02 16:51:57 EDT
So, here's some info on this bug.

It's not exactly a new bug. We've never successfully filtered out the USB stick the system is being installed from as a potential bootloader target.

This specific impact of the bug, however, is more or less new to F16 - or rather, it's much more significant in F16 than in previous releases.

Previously the default action for the bootloader on upgrade was just to update the bootloader configuration - i.e. update grub.conf but don't write the bootloader to disk or anything.

For F16 we changed to the 'write new bootloader configuration' action by default, because we want to migrate people to grub2 when they upgrade.

So, in F15 and earlier, you could likely hit this bug by picking 'write new bootloader configuration' when upgrading, but few people will have done that. In F16, 'write new bootloader configuration' is the only viable option on upgrade. ('skip bootloader' is the other available option, but that will usually leave the system unbootable; it's not a workaround).

This bug does not in fact only affect upgrades, but on non-upgrade paths, its impact is minimized, because in F16 we now always display the 'cleardiskssel' window on install paths when more than one possible target disk is present, which is the one that shows 'data storage devices' on the left and 'install target devices' on the right and asks you to put every disk on the system into one bin or the other, and requires you to pick just one disk from the 'install target devices' set to be the bootloader target. So you don't really experience this issue as a 'bug' on the 'new install' paths, because you always hit that window and pick a sensible bootloader target disk.

Interestingly, in F15, we didn't do this on the 'custom partition layout' path: the cleardiskssel screen wasn't shown on that path. So you could actually hit this bug in F15 just by doing a 'custom partition' install from a non-live USB stick; I've just verified that.

However, in F15, if you change the 'BIOS Drive Order' in the 'Boot loader device' dialog such that the disk you want to get the bootloader installed to was first in the list, that disk would become the one available via the 'Master Boot Record (MBR)' radio button. In F16, this is not the case: the 'Master Boot Record (MBR)' radio button is stuck referring to whichever disk was the preferred stage1 target when the dialog popped up. Even if you change the 'BIOS Drive Order' drop-down, the 'Master Boot Record' radio button target never changes.

So in F15 you could hit this bug on both a fresh install with custom partitioning and an upgrade if you chose the non-default 'write new bootloader configuration' option, but you could at least work around it if you were paying attention when you hit the 'Boot loader device' dialog.

In F16 you can only really experience the bug as a problem when doing an upgrade, but you hit it with the *default* bootloader configuration option, and you cannot work around the bug. If you hit it, it's impossible to convince anaconda to install the bootloader where you actually need it to go, beyond doing it manually once the install is complete.

Interestingly, at least in my testing, this bug does not occur if the USB stick is written with dd: we think this causes anaconda to filter the USB stick out as a potential bootloader target because it considers its disklabel invalid. The bug occurs only if the USB stick is written with livecd-iso-to-disk (or, probably, livecd-creator).

dlehman has come up with a potential fix for this - http://dlehman.fedorapeople.org/updates-750469.0.img - but in my initial testing it appears to be buggy: it causes a crash when you hit "Next" with "install new bootloader configuration" selected. The idea is to filter out:

* the disk that shows up as /dev/live (this catches actual live boot cases)
* any disk with the label LIVE (this should catch livecd-iso-to-disk written sticks)
* any disk with an iso9660 filesystem (this should catch dd-written sticks)

as a valid bootloader target, as I understand the fix. We believe there's no valid use case for installing the bootloader to such a device.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 26 Adam Williamson 2011-11-02 20:12:33 EDT
http://dlehman.fedorapeople.org/updates-750469.1.img contains what seems to be a working fix for this, in initial testing. testing more extensively now.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 27 Fedora Update System 2011-11-02 21:14:44 EDT
anaconda-16.25-1.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/anaconda-16.25-1.fc16
Comment 28 Fedora Update System 2011-11-03 00:59:27 EDT
anaconda-16.25-1.fc16 has been pushed to the Fedora 16 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 29 Samuel Sieb 2011-11-03 13:35:23 EDT
It still doesn't work with RC5.  Looking at the list of conditions from comment 25, none of those match this case.  The label on the disk is the drive label, it's not "LIVE".  Looking in the log, it still considers both drives valid stage1 devices and picks sdb because it is first in the list.
Comment 30 Samuel Sieb 2011-11-03 13:36:58 EDT
Created attachment 531615 [details]
storage.log from RC5
Comment 31 Sandro Mathys 2011-11-03 13:48:46 EDT
Reopening to have this new development discussed in the go/no-go meeting later on.
Comment 32 Adam Williamson 2011-11-03 14:41:56 EDT
So, we did identify a non-fixed case thanks to Samuel.

Anaconda relies on the USB stick to have the label 'LIVE' in order to detect it and filter it out. Currently, livecd-iso-to-disk applies this label when you use --format to format the stick, but that's not the default and is not mentioned in the install guide:

https://docs.fedoraproject.org/en-US/Fedora/15/html/Installation_Guide/Making_USB_Media-UNIX_Linux.html

so we can't expect that people are doing this as a matter of course.

However, we think it's probably best to fix this by having livecd-iso-to-disk change the label even when not formatting the disk (while giving you the option to bail if you don't want to do that). That means we don't have to block Fedora 16 for it. (Fixing it in anaconda would actually be quite difficult, as the only method we've been able to figure for reliably detecting the backing device in the absence of a label doesn't currently work, which was the issue before the RC5 fix).

So we can address the issue by updating livecd-iso-to-disk in F14, F15 and F16 before F16 is officially released, and probably noting in CommonBugs that you might run into this if you don't update livecd-tools prior to writing the stick.

Hence, this doesn't need to block release: dropping from the blocker list.

bcl is working on a livecd-iso-to-disk fix.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 33 Samuel Sieb 2011-11-05 19:18:54 EDT
It turns out that this is very easy to workaround.  If you turned the full DVD installer into a live USB stick, there's no problem now.  The USB drive is disqualified by the recent changes.  If you're using the net installer iso, then when you get the language selection prompt, take out the USB drive.  Now it won't be an option when you get to the bootloader section.
Comment 34 Adam Williamson 2011-11-05 20:31:23 EDT
there should be no difference in behaviour between the net install and the DVD image per se; it's more likely there was a difference in how you burned them.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 35 Samuel Sieb 2011-11-06 01:04:39 EST
I used the same command to write them to the USB drives.  They are on different drives, but neither one has the "LIVE" label.  I will try the DVD one again and get the storage.log from it.
Comment 36 Adam Williamson 2011-11-06 01:28:32 EST
you can look through it for 'is_valid_stage1' and you'll see anaconda's test for whether a device is a valid bootloader stage1 install target. there are various things that can cause a device to fail the test.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 37 Samuel Sieb 2011-11-06 02:13:23 EST
Unfortunately, you are correct.  For whatever reason, they get scanned in a different order.  So in the one case, the hard drive comes first in the list and in the other, the USB drive does.
Comment 38 Christoph Wickert 2011-11-06 14:05:37 EST
Do we have a solution to this problem? anaconda-16.25-1.fc16 correctly excluded the USB stick from the targets, but I still I end up with grub1 instead of grub2 and only "Other" in the menu.

I choose to install grub to the /boot partition rather then the MBR. grub(1) was there, too.
Comment 39 David Lehman 2011-11-07 08:50:26 EST
(In reply to comment #38)
> Do we have a solution to this problem? anaconda-16.25-1.fc16 correctly excluded
> the USB stick from the targets, but I still I end up with grub1 instead of
> grub2 and only "Other" in the menu.
> 
> I choose to install grub to the /boot partition rather then the MBR. grub(1)
> was there, too.

What you're talking about is a completely different bug. "This problem" is failure to filter out the USB device. Any other problem you're having will require a bug report of its own. Thanks.
Comment 40 Brian Lane 2011-11-07 10:29:19 EST
Samuel, you need to make sure that your USB stick's fs is labeled as LIVE. The most recent release of liveusb-creator (not sure if that's in testing or stable) and all stable versions of livecd-iso-to-disk so this. If you are writing a DVD or netinst.iso there should be no difference in the fs label.
Comment 41 Christoph Wickert 2011-11-07 10:44:27 EST
(In reply to comment #39)
> What you're talking about is a completely different bug. "This problem" is
> failure to filter out the USB device.

David, please read the initial bug description, both the reporter and I have exactly the same problem, the summary was changed later. By filtering out the USB stick you may have done your part to fix the bug, however this is only half of what needs to be done to really solve the problem. Anaconda still doesn't write a correct grub2 configuration.

Adam, you keyworded this bug "CommonBugs", however I don't see is on 
https://fedoraproject.org/wiki/Common_F16_bugs
Does this mean there is no workaround?
Comment 42 Samuel Sieb 2011-11-07 11:36:10 EST
Brian, I realize that.  It's just that at first I thought there might be something different with using the DVD as a live disk, but it turned out to just be some sort of ordering difference.
Comment 43 Adam Williamson 2011-11-07 13:22:30 EST
bcl: the livecd-tools updates didn't go stable yet :/ working on that.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 44 David Lehman 2011-11-07 13:25:16 EST
Someone please attach a recent set of logs from the case where the USB media is
not problematic but no bootloader is installed.
Comment 45 Adam Williamson 2011-11-08 00:19:23 EST
christoph: we are not aware that there is another 'half' to this problem. I tested the fix and it worked, and when I say 'worked', I mean 'the upgraded system booted correctly using a grub2 bootloader which was installed as part of the upgrade'. as dlehman says, I suspect your issue is not at all related to the problem we discussed and fixed in this bug.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 46 Adam Williamson 2011-11-08 00:20:03 EST
christoph: it's worth noting that we did just about no programmed testing of bootloader installation outside of the MBR, chainloading, that kinda stuff.
Comment 47 Adam Williamson 2011-11-17 19:54:00 EST
don't really need commonbugs now f14 and f15 are both updated.



-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers
Comment 48 Brian Lane 2011-11-18 11:58:14 EST
If there are other problems with grub1/grub2 or whatever lets open a bug for that and keep this one from becoming a big mess.