Bug 1884042
| Summary: | From Fed32, grub cannot boot Fed33 if Fed33 is installed with live defaults for btrfs and the reverse | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Leslie Satenstein <lsatenstein> | ||||||||||||||||
| Component: | os-prober | Assignee: | Hedayat Vatankhah <hedayatv> | ||||||||||||||||
| Status: | CLOSED WORKSFORME | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||||||||||||
| Severity: | high | Docs Contact: | |||||||||||||||||
| Priority: | unspecified | ||||||||||||||||||
| Version: | 33 | CC: | agk, bugzilla, fmartine, hedayatv, lkundrak, lsatenstein, pjones | ||||||||||||||||
| Target Milestone: | --- | ||||||||||||||||||
| Target Release: | --- | ||||||||||||||||||
| Hardware: | x86_64 | ||||||||||||||||||
| OS: | Linux | ||||||||||||||||||
| Whiteboard: | |||||||||||||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||||||||||
| Doc Text: | Story Points: | --- | |||||||||||||||||
| Clone Of: | Environment: | ||||||||||||||||||
| Last Closed: | 2020-10-23 03:42:12 UTC | Type: | Bug | ||||||||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||||||||
| Documentation: | --- | CRM: | |||||||||||||||||
| Verified Versions: | Category: | --- | |||||||||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||||||
| Embargoed: | |||||||||||||||||||
| Attachments: |
|
||||||||||||||||||
Created attachment 1717989 [details]
some debugging information (grub.cfg x 2) , and a /etc/fstab
22 should be 33 in previous message From Fedora 33, grub2-mkconfig cannot create a grub.cfg containing a second (other) fedora33 distro (both on different disks but same system) It's not a supported configuration. There is a partial work around where you share /boot and /boot/efi for the two Fedora's and then it can work if you also preserve /boot/loader/entries/* and /boot/efi/EFI/fedora/grubenv for the Fedora 32 installation before doing the Fedora 33 installation. And then restore them post-install. This will avoid running into bug 1874724. Hi Chris I write C code and I switch compilers often. I use hard disk, as some of my software looks at the total environment, meaning VMs are out. My /dev/sda has Fedora 32. ext4s /dev/sdb has Fedora 33 btrfs gnome /dev/sdc has Fedora 33 btrfs xfce,kde,... /dev/sdd has Leap 152 btrfs xfce,gnome As long as all systems are ext4/xfs types, grub works as expected. Each system sees the other. When any system is btrfs based, grub2-mkconfig takes note of that system, but does not produce any grub.cfg entries. and the grub.cfg entry for the ext4 system crashes the top (first entry in the menu list, but works fine with the remaining entries. And of course grub.cfg works ok with Debian systems but not with arch based systems. So, do I raise another bug report saying that Fedora 32 grub.cfg fails to create entries for Fedora 33? Argument in favor of change ========================== If my host cpu system had two (Fedora/Centos) btrfs based servers, one backing up the other. I would need the cpu bios password if/when I need to change from one dstro to another what I mean by "not a supported configuration" is that there is no Fedora release criterion for this at all. Dual boot (or mutiboot) Linux isn't supported in any configuration. Dual boot Windows and macOS are supported in that there's a release criterion for them. Strictly speaking it's not GRUB that finds the various ext4 installations, but os-prober. And it's not aware of the "flat" btrfs layout used by Fedora, doing so has some difficulties because how is it supposed to handle possibly dozens or hundreds of identical snapshots? It would require pretty sophisticated support. But this is already a problem for os-prober not finding other kinds of custom installations either. Hence BootLoaderSpec. If the bootloader conforms to that spec, and distros conform to that spec, everything could share one $BOOT volume. But while Fedora is implementing most aspects of BootLoaderSpec, other distros aren't yet. The reality is that mutiboot user-experience is not that friendly and it's difficult to make it work automatically when there are so many ways of booting that aren't discoverable. Strictly speaking it's os-prober that finds OS installs and then the various grub-mkconfig scripts create an entry for it. I don't know if this is in scope for os-prober to: (a) if btrfs (b) search for inode 265 with name "root" or "root??" as candidate subvolumes (c) search for /etc/fstab in only those candidates NOTE 1: inode 265 on btrfs is always a subvolume or subvolume snapshot NOTE 2: Anaconda defaults to naming the subvolume mounted at / as "root" but if it exists already it'll name it "root00" and then "root01" and so on. os-prober shouldn't go searching beyond the top-level of the file system or inside subvolumes with other names because they're could be dozens or even thousands of subvolumes for things like containers or snapshots from snapper and so on. I *think* grub-mkconfig scripts already know from /etc/fstab to use the 'subvol=root' option to add the 'rootflags=subvol=root' parameter and that os-prober doesn't need to worry about that. Just that right now it doesn't know to look inside of subvolumes for an /etc/fstab so it never finds any installed OS's. @Leslie Please run os-prober and then send its output & full logs which appear in system journal. It'd be very helpful. However, let me explain a little about current os-prober state with regard to its upstream & btrfs support: 1. os-prober has a very inactive & unresponsive upstream, and officially doesn't have any special BTRFS support. 2. Fedora has some support for btrfs in its os-prober, contributed by a contributor. The patch was sent upstream years ago but it didn't get any kind of attention (still waiting!). 3. AFAIK, Fedora patches for os-prober was used by some other distros. IIRC, opensuse also used it. However, it is very likely that others might have enhanced support for their distros. Specially since OpenSUSE has been using BTRFS by default for many years, they might have much better btrfs support. I'll try to look into their patches to see if there are things to take back. 4. The patch is expected to look into all BTRFS subvolumes, except snapshot and ro subvolumes, to find another OS. TBH, I have not looked at F33 BTRFS setup, but it is probably expected to find it. It tries to find a GNU/Linux root filesystem, the same way it looks in other file systems (e.g. ext4) for root filesystems. So, sobvolume names doesn't matter. So, first of all, os-prober doesn't have a healthy upstream, so it is better for Fedora to try to not depend on it as much as possible: hopefully BLS will help it happen! Then, I'll try to see if we can find & fix the current problem. Honestly, I don't have a BTRFS system right now, so any help in debugging and solving the issue is highly appreciated. I'll try to create the needed setup too, but I'm not sure I'll be able to do it fast enough before F33 release. os-prober logs are detailed so they might help us find the problem easily. Apparently, you also say that F32 boot entry in F33 grub doesn't work too, right? So, you cannot boot F32 from F33 grub menu? If that's the case, it is a completely different problem than not recognizing other BTRFS OSes on a system. Please explain each problem separately. Actually, it is probably better to be 2 separate bug reports. Had issues os-prober / as root.... rmdir: failed to remove '/var/lib/os-prober/mount': Device or resource busy rmdir: failed to remove '/var/lib/os-prober/mount': Device or resource busy rmdir: failed to remove '/var/lib/os-prober/mount': Device or resource busy /dev/sda2:openSUSE Leap 15.2:openSUSE:linux /dev/sda2:openSUSE Leap 15.2:openSUSE1:linux:btrfs:UUID=acc3bfe6-1a2f-426b-ba3b-e5172f64ef6d:subvol=@/.snapshots/1/snapshot rmdir: failed to remove '/var/lib/os-prober/mount': Device or resource busy rmdir: failed to remove '/var/lib/os-prober/mount': Device or resource busy rmdir: failed to remove '/var/lib/os-prober/mount': Device or resource busy /dev/sda5:openSUSE Leap 15.2:openSUSE2:linux /dev/sda5:openSUSE Leap 15.2:openSUSE3:linux:btrfs:UUID=f3a18eed-6b0d-4879-ab73-2f523d16c2dc:subvol=@/.snapshots/1/snapshot rmdir: failed to remove '/var/lib/os-prober/mount': Device or resource busy rmdir: failed to remove '/var/lib/os-prober/mount': Device or resource busy rmdir: failed to remove '/var/lib/os-prober/mount': Device or resource busy rmdir: failed to remove '/var/lib/os-prober/mount': Device or resource busy rmdir: failed to remove '/var/lib/os-prober/mount': Device or resource busy rmdir: failed to remove '/var/lib/os-prober/mount': Device or resource busy rmdir: failed to remove '/var/lib/os-prober/mount': Device or resource busy rmdir: failed to remove '/var/lib/os-prober/mount': Device or resource busy rmdir: failed to remove '/var/lib/os-prober/mount': Device or resource busy rmdir: failed to remove '/var/lib/os-prober/mount': Device or resource busy rmdir: failed to remove '/var/lib/os-prober/mount': Device or resource busy rmdir: failed to remove '/var/lib/os-prober/mount': Device or resource busy rmdir: failed to remove '/var/lib/os-prober/mount': Device or resource busy Thanks, but this is just the output. Please also send os-prober logs in system journal (run `journal -lf > output` in a cmdline, run os-prober in another, when it finished terminated the journal command and upload the output file). And before running os-prober, try to unmount whatever might be still mounted over /var/lib/os-prober/mount Hi Hedayat. I will comply, tomorrow am. If you have large disk or a two disk system, you can install two fedora 33 systems side by each. Use default gnome or kde or xfce or other. Then try to include the second fedora system from the booted version. sudo grub2-mkconfig -o /tmp/grub.cfg (to see the grub.cfg that is created or used for testingé final: sudo grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg OK I have a reproducer. Setup: * Fedora-Workstation-Live-x86_64-33-20201008.n.0.iso * Install with defaults (automatic partitioning = btrfs) * reboot and shrink the btrfs and GPT to create ~50% free space * reboot to the same installer image and do another automatic install vda 252:0 0 100G 0 disk ├─vda1 252:1 0 600M 0 part /boot/efi ├─vda2 252:2 0 1G 0 part ├─vda3 252:3 0 45G 0 part ├─vda4 252:4 0 1G 0 part /boot └─vda5 252:5 0 52.4G 0 part /home This means the EFI System partition (vda1) is reused but not reformatted; but in effect the fedora directory on it is "wiped" in favor of the most recent installation. There are two separate boot volumes, vda2 and vda4 respectively. And two Btrfs volumes vda3 and vda5 respectively. vda1,2,3 = 1st install. vda1,4,5 = 2nd install. In both cases the Btrfs volume have a subvolume for / and /home. Curiously the subvolumes don't have the same names: the 1st install names them "root" and "home", the 2nd install names them "root00" and "home00". While booted from the 2nd installation: # os-prober /dev/vda3:Fedora 33 (Workstation Edition Prerelease):Fedora:linux:btrfs:UUID=0d7f4e4e-4e0f-40ad-ac3b-3eb3e6cb61bf:subvol=root That is the 1st installation it has found. And yet no grub menu entry. Hmm, maybe it's a bug in grub2 afterall? Attaching the journal next. Created attachment 1721376 [details]
journal osprober
Created attachment 1721377 [details]
installer created grub.cfg
Created attachment 1721378 [details]
bash -x grub2-mkconfig
The previous attachment, and this attachment, are from the 2nd installation.
Created attachment 1721379 [details]
set -x 30_os-prober
Modify /etc/grub.d/30_os-prober s/set -e/set -x/ and then grub2-mkconfig
I'm not sure what's up. It sees vda3 Btrfs and subvol=root. But doesn't seem to figure out that vda2 is the boot volume that goes with it, and no entry is created.
Thanks! Well, it seems that there is a problem with linux-boot-prober command. I forgot about it completely :) Would you please also send the logs/trace of this command: linux-boot-prober btrfs UUID=0d7f4e4e-4e0f-40ad-ac3b-3eb3e6cb61bf subvol=root Thanks again. Unfortunately, I really can't setup something to test right now. Oct 14 10:13:18 chris[2472]: linux-boot-prober: debug: btrfs: partition=/dev/vda3, UUID=0d7f4e4e-4e0f-40ad-ac3b-3eb3e6cb61bf, subvol=root Oct 14 10:13:18 chris[2472]: linux-boot-prober: debug: mounted btrfs /dev/vda3, subvol=root on /var/lib/os-prober/mount Oct 14 10:13:18 chris[2472]: linux-boot-prober: debug: mapped UUID=7c9f0ee3-9ddc-498e-880a-91f754e36a98 to /dev/vda2 Oct 14 10:13:18 chris[2472]: linux-boot-prober: debug: found boot partition /dev/vda2 for linux system on /dev/vda3 Oct 14 10:13:18 chris[2472]: linux-boot-prober: debug: running /usr/libexec/linux-boot-probes/mounted/40grub2 /dev/vda3 /dev/vda2 /var/lib/os-prober/mount btrfs root Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: os-prober: debug: running /usr/libexec/os-probes/50mounted-tests on /dev/nvme0n1p1 Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: 50mounted-tests: debug: mounted using GRUB fat filesystem driver Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: 50mounted-tests: debug: running subtest /usr/libexec/os-probes/mounted/05efi Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: 05efi: debug: /dev/nvme0n1p1 is a FAT partition (mounted by GRUB) Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: 05efi: debug: /dev/nvme0n1p1 partition scheme is gpt Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: 05efi: debug: /dev/nvme0n1p1 partition type is c12a7328-f81f-11d2-ba4b-00a0c93ec93b Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: 05efi: debug: running subtest /usr/libexec/os-probes/mounted/efi/10elilo Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: 05efi: debug: running subtest /usr/libexec/os-probes/mounted/efi/20microsoft Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: 50mounted-tests: debug: running subtest /usr/libexec/os-probes/mounted/10freedos Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: 10freedos: debug: /dev/nvme0n1p1 is a FAT partition (mounted by GRUB) Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: 50mounted-tests: debug: running subtest /usr/libexec/os-probes/mounted/10qnx Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: 10qnx: debug: /dev/nvme0n1p1 is not a QNX4 partition: exiting Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: 50mounted-tests: debug: running subtest /usr/libexec/os-probes/mounted/20macosx Oct 14 13:03:32 Fed33b.localdomain macosx-prober[14371]: debug: /dev/nvme0n1p1 is not an HFS+ partition: exiting Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: 50mounted-tests: debug: running subtest /usr/libexec/os-probes/mounted/20microsoft Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: 20microsoft: debug: Skipping legacy bootloaders on UEFI system Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: 50mounted-tests: debug: running subtest /usr/libexec/os-probes/mounted/30utility Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: 30utility: debug: /dev/nvme0n1p1 is a FAT partition (mounted by GRUB) Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: 50mounted-tests: debug: running subtest /usr/libexec/os-probes/mounted/40lsb Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: 50mounted-tests: debug: running subtest /usr/libexec/os-probes/mounted/70hurd Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: 50mounted-tests: debug: running subtest /usr/libexec/os-probes/mounted/80minix Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: 50mounted-tests: debug: running subtest /usr/libexec/os-probes/mounted/83haiku Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: 83haiku: debug: /dev/nvme0n1p1 is not a BeFS partition: exiting Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: 50mounted-tests: debug: running subtest /usr/libexec/os-probes/mounted/90linux-distro Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: 50mounted-tests: debug: running subtest /usr/libexec/os-probes/mounted/90solaris Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: 50mounted-tests: debug: running subtest /usr/libexec/os-probes/mounted/efi Oct 14 13:03:32 Fed33b.localdomain systemd[1]: var-lib-os\x2dprober-mount.mount: Succeeded. ░░ Subject: Unit succeeded ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ The unit var-lib-os\x2dprober-mount.mount has successfully entered the 'dead' state. Oct 14 13:03:32 Fed33b.localdomain systemd[1840]: var-lib-os\x2dprober-mount.mount: Succeeded. ░░ Subject: Unit succeeded ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ The unit UNIT has successfully entered the 'dead' state. Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: os-prober: debug: running /usr/libexec/os-probes/50mounted-tests on /dev/nvme0n1p6 Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: 50mounted-tests: debug: /dev/nvme0n1p6 is a swap partition; skipping Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: os-prober: debug: btrfs volume uuid=d6a5cfb4-f0de-45de-91f0-80dfe18e8104 partition=/dev/nvme0n1p8 Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: os-prober: debug: running /usr/libexec/os-probes/50mounted-tests on btrfs /dev/nvme0n1p8 Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: 50mounted-tests: debug: begin btrfs processing for d6a5cfb4-f0de-45de-91f0-80dfe18e8104 Oct 14 13:03:32 Fed33b.localdomain leslie[14206]: 50mounted-tests: debug: btrfs volume d6a5cfb4-f0de-45de-91f0-80dfe18e8104 mounted Oct 14 13:03:32 Fed33b.localdomain systemd[1840]: var-lib-os\x2dprober-mount.mount: Succeeded. ░░ Subject: Unit succeeded lines 10115-10160 Created attachment 1721544 [details]
journalctl -xb -| grep os-prober
os-prober journalctl -xb | grep os-prober
linux-boot-prober btrfs UUID=0d7f4e4e-4e0f-40ad-ac3b-3eb3e6cb61bf subvol=root there is one file, one directory labels has the name of two other distros. SUSE Lint 15.2 archo Mount subdirectory is empty. comment 17 reply sudo su journalctl -xb | grep linux-boot-prober Oct 14 15:52:25 Fed33b.localdomain sudo[3513]: leslie : TTY=pts/0 ; PWD=/home/leslie ; USER=root ; COMMAND=/usr/bin/linux-boot-prober btrfs UUID=0d7f4e4e-4e0f-40ad-ac3b-3eb3e6cb61bf subvol=root Oct 14 15:52:25 Fed33b.localdomain leslie[3540]: linux-boot-prober: debug: btrfs: partition=, UUID=0d7f4e4e-4e0f-40ad-ac3b-3eb3e6cb61bf, subvol=root Oct 14 15:52:25 Fed33b.localdomain leslie[3540]: linux-boot-prober: warning: error mounting btrfs subvol=root UUID=0d7f4e4e-4e0f-40ad-ac3b-3eb3e6cb61bf Oct 14 15:52:25 Fed33b.localdomain leslie[3540]: linux-boot-prober: error: umount error, retrying after 1 sec With the Fedora 33 beta dated 20 October, I was able to boot a second Btrfs formatted Fedora. The fix is ok, will it make it into the Live version and will it be backported? I installed with custom settings, includeing mate, gnome, xfce, KDE etc etc. So far with reboot and each selected interface, the grub.cfg shows the other Fedora system Please backport to Fedora 32 Sorry, I wonder why I was not notified for replies after my last comment until this one. But, while os-prober was updated recently, and it is going to be in F33; the change is most probably not related to this bug. And I wonder why it has started to work for you. And I think it should not still work for Chris too. And his logs in comment 18 confirms this. The logs you sent in comment 22 is weird; as it shows some mount errors which is not expected. Did you try anything on your file systems? (e.g. trying to fsck for example?). However, about why it seems to have started to work for you. First of all, you are trying on an EFI system, right? And then, do you have /boot/grub2/grub.cfg in your OSes which are expected to be found by linux-boot-prober/os-prober? >Oct 14 15:52:25 Fed33b.localdomain leslie[3540]: linux-boot-prober: warning: error mounting btrfs subvol=root UUID=0d7f4e4e-4e0f-40ad-ac3b-3eb3e6cb61bf Hard to say what this is without a complete log including kernel messages. If there's no kernel message then it's not a file system problem. Could be a case of "mount: $MOUNTPOINTA: $DEVNODE already mounted on $MOUNTPOINTB" Because it's kinda common to have Btrfs mounted more than once on different mountpoints. This might also explain the subsequent umount error. >And then, do you have /boot/grub2/grub.cfg in your OSes which are expected to be found by linux-boot-prober/os-prober? At least on Fedora, UEFI has $EFISYSTEMPARTITION/EFI/fedora/grub.cfg and /boot/grub2/ only contain lrwxrwxrwx. 1 root root 25 Aug 31 06:12 grubenv -> ../efi/EFI/fedora/grubenv drwxr-xr-x. 1 root root 12 Jun 23 2019 themes |
Created attachment 1717975 [details] fstab from Fedora 32 Description of problem: I have three Fedora systems on my computer Fed32 (ext4 for all files) \ No problem between these setups. Fed33 (ext4 for all files)/ But With Fedora 33 Live setup (full disk defaults) Fedora33 grub menu crashes Fedora 32 if Fedora 32 was selected Fedora32 grub menu does not create entries for Fedora 33 btrfs live full disk installation. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: I will be changing my system bios to boot Fedora 22 btrfs and post that grub.cfg config. Given the two systems are Fedora 32 and Fedora 33 (btrfs), Can You create the two disk setup and perform your own debugging?