Up to date Fedora 20 fails to upgrade with fedup to Fedora 21. Steps to Reproduce: 1. run "fedup --network 21 --product=nonproduct" and then reboot 2. 3. Actual results: Upgrade does not work as /root is not found: [ OK ] Reached target Basic System. [ 127.285515] dracut-initqueue[462]: Warning: Cancelling resume operation. Device not found. [ TIME ] Timed out waiting for device dev-mapper-BCACHE\x2droot.device. [DEPEND] Dependency failed for /Warning: /dev/BCACHE/root does not exist Warning: /dev/BCACHE/swap does not exist Warning: /dev/mapper/BCACHE-root does not exist Expected results: Upgrade runs smoothly Additional info: System has lvm and bcache in use for /root. /boot is on a plain partition: [root@jesse ~]# fdisk -l Disk /dev/sda: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: dos Disk identifier: 0x0007a195 Device Boot Start End Blocks Id System /dev/sda1 2048 158259199 79128576 8e Linux LVM /dev/sda2 158259200 1953525167 897632984 83 Linux Disk /dev/sdb: 238.5 GiB, 256060514304 bytes, 500118192 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x00083c53 Device Boot Start End Blocks Id System /dev/sdb1 * 2048 1026047 512000 83 Linux /dev/sdb2 1026048 500118191 249546072 83 Linux Disk /dev/mapper/BCACHE-swap: 16 GiB, 17179869184 bytes, 33554432 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/mapper/BCACHE-root: 50 GiB, 53687091200 bytes, 104857600 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk /dev/mapper/BCACHE-home: 785 GiB, 842887331840 bytes, 1646264320 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes [root@jesse ~]# [root@jesse ~]# pvdisplay --- Physical volume --- PV Name /dev/bcache0 VG Name BCACHE PV Size 856.05 GiB / not usable 2.70 MiB Allocatable yes PE Size 4.00 MiB Total PE 219148 Free PE 1292 Allocated PE 217856 PV UUID rRcyKe-RTaJ-qMyh-DSkB-Mvas-Wusx-AvBETz "/dev/sda1" is a new physical volume of "75.46 GiB" --- NEW Physical volume --- PV Name /dev/sda1 VG Name PV Size 75.46 GiB Allocatable NO PE Size 0 Total PE 0 Free PE 0 Allocated PE 0 PV UUID D30Qhk-ktIX-1U66-NF3W-IBwL-fW8v-8oohIs [root@jesse ~]# [root@jesse ~]# lvdisplay --- Logical volume --- LV Path /dev/BCACHE/root LV Name root VG Name BCACHE LV UUID 51ktcG-m01V-SRvR-Qgba-9xz7-oV9s-cvVgsM LV Write Access read/write LV Creation host, time localhost.localdomain, 2013-12-10 13:51:20 +0200 LV Status available # open 1 LV Size 50.00 GiB Current LE 12800 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:1 --- Logical volume --- LV Path /dev/BCACHE/swap LV Name swap VG Name BCACHE LV UUID ubkABU-dVtR-lHX1-u8j7-TUuv-2MOh-0H2Mbj LV Write Access read/write LV Creation host, time localhost.localdomain, 2013-12-10 13:51:40 +0200 LV Status available # open 2 LV Size 16.00 GiB Current LE 4096 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:0 --- Logical volume --- LV Path /dev/BCACHE/home LV Name home VG Name BCACHE LV UUID fxJLpE-gVXt-jz50-90DB-LQvW-NmXl-fbrmtD LV Write Access read/write LV Creation host, time localhost.localdomain, 2013-12-10 13:52:43 +0200 LV Status available # open 1 LV Size 785.00 GiB Current LE 200960 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:2 [root@jesse ~]# [root@jesse ~]# bcache-status --- bcache --- UUID c48cd97c-7551-467f-b480-f12a56fb04cf Block Size 512 B Bucket Size 512.00 KiB Congested? False Read Congestion 2.0ms Write Congestion 20.0ms Total Cache Size 238 GiB Total Cache Used 169 GiB (71%) Total Cache Unused 69 GiB (29%) Evictable Cache 238 GiB (100%) Replacement Policy [lru] fifo random Cache Mode [writethrough] writeback writearound none Total Hits 92249 (95%) Total Misses 3974 Total Bypass Hits 21040 (100%) Total Bypass Misses 0 Total Bypassed 1 GiB [root@jesse ~]# Copy paste from emergency mode via serial console: Entering emergency mode. Exit the shell to continue. Type "journalctl" to view system logs. You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot after mounting them and attach it to a bug report. :/# cat /run/initramfs/rdsosreport.txt + cat /lib/dracut/dracut-038-30.git20140903.fc21 dracut-038-30.git20140903.fc21 + cat /proc/cmdline BOOT_IMAGE=/vmlinuz-fedup root=/dev/mapper/BCACHE-root ro rd.lvm.lv=BCACHE/swap vconsole.font=latarcyrheb-sun16 rd.lvm.lv=BCACHE/root rhgb quiet consoleblank=0 crashkernel=128M console=tty0 console=ttyS0,115200 LANG=en_US.UTF-8 + '[' -f /etc/cmdline ']' + for _i in '/etc/cmdline.d/*.conf' + '[' -f /etc/cmdline.d/90lvm.conf ']' + echo /etc/cmdline.d/90lvm.conf /etc/cmdline.d/90lvm.conf + cat /etc/cmdline.d/90lvm.conf rd.lvm.lv=BCACHE/swap rd.lvm.lv=BCACHE/root + for _i in '/etc/cmdline.d/*.conf' + '[' -f /etc/cmdline.d/95resume.conf ']' + echo /etc/cmdline.d/95resume.conf /etc/cmdline.d/95resume.conf + cat /etc/cmdline.d/95resume.conf resume=/dev/mapper/BCACHE-swap + cat /proc/self/mountinfo 0 0 0:1 / / rw shared:1 - rootfs rootfs rw 15 0 0:15 / /sys rw,nosuid,nodev,noexec,relatime shared:2 - sysfs sysfs rw 16 0 0:3 / /proc rw,nosuid,nodev,noexec,relatime shared:7 - proc proc rw 17 0 0:5 / /dev rw,nosuid shared:8 - devtmpfs devtmpfs rw,size=16340264k,nr_inodes=4085066,mode=755 18 15 0:16 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:3 - securityfs securityfs rw 19 17 0:17 / /dev/shm rw,nosuid,nodev shared:9 - tmpfs tmpfs rw 20 17 0:11 / /dev/pts rw,nosuid,noexec,relatime shared:10 - devpts devpts rw,gid=5,mode=620,ptmxmode=000 21 0 0:18 / /run rw,nosuid,nodev shared:11 - tmpfs tmpfs rw,mode=755 22 15 0:19 / /sys/fs/cgroup ro,nosuid,nodev,noexec shared:4 - tmpfs tmpfs ro,mode=755 23 22 0:20 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:5 - cgroup cgroup rw,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 24 15 0:21 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:6 - pstore pstore rw 25 22 0:22 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime shared:12 - cgroup cgroup rw,cpuset 26 22 0:23 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime shared:13 - cgroup cgroup rw,cpu,cpuacct 27 22 0:24 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime shared:14 - cgroup cgroup rw,memory 28 22 0:25 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime shared:15 - cgroup cgroup rw,devices 29 22 0:26 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime shared:16 - cgroup cgroup rw,freezer 30 22 0:27 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime shared:17 - cgroup cgroup rw,net_cls,net_prio 31 22 0:28 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime shared:18 - cgroup cgroup rw,blkio 32 22 0:29 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime shared:19 - cgroup cgroup rw,perf_event 33 22 0:30 / /sys/fs/cgroup/hugetlb rw,nosuid,nodev,noexec,relatime shared:20 - cgroup cgroup rw,hugetlb 34 0 0:31 / /var/lib/nfs/rpc_pipefs rw,relatime shared:21 - rpc_pipefs rpc_pipefs rw 56 15 0:32 / /sys/kernel/config rw,relatime shared:22 - configfs configfs rw + cat /proc/mounts rootfs / rootfs rw 0 0 sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,nosuid,size=16340264k,nr_inodes=4085066,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 rpc_pipefs /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 + blkid /dev/sda1: UUID="D30Qhk-ktIX-1U66-NF3W-IBwL-fW8v-8oohIs" TYPE="LVM2_member" PARTUUID="0007a195-01" /dev/sda2: UUID="e44a2a6b-4b5c-475f-8898-e872d011565b" TYPE="bcache" PARTUUID="0007a195-02" /dev/sdb1: UUID="aa2a7ee9-bda8-45c9-a9be-21e1c77687f7" TYPE="ext4" PARTUUID="00083c53-01" /dev/sdb2: UUID="a19c3d6b-031e-4b46-a779-268dacd404d9" TYPE="bcache" PARTUUID="00083c53-02" + blkid -o udev ID_FS_UUID=D30Qhk-ktIX-1U66-NF3W-IBwL-fW8v-8oohIs ID_FS_UUID_ENC=D30Qhk-ktIX-1U66-NF3W-IBwL-fW8v-8oohIs ID_FS_TYPE=LVM2_member ID_FS_PARTUUID=0007a195-01 ID_FS_UUID=e44a2a6b-4b5c-475f-8898-e872d011565b ID_FS_UUID_ENC=e44a2a6b-4b5c-475f-8898-e872d011565b ID_FS_TYPE=bcache ID_FS_PARTUUID=0007a195-02 ID_FS_UUID=aa2a7ee9-bda8-45c9-a9be-21e1c77687f7 ID_FS_UUID_ENC=aa2a7ee9-bda8-45c9-a9be-21e1c77687f7 ID_FS_TYPE=ext4 ID_FS_PARTUUID=00083c53-01 ID_FS_UUID=a19c3d6b-031e-4b46-a779-268dacd404d9 ID_FS_UUID_ENC=a19c3d6b-031e-4b46-a779-268dacd404d9 ID_FS_TYPE=bcache ID_FS_PARTUUID=00083c53-02 + ls -l /dev/disk/by-id /dev/disk/by-uuid /dev/disk/by-id: total 0 lrwxrwxrwx 1 root 0 9 Mar 6 14:25 ata-ADATA_SSD_SP900_256GB-DL2_7D3020002360 -> ../../sdb lrwxrwxrwx 1 root 0 10 Mar 6 14:25 ata-ADATA_SSD_SP900_256GB-DL2_7D3020002360-part1 -> ../../sdb1 lrwxrwxrwx 1 root 0 10 Mar 6 14:25 ata-ADATA_SSD_SP900_256GB-DL2_7D3020002360-part2 -> ../../sdb2 lrwxrwxrwx 1 root 0 9 Mar 6 14:25 ata-PLDS_DVD+_-RW_DH-16ACS_H_HDD0H7363938553B6A01 -> ../../sr0 lrwxrwxrwx 1 root 0 9 Mar 6 14:25 ata-WDC_WD10EZEX-75ZF5A0_WD-WMC1S5907372 -> ../../sda lrwxrwxrwx 1 root 0 10 Mar 6 14:25 ata-WDC_WD10EZEX-75ZF5A0_WD-WMC1S5907372-part1 -> ../../sda1 lrwxrwxrwx 1 root 0 10 Mar 6 14:25 ata-WDC_WD10EZEX-75ZF5A0_WD-WMC1S5907372-part2 -> ../../sda2 lrwxrwxrwx 1 root 0 10 Mar 6 14:25 lvm-pv-uuid-D30Qhk-ktIX-1U66-NF3W-IBwL-fW8v-8oohIs -> ../../sda1 lrwxrwxrwx 1 root 0 9 Mar 6 14:25 wwn-0x50014ee058f4c3d3 -> ../../sda lrwxrwxrwx 1 root 0 10 Mar 6 14:25 wwn-0x50014ee058f4c3d3-part1 -> ../../sda1 lrwxrwxrwx 1 root 0 10 Mar 6 14:25 wwn-0x50014ee058f4c3d3-part2 -> ../../sda2 /dev/disk/by-uuid: total 0 lrwxrwxrwx 1 root 0 10 Mar 6 14:25 a19c3d6b-031e-4b46-a779-268dacd404d9 -> ../../sdb2 lrwxrwxrwx 1 root 0 10 Mar 6 14:25 aa2a7ee9-bda8-45c9-a9be-21e1c77687f7 -> ../../sdb1 lrwxrwxrwx 1 root 0 10 Mar 6 14:25 e44a2a6b-4b5c-475f-8898-e872d011565b -> ../../sda2 + for _i in '/etc/conf.d/*.conf' + '[' -f /etc/conf.d/fedup.conf ']' + echo /etc/conf.d/fedup.conf /etc/conf.d/fedup.conf + cat /etc/conf.d/fedup.conf UPGRADEBIN=/usr/libexec/system-upgrade-fedora + for _i in '/etc/conf.d/*.conf' + '[' -f /etc/conf.d/systemd.conf ']' + echo /etc/conf.d/systemd.conf /etc/conf.d/systemd.conf + cat /etc/conf.d/systemd.conf systemdutildir="/usr/lib/systemd" systemdsystemunitdir="/usr/lib/systemd/system" systemdsystemconfdir="/etc/systemd/system" + command -v lvm + lvm pvdisplay "/dev/sda1" is a new physical volume of "75.46 GiB" --- NEW Physical volume --- PV Name /dev/sda1 VG Name PV Size 75.46 GiB Allocatable NO PE Size 0 Total PE 0 Free PE 0 Allocated PE 0 PV UUID D30Qhk-ktIX-1U66-NF3W-IBwL-fW8v-8oohIs + lvm vgdisplay No volume groups found + lvm lvdisplay No volume groups found + command -v dmsetup + dmsetup ls --tree No devices found + cat /proc/mdstat Personalities : unused devices: <none> + command -v ip + ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eno1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether b8:ca:3a:9b:dd:94 brd ff:ff:ff:ff:ff:ff + command -v journalctl + journalctl -ab --no-pager -o short-monotonic No journal files were found. :/# [ 312.074237] dracut-initqueue[462]: Warning: Could not boot. [ 312.077595] dracut-initqueue[462]: Warning: /dev/BCACHE/root does not exist [ 312.078094] dracut-initqueue[462]: Warning: /dev/BCACHE/swap does not exist [ 312.078478] dracut-initqueue[462]: Warning: /dev/mapper/BCACHE-root does not exist [ 312.090325] dracut-initqueue[462]: Failed to start dracut-emergency.service: Transaction is destructive. [ 312.093642] dracut-initqueue[462]: Warning: Not all disks have been found. [ 312.094031] dracut-initqueue[462]: Warning: You might want to regenerate your initramfs. :/#
lsinitrd /boot/initramfs-3.18.7-100.fc20.x86_64.img ... dracut modules: bash i18n network ifcfg drm plymouth bcache dm kernel-modules lvm fcoe fcoe-uefi resume rootfs-block terminfo udev-rules biosdevname systemd usrmount base fs-lib shutdown uefi-lib ======================================================================== but lsinitrd /boot/initramfs-fedup.img ... dracut modules: bash modsign mlx4 rdma i18n network ifcfg drm plymouth system-upgrade-fedora btrfs crypt dm dmraid kernel-modules lvm mdraid multipath system-upgrade fcoe fcoe-uefi iscsi nfs resume rootfs-block terminfo udev-rules ostree systemd usrmount base fs-lib shutdown uefi-lib ======================================================================== So is the issue that bcache dracut module is missing from initramfs-fedup.img?
I encountered this doing the exact same thing. The bcache module was loaded but neither the cache device nor the backing device were registered to bcache. I had to manually do: echo /dev/Flash/Cache > /sys/fs/bcache/register echo /dev/md1 > /sys/fs/bcache/register then: lvm pvscan lvm vgchange -ay There is a script that is put in the regular initramfs that performs these actions. It appears that fedup does not put this script in its initramfs.
Thanks for the hint! In my system this worked: Immediately when shell at tty2 was available I issued these commands: modprobe bcache echo /dev/sdb2 > /sys/fs/bcache/register echo /dev/sda2 > /sys/fs/bcache/register Then upgrade ran through normally.
Hrm. bcache isn't in the default dracut package - that module is only there if you install bcache-tools. Which is why it's not in the generic initramfs generated by dracut... ..but fedup tries to pick up anything that's in your initramfs that's not in the generic initramfs. Could you attach full `lsinitrd` output? (Preferably from a F21 system.)
Created attachment 1002438 [details] lsinitrd of initramfs-3.18.9-200.fc21.x86_64.img This lsinitrd of initramfs-3.18.9-200.fc21.x86_64.img is from my system now running F21.
Quick question: How did you install a system with root on bcache if the installer doesn't support bcache?
I did use some instructions for how to test bcache when bcache support was initially introduced to Fedora. Don't have the link available right now, but it involved some manual steps after initial installation.
So, there's a general problem here: You're using a root device that's unsupported by the installer or (generic) dracut. This means your system needs some 3rd-party code to boot.. but the upgrade image doesn't have that code in it. This is a long-standing problem with upgrades. It's not new in fedup - the old installer-based upgrades wouldn't handle this either. I've opened this issue about that general problem: https://github.com/rhinstaller/fedup/issues/62 As for this specific problem with bcache: I *would* say that the bcache module should go into upstream dracut - and bug 1003207 indicates that some versions of dracut *did* have builtin bcache support - but it seems that it was removed (and moved to bcache-tools) in dracut-034 (i.e. sometime before F20). So.. unless someone convinces the dracut/bcache maintainers to put bcache support back into the generic dracut package, you're kind of stuck. For now, if anyone else hits this bug, they're going to have to use the workaround described in comment #2.
This message is a reminder that Fedora 21 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 21. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '21'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 21 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Upgrade of the affected system from 21 directly to 23 using "dnf system-upgrade download --releasever 23" worked just fine.
Okay then - since Fedora 20 is EOL, I can't really fix anything there, but the workaround from comment #2 should still let you get to F21. (F20 users could also try a live yum/dnf upgrade, as described here: https://fedoraproject.org/wiki/Upgrading_Fedora_using_package_manager) If you've got F21 or higher, dnf-plugin-system-upgrade should work just fine.(confirmed in comment #10 - thanks for the feedback!) At this point, both F21 and fedup are (about to be) EOL, so I'm closing this bug accordingly.