Failed to mount /sysroot https://bugs.freedesktop.org/show_bug.cgi?id=90913
I don't see how systemd is involved. Seems to be either nfs.ko or mount.nfs missing in the initrd, or in the kernel. Reassigning to initrd, but could be util-linux or kernel, too.
that's systemd trying: mount nfs:... /sysroot for root=nfs:.... taking the plain root=... as the mount argument.
sorry, it's the fstab-generator taking root=live:CDLABEL... in the sysroot.mount unit as What=..
Status quo?
(In reply to Lennart Poettering from comment #1) > I don't see how systemd is involved. Seems to be either nfs.ko or mount.nfs > missing in the initrd, or in the kernel. Reassigning to initrd, but could be > util-linux or kernel, too. The only major component which in the meantime has changed drastically is systemd, 219 -> 220, not dracut, not util-linux. I'll make a test with the Fedora 21 kernel ...
Created attachment 1037322 [details] Failed to mount /sysroot - sd 202-7 - kernel 4.0.5
Created attachment 1037359 [details] Failed to mount /sysroot - Fedora-Live-Xfce-x86_64-rawhide-20150610.iso
The same happens with dracut-041-12.git20150609.fc23.x86_64 - git 8b52e96
Lowering a couple anaconda versions also doesn't help.
Created attachment 1037797 [details] rdsosreport - SYSTEMeDicine 220-9 - Dr. Acut 042 # livecd-creator ... ... Installing: dracut ##################### [363/529] Installing: dracut-network ##################### [442/529] Installing: dracut-config-rescue ##################### [513/529] ... dracut-install: ERROR: installing '/usr/lib/dracut/modules.d/90dmsquash-live/dmsquash-generator.sh' to '/usr/lib/systemd/system-generators/dracut-dmsquash-generator' /usr/lib/dracut/dracut-install -D /var/tmp/initramfs.Inp0Uv /usr/lib/dracut/modules.d/90dmsquash-live/dmsquash-generator.sh /usr/lib/systemd/system-generators/dracut-dmsquash-generator ... dracut-install: ERROR: installing '/usr/lib/dracut/modules.d/90dmsquash-live/dmsquash-generator.sh' to '/usr/lib/systemd/system-generators/dracut-dmsquash-generator' /usr/lib/dracut/dracut-install -D /var/tmp/initramfs.6PoGmr /usr/lib/dracut/modules.d/90dmsquash-live/dmsquash-generator.sh /usr/lib/systemd/system-generators/dracut-dmsquash-generator ... dracut-042-1.fc23.x86_64 dracut-config-rescue-042-1.fc23.x86_64 dracut-network-042-1.fc23.x86_64
Yeah, and don't forget the tar https://bugzilla.redhat.com/show_bug.cgi?id=1182362 Maybe increment actually becomes, the answer to life, the universe and everything.
As usual with poma I don't understand half of what he's trying to say above, but it is confirmed by multiple reporters (satellit and jpigface) that Rawhide nightly live images do not boot, with this 'Failed to mount /sysroot' error. Nominating as an Alpha blocker, criterion "All release-blocking images must boot in their supported configurations." - https://fedoraproject.org/wiki/Fedora_23_Alpha_Release_Criteria#Release-blocking_images_must_boot
Created attachment 1037801 [details] f23 Fedora-Live-Workstation-x86_64-rawhide-20150611.iso boot failure
Yeah, this definitely looks like systemd. Comparing F22 to F23, the difference is that after dracut-pre-mount in F23, systemd jumps in and tries to mount sysroot as if it were an NFS address and fails. In F22, systemd doesn't try; it's a dracut hook that actually mounts sysroot. Assigning back to systemd. The F23 dracut is not newer than the F22 one (well it is right *now*, but it wasn't until today).
If I were a bettin' man, I'd bet on http://cgit.freedesktop.org/systemd/systemd/commit/src/fstab-generator/fstab-generator.c?id=093c2cfe3b1ae6081f12927ae7906d90d6623534 as the cause of this.
Lennart, just to be clear what's going on here - live images have a cmdline parameter like this: root=live:CDLABEL=Fedora-Live-WS-x86_64-rawhide-20 obviously that's not actually meant to be treated as an NFS location. It's supposed to trigger the dracut dmsquash-live module, which knows how to interpret it and mount the root FS from the live image correctly. systemd jumping in and trying to treat it as an NFS location is the problem.
http://koji.fedoraproject.org/koji/taskinfo?taskID=10027523 is a scratch build to check whether that commit is the problem - it has that commit hand-reverted.
Screwed up that build - try http://koji.fedoraproject.org/koji/taskinfo?taskID=10027679 .
$ cat /proc/cmdline BOOT_IMAGE=vmlinuz0 initrd=initrd0.img root=live:CDLABEL=Rawhide-Live-Base-041 rootfstype=auto ro rd.live.image $ rpm -q anaconda dracut systemd anaconda-23.10-1.fc23.x86_64 dracut-041-10.git20150219.fc23.x86_64 systemd-220-9.1.fc23.x86_64 0001-revert-093c2cfe-breaks-fedora-live-image-booting.patch PASSED
Skip Dr. Acut 042 - it'll break the boot. rdsosreport-042: + cat /lib/dracut/dracut-042-1.fc23 dracut-042-1.fc23 + cat /proc/cmdline BOOT_IMAGE=vmlinuz0 initrd=initrd0.img root=live:CDLABEL=Rawhide-Live-Base-042 rootfstype=auto ro rd.live.image ... [ 1.894289] localhost dracut-cmdline[95]: dracut-23 (Rawhide) dracut-042-1.fc23 ... [ 4.058660] localhost systemd[1]: Reached target Switch Root. [ 4.058901] localhost systemd[1]: Starting Switch Root. [ 4.059996] localhost systemd[1]: Starting Switch Root... [ 4.093799] localhost systemctl[553]: Failed to switch root: Specified switch root path /sysroot does not seem to be an OS tree. os-release file is missing. [ 4.094149] localhost systemd[1]: initrd-switch-root.service: Main process exited, code=exited, status=1/FAILURE [ 4.104710] localhost systemd[1]: Failed to start Switch Root. [ 4.105142] localhost systemd[1]: Startup finished in 1.749s (kernel) + 0 (initrd) + 2.355s (userspace) = 4.104s. [ 4.105580] localhost systemd[1]: initrd-switch-root.service: Unit entered failed state. [ 4.105912] localhost systemd[1]: initrd-switch-root.service: Triggering OnFailure= dependencies. [ 4.106091] localhost systemd[1]: initrd-switch-root.service: Failed with result 'exit-code'. [ 4.106146] localhost audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=initrd-switch-root comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed' [ 4.109609] localhost systemd[1]: Starting Emergency Shell... ...
(In reply to Adam Williamson from comment #18) > Screwed up that build - try > http://koji.fedoraproject.org/koji/taskinfo?taskID=10027679 . Source rpm: https://kojipkgs.fedoraproject.org/work/tasks/7680/10027680/systemd-220-9.1.fc23.src.rpm This is the FOSS - always provide the -source-
Confirming: a live image built with 2015-06-11 Rawhide (that's with dracut-041-10.git20150219.fc23.x86_64) plus my patched systemd boots. A live image built with 2015-06-12 Rawhide (that's with dracut-042-1.fc23.x86_64) plus my patched systemd does not boot, but the cause looks to be something different in dracut. I'll file a new, separate report for that bug. *This* bug clearly seems to be caused by the systemd commit I identified.
Hum, I note this dracut commit: https://git.kernel.org/cgit/boot/dracut/dracut.git/commit?id=8ff624df9f3f300a008711d114a8769464a054db I suspect perhaps that's meant to fix this problem, and so perhaps if we used the 'broken' systemd together with dracut 042, it would actually work? i.e. just use all the latest Rawhide bits? I'll test that.
The regular 2015-06-12 nightly (with dracut 042-1) still has this bug. Still, that dracut commit *definitely* looks related. I have a theory: the dracut commit is *meant* to fix this bug but doesn't quite work. The dracut generator does this: if ! [ -f "$GENERATOR_DIR"/sysroot.mount ]; then { echo "[Unit]" echo "Before=initrd-root-fs.target" [snip more echos] } > "$GENERATOR_DIR"/sysroot.mount fi i.e. it only creates sysroot.mount if it doesn't already exist (that's the "if ! [ -f" bit). The problem is that it *does* already exist: systemd-fstab-generator creates the one that tries to mount it as an NFS location before the dracut generator gets a shot. So the dracut generator sees the sysroot.mount created by systemd-fstab-generator and doesn't overwrite it, hence we still get the failed attempt to mount it as an NFS location. The reason dracut 042 fails to boot even with the reverted systemd is that the commit also stops dracut trying to mount the sysroot itself when it detects systemd - that's the effect of the change to modules.d/90dmsquash-live/dmsquash-live-root.sh . To verify this, boot the 2015-06-12 nightly Workstation live - https://kojipkgs.fedoraproject.org//work/tasks/1779/10031779/Fedora-Live-Workstation-x86_64-rawhide-20150612.iso - and look at /run/systemd/generator/sysroot.mount - it's created by systemd-fstab-generator, not by dracut-systemd's rootfs-generator. So...let's send this back to dracut? :)
Um. Unless I'm missing something, https://git.kernel.org/cgit/boot/dracut/dracut.git/commit?id=8ff624df9f3f300a008711d114a8769464a054db is entirely missing a file. The commit has this: diff --git a/modules.d/90dmsquash-live/module-setup.sh b/modules.d/90dmsquash-live/module-setup.sh index a113c3f..19396a7 100755 --- a/modules.d/90dmsquash-live/module-setup.sh +++ b/modules.d/90dmsquash-live/module-setup.sh @@ -31,6 +31,7 @@ install() { inst_hook pre-pivot 20 "$moddir/apply-live-updates.sh" inst_script "$moddir/dmsquash-live-root.sh" "/sbin/dmsquash-live-root" inst_script "$moddir/iso-scan.sh" "/sbin/iso-scan" + inst_script "$moddir/dmsquash-generator.sh" $systemdutildir/system-generators/dracut-dmsquash-generator # should probably just be generally included inst_rules 60-cdrom_id.rules inst_simple "$moddir/checkisomd5@.service" "/etc/systemd/system/checkisomd5@.service" But there *is* no dmsquash-generator.sh in dracut (or systemd, or any other package) anywhere that I can see. AFAICT $moddir is dracut modules.d/90dmsquash-live , but neither the dracut source tree nor the Fedora dracut package has a dmsquash-generator.sh in that directory.
Proposed as a Freeze Exception for 23-alpha by Fedora user satellit using the blocker tracking app because: All release-blocking images must boot in their supported configurations
I already proposed it as a blocker (as it affects the release-blocking lives), don't see any point to proposing it separately as FE.
dracut-043-1.fc23 to the rescue, now with the dmsquash systemd generator, which should enable dmaquash builds. https://koji.fedoraproject.org/koji/buildinfo?buildID=646070
[liveuser@localhost ~]$ cat /proc/cmdline BOOT_IMAGE=vmlinuz0 initrd=initrd0.img root=live:CDLABEL=Rawhide-Live-Base-043 rootfstype=auto ro rd.live.image $ rpm -q anaconda dracut systemd anaconda-23.10-1.fc23.x86_64 dracut-043-1.fc23.x86_64 systemd-220-9.fc23.x86_64 $ uname -a Linux localhost 4.1.0-0.rc8.git0.1.fc23.x86_64 #1 SMP Mon Jun 15 13:14:22 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux $ du -hs /lib/modules/4.1.0-0.rc8.git0.1.fc23.x86_64/ 438M /lib/modules/4.1.0-0.rc8.git0.1.fc23.x86_64/ Once upon a time the kernel could fit on a floppy. -_- Resolved, Thank you Dr. Acut.
It's not resolved. The official 2015-06-15 nightly still does not boot.
Note these errors during live image creation: DEBUG util.py:378: dracut-install: ERROR: installing '/usr/lib/dracut/modules.d/90dmsquash-live/dmsquash-generator.sh' to '/usr/lib/systemd/system-generators/dracut-dmsquash-generator' DEBUG util.py:378: /usr/lib/dracut/dracut-install -D /var/tmp/initramfs.9IQvoX /usr/lib/dracut/modules.d/90dmsquash-live/dmsquash-generator.sh /usr/lib/systemd/system-generators/dracut-dmsquash-generator DEBUG util.py:378: No '/dev/log' or 'logger' included for syslog logging DEBUG util.py:378: dracut-install: ERROR: installing '/usr/lib/dracut/modules.d/90dmsquash-live/dmsquash-generator.sh' to '/usr/lib/systemd/system-generators/dracut-dmsquash-generator' DEBUG util.py:378: /usr/lib/dracut/dracut-install -D /var/tmp/initramfs.PqbsGc /usr/lib/dracut/modules.d/90dmsquash-live/dmsquash-generator.sh /usr/lib/systemd/system-generators/dracut-dmsquash-generator (that's from https://kojipkgs.fedoraproject.org//work/tasks/485/10060485/root.log , I see the same in a local creation attempt)
$ grep error Rawhide-Live-Base-043.iso.log.txt Installing: libgpg-error ##################### [ 49/526] error reading information on service lm_sensors: No such file or directory error reading information on service mcelog: No such file or directory error reading information on service rngd: No such file or directory libgpg-error-1.19-1.fc23.x86_64 -_-
Good for you, but the official nightlies are not working. Therefore the bug is not fixed. I wondered if it was down to whether the build host has dracut 043 or not, but that doesn't seem to be the difference, I still get the error with local builds even after updating my system to 043.
OK, turns out that is more or less it. You need dracut 043 to be both installed on the build host *and* in the set of packages to be included in the live image. If you have 042 on the host but 043 in the live image package set *or* 043 on the host but 042 in the live image package set, you get that error, it seems. If you have 043 in both, you get no error, and the built image boots. I guess we'll see how the 20150616 image works.
This combination of versions you mentioned, -allegedly- required installed versions on the build host, does not apply here. Live image was compiled properly with lower versions installed on the build host. Test compilation: http://goo.gl/Gm4ffO ISO/Rawhide-Live-Xfce-WiFi.iso
(In reply to poma from comment #29) ... > $ uname -a > Linux localhost 4.1.0-0.rc8.git0.1.fc23.x86_64 #1 SMP Mon Jun 15 13:14:22 > UTC 2015 x86_64 x86_64 x86_64 GNU/Linux > > $ du -hs /lib/modules/4.1.0-0.rc8.git0.1.fc23.x86_64/ > 438M /lib/modules/4.1.0-0.rc8.git0.1.fc23.x86_64/ > > Once upon a time the kernel could fit on a floppy. -_- > "sizeof - kernel core/modules x 10" https://bugzilla.redhat.com/show_bug.cgi?id=1232206
(In reply to Adam Williamson from comment #31) > Note these errors during live image creation: > > DEBUG util.py:378: dracut-install: ERROR: installing > '/usr/lib/dracut/modules.d/90dmsquash-live/dmsquash-generator.sh' to > '/usr/lib/systemd/system-generators/dracut-dmsquash-generator' > DEBUG util.py:378: /usr/lib/dracut/dracut-install -D > /var/tmp/initramfs.9IQvoX > /usr/lib/dracut/modules.d/90dmsquash-live/dmsquash-generator.sh > /usr/lib/systemd/system-generators/dracut-dmsquash-generator > DEBUG util.py:378: No '/dev/log' or 'logger' included for syslog logging > DEBUG util.py:378: dracut-install: ERROR: installing > '/usr/lib/dracut/modules.d/90dmsquash-live/dmsquash-generator.sh' to > '/usr/lib/systemd/system-generators/dracut-dmsquash-generator' > DEBUG util.py:378: /usr/lib/dracut/dracut-install -D > /var/tmp/initramfs.PqbsGc > /usr/lib/dracut/modules.d/90dmsquash-live/dmsquash-generator.sh > /usr/lib/systemd/system-generators/dracut-dmsquash-generator > > (that's from > https://kojipkgs.fedoraproject.org//work/tasks/485/10060485/root.log , I see > the same in a local creation attempt) DEBUG util.py:378: Retrieving http://infrastructure.fedoraproject.org/pub/fedora/linux/development/rawhide/x86_64/os/Packages/d/dracut-042-1.fc23.x86_64.rpm ...OK Seems to be dracut-042 here :-/
I attach a script that also produces a "Failed to switch root" error, which might be related to this report. -----8<----- [FAILED] Failed to start Switch Root. See 'systemctl status initrd-switch-root.service' for details. Warning: /dev/root does not exist Generating "/run/initramfs/rdsosreport.txt" Entering emergency mode. Exit the shell to continue. Type "journalctl" to view system logs. You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot after mounting them and attach it to a bug report. :/# systemctl status initrd-switch-root.service -l ● initrd-switch-root.service - Switch Root Loaded: loaded (/usr/lib/systemd/system/initrd-switch-root.service; static; vendor preset: enabled) Active: failed (Result: exit-code) since Tue 2015-06-16 13:40:42 UTC; 53s ago Process: 1032 ExecStart=/usr/bin/systemctl --no-block --force switch-root /sysroot (code=exited, status=1/FAILURE) Main PID: 1032 (code=exited, status=1/FAILURE) Jun 16 13:40:42 localhost systemd[1]: Starting Switch Root... Jun 16 13:40:42 localhost systemctl[1032]: Failed to switch root: Specified switch root path /sysroot does not seem to be an OS tree. os-release file is missing. Jun 16 13:40:42 localhost systemd[1]: initrd-switch-root.service: Main process exited, code=exited, status=1/FAILURE Jun 16 13:40:42 localhost systemd[1]: Failed to start Switch Root. Jun 16 13:40:42 localhost systemd[1]: initrd-switch-root.service: Unit entered failed state. Jun 16 13:40:42 localhost systemd[1]: initrd-switch-root.service: Triggering OnFailure= dependencies. Jun 16 13:40:42 localhost systemd[1]: initrd-switch-root.service: Failed with result 'exit-code'.
Created attachment 1039499 [details] script to install rawhide with virt-install
Just to mention it: Running the script puts you into the emergency mode shell, and one can go poking around there. Maybe that is useful.
(In reply to Marius Vollmer from comment #38) > I attach a script that also produces a "Failed to switch root" error, which > might be related to this report. > > Jun 16 13:40:42 localhost systemctl[1032]: Failed to switch root: Specified > switch root path /sysroot does not seem to be an OS tree. os-release file is > missing. make sure your live image contains /usr/lib/os-release (was /etc/os-release)
(In reply to Harald Hoyer from comment #41) > (In reply to Marius Vollmer from comment #38) > > I attach a script that also produces a "Failed to switch root" error, which > > might be related to this report. > > > > Jun 16 13:40:42 localhost systemctl[1032]: Failed to switch root: Specified > > switch root path /sysroot does not seem to be an OS tree. os-release file is > > missing. > > > make sure your live image contains /usr/lib/os-release (was /etc/os-release) or your installed OS.
> (In reply to Harald Hoyer from comment #41) > > > make sure your live image contains /usr/lib/os-release (was /etc/os-release) > > or your installed OS. I don't control that. I'll try again a bit later. Sgallagh says that todays compose might fix things.
(In reply to Marius Vollmer from comment #43) > > (In reply to Harald Hoyer from comment #41) > > > > > make sure your live image contains /usr/lib/os-release (was /etc/os-release) > > > > or your installed OS. > > I don't control that. I'll try again a bit later. Sgallagh says that > todays compose might fix things. you might not have install any of the fedora-release rpms
Please don't conflate multiple likely different issues. Rawhide nightly boot.iso 20150615 doesn't boot either, which is more likely the same thing Marius is seeing, but it's not the same problem we have here because there isn't a root=live: cmdline parameter involved there. This report is long and poma-ized enough as it is, please don't make it more confusing. Harald, the os-release thing is a red herring, it just shows up because /sysroot isn't mounted at all so of course os-release is missing from it.
2015-06-16 Workstation nightly live boots, so let's call this fixed. If boot.iso is still broken I'll file a separate bug shortly.
boot.iso failure filed as https://bugzilla.redhat.com/show_bug.cgi?id=1232411 , it may be the same thing affecting Marius' case and cloud images, but please make sure before dumping data into that bug. thanks.
Fedora-Live-Workstation-x86_64-rawhide-20150616.iso boots and installs with #liveinst to f22 VirtualBox, G-I-S works after login
$ cat /proc/cmdline BOOT_IMAGE=vmlinuz0 initrd=initrd0.img root=live:CDLABEL=Fedora-Live-Xfce-x86_64-rawhide- rootfstype=auto ro rd.live.image quiet rhgb rd.luks=0 rd.md=0 rd.dm=0 $ rpm -q kernel anaconda dracut systemd kernel-4.1.0-0.rc8.git0.1.fc23.x86_64 anaconda-23.10-1.fc23.x86_64 dracut-043-1.fc23.x86_64 systemd-220-9.fc23.x86_64 Fedora-Live-Xfce-x86_64-rawhide-20150616.iso 1.5G PASSED: BOOTABLE