Bug 922988

Summary: python-blivet not creating /sys and /run on /mnt/sysimage
Product: [Fedora] Fedora Reporter: Adam Williamson <awilliam>
Component: python-blivetAssignee: David Lehman <dlehman>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 19CC: anaconda-maint-list, bcl, dlehman, dracut-maint, g.kaviyarasu, harald, jonathan, jreznik, kay, kparal, lpoetter, mkolman, p, robatino, sbueno, vanmeeuwen+fedora
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: All   
Whiteboard: AcceptedBlocker
Fixed In Version: python-blivet-0.9-1 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-04-04 21:05:12 EDT Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On:    
Bug Blocks: 834084    
Attachments:
Description Flags
No /sys after installation!
none
systemd cannot mount /sys
none
Proposed patch
none
Proposed patch for anaconda none

Description Adam Williamson 2013-03-18 19:22:56 EDT
I finally managed to build a vaguely working F19 live image today, with latest systemd and dracut and livecd-tools and a bunch of GNOME-level workarounds that shouldn't affect this. I can even run a full install process, which is surprising.

However, the installed system fails to boot. Bit hard to get the journalctl output out, but here are the interesting lines:

systemd-modules-load[51]: Failed to find module 'uinput'
systemd[1]: systemd-modules-load.service: main process exited, code=exited, status=1/FAILURE
systemd[1]: Failed to start Load Kernel Modules.
systemd[1]: Unit systemd-modules-load.service entered failed state
...
systemd[1]: Job dev-mapper-fedora\x2droot.device/start timed out.
systemd[1]: Timed out waiting for device dev-mapper-fedora\x2droot.device.
systemd[1]: Dependency failed for /sysroot.
systemd[1]: Dependency failed for Initrd Root File System.
systemd[1]: Dependency failed for Reload Configuration from the Real Root.

I'm guessing this may have something to do with dracut, but CCing systemd devs too.
Comment 1 Adam Williamson 2013-03-18 19:37:11 EDT
Hmm. So it looks like spice-vdagent package contains:

/etc/modules-load.d/spice-vdagentd.conf

which reads:

# load uinput.ko for spice input
uinput

But this isn't new; my installed F18 system has the same thing. uinput is not in the initramfs either on that (working) system or the (broken) system here. So I don't know what changed that causes this not to work in this test.
Comment 2 Adam Williamson 2013-03-18 19:50:10 EDT
Hm. Moving that config file out of the way doesn't seem to make any difference, it still tries to load uinput. Don't really know where to go from there.
Comment 3 Adam Williamson 2013-03-18 19:59:43 EDT
Proposing as an Alpha Blocker. New-style criterion:

"A system installed with a graphical package set must boot to the 'firstboot' utility on the first boot after installation. The firstboot utility must be able to create a working user account."
Comment 4 Harald Hoyer 2013-03-19 02:56:11 EDT
Where can I get this image?
Comment 5 Adam Williamson 2013-03-19 04:47:45 EDT
For the record, it was a private side build, but I provided Harald with a copy. If anyone's playing along at home, to build a desktop live with current F19 repos, you'll need this kickstart recipe:

%packages
-gnome-initial-setup
firstboot
gnome-screensaver
gnome-session-xsession
%end

and make sure you use the latest f19 livecd-tools (even if you build on f18).
Comment 6 Harald Hoyer 2013-03-19 07:39:17 EDT
(In reply to comment #1)
> Hmm. So it looks like spice-vdagent package contains:
> 
> /etc/modules-load.d/spice-vdagentd.conf
> 
> which reads:
> 
> # load uinput.ko for spice input
> uinput
> 
> But this isn't new; my installed F18 system has the same thing. uinput is
> not in the initramfs either on that (working) system or the (broken) system
> here. So I don't know what changed that causes this not to work in this test.

ignore that
Comment 7 Harald Hoyer 2013-03-19 11:03:42 EDT
Created attachment 712730 [details]
No /sys after installation!

Directory "/sys" does not exist after installing from the Live-CD !
Comment 8 Harald Hoyer 2013-03-19 12:15:42 EDT
Created attachment 712788 [details]
systemd cannot mount /sys
Comment 9 Adam Williamson 2013-03-19 13:11:40 EDT
So Harald and I seem to be seeing different problems...but in my case, we've traced it down to using virtio disk interface. If I switch to an 'IDE' disk in my VM, it boots successfully to firstboot.

lsinitrd | grep virt shows only usr/lib/sysctl.d/libvirtd.conf - no virtio modules.
Comment 10 Adam Williamson 2013-03-19 13:16:43 EDT
after 'dracut -f', lsinitrd shows the virtio modules. So it looks like they're missing from the initramfs after a live install.
Comment 11 Brian Lane 2013-03-19 20:22:11 EDT
harald suggested running this during the postInstall:

new-kernel-pkg --rpmposttrans <kernel_version>
Comment 12 Harald Hoyer 2013-03-20 03:08:12 EDT
(In reply to comment #11)
> harald suggested running this during the postInstall:
> 
> new-kernel-pkg --rpmposttrans <kernel_version>

doesn't help in this case
Comment 13 Harald Hoyer 2013-03-20 03:36:39 EDT
But I see that /sys is not mounted on /mnt/sysimage, while installing.
And I also see, that /mnt/sysimage/sys does not exist.
Comment 14 Harald Hoyer 2013-03-20 04:23:05 EDT
(In reply to comment #13)
> But I see that /sys is not mounted on /mnt/sysimage, while installing.
> And I also see, that /mnt/sysimage/sys does not exist.

And, when I mount bind /mnt/sysimage/sys manually, while rsync syncs, the initramfs contains the proper virtio drivers.
Comment 15 Harald Hoyer 2013-03-20 04:25:24 EDT
started livecd with "selinux=0"

ERR anaconda: unknown selinux state: None
Comment 16 Harald Hoyer 2013-03-20 05:11:16 EDT
And I need:

# mount --bind /run /mnt/sysimage/run

to be able to read the udev database.
Comment 17 Harald Hoyer 2013-03-20 05:33:44 EDT
(In reply to comment #16)
> And I need:
> 
> # mount --bind /run /mnt/sysimage/run
> 
> to be able to read the udev database.

oh, and you want to exclude /run from rsync also
Comment 18 Harald Hoyer 2013-03-20 06:16:23 EDT
dracut-026-72.git20130320.fc19 has a safety check and turns off hostonly mode, if one of the essential filesystems is not mounted or the udev database not available.
Comment 19 Adam Williamson 2013-03-20 12:25:19 EDT
Discussed at 2013-03-20 blocker review meeting: http://meetbot.fedoraproject.org/fedora-blocker-review/2013-03-20/f19alpha-blocker-review-2.2013-03-20-16.00.log.txt . We decided to delay decision on blocker status to find out how bad the impact is (how many modules may be missing), as the impact on virtual machines alone is not enough to constitute Alpha blocker status.

Really, though, we expect the bug to be closed by next week, so we wouldn't have to worry about it: Harald's dracut adjustment should paper over the anaconda issue, and it should not be difficult to fix the anaconda issue, so we ought to be able to fix this from both ends.
Comment 20 Harald Hoyer 2013-03-20 14:33:25 EDT
(In reply to comment #19)
> Discussed at 2013-03-20 blocker review meeting:
> http://meetbot.fedoraproject.org/fedora-blocker-review/2013-03-20/f19alpha-
> blocker-review-2.2013-03-20-16.00.log.txt . We decided to delay decision on
> blocker status to find out how bad the impact is (how many modules may be
> missing), as the impact on virtual machines alone is not enough to
> constitute Alpha blocker status.
> 
> Really, though, we expect the bug to be closed by next week, so we wouldn't
> have to worry about it: Harald's dracut adjustment should paper over the
> anaconda issue, and it should not be difficult to fix the anaconda issue, so
> we ought to be able to fix this from both ends.

The impact is really bad, because hostonly mode would fail completely in this situation. And we don't want to fallback to generic mode, just because the installer does not mount /sys and /run.
Comment 21 Harald Hoyer 2013-03-21 07:02:09 EDT
Created attachment 713771 [details]
Proposed patch
Comment 22 Harald Hoyer 2013-03-21 08:10:33 EDT
Created attachment 713815 [details]
Proposed patch for anaconda

also call new-kernel-pkg with --rpmposttrans

This will build the rescue initramfs image.
Comment 23 Brian Lane 2013-03-21 10:17:20 EDT
This will be fixed in blivet 0.9 and anaconda 19.13
Comment 24 Adam Williamson 2013-03-21 13:04:37 EDT
Just as a note, the dracut-side band-aid does work in Alpha TC1: I was able to install it to a KVM with virtio disk and boot the installed system. But we should certainly fix anaconda-side too.
Comment 25 Adam Williamson 2013-03-27 13:33:02 EDT
Discussed at 2013-03-27 blocker review meeting: http://meetbot.fedoraproject.org/fedora-blocker-review/2013-03-27/f19alpha-blocker-review-3.2013-03-27-16.01.log.txt . Agreed to drop the blocker nomination as the dracut band-aid fixed this effectively for TC1. We can close it when blivet 0.9 goes out.
Comment 26 Pádraig Brady 2013-03-28 14:34:15 EDT
I finally got TC2 installed in a VM, and now it's hanging at boot with:

systemd[1]: Failed to mount /run: No such file or directory

Not sure if it's related to this TBH
Comment 27 Pádraig Brady 2013-03-28 15:50:25 EDT
OK just booted there with init=/bin/sh and did:

mount -o remount,rw /
mkdir /run
mount -o remount,ro /

And now it boots normally.
So this looks like a blocker to me?
Comment 28 Adam Williamson 2013-03-28 16:02:02 EDT
For now that's being tracked separately:

https://bugzilla.redhat.com/show_bug.cgi?id=928339

if anaconda team reckons they should be combined, though, they can be.
Comment 29 Adam Williamson 2013-04-02 12:07:28 EDT
*** Bug 928339 has been marked as a duplicate of this bug. ***
Comment 30 Adam Williamson 2013-04-03 14:22:17 EDT
Discussed at 2013-04-03 blocker review meeting: http://meetbot.fedoraproject.org/fedora-blocker-review/2013-04-03/f19alpha-blocker-review-4.2013-04-03-16.01.log.txt . We can transfer AcceptedBlocker status from 928339. We are unclear if all symptoms of this bug are fixed in blivet 0.9 / TC3, further testing would help clear that up, and we should also check with TC4 when it lands.
Comment 31 Adam Williamson 2013-04-04 21:05:12 EDT
So far as I can tell, everyone complaining in this bug or in one of the ones that has been marked as a dupe says things are fixed as of TC3 (hence blivet 0.9, which is stable). So let's close this.