Bug 1374082

Summary: System installed from Atomic DVD fails to boot
Product: [Fedora] Fedora Reporter: Adam Williamson <awilliam>
Component: ostreeAssignee: Colin Walters <walters>
Status: CLOSED EOL QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 25CC: walters
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-12-12 10:17:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
anaconda.log , program.log and storage.log
none
lvm.log none

Description Adam Williamson 2016-09-07 22:27:54 UTC
since Fedora-25-20160902.n.0, install from the Atomic installer DVD has succeeded (prior to that compose, it was hitting https://bugzilla.redhat.com/show_bug.cgi?id=1331864 ); however, now the installed system fails to boot. It dies in dracut trying to set up the real system root, with:

Warning: /dev/fedora/root does not exist
Warning: /dev/fedora/swap does not exist
Warning: /dev/mapper/fedora-root does not exist

The only file in /dev/mapper is /dev/mapper/control .

I think there may actually be something wrong with the initramfs, because the network interface doesn't appear (it usually would) and there don't actually appear to be any /dev nodes for the hard disk at all. I'm testing in a VM with a virtIO hard disk - as would the openQA test - and there is no /dev/vda , and the only thing in /dev/disk/by-id is the DVD drive.

The boot menu entry lists the initrd as /ostree/fedora-atomic-6cfe14d14ed(...can'tbebotheredtypingitall...)77454/initramfs-4.8.0-0.rc4.git0.1.fc25.x86_64.img , so does that mean Atomic uses a prebuilt initramfs from the ostree on the installer image? If so, I guess something may be wrong with that? There do seem to be quite a lot of errors in the dracut bit of https://kojipkgs.fedoraproject.org/compose/branched/Fedora-25-20160907.n.0/logs/x86_64/Atomic/ostree-1/create-ostree-repo.log , including lots of this:

libkmod: kmod_module_new_from_loaded: could not open /proc/modules: No such file or directory
dracut-install: Could not get list of loaded modules: Unknown error -2

which makes me suspect it may be resulting in the initramfs missing kernel modules it should include?

To reproduce, just grab https://kojipkgs.fedoraproject.org/compose/branched/Fedora-25-20160907.n.0/compose/Atomic/x86_64/iso/Fedora-Atomic-dvd-x86_64-25-20160907.n.0.iso , install it (to a VM with a virtIO disk, I guess? but the bug likely affects all kinds of hardware) and try booting it.

Comment 1 Adam Williamson 2016-09-07 22:28:54 UTC
Atomic image tests in autocloud are also failing - e.g. https://apps.fedoraproject.org/autocloud/jobs/543/output - which I suspect may be caused by the same problem.

Comment 2 Adam Williamson 2016-09-07 22:33:09 UTC
Created attachment 1198836 [details]
anaconda.log , program.log and storage.log

If the problem is what I think it is, anaconda logs probably aren't any use, but just in case, here they are. lvm.log to follow.

Comment 3 Adam Williamson 2016-09-07 22:33:58 UTC
Created attachment 1198837 [details]
lvm.log

Comment 4 Colin Walters 2016-09-08 01:04:53 UTC
This is because we need a newer rpm-ostree which uses bubblewrap to run dracut.  Without having a proper container setup for dracut, /proc/modules doesn't exist, and dracut -> libkmod tries to parse it (even though it shouldn't as we're generating a --no-hostonly initramfs).

You can see this from `lsinitrd /path/to/initramfs.img` showing no kernel modules.

This is a new-in-F25 change, F24 dracut works without a container setup.

The rpm-ostree -> bubblewrap -> dracut is only in 2016.7:
https://bodhi.fedoraproject.org/updates/FEDORA-2016-d5aa429d63
which is blocked on the fact that the dependency on bubblewrap currently break inside  bodhi -> mock.

Comment 5 Adam Williamson 2016-09-08 02:02:30 UTC
could we deal with it in the meantime by fixing dracut not to bother trying to figure out what modules are loaded when it's building a generic initramfs?

Comment 6 Colin Walters 2016-09-08 02:29:39 UTC
Yes, that would probably work too.

Comment 7 Colin Walters 2016-09-08 02:57:45 UTC
https://github.com/fedora-infra/fedmsg-atomic-composer/issues/4 will track the last bit.

Comment 8 Colin Walters 2016-09-08 03:11:32 UTC
I was wondering why lorax wasn't affected by this, turns out:
https://github.com/rhinstaller/lorax/commit/51060f07

Comment 9 Adam Williamson 2016-09-28 22:11:32 UTC
The issue mentioned in #c7 is now marked as closed, but F25 tests are still hitting this bug, e.g.:

https://openqa.fedoraproject.org/tests/36898

what's the current status?

Comment 10 Colin Walters 2016-10-07 18:47:08 UTC
I couldn't find this information in the compose logs, but the problem is that the OSTree commit date from 2016-09-07.  So presumably it hasn't picked up any fixes since then.

I found this by doing:

ostree --repo=/run/install/repo/content/repo show fedora-atomic/25/x86_64/docker-host

in a shell (alt-f2) in the anaconda environment.

Comment 11 Dennis Gilmore 2016-10-07 22:00:44 UTC
(In reply to Colin Walters from comment #7)
> https://github.com/fedora-infra/fedmsg-atomic-composer/issues/4 will track
> the last bit.

that is an area that will have to be fixed, but it is the wrong place  for where we currently stand in the release process, the only f25 ostree repos we have today are made by pungi as part of the nightly branched compose.

Comment 12 Colin Walters 2016-10-10 13:38:49 UTC
Forgot to cross link here: https://pagure.io/releng/issue/120

Comment 13 Colin Walters 2016-10-10 16:37:45 UTC
This should be fixed in 20161010.0

Comment 14 Fedora End Of Life 2017-11-16 19:07:12 UTC
This message is a reminder that Fedora 25 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 25. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '25'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 25 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Comment 15 Fedora End Of Life 2017-12-12 10:17:02 UTC
Fedora 25 changed to end-of-life (EOL) status on 2017-12-12. Fedora 25 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.