Bug 1669256 - netinstall with updates enabled broken: empty machine-id thus no kernel installed
Summary: netinstall with updates enabled broken: empty machine-id thus no kernel insta...
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: systemd
Version: 29
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: systemd-maint
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Keywords:
: 1669055 1669609 1670467 (view as bug list)
Depends On:
Blocks: PPCTracker
TreeView+ depends on / blocked
 
Reported: 2019-01-24 18:26 UTC by Felix Kaechele
Modified: 2019-01-31 11:52 UTC (History)
15 users (show)

(edit)
Clone Of:
(edit)
Last Closed: 2019-01-26 14:16:18 UTC


Attachments (Terms of Use)

Description Felix Kaechele 2019-01-24 18:26:13 UTC
Running a netinstall with the updates repo enabled currently yields an unbootable system. The machine boots right into a grub prompt.

The problem that is causing this is described here: https://www.happyassassin.net/2018/11/05/adamws-debugging-adventures-has-anyone-seen-my-kernel/

The fix is in F30 systemd but not in F29 systemd: https://src.fedoraproject.org/rpms/systemd/c/71e781a09623e04dc1006dcba12838578595d70f?branch=master

Please apply the fix from F30 to F29 as well.

Comment 1 Ben Williams 2019-01-24 19:51:45 UTC
also being able to create updated isos with livemedia-creator or livecd-creator are coming up with the no kernel found in the dracut stage

Comment 2 John Villalovos 2019-01-24 23:53:42 UTC
I have been having an issue like this. Network installs were working on 22-January-2019 (Tuesday). When I tried on 23-January-2019 (Wednesday) the Grub menu would only contain one item that said "System setup". It would then boot to that and end up in BIOS setup.

As a note I am running this on an aarch64 architecture system.

Comment 3 Eveline Raine 2019-01-25 09:20:04 UTC
I am having the same issue (first seen on January 24). 

20-grub.install silently exits if no /etc/machine-id is found, so the kernel does not get copied to /boot from /usr/lib/modules and bootentries are not created. And machine-id is not generated because when post-install script from current systemd rpm runs, there is no ssl installed yet and generation fails silently.

Comment 4 Adam Williamson 2019-01-25 09:36:41 UTC
Thanks for the blog ping - I'm backporting the fix for this now, we'll see if it helps. A new openQA test I just wrote should help with it, in fact: it builds a network installer image and tests an install from it. And indeed, it started failing on F29 updates yesterday. I don't know *what* change was sent to F29 which caused this bug to suddenly show up there (I think the change that triggered this on Rawhide was a fedora-release change, but that change doesn't seem to have showed up in F29...), but whatever, adding the dep will probably fix it and should be safe. The openQA test should confirm whether the fix works, also.

Comment 5 Fedora Update System 2019-01-25 09:43:05 UTC
systemd-239-9.gite339eae.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-827d700f48

Comment 6 Adam Williamson 2019-01-25 11:32:40 UTC
So turns out the openQA test *doesn't* confirm whether the fix works, because of a sort of missing feature in the test: https://pagure.io/fedora-qa/os-autoinst-distri-fedora/issue/89 . So, if someone with the necessary knowledge could grab the updated systemd, create a side repo with it, and run an install with that side repo added to confirm whether the fix works, that'd be great. Thanks!

Comment 7 John Villalovos 2019-01-25 23:03:03 UTC
(In reply to Adam Williamson from comment #6)
> So turns out the openQA test *doesn't* confirm whether the fix works,
> because of a sort of missing feature in the test:
> https://pagure.io/fedora-qa/os-autoinst-distri-fedora/issue/89 . So, if
> someone with the necessary knowledge could grab the updated systemd, create
> a side repo with it, and run an install with that side repo added to confirm
> whether the fix works, that'd be great. Thanks!

TL;DR
I tested it and it fixed the problem. My system booted.

Long version:
I was testing this on our in-house Beaker system and it took me a LONG time to figure out how to get my side-repo added to the ks.cfg via the Beaker job XML. I don't control the Beaker system, so couldn't modify the default ks.cfg. Kind of hacky but I added a %pre section which added a "repo" line to /run/install/ks.cfg. Luckily the ks.cfg gets re-read after the %pre section :)

%pre
echo "repo --name=sysd-test --baseurl=http://example.com/systemd-repo/ > /tmp/my-ks.cfg
cat /run/install/ks.cfg >> /tmp/my-ks.cfg
cp /tmp/my-ks.cfg /run/install/ks.cfg
%end

Hopefully there is some other method that is much easier and more elegant to do this. I would love to know about it! :)

Comment 8 Michel Alexandre Salim 2019-01-26 02:00:28 UTC
We see this at work starting at the latest on January 23 too (our teams in Dublin and London saw it during their workday).

Right now we're mitigating the issue altogether by disabling the updates repo during netinstall, and applying security updates post-install.

Comment 9 Fedora Update System 2019-01-26 03:29:11 UTC
systemd-239-9.gite339eae.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-827d700f48

Comment 10 Adam Williamson 2019-01-26 09:40:22 UTC
John: thanks a lot for the confirmation! The update is queued for stable now, so this problem should go away for everyone soon. Sorry again for the inconvenience.

Comment 11 Fedora Update System 2019-01-26 14:16:18 UTC
systemd-239-9.gite339eae.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.

Comment 12 Vendula Poncova 2019-01-28 10:04:46 UTC
*** Bug 1669609 has been marked as a duplicate of this bug. ***

Comment 13 Michel Normand 2019-01-28 10:31:15 UTC
*** Bug 1669055 has been marked as a duplicate of this bug. ***

Comment 14 Vendula Poncova 2019-01-31 11:52:46 UTC
*** Bug 1670467 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.