Bug 1670467

Summary: Kickstart + repo directive does not install a kernel in /boot under some circumstances.
Product: [Fedora] Fedora Reporter: Nigel Arnot <nigel.arnot>
Component: anacondaAssignee: Anaconda Maintenance Team <anaconda-maint-list>
Status: CLOSED DUPLICATE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 29CC: anaconda-maint-list, extras-qa, jonathan, kellin, mkolman, nigel.arnot, vanmeeuwen+fedora, vponcova, wwoods
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-01-31 11:52:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
kickstart file that failed on 24th and works today 30th none

Description Nigel Arnot 2019-01-29 15:22:34 UTC 
Description of problem:

Kickstart files containing the first two lines below validate OK and are processed without error to what appears to be a successful conclusion. However, reboot drops into the Grub> prompt, because there is no kernel file in /boot

Adding the third line fixes the problem. 

url --mirrorlist="https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-29&arch=x86_64"
repo --name=updates --mirrorlist="https://mirrors.fedoraproject.org/mirrorlist?repo=updates-released-f29&arch=x86_64"
repo --name=updates-modular --mirrorlist="https://mirrors.fedoraproject.org/mirrorlist?repo=updates-released-modular-f29&arch=x86_64"

It may be of note that if one runs the failed kickstart install using the Fedora 28 installer (i.e. boot Fedora28-netinst.iso instead of Fedora29-netinst.iso) it works. (Install Fedora 29 with Fedora 28 installer -- completely unsupported  -- but may point to an unintended change to the functionality )

It is difficult finding out what is causing this problem, because the kickstart file was known good on Fedora 28 and previous, revalidated, and because it can be hours from starting the kickstart install to finding out that the result is not bootable.


Version-Release number of selected component (if applicable):

Fedora-Server-netinst-x86_64-29-1.2.iso

(versus Fedora-Server-netinst-x86_64-28-1.1.iso)


How reproducible:
Completely



Steps to Reproduce:
1. boot Fedora29-netinst.iso, tab to add add ks=sourcefile to the kernel invocation (known good, validated kickstart file).
2. verfify kickstart success if url alone is present, fails if only updates-released-f29 is present, success if both the above repo lines are present


Actual results:
Kickstart install appears to succeed, but boot faills. Drops into Grub> prompt, no kernel present in /boot


Expected results:
Anything but the above!

Suggestions: 

1. installer outputs an error that the updates-released-modular-f29 repo is required if the updates-released-f29 repo is specified
2. ksvalidator does the same (less good)
3. Document the change from previous releases in https://docs.fedoraproject.org/en-US/fedora/f29/install-guide/appendixes/Kickstart_Syntax_Reference/#sect-kickstart-commands-repo (but many won't read it until desperate)
4. Best, implement a kickstart command repo --add-default-update-repositories so that it is not necessary to find out the hard way what exactly these might be
Comment 1 Vendula Poncova 2019-01-30 10:32:44 UTC 
It might be related to the bug 1669256. Please, attach logs from the failed installation. You can find them during the installation in /tmp/*log.
Comment 2 Nigel Arnot 2019-01-30 12:01:10 UTC 
(In reply to Vendula Poncova from comment #1)
> It might be related to the bug 1669256. Please, attach logs from the failed
> installation. You can find them during the installation in /tmp/*log.

Will have to run another install with the bug put back ...

To save me more time working this out, how do I then get the contents of /tmp out of 
a failed install? Presumably, I don't hit enter to complete the install and halt
the system. Will the kernel running at that time (off netinst.iso) recognise a second 
usb stick if I plug it in? Or do I have a live network and ability to run scp?
Comment 3 Martin Kolman 2019-01-30 14:47:46 UTC 
(In reply to Nigel Arnot from comment #2)
> (In reply to Vendula Poncova from comment #1)
> > It might be related to the bug 1669256. Please, attach logs from the failed
> > installation. You can find them during the installation in /tmp/*log.
> 
> Will have to run another install with the bug put back ...
> 
> To save me more time working this out, how do I then get the contents of
> /tmp out of 
> a failed install? 
There should be a root shell running on TTY2.

>Presumably, I don't hit enter to complete the install and
> halt
> the system. Will the kernel running at that time (off netinst.iso) recognise
> a second 
> usb stick if I plug it in? Or do I have a live network and ability to run
> scp?
Both should be possible - a USB drive should be detected & dev node created for it automatically. You will still likely need to mount it somewhere though.
And as long as you have network (I guess you can just configure it manually or over kickstart) it should be possible to scp the log files somewhere else.
Also if the installation is actually successful even though the system is unbootable, the logs should still be there on the installed system in /var/log/anaconda.
So mounting the storage from a working OS might work as well (or using something like libguestfs if installing a VM).
Comment 4 Nigel Arnot 2019-01-30 18:46:56 UTC 
The problem has gone away. (Bangs head on desk). 

I've re-run two of the previously failed kickstart files, and they have succeeded today.
I'm guessing that something in the updates repository (or one of its mirrors?) was causing 
the problem, and it's been replaced since I thought I'd diagnosed the problem.

Unfortunately all the log files for the failures are destroyed, since I was repeatedly
installing onto the same test system (and didn't know they were in /tmp to be saved).

This probably leaves you with very little to go on. I'll attach one of the kickstart
files, but unless you have the means to roll back the updates repository to how it was
last Thursday (24th January) it's probably not much help.

I don't know if this even makes sense, but would it be possible to install a kernel
off the netinst.iso boot media before doing anything at all with downloaded rpms? Then
if the upgraded kernel doesn't install or doesn't work, there is at least something 
bootable.
Comment 5 Nigel Arnot 2019-01-30 18:49:07 UTC 
Created attachment 1525130 [details]
kickstart file that failed on 24th and works today 30th
Comment 6 Vendula Poncova 2019-01-31 11:52:46 UTC 
I am pretty sure that this is a duplicate of the bug 1669256. I think that you installed different versions of systemd, one with the bug and one with the fix.

*** This bug has been marked as a duplicate of bug 1669256 ***