Bug 2193332

Summary: Kickstart stuck after dracut-pre-pivot.service
Product: [Fedora] Fedora Reporter: Jan "Yenya" Kasprzak <kas>
Component: anacondaAssignee: Anaconda Maintenance Team <anaconda-maint-list>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 38CC: anaconda-maint-list, drw_08, hegarty.sam, kkoukiou, swifferm82, vponcova, vslavik, w
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-12-14 13:42:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
VNC console screenshot
none
Generated kickstart file. none

Description Jan "Yenya" Kasprzak 2023-05-05 09:12:53 UTC 
Created attachment 1962520 [details]
VNC console screenshot

Description of problem:

When trying to do a kickstart-based install, it get stuck after dracut-pre-pivot.service. I am trying to prepare an image for use in our OpenNebula. In previous releases of Fedora, I used kickstart for this task. In F38, boot got stuck. So I suspected something being wrong with my kickstart file. Therefore I did a manual install, and tried to use the kickstart.cfg file which gets generated by anaconda. But even with anaconda-generated kickstart file, it also gets stuck as before.

Version-Release number of selected component (if applicable):
Fedora 38

How reproducible:
100 %

Steps to Reproduce:
1. create a small KVM-based virtual machine (mine has 1 VCPU, 8 GB RAM, 10 GB disk over virtio-scsi), connected to a vnet where it can get IPv4 address from DHCP.
2. boot Fedora-38-server-netinst.iso
3. edit the bootloader command line: delete the "quiet" option and add "inst.ks=https://some.where/f38-initial.cfg"
4. boot the installer

Actual results:
It downloads the kickstart config (the network is active and I can ping the VM from the outside), but the last line printed on the system console is "Finished dracut-pre-pivot.service" and remains so forever (I waited at least half an hour).

Expected results:
The machine should get correctly kickstarted.

Additional info:
My OpenNebula nodes are x86_64 running CentOS 8. I will attach a VNC console screenshot and the kickstart file generated by Anaconda which I am trying to use.
Comment 1 Jan "Yenya" Kasprzak 2023-05-05 09:13:44 UTC 
Created attachment 1962521 [details]
Generated kickstart file.
Comment 2 Jan "Yenya" Kasprzak 2023-05-05 11:08:45 UTC 
I did more experiments with kickstart. Note that I am doing the following steps using the very same VM instance, only rebooting it between retries using Ctrl-Alt-Del on a VNC console, not destroying and recreating it again. So the VM configuration should be exactly the same in all the following cases:

- when I boot the VM without an inst.ks=... parameter at all, it enters the graphical install as expected

- when I use the kickstart config file containing a single line which reads either "graphical", or "text", or "# comment", the boot gets stuck after dracut-pre-pivot.service as it did with a complete kickstart.cfg in my original bug report above.

- when I boot the VM with an empty file (length 0 bytes) as the inst.ks=... parameter, or with a file containing only newlines (and no text or comments), it successfully boots into a tmux session, which then dies with "Kickstart file /run/install/ks.cfg is missing".

After destroying the VM, replacing the CD-ROM image with Fedora-37-Server-netinst.iso (37 instead of 38), booting it again, and adding inst.ks=... parameter pointing to a file containing only a single word "graphical", it boots correctly to the graphical installer.

So there is definitely something wrong with F38 kickstart-based install.
Comment 3 Windom WU 2023-07-04 07:15:39 UTC 
Same to me, I have a exactly same issue during the fedora 38 server kickstart installation, hang in the same place
Comment 4 Sam Hegarty 2023-07-04 10:25:33 UTC 
I can also confirm this. Weirdly enough setting systemd.log_target=console and systemd.log_target=debug makes it succeed occasionally - possible race condition somewhere?
Comment 5 Sam Hegarty 2023-07-04 11:18:14 UTC 
Oh and I'm using QEMU (with kvm) + Fedora 38 Server ISO + a disk image with an ext2 filesystem that has the label OEMDRV with my kickstart file located at /ks.cfg rather than fetching the kickstart file over the network.
Comment 6 steppybug 2023-09-04 22:47:35 UTC 
Just ran accross this on a laptop where it was working before UEFI configuration changes were made.  Narrowed it down to hyperthreading.  Once hyperthreading was re-enabled kickstart started working again.
Comment 7 Jan "Yenya" Kasprzak 2023-09-06 06:45:02 UTC 
Interesting. I have re-tested with two virtual CPUs in QEMU instead of one, and this time it worked. Back to one VCPU, and kickstart got stuck after the pre-pivot.service again.

@steppybug, how many CPU cores does your laptop have? Could it be it has single core/two threads?

So, is it a kernel problem, or is a sole CPU stuck somewhere during kickstart-based install, but not during an ordirnary anaconda boot, and having a second (V)CPU helps somehow? I would like to be able to kickstart Fedora even on a smallish instance with only one CPU.
Comment 8 Jan "Yenya" Kasprzak 2023-11-23 07:10:04 UTC 
As far as I can tell, this got fixed in F39 - I was able to kickstart a F39 VM having only one VCPU. Could we get the fix also for F38?
Comment 9 Vladimír Slávik 2023-12-01 19:28:54 UTC 
Sorry, media are not updated after a given Fedora version goes GA, so there is no point backporting the fix.