This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours

Bug 963003

Summary: Kick start install fails on Fedora 19
Product: [Fedora] Fedora Reporter: Jeevan Savant <savjee>
Component: anacondaAssignee: Anaconda Maintenance Team <anaconda-maint-list>
Status: CLOSED DUPLICATE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 19CC: anaconda-maint-list, dshea, g.kaviyarasu, jonathan, loleary, mkolman, q2dg, sbueno, techtonik, vanmeeuwen+fedora
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-06-24 16:14:18 EDT Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Attachments:
Description Flags
Screen shot of failure
none
sosreport.txt
none
cat /proc/cmdline
none
isolinux.cfg
none
sosreport.txt
none
Kickstart file
none
Screenshot of failure after ks=cdrom::/ks.cfg
none
sosreport.txt none

Description Jeevan Savant 2013-05-14 19:47:03 EDT
Created attachment 747967 [details]
Screen shot of failure

Description of problem:
Install using kick start script fails for Fedora 19 - Alpha

Version-Release number of selected component (if applicable):
Fedora-19-Alpha-x86_64-DVD

How reproducible:
Completely reproducible, every time.

Steps to Reproduce:
1. Do a non kick start install (i.e. manual install) of Fedora-19-Alpha-x86_64-DVD
2. Save kick start file anaconda-ks.cfg as ks.cfg
3. Re spin Fedora-19-Alpha-x86_64-DVD to use ks.cfg
4. Install fails and drops you to the dracut shell. Only error/warning message is dracut-initqueue[477]: Warning: could not boot.
  
Actual results:
Install does not work


Expected results:
Install should go through using the provided kick start file

Additional info:
1. I also tried the same by generating the kick start file using system-config-kickstart utility, same results.
2. Tried the same respun ISO without kick start and that works correctly.
Comment 1 Jeevan Savant 2013-05-14 19:48:09 EDT
Created attachment 747968 [details]
sosreport.txt
Comment 2 Jeevan Savant 2013-05-14 19:51:26 EDT
Created attachment 747972 [details]
cat /proc/cmdline
Comment 3 Jeevan Savant 2013-05-14 19:53:44 EDT
Created attachment 747973 [details]
isolinux.cfg
Comment 4 Larry O'Leary 2013-05-25 02:14:37 EDT
Created attachment 752939 [details]
sosreport.txt

Same problem here.
Comment 5 Jeevan Savant 2013-05-30 19:41:27 EDT
Reproduced the same issue with Fedora 19-Beta,
using "Fedora-19-Beta-x86_64-DVD.iso" as well

Bottom line kick start install on Fedora 19 is broken.
Comment 6 Larry O'Leary 2013-06-02 23:44:32 EDT
Not sure what is going on here but I got Anaconda to start by adding a repo to the boot parameters:

    repo=http://download.fedoraproject.org/pub/fedora/linux/development/19/x86_64/os/

Not sure how this has anything to do with the boot problem. In either case, this is not normal and needs to be addressed prior to release. I expect kickstart to just work.
Comment 7 Samantha N. Bueno 2013-06-03 11:01:01 EDT
Please attach your kickstart file as a text/plain attachment.
Comment 8 Jeevan Savant 2013-06-03 12:29:08 EDT
Created attachment 756406 [details]
Kickstart file

Attached ks.cfg.
Comment 9 techtonik 2013-06-13 08:09:51 EDT
Same problem here. But 969076 seems to be a duplicate.
Comment 10 Jeevan Savant 2013-06-24 14:53:02 EDT
I see the Fedora 19 is scheduled to release in a week from now, is this issue going to be addressed? I cannot edit the priority on this bug to high, if that's what it takes to have someone look at this. Without kickstart support I imagine Fedora 19 would be severely limited in places it can be deployed in my opinion.
Comment 11 techtonik 2013-06-24 14:56:10 EDT
Should we open seven new bugs with critical priority? Or just live with it?
Comment 12 Samantha N. Bueno 2013-06-24 16:14:18 EDT
(In reply to techtonik from comment #11)
> Should we open seven new bugs with critical priority? Or just live with it?

Please don't open multiple bugs regarding the same issue, especially if you know there already exists a bug for your problem.

If this bug hasn't received any attention, it's because the only changes being accepted at this point are for bugs marked as "Accepted Blocker" or "Accepted Freeze Exception"--neither of which this bug is--so we're focusing the bulk of our attention on fixing those. See: https://fedoraproject.org/wiki/Releases/19/Schedule

Lastly, the attached sosreports show that installation failed in two completely different ways.

(attachment 747968 [details])
[   14.339924] localhost dracut-initqueue[477]: anaconda using disk root at /dev/sr1
[   14.481946] localhost dracut-initqueue[477]: mount: /dev/sr1 is write-protected, mounting read-only
[   14.749027] localhost kernel: ISO 9660 Extensions: RRIP_1991A
[   14.588100] localhost dracut-initqueue[477]: anaconda: found /run/install/repo//LiveOS/squashfs.img
[   15.438723] localhost kernel: bio: create slab <bio-1> at 1
[  211.260044] localhost dracut-initqueue[477]: Warning: Could not boot.
[  211.288427] localhost systemd[1]: Starting Dracut Emergency Shell...

Actually, from further up in the report, it doesn't even look like your ks file is being detected. According to the wiki page, you are passing the incorrect boot option to tell anaconda your ks file lives on cdrom: http://fedoraproject.org/wiki/Anaconda/Kickstart#Boot_CD-ROM


(attachment 752939 [details])
[   19.917877] localhost dracut-initqueue[286]: dhcp failed
[   20.583153] localhost dracut-initqueue[286]: anaconda fetching kickstart from http://home01.oleary.home/ks/base.ks
[   20.627366] localhost dracut-initqueue[286]: % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[   20.639803] localhost dracut-initqueue[286]: Dload  Upload   Total   Spent    Left  Speed
[   20.689808] localhost dracut-initqueue[286]: 0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 28013  100 28013    0     0   423k      0 --:--:-- --:--:-- --:--:--  427k
[   22.742959] localhost dracut-initqueue[286]: anaconda using disk root at /dev/sr0
[   22.796683] localhost dracut-initqueue[286]: mount: /dev/sr0 is write-protected, mounting read-only
[   22.850553] localhost dracut-initqueue[286]: Warning: no suitable images
[  230.115862] localhost dracut-initqueue[286]: Warning: Could not boot.
[  230.174730] localhost dracut-initqueue[286]: Warning: /dev/root does not exist
[  230.185810] localhost dracut-initqueue[286]: Warning: /dev/root does not exist
[  230.214731] localhost systemd[1]: Starting Dracut Emergency Shell...

This didn't even dhcp, so it's failing to even pull down your ks file.
Comment 13 Jeevan Savant 2013-06-24 16:54:55 EDT
If you see the attached /proc/cmdline, you will see how ks.cfg was passed.
initrd=initrd.img inst.stage2=hd:LABEL=Fedora\x2019-Alpha\x20x86_64 quiet ks=cdrom:/ks.cfg BOOT_IMAGE=vmlinuz

The document you referred says following...

ks=cdrom:/<path> or in newer versions ks=cdrom:<cdrom device>:/<path>
The installation program will look for the kickstart file on CD-ROM, as file <path>.

I fail to understand your diagnosis "you are passing the incorrect boot option to tell anaconda your ks file lives on cdrom:", 
Please explain how is it not within what the documents states. Particularly if both "ks=cdrom:/<path>" and "ks=cdrom:<cdrom device>:/<path>" are allowed?

Now I find the new anaconda in Fedora 19 no longer supports "ks=cdrom:/<path>". Is this a bug or functionality was removed on purpose.

This essentially breaks having a single custom Fedora 19 image across multiple hardware configurations where the "cdrom" device varies. Earlier anaconda was nice enough to scan for all possible cdrom devices and find ks.cfg. 

i.e. on some boxes you have to pass ks=cdrom:/dev/sr0:/ks.cfg, on others ks=cdrom:/dev/sr1:ks.cfg. In past ks=cdrom:/ks.cfg was sufficient.

Now this change means you have to build separate images for each platform. Why break functionality that was perfectly working for last 18 Fedora versions?
Comment 14 q2dg 2013-06-24 16:57:19 EDT
If it's not a blocker bug, it should be. 
I don't know the bureaucracy, but I do believe that it would be worthy to spend a minute in this bug. It's really really really disappointing and annoying
Comment 15 Samantha N. Bueno 2013-06-24 17:58:43 EDT
(In reply to Jeevan Savant from comment #13)
>
> Now I find the new anaconda in Fedora 19 no longer supports
> "ks=cdrom:/<path>". Is this a bug or functionality was removed on purpose.
> 
> This essentially breaks having a single custom Fedora 19 image across
> multiple hardware configurations where the "cdrom" device varies. Earlier
> anaconda was nice enough to scan for all possible cdrom devices and find
> ks.cfg. 
> 
> i.e. on some boxes you have to pass ks=cdrom:/dev/sr0:/ks.cfg, on others
> ks=cdrom:/dev/sr1:ks.cfg. In past ks=cdrom:/ks.cfg was sufficient.
> 
> Now this change means you have to build separate images for each platform.
> Why break functionality that was perfectly working for last 18 Fedora
> versions?

We don't just arbitrarily decide to change something because it sounds like a good idea. As far as why, that would be in commit 665f01f85ce85653b5d37af19ad87e7f0064bd7d, if you want to see the code (bug#828589). The log message is:

"""
We weren't handling the "empty $ksdev" case, which (as it turns out) is
legit for 'ks=cdrom'.
    
Note that the old option was "ks=cdrom[:<path>]". Rather than trying
to cleverly detect the difference between a device and path, we're
making it "cdrom:<dev>:<path>", which matches the "hd:.." argument.
    
Thus if you want to specify a path for the kickstart but not the CDROM
device, do: ks=cdrom::<path>
"""

Also, that commit is from June 2012, meaning this change was already in F18.

(In reply to q2dg from comment #14)
> If it's not a blocker bug, it should be. 
> I don't know the bureaucracy, but I do believe that it would be worthy to
> spend a minute in this bug. It's really really really disappointing and
> annoying

If that's the way you feel, then it is probably worth your time to learn "the bureaucracy" so you can work to implement changes, especially if what you find important is not in line with what other contributors find important. 

I spent more than a minute of my time on this bug testing and looking over the provided ks file, combing through the sosreports to provide feedback to those who attached them, and composing responses.
Comment 16 q2dg 2013-06-24 18:37:47 EDT
Ok. I didn't want to be rude, I'm very sorry if you could think so.

I just wanted to express my surprise to note that several features that can help Fedora / Red Hat to spread in production environments with multiple machines (offices, schools...) are apparently not sufficiently taken into account (or aren't well documented). Another typical case is LTSP (K12Linux).

In any case, thank you very much for the answer
Comment 17 Jeevan Savant 2013-06-24 19:40:30 EDT
>Thus if you want to specify a path for the kickstart but not the CDROM
>device, do: ks=cdrom::<path>

I attempted the workaround suggested on 4 boxes
Two of these boxes have *no* physical cdroms, I am using Virtual CD over IPMI to mount the ISO's. Both of these resulted in to a crash, which has been previously reported through
https://bugzilla.redhat.com/show_bug.cgi?id=967527

The bug resolution mentions "Verified fixed with F19 TC5 DVD.". I will give the TC6 a shot and see how it fares.

The other two boxes have physical cdroms, here too I am using Virtual CD over IPMI. Both of these boxes drop me to the dracut shell prompt, similar to one in the originally reported bug. 

This is because the CD device is /dev/sr1, but installer is looking for ks.cfg on /dev/sr0. See attached screenshot, particularly lines after "Reached basic target system". I am also attaching the sosreport from one of these failed boxes.

Based on this I am thinking the workaround may only work if the install iso is mounted on the first CD device of the box. In case where there are multiple CD devices then it is not working. 

It would be great if ks=cdrom::<path> work.
Comment 18 Jeevan Savant 2013-06-24 19:41:27 EDT
Created attachment 764813 [details]
Screenshot of failure after ks=cdrom::/ks.cfg
Comment 19 Jeevan Savant 2013-06-24 19:43:42 EDT
Created attachment 764814 [details]
sosreport.txt
Comment 20 Jeevan Savant 2013-06-24 20:05:55 EDT
Additionally looking at the link you sent for(bug#828589) https://bugzilla.redhat.com/show_bug.cgi?id=828589, I noticed 

1) In comment 1 it says "[1] as long as you only have one CDROM device, anyway.". This matches my finding, i.e. the workaround does not work when you have multiple CDROM devices.

If you have hardware configurations with multiple CD devices, typically both physical CD and virtual CD, then you will end up with multiple ISO images each specific for configuration. This is a problem for data centers having multitudes of boxes of various types when it comes to roll out custom FC19 deployment.
Comment 21 Samantha N. Bueno 2013-06-25 11:39:51 EDT
(In reply to Jeevan Savant from comment #17)
> >Thus if you want to specify a path for the kickstart but not the CDROM
> >device, do: ks=cdrom::<path>
> 
> I attempted the workaround suggested on 4 boxes
> Two of these boxes have *no* physical cdroms, I am using Virtual CD over
> IPMI to mount the ISO's. Both of these resulted in to a crash, which has
> been previously reported through
> https://bugzilla.redhat.com/show_bug.cgi?id=967527
> 
> The bug resolution mentions "Verified fixed with F19 TC5 DVD.". I will give
> the TC6 a shot and see how it fares.

I'm sure you've already found it by now, but if not, F19 final RC01 was released this morning: http://dl.fedoraproject.org/pub/alt/stage/19-RC1/

> The other two boxes have physical cdroms, here too I am using Virtual CD
> over IPMI. Both of these boxes drop me to the dracut shell prompt, similar
> to one in the originally reported bug. 
> 
> This is because the CD device is /dev/sr1, but installer is looking for
> ks.cfg on /dev/sr0. See attached screenshot, particularly lines after
> "Reached basic target system". I am also attaching the sosreport from one of
> these failed boxes.
> 
> Based on this I am thinking the workaround may only work if the install iso
> is mounted on the first CD device of the box. In case where there are
> multiple CD devices then it is not working. 
> 
> It would be great if ks=cdrom::<path> work.

Ahh--ok then. I was not aware of the multiple optical drives. It might be appropriate to chime in on bug 828589 (since it's the original one) and note that you're still experiencing issues--especially since QA is specifically asking (in comment 14) for feedback and hasn't received any. I'll change this bug to mark it as a duplicate.

At this point, it is likely too late to get a fix for F19, but for F20 I'm sure it'll get looked at.

*** This bug has been marked as a duplicate of bug 828589 ***
Comment 22 Jeevan Savant 2013-06-25 20:31:50 EDT
Thanks for your insights they were very helpful for me to make progress.

I will start chasing 828589 as it (failures on boxes with multiple CDROM like devices) is still happening with the FC19 RC01 that I tried today.