Bug 746928 - Fedora-16 guest on a Fedora-16 host freezes while starting anaconda
Fedora-16 guest on a Fedora-16 host freezes while starting anaconda
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: anaconda (Show other bugs)
16
Unspecified Linux
unspecified Severity medium
: ---
: ---
Assigned To: Anaconda Maintenance Team
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2011-10-18 05:58 EDT by Kashyap Chamarthy
Modified: 2012-01-16 15:50 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-11-28 00:49:27 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Output of virt-install (18.30 KB, text/plain)
2011-10-23 08:28 EDT, Kyle
no flags Details
Kickstart file for Chris (2.74 KB, application/octet-stream)
2011-10-24 21:01 EDT, Kyle
no flags Details

  None (edit)
Description Kashyap Chamarthy 2011-10-18 05:58:50 EDT
Description of problem:

I was trying an unattended virt-install an F16 guest on an Fedora-16 latest host. It just basically freezes while starting anaconda. Refer the virsh serial console o/p below for the exact hang.

Also, an F15 guest works just fine on the same f16 host.



 ==> Version Info <==
######################################################################
[root at moon ~]# uname -r ; rpm -q qemu-kvm libvirt python-virtinst
3.1.0-0.rc9.git0.0.fc16.x86_64
qemu-kvm-0.15.0-5.fc16.x86_64
libvirt-0.9.6-2.fc16.x86_64
python-virtinst-0.600.0-5.fc16.noarch
[root at moon ~]#
######################################################################




Below is a snippet of serial console o/p where the virt-install hung.
#######################################################################
.
.
.
.
Starting Initialize storage subsystems (RAID, LVM, etc.) failed, see 'systemctl status
fedora-storage-init-late.service' for details.
[0/386]
[    4.235192] systemd[1]: Unit fedora-storage-init-late.service entered failed state.
Started POSIX Message Queue File System.
Started Debug File System.
Started Huge Pages File System.
Started Security File System.
Starting Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...
Started Load legacy module configuration.
Starting File System Check on Root Device...
Starting Setup Virtual Console...
Started Load Kernel Modules.
Started FUSE Control File System.
Started Configuration File System.
[    4.350713] systemd-fsck[297]: Root directory is writable, skipping check.
Starting Apply Kernel Variables...
Started Set Up Additional Binary Formats.
Starting STDOUT Syslog Bridge...
[    4.405194] systemd-vconsole-setup[303]: /bin/loadkeys failed with error code 1.
[    4.408727] systemd-vconsole-setup[303]: /bin/setfont failed with error code 1.
Started STDOUT Syslog Bridge.
Started udev Coldplug all Devices.
Started File System Check on Root Device.
Started Setup Virtual Console.
Started Apply Kernel Variables.
Starting Remount Root FS...
Starting udev Wait for Complete Device Initialization...
[    4.517828] mount[328]: warning: can't open /etc/fstab: No such file or directory
[    4.551845] EXT4-fs (dm-0): re-mounted. Opts: user_xattr,acl,barrier=1,data=ordered
Started Remount Root FS.
[    4.586326] lvm[296]: No volume groups found
Started Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling.
Started Reconfigure the system on administrator request.
Started Relabel all filesystems, if necessary.
Started Mark the need to relabel after reboot.
Starting Load Random Seed...
Starting Recreate Volatile Files and Directories...
Started Load Random Seed.
[    4.703010] systemd-tmpfiles[345]: Successfully loaded SELinux database in 39ms 665us,
size on heap is 468K.
Started Recreate Volatile Files and Directories.
[    5.424048]  vda: unknown partition table
Started udev Wait for Complete Device Initialization.
Starting Software RAID Monitor Takeover...
Started anaconda performance monitor.
Starting System Logging Service...
Starting Shell on tty2...
[    5.879702] udevd[298]: setfilecon /dev/vcs2 failed: No such file or directory
Started Shell on tty2.
Looking for the virtio ports... done.
Starting Anaconda version 16.20.
#######################################################################

==> Script I used <==
# virt-install --connect=qemu:///system \
    --network=bridge:br0 \
    --initrd-inject=/export/fed.ks \
    --extra-args="ks=file:/fed.ks console=tty0 console=ttyS0,115200" \
    --name=$domname \
    --disk /export/vmimgs/$domname.img,size=20 \
    --ram 2048 \
    --vcpus=2 \
    --check-cpu \
    --accelerate \
    --hvm \
    --location=$location1 \
    --nographics
#######################################################################

==> Kickstart <==

# cat fed.ks
install
text
reboot
lang en_US.UTF-8
keyboard us
network --bootproto dhcp
rootpw redhat
firewall --enabled --ssh
selinux --enforcing
timezone --utc America/New_York
bootloader --location=mbr --append="console=tty0 console=ttyS0,115200 rd_NO_PLYMOUTH"
zerombr
clearpart --all --initlabel
autopart

%packages
@core
%end
#######################################################################
Comment 1 Chris Lumens 2011-10-18 09:39:58 EDT
Does it work if you also add "serial" as a command line argument for anaconda?
Comment 2 Kashyap Chamarthy 2011-10-18 12:03:06 EDT
Yeah, I tried that too. 

I added the 'serial' cmdline argument in the kickstart (refer below), to no avail.

--append="console=tty0 console=ttyS0,115200 rd_NO_PLYMOUTH serial text"
Comment 3 Ales Kozumplik 2011-10-19 01:45:26 EDT
(In reply to comment #2)
> Yeah, I tried that too. 
> 
> I added the 'serial' cmdline argument in the kickstart (refer below), to no
> avail.
> 
> --append="console=tty0 console=ttyS0,115200 rd_NO_PLYMOUTH serial text"

This line in kickstart only tells anaconda what parameters should the bootloader use once the system is installed.

What Chris meant in comment 1 was adding "serial" to the kernel boot line before booting anaconda.
Comment 4 Kyle 2011-10-23 08:25:06 EDT
If it helps, I've run into exactly the same problem, pulling down 3.1.0-0.rc10.git0.1.fc16.x86_64 from JAIST.

virt-install command used:
virt-install -p -r 512 -n rawhide -f /dev/vg_caesium_domUs/rawhide-boot -f /dev/vg_caesium_domUs/rawhide-root -f /dev/vg_caesium_domUs/rawhide-home -f /dev/vg_caesium_domUs/rawhide-swap -l ftp://ftp.jaist.ac.jp/pub/Linux/Fedora/development/16/x86_64/os/ -x serial,ks="http://hydrogen/~kyl191/ks/ks.php?hostname=rawhide"

If I read comment 3 correctly, I've got 'serial' in the right place. Not much difference though - My last few messages are only slightly different:
Started udev Wait for Complete Device Initialization.
Starting Software RAID Monitor Takeover...
Started anaconda performance monitor.
Starting System Logging Service...
Starting Shell on tty2...
Started Shell on tty2.
Starting Shell on hvc1...
Started Shell on hvc1.
Starting Anaconda version 16.22.

Essentially, no mention of setfilecon, or virtio ports, but I've got hvc1 instead. Also, Anaconda version 16.22.

Also, this output was the same without 'serial' in the kernel boot line.
Comment 5 Kyle 2011-10-23 08:28:07 EDT
Created attachment 529676 [details]
Output of virt-install
Comment 6 Ales Kozumplik 2011-10-24 02:27:13 EDT
What happens when you don't do serial install and connect to the virtual box through vnc instead?
Comment 7 Kyle 2011-10-24 08:49:05 EDT
This is weird. Running it through virt-manager, it boots, complains about my kickstart file "not having a closing quotation", then goes into the normal install.

So the freeze seems to be limited to the console somehow. I'm testing --vnc next.

Now I've got to figure out why my kickstart file passes in pykickstart-1.82-1.fc15, but fails in pykickstart-1.99.4-1.fc16 too.
Comment 8 Chris Lumens 2011-10-24 10:10:14 EDT
Kyle - if you attach your kickstart file, I can take a look.  Remember to remove any passwords or similar sensitive information.
Comment 9 Ales Kozumplik 2011-10-24 10:11:19 EDT
Hm, I tried 'f16 guest on f16 host' and serial install works for me.  I think you still haven't appended 'serial' to virt-install's --extra-args.
Comment 10 Kyle 2011-10-24 21:01:15 EDT
Created attachment 529978 [details]
Kickstart file for Chris

As requested, the kickstart file I'm using, less the crypted passwords & my SSH key.
Comment 11 Kyle 2011-10-24 21:26:27 EDT
Er... I did virt-install ... -x serial,ks="...", so it should have passed through to the install.

I'm just wondering if it's my system. I'm going to try doing a virt-install of F16 on my known-working F14 system, and try doing virt-installs of F14/15 on the F16 system.

I'm on call at work for the next 2 days so I won't be able to try until then though.
Comment 12 Ales Kozumplik 2011-10-25 03:09:28 EDT
(In reply to comment #11)
> Er... I did virt-install ... -x serial,ks="...", so it should have passed
> through to the install.

Well and this one misses the 'console=ttyS0,112500' part. This is what your extra args should look like:

--extra-args="ks=file:/fed.ks console=ttyS0,115200 serial"
Comment 13 Kyle 2011-10-25 03:55:24 EDT
(In reply to comment #12)
> Well and this one misses the 'console=ttyS0,112500' part. This is what your
> extra args should look like:
> 
> --extra-args="ks=file:/fed.ks console=ttyS0,115200 serial"
Er... 
I'm not the guy who needed that config - I came in in comment 4, after you posted that.
Comment 14 Ales Kozumplik 2011-10-25 08:09:53 EDT
(In reply to comment #13)

> Er... 
> I'm not the guy who needed that config - I came in in comment 4, after you
> posted that.

Ah.
Comment 15 Kyle 2011-10-26 00:54:12 EDT
Ok, as an additional data point, virt-install of f16 from JAIST fails on a f14 host too. Trying an f15 install on the same system works though, so it looks (to me) like something's up with f16 interacting with virt-install (since using virt-manager works).

One question - F15 uses 'hvc0', while F16 is using 'hvc1'. Is that significant? I've tried using console=hvc0 in the boot line, but that didn't work/change anything obvious.
Comment 16 Kyle 2011-10-26 00:56:58 EDT
Wait wait wait. I'm running this on a Xen hypervisor.

Does "Looking for the virtio ports... done." imply use of KVM?
Comment 17 Ales Kozumplik 2011-10-26 03:45:01 EDT
(In reply to comment #16)
> Wait wait wait. I'm running this on a Xen hypervisor.
> 
> Does "Looking for the virtio ports... done." imply use of KVM?

It does, it means we found the string "QEMU virtual CPU"  in /proc/cpuinfo.
Comment 18 Kyle 2011-10-26 04:37:05 EDT
Hmm. When you mentioned your success, did you do your testing with KVM or Xen?

Because I'm not getting it working on Xen.
Comment 19 Ales Kozumplik 2011-10-26 11:43:30 EDT
I am on KVM.
Comment 21 Chris Lumens 2011-10-31 11:37:05 EDT
Kyle - the problem lines in your kickstart file are:

echo "# Added by Kickstart install options
%wheel ALL=(ALL) ALL" >> /etc/sudoers

I guess pykickstart doesn't like the fact that it's split over lines, but it shouldn't be attempting to parse lines inside a script like this anyway.  I'll take a look.  Also, I don't think I can generate a better error message there.  I'm just getting a ValueError and reprinting it.
Comment 22 Chris Lumens 2011-10-31 13:02:05 EDT
I've pushed a fix to pykickstart to take care of this, including a test case so it doesn't happen again:

http://git.fedorahosted.org/git/?p=pykickstart.git;a=commit;h=67e0c268abbb3f53c9ec574c34f718bf1f69897c
Comment 23 Kashyap Chamarthy 2011-11-21 02:54:33 EST
Ales, and rest: Sorry for this late response, I was drowned w/ a torrent of other bugs and lost track of this.

From comment #9, seems like it works for you.


I just retried with the latest qemu-kvm and it works now for me.
==================
[root@moon ~]# uname -r ; rpm -q qemu-kvm libvirt python-virtinst
3.1.0-0.rc9.git0.0.fc16.x86_64
qemu-kvm-0.15.1-3.fc16.x86_64
libvirt-0.9.6-2.fc16.x86_64
python-virtinst-0.600.0-5.fc16.noarch
[root@moon ~]# 
================

But, access to serial console seems to be distorted. While I type password, it just escapes the prompt and throws error(I'm seeing this behaviour for f16 guests as well.

In a previous discussion w/ Ray Strode, he identified that plymouth was interfering. So I disable it pretty much using the rd_NO_PLYMOUTH option.

(though this is not be related to this bug. I think the current bz can be closed if there are no objections)
================================================
[root@moon ~]# virsh console pkifedora1
Connected to domain pkifedora1
Escape character is ^]

Fedora release 16 (Verne)
Kernel 3.1.0-7.fc16.x86_64 on an x86_64 (ttyS0)

dhcp201-193 login: root
Password: 
Login incorrect

login: 
Password: 
Login incorrect

login: 
================================================

Ales, can you please post your working version of kernel, qemu-kvm libvirt python-virtinst ? Just for comparison?
Comment 24 Ales Kozumplik 2011-11-21 05:15:00 EST
(In reply to comment #23)
> (though this is not be related to this bug. I think the current bz can be
> closed if there are no objections)

no objections from me.

> 
> Ales, can you please post your working version of kernel, qemu-kvm libvirt
> python-virtinst ? Just for comparison?


These are mine versions:

python-virtinst-0.600.0-5.fc16.noarch
qemu-kvm-0.15.1-1.fc16.x86_64
libvirt-0.9.6-2.fc16.x86_64

I never tested entering password in plymount via serial console so these won't guarantee anything.
Comment 25 Niranjan Mallapadi Raghavender 2011-11-21 06:32:34 EST
Hi 

I just tried with these versions

[root@dhcp209-47 ~]# rpm -q libvirt qemu-kvm python-virtinst
libvirt-0.9.6-2.fc16.x86_64
qemu-kvm-0.15.0-4.fc16.x86_64
python-virtinst-0.600.0-5.fc16.noarch

uname-r:

3.1.0-0.rc8.git0.1.fc16.x86_64

I tried the exact same command-line as mentioned in description on my f16 host, and it got stuck with:

....
.....
.
.
.

Starting Load Random Seed...
Starting Recreate Volatile Files and Directories...
Started Load Random Seed.
[   27.982542] systemd-tmpfiles[321]: Successfully loaded SELinux database in 107ms 781us, size on heap is 476K.
Started Recreate Volatile Files and Directories.
[   30.746144]  vda: unknown partition table
Started udev Wait for Complete Device Initialization.
Started anaconda performance monitor.
Starting System Logging Service...
Starting Shell on tty2...
Started Shell on tty2.
Looking for the virtio ports... done.
Starting Anaconda version 16.25.

It got stuck as mentioned in description
Comment 26 Niranjan Mallapadi Raghavender 2011-11-21 06:39:52 EST
We just re-verified by adding "serial" to extra args 

--extra-args="ks=file:/fed.ks console=tty0 console=ttyS0,115200 serial rd_NO_PLYMOUTH"

I think Bug can be closed
Comment 27 Kyle 2011-11-23 04:36:49 EST
Yep, working here too. Xen's working with -x "console=hvc0 serial rd_NO_PLYMOUTH". 

Might be able to drop either the console=hvc0 or serial, but I haven't tested it yet.

+1 to close as errata.
Comment 28 Kashyap Chamarthy 2011-11-28 00:49:27 EST
Closing per comments #23, #26, #27
Comment 29 purpleidea 2012-01-16 15:50:19 EST
FWIW, this affects RHEL and Centos 6+ hosts.

Note You need to log in before you can comment on or make changes to this bug.