Bug 1292710 - i686 qemu guest install of Fedora 23 fails or does not boot properly
i686 qemu guest install of Fedora 23 fails or does not boot properly
Status: CLOSED EOL
Product: Fedora
Classification: Fedora
Component: libguestfs (Show other bugs)
25
i686 Linux
unspecified Severity low
: ---
: ---
Assigned To: Richard W.M. Jones
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-12-18 02:17 EST by Matthew McGillis
Modified: 2017-01-27 17:16 EST (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-12-20 12:08:13 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
dmesg output (48.13 KB, text/plain)
2016-02-16 21:45 EST, Matthew McGillis
no flags Details
boot.log output (21.11 KB, application/octet-stream)
2016-02-16 21:45 EST, Matthew McGillis
no flags Details
dmesg output (48.92 KB, text/plain)
2016-05-05 14:40 EDT, Matthew McGillis
no flags Details
boot.log output (22.71 KB, application/octet-stream)
2016-05-05 14:41 EDT, Matthew McGillis
no flags Details
boot.log output (33.22 KB, text/plain)
2016-05-05 18:24 EDT, Matthew McGillis
no flags Details
libvirt test xml dump (3.52 KB, text/plain)
2016-05-06 14:09 EDT, Matthew McGillis
no flags Details
boot.log with virt-customize and systemd debug (32.03 KB, text/plain)
2016-05-15 22:15 EDT, Matthew McGillis
no flags Details

  None (edit)
Description Matthew McGillis 2015-12-18 02:17:48 EST
Description of problem:
Can't install Fedora 23 guest on a i686 Fedora 23 box using qemu

Host environment:
Hardware: Intel Pentium 4 2.8GHZ 32bit 4G memory
OS Software: Fedora 23 Server Install with Headless Virtualization
Upgraded to current available patches (dnf upgrade).

How reproducible:
Attempted to use vert-manger to create Fedora 23 guest server using new. Then selecting local iso image of Fedora 23 server with 1.7G memory 1 CPU and 20G drive.

Attempted to use both the graphical and text modes of install. Ignoring the issue of slowness (takes about a day to get through to clearly failing) the installs never succeed and in the end on reboot just attempt to boot from the hard drive and do nothing won't even get to grub.

Using:
virt-builder fedora-23 --root-password password:pass -o test.qcow2 --format qcow2 --hostname test.domain.com

Was able to get something that (again ignoring the issue of slowness) would eventually boot. It always initially boots into maintenance mode with errors like:

Dec 17 18:12:27 backup.jenika.com audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-udev-trigger 
Dec 17 18:12:41 backup.jenika.com kernel: piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0
Dec 17 18:12:54 backup.jenika.com systemd[1]: dev-ttyS0.device: Job dev-ttyS0.device/start timed out.
Dec 17 18:12:54 backup.jenika.com systemd[1]: Timed out waiting for device dev-ttyS0.device.
Dec 17 18:12:54 backup.jenika.com systemd[1]: local-fs.target: Triggering OnFailure= dependencies.
Dec 17 18:12:54 backup.jenika.com systemd[1]: Dependency failed for Serial Getty on ttyS0.
Dec 17 18:12:55 backup.jenika.com systemd[1]: dev-disk-by\x2duuid-58cda93d\x2df577\x2d4c9b\x2d8a54\x2d23e0335b20ff.device: Job dev-disk-by\x2duuid-58cda93d\x2df577\x
Dec 17 18:12:54 backup.jenika.com systemd[1]: serial-getty@ttyS0.service: Job serial-getty@ttyS0.service/start failed with result 'dependency'.
Dec 17 18:12:54 backup.jenika.com systemd[1]: dev-ttyS0.device: Job dev-ttyS0.device/start failed with result 'timeout'.
Dec 17 18:12:54 backup.jenika.com systemd[1]: dev-disk-by\x2duuid-51816408\x2d7bd8\x2d415e\x2db48c\x2d6796ad2c8687.device: Job dev-disk-by\x2duuid-51816408\x2d7bd8\x
Dec 17 18:12:54 backup.jenika.com systemd[1]: Timed out waiting for device dev-disk-by\x2duuid-51816408\x2d7bd8\x2d415e\x2db48c\x2d6796ad2c8687.device.
Dec 17 18:12:54 backup.jenika.com systemd[1]: Dependency failed for /boot.
Dec 17 18:12:54 backup.jenika.com systemd[1]: Dependency failed for Local File Systems.
Dec 17 18:12:54 backup.jenika.com systemd[1]: Dependency failed for Relabel all filesystems, if necessary.
Dec 17 18:12:54 backup.jenika.com systemd[1]: fedora-autorelabel.service: Job fedora-autorelabel.service/start failed with result 'dependency'.
Dec 17 18:12:54 backup.jenika.com systemd[1]: Dependency failed for Mark the need to relabel after reboot.
Dec 17 18:12:54 backup.jenika.com systemd[1]: fedora-autorelabel-mark.service: Job fedora-autorelabel-mark.service/start failed with result 'dependency'.
Dec 17 18:12:54 backup.jenika.com systemd[1]: boot.mount: Job boot.mount/start failed with result 'dependency'.
Dec 17 18:12:55 backup.jenika.com systemd[1]: Timed out waiting for device dev-disk-by\x2duuid-58cda93d\x2df577\x2d4c9b\x2d8a54\x2d23e0335b20ff.device.
Dec 17 18:12:55 backup.jenika.com systemd[1]: Dependency failed for Swap.
Dec 17 18:12:56 backup.jenika.com systemd[1]: Starting Restore /run/initramfs on shutdown...

If you ignore the above and simply press ctrl-d when promoted for maintenance mode it will complete boot up and function.

Completed a "dnf upgrade" on the guest and reboots still get the above messages during boot and try to kick you into maintenance mode.

Actual results:
Installs either fail or can't boot properly.
Perfomrance is ridiculously slow.

Expected results:
Installing Fedora 23 as guest using any of the above three methods (default install, text install, or vert-builder install) would succeed and os boots cleanly. 

Additional info:
I expect this is not a very common use case and I fully expected things to be slow (not as slow as they are) but given that all the documentation I can find indicates this should work I didn't expect things to fail this badly or be this slow.

I have identical hardware running ubuntu and vmware-server with a Fedora 23 guest that has not had the same issues and is not nearly as slow. Although the Fedora guest has been around since much older versions of Fedora and has only gone through fedup upgrades. But it doesn't demonstrate any of the boot issues identified above.
Comment 1 Richard W.M. Jones 2015-12-18 05:25:59 EST
Re slowness I expect that KVM is somehow disabled on your
host.  This page has some tips:
http://virt-tools.org/learning/check-hardware-virt/

After using virt-builder to build the guest, what precise
commands are you using to import the guest into libvirt and
boot it?  Assuming you are using libvirt - anyway, I need to
see the precise commands you are using.
Comment 2 Richard W.M. Jones 2015-12-18 05:27:15 EST
Oh wait, it's a 32 bit *host*??  No wonder it's slow.  You're running
a 64 bit guest on 32 bit, so it has to emulate everything.
Comment 3 Matthew McGillis 2015-12-18 12:42:48 EST
It is a 32bit guest on a 32bit host and full emulation is expected. Again I expect it to be slow but the level of slowness that I see seems beyond just a emulation issue but the only comparison I have to guide that is the same hardware with vmware-server.

Once I have the image with virt-builder I just used virt-manager new to create a virtual using the vert-manager qcow2 file, 1.7G M 1 CPU nothing else adjusted from defaults.
Comment 4 Matthew McGillis 2015-12-18 12:48:29 EST
This is what the running process looks like on the host for the qemu guest.

qemu      1499     1  6 Dec17 ?        00:58:03 /usr/bin/qemu-system-i386 -name backup -S -machine pc-i440fx-2.4,accel=tcg,usb=off,vmport=off -m 1536 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 76eca44f-3887-4cc1-861a-3e7e98b8b6c2 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/backup.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/var/lib/libvirt/images/backup.qcow2,if=none,id=drive-virtio-disk0,format=qcow2 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=22,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:7e:3c:52,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/backup.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel1,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 -device usb-tablet,id=input0 -spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on
Comment 5 Richard W.M. Jones 2015-12-18 13:52:59 EST
I wouldn't expect qemu to be fast on a Pentium 4, as I don't believe
any common models had hardware virtualization support.  So even though
the guest is 32 bit, it is still doing software emulation (TCG)
and that's going to be dead slow.  VMware (and indeed Xen PV) work
in quite different ways and would probably perform better in this
situation.  No one is doing virt on 32 bit hosts these days so
it's not an area of concern.

Anyway on the question of why this guest doesn't boot all the way,
I ran this virt-builder command (on 64 bit host):

$ virt-builder --arch i686 fedora-23 

and then I imported it into libvirt and booted it, but making sure
to use software emulation (of course my machine is capable of
hardware virt, but I wanted to try to simulate the situation
described in this bug to some extent):

$ virt-install --import --name test-bz1292710 --ram 2048 --disk path=fedora-23.img,format=raw --os-variant fedora22 --arch i686 --virt-type qemu 

That boots OK for me, albeit a bit slowly.

It might be that the emulation on the P4 is so slow that udev times
out, so you could try editing the kernel config (in grub, or offline
using virt-edit) and add udev.event-timeout=6000 to the kernel command
line.
Comment 6 Matthew McGillis 2015-12-18 15:12:05 EST
Inside guest modified grub:

vi /boot/grub2/grub.cfg
replaced line:
linux16 /vmlinuz-4.2.7-300.fc23.i686+PAE root=UUID=d1209578-8227-47c9-bf15-56e3c3c6aac0 ro console=tty0 rd_NO_PLYMOUTH console=ttyS0,115200 LANG=en_US.UTF-8
with:
linux16 /vmlinuz-4.2.7-300.fc23.i686+PAE root=UUID=d1209578-8227-47c9-bf15-56e3c3c6aac0 ro console=tty0 rd_NO_PLYMOUTH console=ttyS0,115200 LANG=en_US.UTF-8 udev.event-timeout=<value>

Where <value> was 6000,10000,100000,1000000,10000000

Shutdown guest. Started guest and in all cases still end up in maintenance mode with same error messages as above.

I realize this is not a common use case and I"m fine letting it go. But, I'm also happy to provide what ever details may help resolve this if interested.
Comment 7 Richard W.M. Jones 2015-12-18 15:41:46 EST
That was my best guess.  My only other guess is that you need to
make sure the --os-variant is set to Fedora 22 (not sure how
that is expressed in virt-manager, but there will be an equivalent
setting somewhere).

It fails on the ISA serial device, although (a) why it would fail
rather than just warning I don't know and (b) you're passing an
ISA serial device, so it should be OK.  Try debugging systemd/udev
issues I guess.

In any case this doesn't seem to be a virt-builder or libguestfs
or qemu bug as far as I can tell.
Comment 8 Matthew McGillis 2015-12-18 16:06:06 EST
> In any case this doesn't seem to be a virt-builder or libguestfs
> or qemu bug as far as I can tell.

Could be true I wonder if this would be obvious for the systemd group. Perhaps switch the bug to that and see if anything turns up?

Although it could be an issue with what qemu emulates vs what systemd expects. Number of discussions seem to exist related to this error "Timed out waiting for device dev-ttyS0.device." and containers.
Comment 9 Pino Toscano 2016-02-16 06:44:14 EST
Since it's a Fedora guest, did you tell virt-builder to properly fix the SELinux attributes with --selinux-relabel?  I haven't seen that flag specified in all the invocations written so far, so making sure SELinux attributes are fixed should help getting this further on.
Comment 10 Matthew McGillis 2016-02-16 15:10:27 EST
I will give that a try this evening and let you know how it goes.
Comment 11 Matthew McGillis 2016-02-16 21:44:42 EST
# virt-builder fedora-23 --root-password password:##### -o test.qcow2 --format qcow2 --hostname host.domain.org --selinux-relabel
[   4.8] Downloading: http://libguestfs.org/download/builder/fedora-23-i686.xz
######################################################################## 100.0%
[ 224.1] Planning how to build this image
[ 224.2] Uncompressing
[ 311.6] Converting raw to qcow2
[ 394.1] Opening the new disk
[ 583.8] Setting a random seed
[ 584.0] Setting the hostname: backup.jenika.com
[ 584.1] Setting passwords
[ 651.2] SELinux relabelling
[ 658.0] Finishing off
                   Output file: test.qcow2
                   Output size: 6.0G
                 Output format: qcow2
            Total usable space: 5.4G
                    Free space: 4.6G (85%)
# virt-manager
click new
click import existing disk image
select test.qcow2 image
OS Type Linux
Version Fedora 22
1.5G of memroy
1 CPU
Name test
finish

watch it boot end up at prompt for maintenance mode.
log in with root password
copied the boot.log and dmesg output and attached to this.
Comment 12 Matthew McGillis 2016-02-16 21:45 EST
Created attachment 1127796 [details]
dmesg output
Comment 13 Matthew McGillis 2016-02-16 21:45 EST
Created attachment 1127797 [details]
boot.log output
Comment 14 Pino Toscano 2016-05-05 05:36:10 EDT
Sorry for the late reply.

I think the problem might be another case of bug #1049656 -- i.e. the systemd autorelabel service fails to start with a missing tty.

Can you please check using the hack suggested in
  https://bugzilla.redhat.com/show_bug.cgi?id=1049656#c33
whether it makes the VMs boot for you?
Comment 15 Matthew McGillis 2016-05-05 14:38:45 EDT
# virt-builder fedora-23 --root-password password:******* -o test.qcow2 --format qcow2 --hostname test.jenika.com --selinux-relabel --edit '/usr/lib/systemd/system/fedora-autorelabel.service: $_ = "" if /StandardInput=tty/'
[   1.7] Downloading: http://libguestfs.org/download/builder/fedora-23-i686.xz
[   5.8] Planning how to build this image
[   5.8] Uncompressing
[  98.1] Converting raw to qcow2
[ 156.0] Opening the new disk
[ 285.8] Setting a random seed
[ 286.0] Setting the hostname: test.jenika.com
[ 286.1] Editing: /usr/lib/systemd/system/fedora-autorelabel.service
[ 286.9] Setting passwords
[ 353.2] SELinux relabelling
[ 360.1] Finishing off
                   Output file: test.qcow2
                   Output size: 6.0G
                 Output format: qcow2
            Total usable space: 5.4G
                    Free space: 4.5G (83%)
# virt-manager
...

No change still end up in maintenance mode.
Comment 16 Matthew McGillis 2016-05-05 14:40 EDT
Created attachment 1154360 [details]
dmesg output
Comment 17 Matthew McGillis 2016-05-05 14:41 EDT
Created attachment 1154361 [details]
boot.log output
Comment 18 Richard W.M. Jones 2016-05-05 16:41:15 EDT
The error does seem to be a bit different now, in that it
is now failing to find /boot and/or the root filesystem.
ISTR previously it was failing to find a serial device.

I'm not really any wiser, but systemd has a way to enable
lots more debugging.

https://fedoraproject.org/wiki/How_to_debug_Systemd_problems
https://freedesktop.org/wiki/Software/systemd/Debugging/

I think you need to break into the boot at the grub console
and add:

 systemd.log_level=debug systemd.log_target=console

to the kernel command line.
Comment 19 Matthew McGillis 2016-05-05 18:24 EDT
Created attachment 1154397 [details]
boot.log output

boot.log when boot command include systemd.log_level=debug systemd.log_target=console
Comment 20 Pino Toscano 2016-05-06 05:04:33 EDT
Can you please provide the libvirt XML configuration for the VM?
See https://www.centos.org/docs/5/html/5.2/Virtualization/chap-Virtualization-Managing_guests_with_virsh.html for example how to do it.
Comment 21 Matthew McGillis 2016-05-06 14:09 EDT
Created attachment 1154721 [details]
libvirt test xml dump
Comment 22 Pino Toscano 2016-05-10 05:14:05 EDT
Ah thanks, now I was able to reproduce the issue.

The templates we provide (at least, the fedora-23-i386) are not correctly labelled: among the mislabelled files there are /etc/ld.so.cache and /usr/lib/modules/*/modules.*, causing issues even before the systemd autorelabel service is started.

Easy fix locally is:
$ virt-builder fedora-23 --arch i686 -o test.qcow2 [etc...]
$ virt-customize -a test.qcow2 --no-network --run-command "setfiles /etc/selinux/targeted/contexts/files/file_contexts /"
then test.qcow2 should boot fine now.

I just sent a patch to fix/improve the SELinux handling of templates:
  https://www.redhat.com/archives/libguestfs/2016-May/msg00040.html

See also bug #1089100 for issues/improvements in the current SELinux relabelling.
Comment 23 Matthew McGillis 2016-05-15 22:13:00 EDT
Still no luck, end up in maintenance mode, attached new boot.log output.

# virt-customize --no-network --run-command "setfiles /etc/selinux/targeted/contexts/files/file_contexts /" -a test.qcow2
[   0.0] Examining the guest ...
[ 103.6] Setting a random seed
[ 103.7] Running: setfiles /etc/selinux/targeted/contexts/files/file_contexts /
[ 622.3] Finishing off
Comment 24 Matthew McGillis 2016-05-15 22:15 EDT
Created attachment 1157782 [details]
boot.log with virt-customize and systemd debug
Comment 25 Pino Toscano 2016-05-16 04:32:06 EDT
(In reply to Matthew McGillis from comment #23)
> Still no luck, end up in maintenance mode, attached new boot.log output.

The patch fixing the generation of Fedora templates was pushed few days ago, but the templates on libguestfs.org haven't been updated yet.
Comment 26 Matthew McGillis 2016-09-16 13:30:39 EDT
Not sure if anyone is interested but I upgraded this server to fedora-24 and then tried the same basic steps to create a fedora-24 virtual.

Things seem to have gotten worse the virtual goes into grub and starts fedora then kernel panics. I can generated a log if someone wants it.
Comment 27 Fedora End Of Life 2016-11-24 09:19:37 EST
This message is a reminder that Fedora 23 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 23. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '23'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 23 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.
Comment 28 Fedora End Of Life 2016-12-20 12:08:13 EST
Fedora 23 changed to end-of-life (EOL) status on 2016-12-20. Fedora 23 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.
Comment 29 Matthew McGillis 2017-01-27 17:16:55 EST
The basic issue of Fedora not being able to run a virtual on 32 bit hardware as outlined above is still the case. I get different errors/failures but the result is still the same the virtual will not boot up and be usable.

Happy to provide any details if someone is interested.

Note You need to log in before you can comment on or make changes to this bug.