Red Hat Bugzilla – Bug 1292710
i686 qemu guest install of Fedora 23 fails or does not boot properly
Last modified: 2017-01-27 17:16:55 EST
Description of problem:
Can't install Fedora 23 guest on a i686 Fedora 23 box using qemu
Hardware: Intel Pentium 4 2.8GHZ 32bit 4G memory
OS Software: Fedora 23 Server Install with Headless Virtualization
Upgraded to current available patches (dnf upgrade).
Attempted to use vert-manger to create Fedora 23 guest server using new. Then selecting local iso image of Fedora 23 server with 1.7G memory 1 CPU and 20G drive.
Attempted to use both the graphical and text modes of install. Ignoring the issue of slowness (takes about a day to get through to clearly failing) the installs never succeed and in the end on reboot just attempt to boot from the hard drive and do nothing won't even get to grub.
virt-builder fedora-23 --root-password password:pass -o test.qcow2 --format qcow2 --hostname test.domain.com
Was able to get something that (again ignoring the issue of slowness) would eventually boot. It always initially boots into maintenance mode with errors like:
Dec 17 18:12:27 backup.jenika.com audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-udev-trigger
Dec 17 18:12:41 backup.jenika.com kernel: piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0
Dec 17 18:12:54 backup.jenika.com systemd: dev-ttyS0.device: Job dev-ttyS0.device/start timed out.
Dec 17 18:12:54 backup.jenika.com systemd: Timed out waiting for device dev-ttyS0.device.
Dec 17 18:12:54 backup.jenika.com systemd: local-fs.target: Triggering OnFailure= dependencies.
Dec 17 18:12:54 backup.jenika.com systemd: Dependency failed for Serial Getty on ttyS0.
Dec 17 18:12:55 backup.jenika.com systemd: dev-disk-by\x2duuid-58cda93d\x2df577\x2d4c9b\x2d8a54\x2d23e0335b20ff.device: Job dev-disk-by\x2duuid-58cda93d\x2df577\x
Dec 17 18:12:54 backup.jenika.com systemd: serial-getty@ttyS0.service: Job serial-getty@ttyS0.service/start failed with result 'dependency'.
Dec 17 18:12:54 backup.jenika.com systemd: dev-ttyS0.device: Job dev-ttyS0.device/start failed with result 'timeout'.
Dec 17 18:12:54 backup.jenika.com systemd: dev-disk-by\x2duuid-51816408\x2d7bd8\x2d415e\x2db48c\x2d6796ad2c8687.device: Job dev-disk-by\x2duuid-51816408\x2d7bd8\x
Dec 17 18:12:54 backup.jenika.com systemd: Timed out waiting for device dev-disk-by\x2duuid-51816408\x2d7bd8\x2d415e\x2db48c\x2d6796ad2c8687.device.
Dec 17 18:12:54 backup.jenika.com systemd: Dependency failed for /boot.
Dec 17 18:12:54 backup.jenika.com systemd: Dependency failed for Local File Systems.
Dec 17 18:12:54 backup.jenika.com systemd: Dependency failed for Relabel all filesystems, if necessary.
Dec 17 18:12:54 backup.jenika.com systemd: fedora-autorelabel.service: Job fedora-autorelabel.service/start failed with result 'dependency'.
Dec 17 18:12:54 backup.jenika.com systemd: Dependency failed for Mark the need to relabel after reboot.
Dec 17 18:12:54 backup.jenika.com systemd: fedora-autorelabel-mark.service: Job fedora-autorelabel-mark.service/start failed with result 'dependency'.
Dec 17 18:12:54 backup.jenika.com systemd: boot.mount: Job boot.mount/start failed with result 'dependency'.
Dec 17 18:12:55 backup.jenika.com systemd: Timed out waiting for device dev-disk-by\x2duuid-58cda93d\x2df577\x2d4c9b\x2d8a54\x2d23e0335b20ff.device.
Dec 17 18:12:55 backup.jenika.com systemd: Dependency failed for Swap.
Dec 17 18:12:56 backup.jenika.com systemd: Starting Restore /run/initramfs on shutdown...
If you ignore the above and simply press ctrl-d when promoted for maintenance mode it will complete boot up and function.
Completed a "dnf upgrade" on the guest and reboots still get the above messages during boot and try to kick you into maintenance mode.
Installs either fail or can't boot properly.
Perfomrance is ridiculously slow.
Installing Fedora 23 as guest using any of the above three methods (default install, text install, or vert-builder install) would succeed and os boots cleanly.
I expect this is not a very common use case and I fully expected things to be slow (not as slow as they are) but given that all the documentation I can find indicates this should work I didn't expect things to fail this badly or be this slow.
I have identical hardware running ubuntu and vmware-server with a Fedora 23 guest that has not had the same issues and is not nearly as slow. Although the Fedora guest has been around since much older versions of Fedora and has only gone through fedup upgrades. But it doesn't demonstrate any of the boot issues identified above.
Re slowness I expect that KVM is somehow disabled on your
host. This page has some tips:
After using virt-builder to build the guest, what precise
commands are you using to import the guest into libvirt and
boot it? Assuming you are using libvirt - anyway, I need to
see the precise commands you are using.
Oh wait, it's a 32 bit *host*?? No wonder it's slow. You're running
a 64 bit guest on 32 bit, so it has to emulate everything.
It is a 32bit guest on a 32bit host and full emulation is expected. Again I expect it to be slow but the level of slowness that I see seems beyond just a emulation issue but the only comparison I have to guide that is the same hardware with vmware-server.
Once I have the image with virt-builder I just used virt-manager new to create a virtual using the vert-manager qcow2 file, 1.7G M 1 CPU nothing else adjusted from defaults.
This is what the running process looks like on the host for the qemu guest.
qemu 1499 1 6 Dec17 ? 00:58:03 /usr/bin/qemu-system-i386 -name backup -S -machine pc-i440fx-2.4,accel=tcg,usb=off,vmport=off -m 1536 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 76eca44f-3887-4cc1-861a-3e7e98b8b6c2 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/backup.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/var/lib/libvirt/images/backup.qcow2,if=none,id=drive-virtio-disk0,format=qcow2 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=22,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:7e:3c:52,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/backup.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel1,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 -device usb-tablet,id=input0 -spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on
I wouldn't expect qemu to be fast on a Pentium 4, as I don't believe
any common models had hardware virtualization support. So even though
the guest is 32 bit, it is still doing software emulation (TCG)
and that's going to be dead slow. VMware (and indeed Xen PV) work
in quite different ways and would probably perform better in this
situation. No one is doing virt on 32 bit hosts these days so
it's not an area of concern.
Anyway on the question of why this guest doesn't boot all the way,
I ran this virt-builder command (on 64 bit host):
$ virt-builder --arch i686 fedora-23
and then I imported it into libvirt and booted it, but making sure
to use software emulation (of course my machine is capable of
hardware virt, but I wanted to try to simulate the situation
described in this bug to some extent):
$ virt-install --import --name test-bz1292710 --ram 2048 --disk path=fedora-23.img,format=raw --os-variant fedora22 --arch i686 --virt-type qemu
That boots OK for me, albeit a bit slowly.
It might be that the emulation on the P4 is so slow that udev times
out, so you could try editing the kernel config (in grub, or offline
using virt-edit) and add udev.event-timeout=6000 to the kernel command
Inside guest modified grub:
linux16 /vmlinuz-4.2.7-300.fc23.i686+PAE root=UUID=d1209578-8227-47c9-bf15-56e3c3c6aac0 ro console=tty0 rd_NO_PLYMOUTH console=ttyS0,115200 LANG=en_US.UTF-8
linux16 /vmlinuz-4.2.7-300.fc23.i686+PAE root=UUID=d1209578-8227-47c9-bf15-56e3c3c6aac0 ro console=tty0 rd_NO_PLYMOUTH console=ttyS0,115200 LANG=en_US.UTF-8 udev.event-timeout=<value>
Where <value> was 6000,10000,100000,1000000,10000000
Shutdown guest. Started guest and in all cases still end up in maintenance mode with same error messages as above.
I realize this is not a common use case and I"m fine letting it go. But, I'm also happy to provide what ever details may help resolve this if interested.
That was my best guess. My only other guess is that you need to
make sure the --os-variant is set to Fedora 22 (not sure how
that is expressed in virt-manager, but there will be an equivalent
It fails on the ISA serial device, although (a) why it would fail
rather than just warning I don't know and (b) you're passing an
ISA serial device, so it should be OK. Try debugging systemd/udev
issues I guess.
In any case this doesn't seem to be a virt-builder or libguestfs
or qemu bug as far as I can tell.
> In any case this doesn't seem to be a virt-builder or libguestfs
> or qemu bug as far as I can tell.
Could be true I wonder if this would be obvious for the systemd group. Perhaps switch the bug to that and see if anything turns up?
Although it could be an issue with what qemu emulates vs what systemd expects. Number of discussions seem to exist related to this error "Timed out waiting for device dev-ttyS0.device." and containers.
Since it's a Fedora guest, did you tell virt-builder to properly fix the SELinux attributes with --selinux-relabel? I haven't seen that flag specified in all the invocations written so far, so making sure SELinux attributes are fixed should help getting this further on.
I will give that a try this evening and let you know how it goes.
# virt-builder fedora-23 --root-password password:##### -o test.qcow2 --format qcow2 --hostname host.domain.org --selinux-relabel
[ 4.8] Downloading: http://libguestfs.org/download/builder/fedora-23-i686.xz
[ 224.1] Planning how to build this image
[ 224.2] Uncompressing
[ 311.6] Converting raw to qcow2
[ 394.1] Opening the new disk
[ 583.8] Setting a random seed
[ 584.0] Setting the hostname: backup.jenika.com
[ 584.1] Setting passwords
[ 651.2] SELinux relabelling
[ 658.0] Finishing off
Output file: test.qcow2
Output size: 6.0G
Output format: qcow2
Total usable space: 5.4G
Free space: 4.6G (85%)
click import existing disk image
select test.qcow2 image
OS Type Linux
Version Fedora 22
1.5G of memroy
watch it boot end up at prompt for maintenance mode.
log in with root password
copied the boot.log and dmesg output and attached to this.
Created attachment 1127796 [details]
Created attachment 1127797 [details]
Sorry for the late reply.
I think the problem might be another case of bug #1049656 -- i.e. the systemd autorelabel service fails to start with a missing tty.
Can you please check using the hack suggested in
whether it makes the VMs boot for you?
# virt-builder fedora-23 --root-password password:******* -o test.qcow2 --format qcow2 --hostname test.jenika.com --selinux-relabel --edit '/usr/lib/systemd/system/fedora-autorelabel.service: $_ = "" if /StandardInput=tty/'
[ 1.7] Downloading: http://libguestfs.org/download/builder/fedora-23-i686.xz
[ 5.8] Planning how to build this image
[ 5.8] Uncompressing
[ 98.1] Converting raw to qcow2
[ 156.0] Opening the new disk
[ 285.8] Setting a random seed
[ 286.0] Setting the hostname: test.jenika.com
[ 286.1] Editing: /usr/lib/systemd/system/fedora-autorelabel.service
[ 286.9] Setting passwords
[ 353.2] SELinux relabelling
[ 360.1] Finishing off
Output file: test.qcow2
Output size: 6.0G
Output format: qcow2
Total usable space: 5.4G
Free space: 4.5G (83%)
No change still end up in maintenance mode.
Created attachment 1154360 [details]
Created attachment 1154361 [details]
The error does seem to be a bit different now, in that it
is now failing to find /boot and/or the root filesystem.
ISTR previously it was failing to find a serial device.
I'm not really any wiser, but systemd has a way to enable
lots more debugging.
I think you need to break into the boot at the grub console
to the kernel command line.
Created attachment 1154397 [details]
boot.log when boot command include systemd.log_level=debug systemd.log_target=console
Can you please provide the libvirt XML configuration for the VM?
See https://www.centos.org/docs/5/html/5.2/Virtualization/chap-Virtualization-Managing_guests_with_virsh.html for example how to do it.
Created attachment 1154721 [details]
libvirt test xml dump
Ah thanks, now I was able to reproduce the issue.
The templates we provide (at least, the fedora-23-i386) are not correctly labelled: among the mislabelled files there are /etc/ld.so.cache and /usr/lib/modules/*/modules.*, causing issues even before the systemd autorelabel service is started.
Easy fix locally is:
$ virt-builder fedora-23 --arch i686 -o test.qcow2 [etc...]
$ virt-customize -a test.qcow2 --no-network --run-command "setfiles /etc/selinux/targeted/contexts/files/file_contexts /"
then test.qcow2 should boot fine now.
I just sent a patch to fix/improve the SELinux handling of templates:
See also bug #1089100 for issues/improvements in the current SELinux relabelling.
Still no luck, end up in maintenance mode, attached new boot.log output.
# virt-customize --no-network --run-command "setfiles /etc/selinux/targeted/contexts/files/file_contexts /" -a test.qcow2
[ 0.0] Examining the guest ...
[ 103.6] Setting a random seed
[ 103.7] Running: setfiles /etc/selinux/targeted/contexts/files/file_contexts /
[ 622.3] Finishing off
Created attachment 1157782 [details]
boot.log with virt-customize and systemd debug
(In reply to Matthew McGillis from comment #23)
> Still no luck, end up in maintenance mode, attached new boot.log output.
The patch fixing the generation of Fedora templates was pushed few days ago, but the templates on libguestfs.org haven't been updated yet.
Not sure if anyone is interested but I upgraded this server to fedora-24 and then tried the same basic steps to create a fedora-24 virtual.
Things seem to have gotten worse the virtual goes into grub and starts fedora then kernel panics. I can generated a log if someone wants it.
This message is a reminder that Fedora 23 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 23. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora 'version'
Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.
Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 23 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.
Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.
Fedora 23 changed to end-of-life (EOL) status on 2016-12-20. Fedora 23 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.
If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
Thank you for reporting this bug and we are sorry it could not be fixed.
The basic issue of Fedora not being able to run a virtual on 32 bit hardware as outlined above is still the case. I get different errors/failures but the result is still the same the virtual will not boot up and be usable.
Happy to provide any details if someone is interested.