Created attachment 1825695 [details] dmesg output from latest boot to run level 3 Description of problem: I have a Fedora 34 VM that was running perfectly under Fedora 33 host. I recently upgraded the host to Fedora 35 and now when I try to start the VM it will not launch in graphical mode. It first launches as a black screen, and if I wait 1 - 2 minutes after starting it the guest window goes away and the manager shows the guest is disconnected. If I edit the grub command line and add the number 3 at the end of the line beginning with linux the VM will start in console mode. From within the console mode trying to start graphics with "startx" also causes the crash. Version-Release number of selected component (if applicable): 2:6.1.0-5.fc35 How reproducible: Attempt to power on VM in graphics mode or attempt to "startx" from console mode (run level 3) Steps to Reproduce: 1. Open VMM 2. boot to graphics mode OR 3. boot to run level 3 then attempt to "startx" Actual results: VM crashes Expected results: VM should start in graphics mode Additional info: Host machine running Fedora 35 fully updated as of 9/23/2021 Host using nvidia graphics version 470.63 and also updated to 470.74 with same results. VM running Fedora 34 fully updated as of 9/23/2021. VM has 2 CPUs and 4096 MB memory allocated. Watching the CPU graph in the VMM management window, the CPUs show ~50% for a few moments then just before the crash jump to ~100% and crash.
Created attachment 1825696 [details] journalctl -b -1 output for last attempt to boot in graphical mode
Created attachment 1825697 [details] bootlog files for every attempt to boot VM with host at fedora 35
Created attachment 1825698 [details] bootlog files for every attempt to boot VM with host at fedora 35
Created attachment 1825699 [details] bootlog files for every attempt to boot VM with host at fedora 35
Lot of information here, but an unclear picture. What exactly are the log files? Are they from the guest or the host? If they're from the guest, how did you obtain them if the guest just shuts down (virt-log maybe)? The main thing to look for would be the *host* dmesg, so you can see if qemu is crashing. If qemu is crashing I would also expect to see a core dump on the host - use "coredumpctl list" to see. Also the contents of /var/log/libvirt/qemu/log/<guestname>.log may be interesting. Also: https://fedoraproject.org/wiki/How_to_debug_Virtualization_problems
All previously sent logs are from the VM. The one attached is the log from the host that shows an entry series for the last boot in run level 3 dated 2021-09-23 and one following dated 2021-09-24 as an attempt to boot into normal graphics mode run level 5. It ends with a spice server bug entry. Every attempt to start graphics mode causes the vm to crash graphically yet qemu does not crash and there are no core dumps on either the host or the vm. With the crash (possibly the spice server) the VM window disappears and the qemu vmm window shows the vm as disconnected. I can reconnect and it shows as running, but once I do so the only actions that do not cause qemu to lock up are a normal shutdown or a reset of that VM. (I only have the one VM on this machine, which is my laptop) The logs I am sending now are from the host. The guest vm log and dmesg output. Also the current journalctl output for the latest boot from the host, trimmed to the relevant time of the last attempt to boot the vm. I have left the VM running in runlevel 3 after the failed graphical boot.
Created attachment 1825958 [details] Host log for VM
Created attachment 1825959 [details] Dmesg output from host
Created attachment 1825961 [details] journalctl log for relevant time last attempt to boot in graphical mode was at about 23:10 2021-09-23 CDT.
There is this message in the qemu log: qxl_send_events: spice-server bug: guest stopped, ignoring A search points to a possible recent regression in spice: https://bugs.archlinux.org/task/71473 You could try seeing if upgrading or downgrading spice-server helps.
Created attachment 1826311 [details] virsh capavilities
Created attachment 1826312 [details] cpuinfo
Created attachment 1826313 [details] Dmesg output from host
Created attachment 1826314 [details] VM xml config
Created attachment 1826315 [details] VM log /var/log/libvirt/qemu/fedora34.log trimmed to last 2 boots only
Created attachment 1826316 [details] virt-host-validate
Created attachment 1826317 [details] lspci -vvv
Created attachment 1826318 [details] versions of installed packages plus the uname -a output
I attached several log files from the host according to that suggested by the referenced web page about trouble shooting VM problems. I cannot specifically say what the error is but hopefully someone with more experience can identify it. I have tried using spice-server version 0.14.3 from fedora 34 as well as version 0.15.0 from fedora 35 with no changes in behavior. Trying to boot to run level 5, with the progress shown on screen, as near as I can tell the last text displayed references starting the graphics, possibly gdm, then the window disappears. In all cases the VM boots. In run level 3 there are no errors at all except the inability to launch graphics using startx. When booted to run level 5 the viewer window dies and the virt-manager disconnects. Reconnecting the virt-manager allows a shutdown but no other actions, including no functionality in the viewer window -- mouse, keyboard, or other. The VM can be accessed in this state only by using ssh from the host and using the ssh session to access the VM system. The command "runlevel" in this state returns "N 5". I have reinstalled all libvirt packages, all qemu packages, and all spice-* packages with no changes.
If anyone else is hitting this, please report if you are using nvidia driver or not. Jeff's VM config shows stock spice and qxl setup created by virt-manager, so nothing interestin there. Jeff, does switching guest video device to VGA or Virtio help? Also please confirm that switching from SPICE to VNC graphics helps.
With Spice graphics it will boot to run level 3 using QXL, VGA, and Virtio video. All those give me the console login screen. With VNC graphics it will not even power on. I click the on button and absolutely nothing happens. This is with the kernel command line having the number 3 at the end for run level 3. I have just tried again with booting to graphics, (I remove "rhgb quiet 3" from the kernel command line) and got the following errors. Spice & QXL the system gives me the text thru (I think starting GDM) then a gray window appears and hangs for about 30 seconds to a minute and the graphics window disappears Spice & Virtio - The window never displays the gray window, it stays totally black, and with about the same delay also disappears. Spice & VGA - the same as for QXL. With all configs where it boots, the main VMM window shows disconnected (QEMU/KVM - Not Connected) after the viewer crashes. With all the VM is actually running and I can access it with ssh from the host. With all runlevel shows "N 5". The text display shows the same for booting with all modes, and the graphical viewer window disappears and cannot be reconnected in all modes. Remember this same VM and the VMM worked perfectly with the host at Fedora 33. The upgrade of the host from Fedora 33 to Fedora 35 seems to be what triggered the problems. It is not that the VM is not booting, it is that VMM loses connection and the graphical viewer is not functional and actually dies when attempting to start graphics. A copy of the VM qcow file also boots properly on a fedora 34 host.
Thanks for the logs and tests Jeff. Based on comment #10 and that spice server in Fedora 33 is 0.14.3 and in F35 is 0.15.0, I think it is fair to consider spice regression for now and investigate. I opened the bug upstream: https://gitlab.freedesktop.org/spice/spice/-/issues/64 I'll report back when I have more to share.
Note in comment # 19 that I reverted back to spice server 0.14.3 from F34 and that did not change my results in testing. It appears that either it is some component of spice that I overlooked and did not get switched back to 0.14.3 or something else (possibly one of the dependencies) is the actual cause. Spice components currently installed on the F35 host ------------------- # dnf list installed | grep spice qemu-audio-spice.x86_64 2:6.1.0-5.fc35 @updates-testing qemu-char-spice.x86_64 2:6.1.0-5.fc35 @updates-testing qemu-ui-spice-app.x86_64 2:6.1.0-5.fc35 @updates-testing qemu-ui-spice-core.x86_64 2:6.1.0-5.fc35 @updates-testing spice-glib.x86_64 0.39-4.fc35 @fedora spice-gtk3.x86_64 0.39-4.fc35 @fedora spice-server.x86_64 0.15.0-2.fc35 @fedora spice-vdagent.x86_64 0.21.0-5.fc35 @fedora ---------------- Spice components installed on the F34 host where the VM still functions ---------------- # dnf list installed | grep spice qemu-audio-spice.x86_64 2:5.2.0-8.fc34 @updates qemu-char-spice.x86_64 2:5.2.0-8.fc34 @updates qemu-ui-spice-app.x86_64 2:5.2.0-8.fc34 @updates qemu-ui-spice-core.x86_64 2:5.2.0-8.fc34 @updates spice-glib.x86_64 0.39-2.fc34 @fedora spice-gtk3.x86_64 0.39-2.fc34 @fedora spice-server.x86_64 0.14.3-3.fc34 @fedora spice-vdagent.x86_64 0.21.0-3.fc34 @updates ---------------- I note that in addition to the version update on spice-server all the qemu portions are version updated as well.
So, it might be a qemu or guest bug. I should have thought because it is not limited to Spice only: > With VNC graphics it will not even power on. I click the on button and absolutely nothing happens. but I fail to see a problem in the guest's logs and the qemu log is short to the minimum. We either need to be able to reproduce this or more information related to the issue. Would it be possible to try the options from the link below? The desired outcome is to have /var/log/libvirt/qemu/log/<guestname>.log to be verbose when the bug happens. https://fedoraproject.org/wiki/How_to_debug_Virtualization_problems
For the VNC issue, try updating qemu, I backported a patch recently that may fix it. -8 build
I waited a month and never actually noted the patch with qemu that was mentioned in post 25, but looking today I see that qemu-* is now at 6.1.0-10-f35 and spice-server is at 0.15.0-2-f35 I tried again today to start the VM in graphical mode and apparently the F35 updates have solved my issue because the vm was able to start in graphical mode. I had left it with the config at spice server & VGA and it booted that way when running the 5.14.17 F35 kernel on the host. I then reset the config to default with spice server & QXL, did a restart of VMM and the guest VM and it also booted to run level 5 properly. I have no clue what exactly was changed, but the sum of the updates over the last month on my host seems to have fixed the issue with the VM not booting into graphical mode. Thank you to all who have looked at this issue and for fixing the hangup.