Bug 856052 - KVM guest using spice crashes on a host running RHEL 6.3
KVM guest using spice crashes on a host running RHEL 6.3
Status: CLOSED INSUFFICIENT_DATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: virt-manager (Show other bugs)
6.3
x86_64 Linux
unspecified Severity medium
: rc
: ---
Assigned To: Martin Kletzander
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-09-11 01:13 EDT by Akemi Yagi
Modified: 2013-06-25 09:44 EDT (History)
15 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-06-25 09:44:46 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Akemi Yagi 2012-09-11 01:13:17 EDT
Description of problem:

Both the host machine and the guest VM are running RHEL 6.3 x86_64.

When a KVM guest using spice was started in a graphical console in virt-manager, the VM freezes either immediately or after some moments. It does not respond to ping. The host's desktop may also become unresponsive. However, it is still possible to access the host through ssh. After some minutes, the VM partly comes back in the sense that the mouse now moves. Everything else remains frozen.

At this point the following lines appear in the log file ( /var/log/libvirt/qemu/guest.log ) on the host:

flush_display_commands: update timeout
red_channel_client_disconnect: 0x7ff78006a200 (channel 0x7ff7800458d0 type 2 id 0)
display_channel_client_on_disconnect: 

Note that the crash occurs only with the virt-manager's console, NOT when connecting to the VM using spice client or by using virt-viewer.

Version-Release number of selected component (if applicable):
Not sure which component is the culprit, so will list some potential candidates.

$ rpm -qa |egrep "spice|qemu|kvm|libvirt|qxl|usbredir" | sort
cairo-spice-1.8.7.1-4.el6.x86_64
ffmpeg-spice-libs-0.4.9-0.15.5spice.20080908.el6.x86_64
gpxe-roms-qemu-0.9.7-6.9.el6.noarch
libvirt-0.9.10-21.el6_3.4.x86_64
libvirt-client-0.9.10-21.el6_3.4.x86_64
libvirt-python-0.9.10-21.el6_3.4.x86_64
pixman-spice-0.13.3-5.el6.x86_64
qemu-img-0.12.1.2-2.295.el6_3.2.x86_64
qemu-kvm-0.12.1.2-2.295.el6_3.2.x86_64
spice-client-0.8.2-15.el6.x86_64
spice-glib-0.11-11.el6.x86_64
spice-gtk-0.11-11.el6.x86_64
spice-gtk-python-0.11-11.el6.x86_64
spice-gtk-tools-0.11-11.el6.x86_64
spice-protocol-0.10.1-5.el6.noarch
spice-server-0.10.1-10.el6.x86_64
spice-vdagent-0.8.1-3.el6.x86_64
spice-xpi-2.7-20.el6.x86_64
usbredir-0.4.3-1.el6.x86_64
xorg-x11-drv-qxl-0.0.14-13.el6_2.x86_64

How reproducible: Always.

Steps to Reproduce:
1. Install a KVM guest on a RHEL 6 host setting it to use spice.
2. Start the guest.
3.
  
Actual results:
The guest crashes.

Expected results:
No crash/freezing.

Additional info:
When the same VM is set to use VNC, there is no issue.
Comment 2 Marian Krcmarik 2012-09-11 04:44:33 EDT
I would definatly blame virt-manager, I guess the virt-manager is freezing.
Comment 3 Akemi Yagi 2012-09-11 08:18:11 EDT
I meant to add virt-manager to the list but forgot:

virt-manager-0.9.0-14.el6.x86_64

However, there is a preliminary finding that seems to indicate that downgrading qemu-kvm and qemu-img to 0.12.1.2-2.209.el6_2.4 resolves the issue.
Comment 4 AndrewSerk 2012-09-11 23:13:55 EDT
I am experiencing similar issues.

Host is 6.2 Guest is 6.3
When a guest has a spice display installed and boots to runlevel 5, the guest crashes to the poweroff state.  This happens when starting the guest with “virsh” and “virt-manager” whether connecting with “virt-viewer” or “spice-client” or not connecting at all.

Snippet from host dmesg after crash:
qemu-kvm[4600]: segfault at 50 ip 00007fe01640c225 sp 00007fe00e8cd9a0 error 4 in libspice-server.so.1.0.2[7fe0163d8000+e2000]

$ rpm -qa |egrep "spice|qemu|kvm|libvirt|usbredir|virt-manager|qxl" | sort
ffmpeg-spice-libs-0.4.9-0.15.5spice.20080908.el6.x86_64
gpxe-roms-qemu-0.9.7-6.9.el6.noarch
libvirt-0.9.10-21.el6_3.4.x86_64
libvirt-client-0.9.10-21.el6_3.4.x86_64
libvirt-python-0.9.10-21.el6_3.4.x86_64
pixman-spice-0.13.3-5.el6.x86_64
qemu-img-0.12.1.2-2.295.el6_3.2.x86_64
qemu-kvm-0.12.1.2-2.295.el6_3.2.x86_64
qemu-kvm-debuginfo-0.12.1.2-2.295.el6_3.2.x86_64
qemu-kvm-tools-0.12.1.2-2.295.el6_3.2.x86_64
spice-client-0.8.2-7.el6.x86_64
spice-glib-0.6-2.el6.x86_64
spice-gtk-0.6-2.el6.x86_64
spice-gtk-python-0.6-2.el6.x86_64
spice-gtk-tools-0.6-2.el6.x86_64
spice-protocol-0.8.1-2.el6.noarch
spice-server-0.8.2-5.el6.x86_64
spice-server-devel-0.8.2-5.el6.x86_64
spice-vdagent-0.8.1-3.el6.x86_64
usbredir-0.4.3-1.el6.x86_64
virt-manager-0.9.0-7.el6.x86_64
xorg-x11-drv-qxl-0.0.14-13.el6_2.x86_64

After the following downgrades the issue is resolved here:
Packages Altered:
    Downgrade  qemu-img-2:0.12.1.2-2.209.el6_2.4.x86_64       @sl-security
    Downgraded          2:0.12.1.2-2.295.el6_3.2.x86_64       @sl-security
    Downgrade  qemu-kvm-2:0.12.1.2-2.209.el6_2.4.x86_64       @sl-security
    Downgraded          2:0.12.1.2-2.295.el6_3.2.x86_64       @sl-security
    Downgrade  qemu-kvm-tools-2:0.12.1.2-2.209.el6_2.4.x86_64 @sl-security
    Downgraded                2:0.12.1.2-2.295.el6_3.2.x86_64 @sl-security
    Erase      usbredir-0.4.3-1.el6.x86_64                    @sl-security
Comment 5 Marian Krcmarik 2012-09-12 07:16:00 EDT
Akemi,

I can reproduce what you observed, but It seems like virt-manager is being frozen, VM is still alive and fine, once I kill virt-manager I can connect to the VM and the VM behaves normally. 
That's why I move to virt-manager. The truth is that It's easily reproducible on qemu-kvm-0.12.1.2-2.295.el6_3.2.

Andrew,

It seems you are using 6.2 with qemu-kvm from 6.3, This may not be supported, either keep 6.2 packages on 6.2 or try to upgrade all packages to 6.3 on your host especially spice-server.
Comment 6 Uri Lublin 2012-09-12 10:13:40 EDT
With newer packages, I could not reproduce 

guest: nightly RHEL-6.4 iso ("Minimal Desktop" installed).
host: RHEL-6.3 with the following packages:
  spice-server-0.11.3-1.el6_3.x86_64
  qemu-kvm-0.12.1.2-2.295.el6_3.1.x86_64 (also works with 295.el6_3.2)
  libvirt-0.9.10-21.el6_3.3.x86_64
  virt-manager-0.9.0-14.el6.x86_64
  spice-gtk-0.11-11.el6.x86_64
Comment 7 AndrewSerk 2012-09-12 10:50:45 EDT
Thanks for the responses, 
I upgraded host to 6.3 and now I can reproduce same issue as Akemi and Marian. The issue is only present when using virt-manager.

rpm -qa |egrep "spice|qemu|kvm|libvirt|usbredir|virt-manager|qxl" | sort
ffmpeg-spice-libs-0.4.9-0.15.5spice.20080908.el6.x86_64
gpxe-roms-qemu-0.9.7-6.9.el6.noarch
libvirt-0.9.10-21.el6_3.4.x86_64
libvirt-client-0.9.10-21.el6_3.4.x86_64
libvirt-python-0.9.10-21.el6_3.4.x86_64
pixman-spice-0.13.3-5.el6.x86_64
qemu-img-0.12.1.2-2.295.el6_3.2.x86_64
qemu-kvm-0.12.1.2-2.295.el6_3.2.x86_64
qemu-kvm-debuginfo-0.12.1.2-2.295.el6_3.2.x86_64
qemu-kvm-tools-0.12.1.2-2.295.el6_3.2.x86_64
spice-client-0.8.2-15.el6.x86_64
spice-glib-0.11-11.el6.x86_64
spice-gtk-0.11-11.el6.x86_64
spice-gtk-python-0.11-11.el6.x86_64
spice-gtk-tools-0.11-11.el6.x86_64
spice-protocol-0.10.1-5.el6.noarch
spice-server-0.10.1-10.el6.x86_64
spice-server-devel-0.10.1-10.el6.x86_64
spice-vdagent-0.8.1-3.el6.x86_64
usbredir-0.4.3-1.el6.x86_64
virt-manager-0.9.0-14.el6.x86_64
xorg-x11-drv-qxl-0.0.14-13.el6_2.x86_64
Comment 8 Akemi Yagi 2012-09-12 11:07:25 EDT
(In reply to comment #6)
> With newer packages, I could not reproduce 
> 
> guest: nightly RHEL-6.4 iso ("Minimal Desktop" installed).
> host: RHEL-6.3 with the following packages:
>   spice-server-0.11.3-1.el6_3.x86_64
>   qemu-kvm-0.12.1.2-2.295.el6_3.1.x86_64 (also works with 295.el6_3.2)
>   libvirt-0.9.10-21.el6_3.3.x86_64
>   virt-manager-0.9.0-14.el6.x86_64
>   spice-gtk-0.11-11.el6.x86_64

If I compare the above list with the one that causes the problem:

spice-server-0.10.1-10.el6.x86_64
qemu-kvm-0.12.1.2-2.295.el6_3.2.x86_64
libvirt-0.9.10-21.el6_3.4.x86_64
virt-manager-0.9.0-14.el6.x86_64
spice-gtk-0.11-11.el6.x86_64

What stands out is spice-server. Others are either the same or older on the list that does not reproduce the issue.
Comment 9 Alon Levy 2012-09-13 07:54:02 EDT
The block sounds like a known issue, blocking of virt-manager because qemu is stuck writing to spice client, preventing libvirtd from getting result from qemu monitor: virt-manager blocks on qemu that blocks on spice that blocks on virt-manager.

Akemi, please check:
 1. what revision qemu is showing from the qxl device? it should be 3:
virsh qemu-monitor-command --hmp info qtree | grep revision
 2. what version xorg-x11-drv-qxl are you running in the guest?

The second issue (comment 4) doesn't sound at all like the original poster's problem, since it is a crash. Andrew, please provide a stack trace of this issue.

Alon
Comment 10 Akemi Yagi 2012-09-13 10:51:25 EDT
(In reply to comment #9)
> The block sounds like a known issue, blocking of virt-manager because qemu
> is stuck writing to spice client, preventing libvirtd from getting result
> from qemu monitor: virt-manager blocks on qemu that blocks on spice that
> blocks on virt-manager.
> 
> Akemi, please check:
>  1. what revision qemu is showing from the qxl device? it should be 3:
> virsh qemu-monitor-command --hmp info qtree | grep revision

dev-prop: revision = 3

>  2. what version xorg-x11-drv-qxl are you running in the guest?

0.0.14-13.el6_2.x86_64
Comment 11 AndrewSerk 2012-09-17 12:50:49 EDT
(in reply to comment #9)

I would gladly provide a stack trace but am unable to because I updated my host from 6.2 to 6.3. I now have the same issue as Akemi describes in his first post.

I also show the same qemu revision for the qxl device and am using the same version xorg-x11-drv-qxl in my guests as Akemi.
Comment 12 Geyang Kong 2012-10-15 02:14:31 EDT
I think this bug is same as bug 816451. I suggest you retry this bug on latest libvirt packages, since from libvirt-0.10.0-0rc1.el6.x86_64, this issue has gone for me.
Comment 13 Akemi Yagi 2012-10-15 08:02:16 EDT
I tried again and this time there was no crash. The host now has:

libvirt-0.9.10-21.el6_3.5.x86_64

So, yes, it seems like this bug is the same as (or related to) bug 816451.

Because libvirt-0.9.10-21.el6_3.4.x86_64 had the issue, the fix must be in the latest update. Changelog shows:

* Wed Sep 26 2012 Jiri Denemark <jdenemar@redhat.com> - libvirt-0.9.10-21.el6_3.5                    
- security: Fix libvirtd crash possibility (CVE-2012-4423)                                           
- Fix augeas test of shared sanlock leases (rhbz#858988)
- qemu augeas: Add spice_tls/spice_tls_x509_cert_dir (rhbz#858988)
- Fix mistakes in augeas lens (rhbz#858988)
- qemu: Fix failure path in disk hotplug (rhbz#859376)
- blockjob: Relabel entire existing chain (rhbz#860720)
Comment 14 Akemi Yagi 2012-10-15 08:34:29 EDT
I spoke too soon. The guest froze after a 'yum update' & 'reboot' with the same symptoms.
Comment 15 Akemi Yagi 2012-10-16 03:32:36 EDT
I tested using yet another host updated with libvirt-0.9.10-21.el6_3.4 and confirm that the problem persists.
Comment 16 Geyang Kong 2012-10-16 23:36:23 EDT
If you still hit this error for libvirt-0.9.10-21.el6_3.5, I suggest you try with this link: http://veillard.com/libvirt/6.4/x86_64/ , you can download the libvirt packages from here and try again. On my machine, all of libvirt-0.10.x packages works well.
Comment 17 Akemi Yagi 2012-10-17 08:08:10 EDT
I installed:

libvirt.x86_64 0:0.10.2-1.el6   
libvirt-client.x86_64 0:0.10.2-1.el6   
libvirt-python.x86_64 0:0.10.2-1.el6

But I got the same guest freeze as before. Will try with another machine and report back.
Comment 18 Akemi Yagi 2012-12-14 14:20:46 EST
(In reply to comment #6)
> With newer packages, I could not reproduce 
> 
> guest: nightly RHEL-6.4 iso ("Minimal Desktop" installed).
> host: RHEL-6.3 with the following packages:
>   spice-server-0.11.3-1.el6_3.x86_64
>   qemu-kvm-0.12.1.2-2.295.el6_3.1.x86_64 (also works with 295.el6_3.2)
>   libvirt-0.9.10-21.el6_3.3.x86_64
>   virt-manager-0.9.0-14.el6.x86_64
>   spice-gtk-0.11-11.el6.x86_64

Now that RHEL6.4beta is out, should I go ahead and try it? Or has this already been tested as indicated by Uri Lublin?
Comment 20 Martin Kletzander 2013-03-14 10:59:22 EDT
Please try to reproduce that on 6.4 with current updated packages (mainly spice-server), thanks.
Comment 21 Akemi Yagi 2013-03-14 12:11:42 EDT
I updated the host to RHEL 6.4 when it came out. The issue reported here seemed to be gone. I have yet to update the guest but thought this would add a useful piece of information.
Comment 22 Martin Kletzander 2013-06-25 09:44:46 EDT
Going through old unresolved bugs this seems like an issue which was fixed on few levels.  I'm closing this as INSUFFICIENT_DATA as there is no information we can use to track down the crash/freeze, but feel free to reopen this BZ if you'll see the problem appearing again.  In that case, post a backtrace of crashed/frozen processes, please.

Thank you, Martin

Note You need to log in before you can comment on or make changes to this bug.