Bug 679127

Summary: Multiple running applications cause hanging/failing
Product: Red Hat Enterprise Linux 6 Reporter: Marian Krcmarik <mkrcmari>
Component: qemu-kvmAssignee: Uri Lublin <uril>
Status: CLOSED DUPLICATE QA Contact: Virtualization Bugs <virt-bugs>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 6.1CC: khong, lkocman, mhasko, mkenneth, tburke, uril, virt-maint
Target Milestone: rcKeywords: Regression, TestBlocker, Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-03-03 13:36:39 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
multi-threaded backtrace none

Description Marian Krcmarik 2011-02-21 16:54:58 UTC
Description of problem:
qemu-kvm fails/hangs after running several applications on the Guest. I connect to the guest with using spicec. Qemu-kvm is started through cli:
/usr/libexec/qemu-kvm -vga qxl -m 1024 -smp 2 -spice port=3001,disable-ticketing /dev/rootvm/RHEL61_x64_test. RHEL61_x64_test is lv with installed guest. After connect to the guest with using spicec (spicec --host localhost --port 3001) and starting several applications qemu-kvm hangs/fails with output:
handle_dev_destroy_surfaces: 
red_wait_outgoing_item: blocked
/
handle_dev_destroy_surfaces: 
red_wait_outgoing_item: blocked
qemu-kvm: /builddir/build/BUILD/qemu-kvm-0.12.1.2/qemu-kvm.c:2574: kvm_mutex_unlock: Assertion `!cpu_single_env' failed.


Version-Release number of selected component (if applicable):
qemu-kvm-0.12.1.2-2.147.el6.x86_64
Host: Linux 2.6.32-114.0.1.el6.x86_64
Guest: Linux 2.6.32-114.0.1.el6.x86_64
spice-client spice-client-0.7.3-1.el6.x86_64


How reproducible:
Always

Steps to Reproduce:
1. Run through cli qemu-kvm with installed guest (/usr/libexec/qemu-kvm -vga qxl -m 1024 -smp 2 -spice port=3001,disable-ticketing /dev/rootvm/RHEL61_x64_test) with spice options.
2. Connect to the guest using spicec. (spicec --host localhost --port 3001)
3. Keep starting different applications (FF, OOO, etc.) till qemu hangs/fails with output such as:
handle_dev_destroy_surfaces: 
red_wait_outgoing_item: blocked
handle_dev_destroy_surfaces: 
qemu-kvm: /builddir/build/BUILD/qemu-kvm-0.12.1.2/qemu-kvm.c:2574: kvm_mutex_unlock: Assertion `!cpu_single_env' faileds:
or
handle_dev_destroy_surfaces: 
red_wait_outgoing_item: blocked.

Additional info:
Sometimes only session is logged out instead of failing/hanging.

Comment 2 Marian Krcmarik 2011-02-22 09:11:43 UTC
From bt:
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
#1  0x0000003fe2e09345 in _L_lock_870 () from /lib64/libpthread.so.0
#2  0x0000003fe2e09217 in __pthread_mutex_lock (mutex=0x8e39a0) at pthread_mutex_lock.c:61
#3  0x000000000042af4e in kvm_mutex_lock () at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2580
#4  0x000000000040b8d8 in main_loop_wait (timeout=1000) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4418
#5  0x000000000042b2fa in kvm_main_loop () at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2165
#6  0x000000000040ef0f in main_loop (argc=<value optimized out>, argv=<value optimized out>, 
    envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4634
#7  main (argc=<value optimized out>, argv=<value optimized out>, envp=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6848

Comment 4 Lubos Kocman 2011-02-23 16:58:50 UTC
Seems like it happens only with -vga qxl set (guest driver doesn't matter) I got the same freeze with vesa and qxl.

Comment 5 Lubos Kocman 2011-02-23 16:59:46 UTC
I'm getting kicked off the session after like 20 seconds of usage.

Comment 6 Lubos Kocman 2011-02-24 10:00:22 UTC
This does happen also with single vcpu

Comment 7 David Jaša 2011-02-24 10:55:38 UTC
Created attachment 480718 [details]
multi-threaded backtrace

Attached multi-threaded backtrace, qemu-kvm hanged right after start of guest's X. CL was:
$ qemu-kvm -smp 2 -m 1024 -name rhel6_32 -enable-kvm -spice port=3001,disable-ticketing -soundhw ac97 -vga qxl /dev/root_vg/q-rhel6_32

Comment 8 Lubos Kocman 2011-02-24 12:22:06 UTC
From origin:
> Additional info:
> Sometimes only session is logged out instead of failing/hanging.

reported as separate bug:

https://bugzilla.redhat.com/show_bug.cgi?id=680120

Comment 9 Uri Lublin 2011-02-24 18:22:35 UTC
I could not reproduce with a single vcpu and some applications running (did not
try too hard).
Easily reproduced the "hanging" part of the bug with "chvt 4" and trying to
chvt back to the X console.
It seems that the problem is missing mutex unlocks. We have a patch upstream
(spice upstream not qemu) that fixes such issues; and specifically, the exact
same issue that we've encountered -- in qxl_add_memslot.

Upstream in (spice) patch: 4855161da1f6dbc129593487814cd8518016de20

Comment 10 Uri Lublin 2011-02-24 19:28:11 UTC
(In reply to comment #9)
> I could not reproduce with a single vcpu and some applications running (did not
> try too hard).
scratch that. Easily reproduced following the instructions in #comment0 ("Steps to Reproduce"), sometimes even before that when X starts. I guess I was just lucky earlier and I did not try too hard (used chvt instead of installing OOO).

Comment 11 Uri Lublin 2011-03-03 13:36:39 UTC
This bug looks like a duplicate of 678208 to me, especially when looking at the stack trace (comment#2).

*** This bug has been marked as a duplicate of bug 678208 ***