Bug 1367029

Summary: Qemu crash after several times rebooting if guest has GUI and multiple videos
Product: Red Hat Enterprise Linux 7 Reporter: Fangge Jin <fjin>
Component: spiceAssignee: Default Assignee for SPICE Bugs <rh-spice-bugs>
Status: CLOSED ERRATA QA Contact: SPICE QE bug list <spice-qe-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.3CC: cfergeau, chayang, dblechte, dyuan, fjin, fziglio, juzhang, knoel, mzhan, pgrunt, philipp, rduda, tpelka, virt-maint, yafu, yanqzhan, zhguo, zpeng
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: spice-0.12.8-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-01 16:06:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1388947    
Bug Blocks:    
Attachments:
Description Flags
Qemu log
none
Guest xml
none
The full backtrace of coredump none

Description Fangge Jin 2016-08-15 10:11:01 UTC
Created attachment 1190844 [details]
Qemu log

Description of problem:
Start a guest with multiple video devices, when guest boots up fully, reboot guest, and repeating booting for several times, qemu process will crash.

Version-Release number of selected component:
spice-server-0.12.4-18.el7.x86_64
qemu-kvm-rhev-2.6.0-20.el7.x86_64
libvirt-2.0.0-5.el7.x86_64
kernel-3.10.0-489.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Install a guest with GUI and add multiple videos into guest xml:
# virsh dumpxml rhel7
...
    <video>
      <model type='qxl' ram='1' vram='32768' vgamem='16384' heads='2' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </video>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </video>
    <video>
      <model type='qxl' ram='65536' vram='16384' vgamem='16384' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>

2. Start guest

3. when guest boots up fully, reboot guest
# virsh reboot rhel7

4. Repeat step 3 for several times, qemu will crash

Actual results:
Qemu crash after rebooting several times

Expected results:
Qemu doesn't crash

Additional info:
Backtrace of coredump:
(gdb) bt
#0  0x00007faeeb0e91d7 in raise () from /lib64/libc.so.6
#1  0x00007faeeb0ea8c8 in abort () from /lib64/libc.so.6
#2  0x00007faeedefb5fc in spice_logv (log_domain=0x7faeedf78d3d "SpiceWorker", log_level=SPICE_LOG_LEVEL_ERROR, strloc=0x7faeedf7a4a4 "red_worker.c:4091",
    function=0x7faeedf7c600 <__FUNCTION__.27093> "free_one_drawable", format=0x7faeedf71b9e "assertion `%s' failed", args=args@entry=0x7fae857fc5c0)
    at log.c:109
#3  0x00007faeedefb755 in spice_log (log_domain=log_domain@entry=0x7faeedf78d3d "SpiceWorker", log_level=log_level@entry=SPICE_LOG_LEVEL_ERROR,
    strloc=strloc@entry=0x7faeedf7a4a4 "red_worker.c:4091", function=function@entry=0x7faeedf7c600 <__FUNCTION__.27093> "free_one_drawable",
    format=format@entry=0x7faeedf71b9e "assertion `%s' failed") at log.c:123
#4  0x00007faeedece21d in free_one_drawable (worker=0x7faf0b378000, force_glz_free=0) at red_worker.c:4091
#5  0x00007faeeded004e in red_process_commands (worker=<optimized out>, ring_is_empty=<optimized out>, max_pipe_size=50) at red_worker.c:4115
#6  0x00007faeededa75a in red_worker_main (arg=<optimized out>) at red_worker.c:12294
#7  0x00007faeeca70dc5 in start_thread () from /lib64/libpthread.so.0
#8  0x00007faeeb1ab73d in clone () from /lib64/libc.so.6

Comment 1 Fangge Jin 2016-08-15 10:12:40 UTC
Created attachment 1190845 [details]
Guest xml

Comment 2 Fangge Jin 2016-08-15 10:13:08 UTC
Created attachment 1190846 [details]
The full backtrace of coredump

Comment 4 Gerd Hoffmann 2016-09-05 09:06:26 UTC
spice worker thread crashing, reassigning to spice.

Comment 5 Pavel Grunt 2016-09-14 15:25:24 UTC
Hi,

what is purpose of the configuration (multiple devices and multiple heads) ?

(In reply to JinFangge from comment #0)
> # virsh dumpxml rhel7
> ...
>     <video>
>       <model type='qxl' ram='1' vram='32768' vgamem='16384' heads='2'
> primary='yes'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
> function='0x0'/>
>     </video>
>     <video>
>       <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='3'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
> function='0x0'/>
>     </video>
>     <video>
>       <model type='qxl' ram='65536' vram='16384' vgamem='16384' heads='1'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> function='0x0'/>
>     </video>
>

Comment 6 Pavel Grunt 2016-09-14 15:38:07 UTC
the assertion is removed in the commit 5c7e248445f95c3fa2627532780950cf604b9e20

https://cgit.freedesktop.org/spice/spice/commit/?id=5c7e248445f95c3fa2627532780950cf604b9e20

Comment 7 Fangge Jin 2016-09-14 15:47:51 UTC
(In reply to Pavel Grunt from comment #5)
> Hi,
> 
> what is purpose of the configuration (multiple devices and multiple heads) ?
> 
> (In reply to JinFangge from comment #0)
> > # virsh dumpxml rhel7
> > ...
> >     <video>
> >       <model type='qxl' ram='1' vram='32768' vgamem='16384' heads='2'
> > primary='yes'/>
> >       <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
> > function='0x0'/>
> >     </video>
> >     <video>
> >       <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='3'/>
> >       <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
> > function='0x0'/>
> >     </video>
> >     <video>
> >       <model type='qxl' ram='65536' vram='16384' vgamem='16384' heads='1'/>
> >       <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > function='0x0'/>
> >     </video>
> >

It's just for testing, and "heads" doesn't take effect for now actually.

Comment 8 David Blechter 2016-09-14 15:57:45 UTC
Incorrect setting - there is only one driver for rhel 7 VMs. Need to investigate to be sure the only incorrect setting causes the crash.

Comment 9 Christophe Fergeau 2016-09-14 16:09:25 UTC
A "minimal" reproducer is to just add an additional
<video><model type='qxl'/></video> node to an existing RHEL7 VM, and then use virsh reboot. The specifics of the <video> XML nodes did not make a difference here for reproduction.

Comment 10 Frediano Ziglio 2016-12-13 15:37:49 UTC
Pavel, should we backport the patch?

Comment 11 Pavel Grunt 2016-12-13 19:15:59 UTC
(In reply to Frediano Ziglio from comment #10)
> Pavel, should we backport the patch?

Yes. It is in the upstream stable branch as well - ie it can be consider fixed by resolving the bug 1388947

Comment 12 Philip Prindeville 2017-01-16 21:54:36 UTC
On CentOS we're seeing a similar bug:

https://bugs.centos.org/view.php?id=12666

but that's got the following calling stack:

 {   "crash_thread": true
        ,   "frames":
              [ {   "address": 140542346469847
                ,   "build_id": "8b2c421716985b927aa0caf2a05d0b1f452367f7"
                ,   "build_id_offset": 217559
                ,   "function_name": "raise"
                ,   "file_name": "/lib64/libc.so.6"
                }
              , {   "address": 140542346475720
                ,   "build_id": "8b2c421716985b927aa0caf2a05d0b1f452367f7"
                ,   "build_id_offset": 223432
                ,   "function_name": "abort"
                ,   "file_name": "/lib64/libc.so.6"
                }
              , {   "address": 140542360880444
                ,   "build_id": "279f86639ce10689c4b461a2a2075f01480dc8b1"
                ,   "build_id_offset": 423228
                ,   "function_name": "spice_logv"
                ,   "file_name": "/lib64/libspice-server.so.1"
                }
              , {   "address": 140542360880789
                ,   "build_id": "279f86639ce10689c4b461a2a2075f01480dc8b1"
                ,   "build_id_offset": 423573
                ,   "function_name": "spice_log"
                ,   "file_name": "/lib64/libspice-server.so.1"
                }
              , {   "address": 140542360695117
                ,   "build_id": "279f86639ce10689c4b461a2a2075f01480dc8b1"
                ,   "build_id_offset": 237901
                ,   "function_name": "free_one_drawable"
                ,   "file_name": "/lib64/libspice-server.so.1"
                }
              , {   "address": 140542360702846
                ,   "build_id": "279f86639ce10689c4b461a2a2075f01480dc8b1"
                ,   "build_id_offset": 245630
                ,   "function_name": "red_process_commands.constprop.139"
                ,   "file_name": "/lib64/libspice-server.so.1"
                }
              , {   "address": 140542360745610
                ,   "build_id": "279f86639ce10689c4b461a2a2075f01480dc8b1"
                ,   "build_id_offset": 288394
                ,   "function_name": "red_worker_main"
                ,   "file_name": "/lib64/libspice-server.so.1"
                }
              , {   "address": 140542473665989
                ,   "build_id": "c3deb1fa27cd0c1c3cc575b944abacba0698b0f2"
                ,   "build_id_offset": 32197
                ,   "function_name": "start_thread"
                ,   "file_name": "/lib64/libpthread.so.0"
                }
              , {   "address": 140542347265853
                ,   "build_id": "8b2c421716985b927aa0caf2a05d0b1f452367f7"
                ,   "build_id_offset": 1013565
                ,   "function_name": "__clone"
                ,   "file_name": "/lib64/libc.so.6"
                } ]
        }

Comment 15 errata-xmlrpc 2017-08-01 16:06:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1866