Bug 1318574

Summary: virt-viewer SIGSEGV when migrate from qemu-kvm 2.5 to qemu-kvm 2.3
Product: Red Hat Enterprise Linux 7 Reporter: Han Han <hhan>
Component: spice-gtkAssignee: Pavel Grunt <pgrunt>
Status: CLOSED ERRATA QA Contact: SPICE QE bug list <spice-qe-bugs>
Severity: low Docs Contact:
Priority: low    
Version: 7.3CC: dblechte, dyuan, hhan, mxie, pgrunt, rbalakri, rduda, tpelka, tzheng, xiaodwan, xuzhang
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: spice-gtk-0.31-1.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-04 01:15:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1329973    
Bug Blocks:    
Attachments:
Description Flags
The virtviewer debug msg
none
Backtrace and logs none

Description Han Han 2016-03-17 09:34:37 UTC
Description of problem:
When migrating a guest from qemu2.5 host to qemu2.3 host with virt-viewer on, virt-viewer gets SIGSEGV

Version-Release number of selected component (if applicable):
Src host:
libvirt-1.3.2-1.el7.x86_64
qemu-kvm-rhev-2.5.0-2.el7.x86_64
virt-viewer-2.0-6.el7.x86_64
Dst host:
libvirt-1.3.2-1.el7.x86_64
qemu-kvm-rhev-2.3.0-31.el7_2.10.x86_64
virt-viewer-2.0-6.el7.x86_64

How reproducible:
90%

Steps to Reproduce:
1. On src&dst machines, set qemu spice listen on:
# vim /etc/libvirt/qemu.conf
spice_listen = "0.0.0.0"
# systemctl restart libvirtd

2. Prepare a healthy guest and start virt-viewer
# virsh list
 Id    Name                           State
----------------------------------------------------
 1     graph-1                        running

# virsh dumpxml graph-1|awk '/<grap/,/<\/grap/' 
    <graphics type='spice' port='5900' autoport='no' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
# virt-viewer graph-1

3. migrate with --live  --p2p --tunnelled options:
# virsh migrate --live  --p2p --tunnelled graph-1 qemu+ssh://lab.work.me/system --verbose  --unsafe
Migration: [  0 %]error: operation failed: Lost connection to destination host

[1]    11797 segmentation fault (core dumped)  virt-viewer graph-1

Actual results:
As step 3.

Expected results:
No SIGSEGV

Additional info:
The bug can also be reporduced without --p2p --tunnelled options in step3.

Comment 1 Han Han 2016-03-17 09:39:32 UTC
The backtrace:
#0  0x00007fc34c814959 in coroutine_yieldto (to=0x0, arg=0x0) at coroutine_ucontext.c:122
        _g_boolean_var_ = <optimized out>
        __func__ = "coroutine_yieldto"
#1  0x00007fc34b1ae60c in g_cclosure_marshal_VOID__ENUMv (closure=<optimized out>, return_value=<optimized out>, instance=<optimized out>, args=<optimized out>, marshal_data=<optimized out>, n_params=<optimized out>, param_types=0x288c6a0) at gmarshal.c:706
        cc = <optimized out>
        data1 = <optimized out>
        data2 = <optimized out>
        callback = <optimized out>
        arg0 = <optimized out>
        args_copy = {{gp_offset = 32, fp_offset = 48, overflow_arg_area = 0x7fc317fffae0, reg_save_area = 0x7fc317fffa00}}
#2  0x00007fc34b1ac067 in _g_closure_invoke_va (closure=closure@entry=0x29ecbc0, return_value=return_value@entry=0x0, instance=instance@entry=0x27afc60, args=args@entry=0x7fc317fff9d8, n_params=1, param_types=0x288c6a0) at gclosure.c:831
        marshal = 0x7fc34b1ae5b0 <g_cclosure_marshal_VOID__ENUMv>
        marshal_data = 0x0
        in_marshal = 0
        real_closure = 0x29ecba0
        __FUNCTION__ = "_g_closure_invoke_va"
#3  0x00007fc34b1c4b27 in g_signal_emit_valist (instance=0x27afc60, signal_id=<optimized out>, detail=0, var_args=var_args@entry=0x7fc317fff9d8) at gsignal.c:3218
        return_accu = 0x0
        accu = 
              {g_type = 0, data = {{v_int = 0, v_uint = 0, v_long = 0, v_ulong = 0, v_int64 = 0, v_uint64 = 0, v_float = 0, v_double = 0, v_pointer = 0x0}, {v_int = 0, v_uint = 0, v_long = 0, v_ulong = 0, v_int64 = 0, v_uint64 = 0, v_float = 0, v_double = 0, v_pointer = 0x0}}}
        accumulator = 0x0
        emission = {next = 0x0, instance = 0x27afc60, ihint = {signal_id = 239, detail = 0, run_type = G_SIGNAL_RUN_FIRST}, state = EMISSION_RUN, chain_type = 40570912}
        signal_id = 239
        instance_type = 40570912
        emission_return = 
              {g_type = 0, data = {{v_int = 0, v_uint = 0, v_long = 0, v_ulong = 0, v_int64 = 0, v_uint64 = 0, v_float = 0, v_double = 0, v_pointer = 0x0}, {v_int = 0, v_uint = 0, v_long = 0, v_ulong = 0, v_int64 = 0, v_uint64 = 0, v_float = 0, v_double = 0, v_pointer = 0x0}}}
        rtype = 4
        static_scope = 0
        fastpath_handler = <optimized out>
        closure = 0x29ecbc0
        run_type = <optimized out>
        l = <optimized out>
        fastpath = <optimized out>
        instance_and_params = <optimized out>
        signal_return_type = <optimized out>
        param_values = <optimized out>
        node = 0x288c6c0
        i = <optimized out>
        n_params = <optimized out>
        __FUNCTION__ = "g_signal_emit_valist"
#4  0x00007fc34c7edae6 in emit_main_context (opaque=0x7fc317fff9b0) at gio-coroutine.c:200
        signal = 0x7fc317fff9b0
#5  0x00007fc34aeae79a in g_main_context_dispatch (context=0x264c290) at gmain.c:3109
        dispatch = 0x7fc34aeab4b0 <g_idle_dispatch>
        prev_source = 0x0
        was_in_call = 0
        user_data = 0x7fc317fff9b0
        callback = 0x7fc34c7edad0 <emit_main_context>
        cb_funcs = 0x7fc34b19a8a0 <g_source_callback_funcs>
        cb_data = 0x29e9500
        need_destroy = <optimized out>
        source = 0x292fe70
        current = 0x290b630
        i = 1
#6  0x00007fc34aeae79a in g_main_context_dispatch (context=context@entry=0x264c290) at gmain.c:3708
#7  0x00007fc34aeaeae8 in g_main_context_iterate (context=0x264c290, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at gmain.c:3779
        max_priority = 200
        timeout = 0
        some_ready = 1
        nfds = <optimized out>
        allocated_nfds = 17
        fds = 0x29ab8f0
#8  0x00007fc34aeaedba in g_main_loop_run (loop=0x290bb00) at gmain.c:3973
        __FUNCTION__ = "g_main_loop_run"
#9  0x00007fc34ccd8045 in gtk_main () at gtkmain.c:1207
        loop = 0x290bb00
#10 0x0000000000410d63 in main (argc=1, argv=0x7ffd822d8868) at virt-viewer-main.c:129
        context = <optimized out>
        error = 0x0
        ret = 1
        uri = 0x0
        args = 0x25f1650
        direct = 0
        attach = 0
        waitvm = 0
        reconnect = 0
        viewer = 0x262b170 [VirtViewer]
        base_name = <optimized out>
        help_msg = 0x2609c70 "Run 'virt-viewer --help' to see a full list of available command line options"
        options = 
            {{long_name = 0x42bc95 "version", short_name = 86 'V', flags = 8, arg = G_OPTION_ARG_CALLBACK, arg_data = 0x429a70 <virt_viewer_version>, description = 0x430117 "Display version information", arg_description = 0x0}, {long_name = 0x430133 "direct", short_name = 100 'd', flags = 0, arg = G_OPTION_ARG_NONE, arg_data = 0x7ffd822d85b8, description = 0x4301b0 "Direct connection with no automatic tunnels", arg_description = 0x0}, {long_name = 0x43013a "attach", short_name = 97 'a', flags = 0, arg = G_OPTION_ARG_NONE, arg_data = 0x7ffd822d85bc, description = 0x4301e0 "Attach to the local display using libvirt", arg_description = 0x0}, {long_name = 0x42fb96 "connect", short_name = 99 'c', flags = 0, arg = G_OPTION_ARG_STRING, arg_data = 0x7ffd822d85d0, description = 0x430141 "Connect to hypervisor", arg_description = 0x42a53f "URI"}, {long_name = 0x430157 "wait", short_name = 119 'w', flags = 0, arg = G_OPTION_ARG_NONE, arg_data = 0x7ffd822d85c0, description = 0x43015c "Wait for domain to start", arg_description = 0x0}, {long_name = 0x430175 "reconnect", short_name = 114 'r', flags = 0, arg = G_OPTION_ARG_NONE, arg_data = 0x7ffd822d85c4, description = 0x430210 "Reconnect to domain upon restart", arg_description = 0x0}, {long_name = 0x42cfcf "", short_name = 0 '\000', flags = 0, arg = G_OPTION_ARG_STRING_ARRAY, arg_data = 0x7ffd822d85d8, description = 0x0, arg_description = 0x43017f "-- DOMAIN-NAME|ID|UUID"}, {long_name = 0x0, short_name = 0 '\000', flags = 0, arg = G_OPTION_ARG_NONE, arg_data = 0x0, description = 0x0, arg_description = 0x0}}
        app_options = <optimized out>

Comment 3 Pavel Grunt 2016-03-18 17:22:29 UTC
Do you know why the migration failed ?
Does it happen only qemu 2.5 -> qemu 2.3 ? 2.5 -> 2.5, 2.3->2.5, 2.3 ->2.3 works?
How was the vm created - default options of virt-manager ?
Do you need to run virt-viewer as root ?

Thanks

Comment 4 Han Han 2016-03-24 08:27:31 UTC
well, in comment1 I start virt-viewer on src host as root user.
Because it's not a common usage, so I test virt-viewer on the third machine as a common user:
First copy then ssh pub key to src and dst hosts.
Then on the third host:
$ virt-viewer -c qemu+ssh://root.6.236/system nfs --debug -v
(virt-viewer:15560): GSpice-CRITICAL **: spice_inputs_key_press: assertion 'SPICE_CHANNEL(channel)->priv->state != SPICE_CHANNEL_STATE_UNCONNECTED' failed
(virt-viewer:15560): GSpice-CRITICAL **: spice_inputs_key_release: assertion 'SPICE_CHANNEL(channel)->priv->state != SPICE_CHANNEL_STATE_UNCONNECTED' failed
I tried qemu qemu 2.5 -> qemu 2.3, 2.5 -> 2.5, 2.3->2.5, 2.3 ->2.3. All of them sometimes cause virt-viewer hang.

For 2.5->2.5:
# virsh migrate --live --p2p --tunnelled nfs qemu+ssh://10.66.7.113/system  --verbose --unsafe
Migration: [100 %]
Qemu hangs at 100%.

2.3<->2.5:
# virsh migrate --live  --p2p --tunnelled nfs qemu+ssh://lab.work.me/system --verbose  --unsafe
Migration: [  0 %]error: operation failed: Lost connection to destination host
Refer to https://bugzilla.redhat.com/show_bug.cgi?id=1320847

2.3->2.3 migration is OK. But virt-viewer hangs.

I will add the debug logs of virt-viewer.

Comment 5 Han Han 2016-03-24 08:31:20 UTC
Created attachment 1139870 [details]
The virtviewer debug msg

Comment 6 Han Han 2016-03-24 10:03:58 UTC
For the bug of qemu migrate hangs at 100%, refer to https://bugzilla.redhat.com/show_bug.cgi?id=1320904

Comment 7 Pavel Grunt 2016-03-24 10:08:48 UTC
(In reply to Han Han from comment #5)
> Created attachment 1139870 [details]
> The virtviewer debug msg

Thanks for the logs, can you please run also with --spice-debug

Also what is the spice-gtk version ?

Comment 8 Han Han 2016-03-25 02:28:04 UTC
(In reply to Pavel Grunt from comment #7)
> (In reply to Han Han from comment #5)
> > Created attachment 1139870 [details]
> > The virtviewer debug msg
> 
> Thanks for the logs, can you please run also with --spice-debug
> 
> Also what is the spice-gtk version ?
My spice version is:
spice-glib-0.26-6.el7.x86_64
package spice-gtk is not installed
spice-gtk3-0.26-6.el7.x86_64
virt-viewer-2.0-7.el7.x86_64

When virt-viewer with --direct, qemu 2.5->2.3 migrate fail and virt-viewer SIGSEGV; Qemu 2.5-> 2.5 all is ok; Qemu 2.3—>2.5 only migrate fail.

Comment 9 Han Han 2016-03-25 02:38:36 UTC
Created attachment 1140220 [details]
Backtrace and logs

The coredump backtrace and dbeug logs with --debug --debug-spice

Comment 10 David Blechter 2016-04-01 13:59:06 UTC
(In reply to Han Han from comment #9)
> Created attachment 1140220 [details]
> Backtrace and logs
> 
> The coredump backtrace and dbeug logs with --debug --debug-spice

You have selected 7.3 in the "Version". It does not make any sense, as we have no builds for 7.3

Comment 11 tingting zheng 2016-04-05 02:51:30 UTC
(In reply to David Blechter from comment #10)
> (In reply to Han Han from comment #9)
> > Created attachment 1140220 [details]
> > Backtrace and logs
> > 
> > The coredump backtrace and dbeug logs with --debug --debug-spice
> 
> You have selected 7.3 in the "Version". It does not make any sense, as we
> have no builds for 7.3

comment 8 use virt-viewer-2.0-7.el7.x86_64 and it is targeted on rhel7.3.

Comment 12 Pavel Grunt 2016-04-14 12:21:20 UTC
the bug is in spice-gtk, I'm changing the component

Comment 13 Pavel Grunt 2016-04-22 15:19:11 UTC
Patch posted upstream: https://lists.freedesktop.org/archives/spice-devel/2016-April/028452.html

Comment 17 errata-xmlrpc 2016-11-04 01:15:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2229.html