Bug 1713600
| Summary: | remote-viewer core dumped sometimes when doing live migration with guest having spice_tls setting | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | yafu <yafu> | ||||
| Component: | virt-viewer | Assignee: | Victor Toso <victortoso> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 7.7 | CC: | berrange, fjin, juzhou, mzhan, tzheng, victortoso, xiaodwan | ||||
| Target Milestone: | rc | ||||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | virt-viewer-5.0-15.el7 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2020-03-31 20:09:14 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
yafu
2019-05-24 08:31:52 UTC
Created attachment 1573767 [details]
remote-viewer log
I met the remote-viewer coredump issue during live migration(for both plaintext and tls connection) frequently recently, my backtrace is as below:
(gdb) t a a bt
Thread 9 (Thread 0x7f8ff7dec700 (LWP 6021)):
#0 0x00007f903658f1c9 in syscall () from /lib64/libc.so.6
#1 0x00007f9036b110de in g_cond_wait_until () from /lib64/libglib-2.0.so.0
#2 0x00007f9036a9fc21 in g_async_queue_pop_intern_unlocked () from /lib64/libglib-2.0.so.0
#3 0x00007f9036af3e62 in g_thread_pool_thread_proxy () from /lib64/libglib-2.0.so.0
#4 0x00007f9036af34f0 in g_thread_proxy () from /lib64/libglib-2.0.so.0
#5 0x00007f903686bdd5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f9036594ead in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f901cf6f700 (LWP 19723)):
#0 0x00007f903658a20d in poll () from /lib64/libc.so.6
#1 0x00007f90329d17a0 in poll (__timeout=60000, __nfds=2, __fds=0x7f90140008c0) at /usr/include/bits/poll2.h:46
#2 handle_events (ctx=ctx@entry=0x557e73c78460, tv=tv@entry=0x7f901cf6eb30) at io.c:2151
#3 0x00007f90329d2689 in handle_events (tv=0x7f901cf6eb30, ctx=0x557e73c78460) at io.c:2084
#4 libusb_handle_events_timeout_completed (ctx=0x557e73c78460, tv=tv@entry=0x7f901cf6eb80,
completed=completed@entry=0x0) at io.c:2345
#5 0x00007f90329d276f in libusb_handle_events (ctx=<optimized out>) at io.c:2421
#6 0x00007f9037e5e151 in spice_usb_device_manager_usb_ev_thread (user_data=<optimized out>)
at usb-device-manager.c:1175
#7 0x00007f9036af34f0 in g_thread_proxy () from /lib64/libglib-2.0.so.0
#8 0x00007f903686bdd5 in start_thread () from /lib64/libpthread.so.0
#9 0x00007f9036594ead in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f901f431700 (LWP 19709)):
#0 0x00007f903658a20d in poll () from /lib64/libc.so.6
#1 0x00007f9036acc32c in g_main_context_iterate.isra.19 () from /lib64/libglib-2.0.so.0
#2 0x00007f9036acc67a in g_main_loop_run () from /lib64/libglib-2.0.so.0
#3 0x00007f90370b59a6 in gdbus_shared_thread_func () from /lib64/libgio-2.0.so.0
---Type <return> to continue, or q <return> to quit---
#4 0x00007f9036af34f0 in g_thread_proxy () from /lib64/libglib-2.0.so.0
#5 0x00007f903686bdd5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f9036594ead in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f901fc32700 (LWP 19708)):
#0 0x00007f903658a20d in poll () from /lib64/libc.so.6
#1 0x00007f9036acc32c in g_main_context_iterate.isra.19 () from /lib64/libglib-2.0.so.0
#2 0x00007f9036acc45c in g_main_context_iteration () from /lib64/libglib-2.0.so.0
#3 0x00007f9036acc499 in glib_worker_main () from /lib64/libglib-2.0.so.0
#4 0x00007f9036af34f0 in g_thread_proxy () from /lib64/libglib-2.0.so.0
#5 0x00007f903686bdd5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f9036594ead in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f901d770700 (LWP 19722)):
#0 0x00007f903658a20d in poll () from /lib64/libc.so.6
#1 0x00007f90329d7a7c in poll (__timeout=-1, __nfds=2, __fds=0x7f901d76fba0) at /usr/include/bits/poll2.h:46
#2 linux_udev_event_thread_main (arg=<optimized out>) at os/linux_udev.c:175
#3 0x00007f903686bdd5 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f9036594ead in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f8fe77fc700 (LWP 6022)):
#0 0x00007f903658f1c9 in syscall () from /lib64/libc.so.6
#1 0x00007f9036b110de in g_cond_wait_until () from /lib64/libglib-2.0.so.0
#2 0x00007f9036a9fc21 in g_async_queue_pop_intern_unlocked () from /lib64/libglib-2.0.so.0
#3 0x00007f9036af3e62 in g_thread_pool_thread_proxy () from /lib64/libglib-2.0.so.0
#4 0x00007f9036af34f0 in g_thread_proxy () from /lib64/libglib-2.0.so.0
#5 0x00007f903686bdd5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f9036594ead in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
Thread 3 (Thread 0x7f8fe67fa700 (LWP 6023)):
#0 0x00007f903658f1c9 in syscall () from /lib64/libc.so.6
#1 0x00007f9036b110de in g_cond_wait_until () from /lib64/libglib-2.0.so.0
#2 0x00007f9036a9fc21 in g_async_queue_pop_intern_unlocked () from /lib64/libglib-2.0.so.0
#3 0x00007f9036af3e62 in g_thread_pool_thread_proxy () from /lib64/libglib-2.0.so.0
#4 0x00007f9036af34f0 in g_thread_proxy () from /lib64/libglib-2.0.so.0
#5 0x00007f903686bdd5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f9036594ead in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f8fe7ffd700 (LWP 6010)):
#0 0x00007f903658f1c9 in syscall () from /lib64/libc.so.6
#1 0x00007f9036b110de in g_cond_wait_until () from /lib64/libglib-2.0.so.0
#2 0x00007f9036a9fc21 in g_async_queue_pop_intern_unlocked () from /lib64/libglib-2.0.so.0
#3 0x00007f9036af3e62 in g_thread_pool_thread_proxy () from /lib64/libglib-2.0.so.0
#4 0x00007f9036af34f0 in g_thread_proxy () from /lib64/libglib-2.0.so.0
#5 0x00007f903686bdd5 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f9036594ead in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f903aa4da80 (LWP 19701)):
#0 0x00007f90364cd207 in raise () from /lib64/libc.so.6
#1 0x00007f90364ce8f8 in abort () from /lib64/libc.so.6
#2 0x00007f9036af2665 in g_assertion_message () from /lib64/libglib-2.0.so.0
#3 0x00007f9036af26ca in g_assertion_message_expr () from /lib64/libglib-2.0.so.0
#4 0x00007f9033cc4e61 in glib_defer_enable (e=<optimized out>, b=<optimized out>) at pulse/glib-mainloop.c:394
#5 0x00007f902bdfb496 in pa_pstream_send_tagstruct_with_ancil_data (p=0x557e73ee3400, t=0x557e7404f340,
ancil_data=ancil_data@entry=0x0) at pulsecore/pstream-util.c:45
#6 0x00007f902bdfb617 in pa_pstream_send_tagstruct_with_creds (p=<optimized out>, t=t@entry=0x557e7404f340,
creds=creds@entry=0x0) at pulsecore/pstream-util.c:61
#7 0x00007f9033a94b4a in pa_context_set_sink_input_volume (c=0x557e73ee1540, idx=38,
---Type <return> to continue, or q <return> to quit---
volume=volume@entry=0x7ffe2b20f7f0, cb=cb@entry=0x0, userdata=userdata@entry=0x0) at pulse/introspect.c:1514
#8 0x00007f9037e6d5ad in playback_volume_changed (object=<optimized out>, pspec=<optimized out>,
data=<optimized out>) at spice-pulse.c:624
#9 0x00007f9036da5988 in g_closure_invoke () from /lib64/libgobject-2.0.so.0
#10 0x00007f9036db805d in signal_emit_unlocked_R () from /lib64/libgobject-2.0.so.0
#11 0x00007f9036dbfff1 in g_signal_emit_valist () from /lib64/libgobject-2.0.so.0
#12 0x00007f9036dc02df in g_signal_emit () from /lib64/libgobject-2.0.so.0
#13 0x00007f9036daa254 in g_object_dispatch_properties_changed () from /lib64/libgobject-2.0.so.0
#14 0x00007f9036dac669 in g_object_notify () from /lib64/libgobject-2.0.so.0
#15 0x00007f9037e47ff0 in notify_main_context (opaque=0x7f9008fff9d0) at gio-coroutine.c:238
#16 0x00007f9036ac8c77 in g_idle_dispatch () from /lib64/libglib-2.0.so.0
#17 0x00007f9036acc049 in g_main_context_dispatch () from /lib64/libglib-2.0.so.0
#18 0x00007f9036acc3a8 in g_main_context_iterate.isra.19 () from /lib64/libglib-2.0.so.0
#19 0x00007f9036acc45c in g_main_context_iteration () from /lib64/libglib-2.0.so.0
#20 0x00007f9037089355 in g_application_run () from /lib64/libgio-2.0.so.0
#21 0x0000557e7343251c in main (argc=5, argv=0x7ffe2b20ffa8) at remote-viewer-main.c:42
Sometimes the backtrace is as below:
(gdb) bt
#0 0x00007f926af95207 in raise () from /lib64/libc.so.6
#1 0x00007f926af968f8 in abort () from /lib64/libc.so.6
#2 0x00007f926afd7d27 in __libc_message () from /lib64/libc.so.6
#3 0x00007f926afe0489 in _int_free () from /lib64/libc.so.6
#4 0x00007f926b59979e in g_free () from /lib64/libglib-2.0.so.0
#5 0x000055b3906fa884 in remote_viewer_session_connected (session=<optimized out>,
guri=0x55b391b6cbc0 "\360]\302\221\263U") at remote-viewer.c:670
#6 0x00007f926b86d988 in g_closure_invoke () from /lib64/libgobject-2.0.so.0
#7 0x00007f926b88005d in signal_emit_unlocked_R () from /lib64/libgobject-2.0.so.0
#8 0x00007f926b887ff1 in g_signal_emit_valist () from /lib64/libgobject-2.0.so.0
#9 0x00007f926b888828 in g_signal_emit_by_name () from /lib64/libgobject-2.0.so.0
#10 0x000055b3907107db in virt_viewer_session_spice_main_channel_event (channel=0x55b391c86bd0,
event=SPICE_CHANNEL_OPENED, session=0x55b391c71890) at virt-viewer-session-spice.c:684
---Type <return> to continue, or q <return> to quit---
#11 0x00007f926b86d988 in g_closure_invoke () from /lib64/libgobject-2.0.so.0
#12 0x00007f926b88005d in signal_emit_unlocked_R () from /lib64/libgobject-2.0.so.0
#13 0x00007f926b887ff1 in g_signal_emit_valist () from /lib64/libgobject-2.0.so.0
#14 0x00007f926c90ffc6 in emit_main_context (opaque=0x7f9243fff990) at gio-coroutine.c:198
#15 0x00007f926b590c77 in g_idle_dispatch () from /lib64/libglib-2.0.so.0
#16 0x00007f926b594049 in g_main_context_dispatch () from /lib64/libglib-2.0.so.0
#17 0x00007f926b5943a8 in g_main_context_iterate.isra.19 () from /lib64/libglib-2.0.so.0
#18 0x00007f926b59445c in g_main_context_iteration () from /lib64/libglib-2.0.so.0
#19 0x00007f926bb51355 in g_application_run () from /lib64/libgio-2.0.so.0
#20 0x000055b3906fa51c in main (argc=5, argv=0x7ffe8b0df278) at remote-viewer-main.c:42
Backtrace from comment #4 and posted patch [0] are similar, should be fixed. [0] https://www.redhat.com/archives/virt-tools-list/2019-September/msg00016.html Commit that fixes backtrace from comment #4 is: https://pagure.io/virt-viewer/c/a13173ae649412d06106a0d9c6d29e6a45d5bf57?branch=master and a scratch-build for it is available at: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=23857190 I haven't yet reproduced backtrace from comment #3. Build is on the making, moving to MODIFIED to add to errata. (In reply to Victor Toso from comment #9) > Build is on the making, moving to MODIFIED to add to errata. Hi Victor Toso, Please help add this bug to errata: https://errata.devel.redhat.com/advisory/47221 Thanks very much. BR, juzhou. Reproduce with package: virt-viewer-5.0-15.el7.x86_64
Reproduce rate: only once, less 5%
Connection type: plain-text connection
The backtrack is as follows:
(gdb) bt
#0 0x00007efe8e23a377 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1 0x00007efe8e23ba68 in __GI_abort () at abort.c:90
#2 0x00007efe8e27cec7 in __libc_message (do_abort=do_abort@entry=2,
fmt=fmt@entry=0x7efe8e38f3f8 "*** Error in `%s': %s: 0x%s ***\n")
at ../sysdeps/unix/sysv/linux/libc_fatal.c:196
#3 0x00007efe8e283804 in malloc_printerr (action=<optimized out>,
str=0x7efe8e38f5c0 "malloc(): smallbin double linked list corrupted", ptr=<optimized out>,
ar_ptr=<optimized out>) at malloc.c:4967
#4 0x00007efe8e286f40 in _int_malloc (av=av@entry=0x7efe8e5cb760 <main_arena>, bytes=bytes@entry=32)
at malloc.c:3383
#5 0x00007efe8e28a5a4 in __libc_calloc (n=<optimized out>, elem_size=<optimized out>) at malloc.c:3241
#6 0x00007efe8e83f6e6 in g_malloc0 (n_bytes=32) at gmem.c:129
#7 0x00007efe8e83f901 in g_malloc0_n (n_blocks=<optimized out>, n_block_bytes=n_block_bytes@entry=4)
at gmem.c:360
#8 0x00007efe8e828c35 in g_hash_table_new_full (hash_func=0x7efe8e828230 <g_direct_hash>,
key_equal_func=0x7efe8e829ac0 <g_direct_equal>, key_destroy_func=key_destroy_func@entry=0x0,
value_destroy_func=value_destroy_func@entry=0x7efe8fbb90f0 <destroy_surface>) at ghash.c:731
#9 0x00007efe8fbb9d13 in spice_display_channel_init (channel=0x560fa796a4b0) at channel-display.c:883
#10 0x00007efe8eb354eb in g_type_create_instance (type=94625233904736) at gtype.c:1866
#11 0x00007efe8eb191fd in g_object_new_internal (class=class@entry=0x560fa7967050,
params=params@entry=0x7ffe5055f350, n_params=3) at gobject.c:1799
#12 0x00007efe8eb1b121 in g_object_new_valist (object_type=<optimized out>,
first_property_name=first_property_name@entry=0x7efe8fc6590f "spice-session",
var_args=var_args@entry=0x7ffe5055f4a0) at gobject.c:2122
#13 0x00007efe8eb1b469 in g_object_new (object_type=object_type@entry=94625233904736,
first_property_name=first_property_name@entry=0x7efe8fc6590f "spice-session") at gobject.c:1642
#14 0x00007efe8fbb12ca in spice_channel_new (s=0x560fa750d6a0, type=2, id=0) at spice-channel.c:2266
#15 0x00007efe8fbc04e7 in _channel_new (c=c@entry=0x560fa7591450) at channel-main.c:1732
#16 0x00007efe8e836c77 in g_idle_dispatch (source=0x560fa7579cd0,
callback=0x7efe8fbc04d0 <_channel_new>, user_data=0x560fa7591450) at gmain.c:5533
#17 0x00007efe8e83a049 in g_main_dispatch (context=0x560fa74db0a0) at gmain.c:3175
#18 g_main_context_dispatch (context=context@entry=0x560fa74db0a0) at gmain.c:3828
#19 0x00007efe8e83a3a8 in g_main_context_iterate (context=context@entry=0x560fa74db0a0,
block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at gmain.c:3901
#20 0x00007efe8e83a45c in g_main_context_iteration (context=context@entry=0x560fa74db0a0,
may_block=may_block@entry=1) at gmain.c:3962
#21 0x00007efe8edf7355 in g_application_run (application=application@entry=0x560fa74d8300,
argc=argc@entry=5, argv=argv@entry=0x7ffe5055f858) at gapplication.c:2470
#22 0x0000560fa60be7fc in main (argc=5, argv=0x7ffe5055f858) at remote-viewer-main.c:42
Verify this bug with packages:
virt-viewer-5.0-17.el7.x86_64
spice-gtk3-0.35-5.el7.x86_64
spice-server-0.14.0-8.el7.x86_64
gtk3-3.22.30-5.el7.x86_64
glib2-2.56.1-5.el7.x86_64
libvirt-4.5.0-28.el7.x86_64
qemu-kvm-rhev-2.12.0-38.el7.x86_64
Steps:
Testing scenario-1: plain-text connection
1.1 Prepare migration env
1.2 Start a vm with spice graphic on src host
...
<graphics type='spice' port='5900' autoport='yes' listen='0.0.0.0'>
<listen type='address' address='0.0.0.0'/>
<image compression='off'/>
</graphics>
1.3 Connect to vm by remote-viewer:
$ remote-viewer spice://10.73.xx.xx:5900
1.4 Migrate vm to dest host:
# virsh migrate $vm qemu+ssh://$DST/system --live --verbose --p2p
1.5 Migrate vm back to src host, then repeat for several times, remote-viewer works well, no crash occurs.
Testing scenario-1: TLS connection
2.1 Prepare migration env and TLS env
2.2 Start a vm which has spice_tls setting on src host
...
<graphics type='spice' port='5900' tlsPort='5901' autoport='yes' listen='0.0.0.0'>
<listen type='address' address='0.0.0.0'/>
<image compression='off'/>
</graphics>
2.3 Connect to vm by remote-viewer:
$ remote-viewer spice://10.73.xx.xx/?port=5900\&tls-port=5901 --spice-ca-file=/etc/pki/libvirt-spice/ca-cert.pem --spice-host-subject=" C=IL, L=Raanana, O=Red Hat, CN=my server "
2.4 Migrate vm to dest host:
# virsh migrate $vm qemu+ssh://$DST/system --live --verbose --p2p
2.5 Migrate vm back to src host, then repeat for several times, remote-viewer works well, no crash occurs.
======================================================================================
Make a summary, remote-viewer works well during migration.
@yafu, while you do related migration testing, please also help check it again.
And if it also works well for you, then I will move this bug from ON_QA to VERIFIED, thanks.
(In reply to zhoujunqin from comment #13) > Verify this bug with packages: > virt-viewer-5.0-17.el7.x86_64 > spice-gtk3-0.35-5.el7.x86_64 > spice-server-0.14.0-8.el7.x86_64 > gtk3-3.22.30-5.el7.x86_64 > glib2-2.56.1-5.el7.x86_64 > libvirt-4.5.0-28.el7.x86_64 > qemu-kvm-rhev-2.12.0-38.el7.x86_64 > > > Steps: > Testing scenario-1: plain-text connection > 1.1 Prepare migration env > > 1.2 Start a vm with spice graphic on src host > ... > <graphics type='spice' port='5900' autoport='yes' listen='0.0.0.0'> > <listen type='address' address='0.0.0.0'/> > <image compression='off'/> > </graphics> > > 1.3 Connect to vm by remote-viewer: > $ remote-viewer spice://10.73.xx.xx:5900 > > 1.4 Migrate vm to dest host: > # virsh migrate $vm qemu+ssh://$DST/system --live --verbose --p2p > > 1.5 Migrate vm back to src host, then repeat for several times, > remote-viewer works well, no crash occurs. > > > Testing scenario-1: TLS connection > > > 2.1 Prepare migration env and TLS env > > 2.2 Start a vm which has spice_tls setting on src host > ... > <graphics type='spice' port='5900' tlsPort='5901' autoport='yes' > listen='0.0.0.0'> > <listen type='address' address='0.0.0.0'/> > <image compression='off'/> > </graphics> > > 2.3 Connect to vm by remote-viewer: > $ remote-viewer spice://10.73.xx.xx/?port=5900\&tls-port=5901 > --spice-ca-file=/etc/pki/libvirt-spice/ca-cert.pem --spice-host-subject=" > C=IL, L=Raanana, O=Red Hat, CN=my server " > > 2.4 Migrate vm to dest host: > # virsh migrate $vm qemu+ssh://$DST/system --live --verbose --p2p > > 2.5 Migrate vm back to src host, then repeat for several times, > remote-viewer works well, no crash occurs. > > ============================================================================= > ========= > Make a summary, remote-viewer works well during migration. > > @yafu, while you do related migration testing, please also help check it > again. > And if it also works well for you, then I will move this bug from ON_QA to > VERIFIED, thanks. Also works for me. Please move the bug to VERIFIED. I move this bug from ON_QA to VERIFIED based on Comment 13 and Comment 14 testing. Thanks for yafu's help. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:1170 |