Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 679006

Summary: QEMU/spice server crashes when saving to a file due to no client migrate info
Product: Red Hat Enterprise Linux 6 Reporter: Johnny Liu <jialiu>
Component: spice-serverAssignee: Uri Lublin <uril>
Status: CLOSED DUPLICATE QA Contact: Desktop QE <desktop-qa-list>
Severity: high Docs Contact:
Priority: high    
Version: 6.1CC: berrange, dblechte, dyuan, eblake, jyang, llim, mkenneth, xen-maint
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-02-23 23:23:08 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
libvirtd log none

Description Johnny Liu 2011-02-21 09:26:45 UTC
Description of problem:
Create a domain with spice graphics:
<graphics type='spice' autoport='yes' listen='0'/>

When save this domain, it failed.
# virsh save rhel55 /tmp/rhel55.save
error: Failed to save domain rhel55 to /tmp/rhel55.save
error: cannot send monitor command '{"execute":"query-migrate"}': Connection reset by peer


Version-Release number of selected component (if applicable):
libvirt-0.8.7-7.el6.x86_64
kernel-2.6.32-113.el6.x86_64
qemu-kvm-0.12.1.2-2.147.el6.x86_64
spice-server-0.7.2-4.el6.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Create a domain with spice graphics.
2. Save the domain
3.
  
Actual results:
Save operation failed.

log:
2011-02-21 12:17:35.271: starting up
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -S -M rhel6.0.0 -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -name rhel55 -uuid 2814d348-ee12-9dd3-2373-99bb9301e8a7 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/rhel55.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -boot c -drive file=/var/lib/libvirt/images/rhel55.img,if=none,id=drive-ide0-0-0,format=raw,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=22,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:8e:71:23,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -spice port=5900,addr=0,disable-ticketing -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
char device redirected to /dev/pts/1
do_spice_init: starting 0.7.2
spice_server_add_interface: SPICE_INTERFACE_KEYBOARD
spice_server_add_interface: SPICE_INTERFACE_MOUSE
Using CPU model "cpu64-rhel6"
spice_server_add_interface: SPICE_INTERFACE_QXL
red_worker_main: begin
handle_dev_input: start
handle_dev_input: stop
0+0 records in
0+0 records out
0 bytes (0 B) copied, 6.375e-06 s, 0.0 kB/s
0+684 records in
0+684 records out
39426630 bytes (39 MB) copied, 1.13367 s, 34.8 MB/s
reds_mig_switch:
2011-02-21 12:17:52.312: shutting down


Expected results:
Save should be work fine.

Additional info:
When modify spice graphics to vnc, save operation works fine.

Comment 2 Daniel Berrangé 2011-02-21 12:10:14 UTC
"Connection reset by peer" is an indication the QEMU exited unexpectedly, most likely a crash. We need to capture a little more logging to libvirt so we can see what we're telling qemu todo

Can you enable logging in libvirtd.conf

  log_filters="1:qemu 1:util 1:security 1:libvirt"
  log_outputs="1:file:/var/log/libvirt/libvirtd.log"

And restart libvirtd, and then reproduce the crash & attach the libvirtd.log file

Comment 3 Johnny Liu 2011-02-22 02:21:49 UTC
Yes, indeed. I can see qemu-kvm segfault in /var/log/messages.
...
Feb 22 05:18:07 dhcp-93-91 kernel: qemu-kvm[14747]: segfault at b8 ip 000000361b2165d6 sp 00007fff42af2f90 error 4 in libspice-server.so.1.0.2[361b200000+dd000]
...

libvirtd log have been attached.

Comment 4 Johnny Liu 2011-02-22 02:23:01 UTC
Created attachment 480036 [details]
libvirtd log

Comment 5 Daniel Berrangé 2011-02-22 10:45:35 UTC
I can't reproduce the crash myself. Can you try capture a stack trace of QEMU crashing 

 1. virsh start $GUEST
 2. gdb /usr/libexec/qemu-kvm $GUEST_PID
 3. virsh save $GUEST /tmp/$GUEST.save

And then 'thread apply all bt' when QEMU crashes.

Comment 6 Johnny Liu 2011-02-23 06:37:03 UTC
(In reply to comment #5)
> I can't reproduce the crash myself. Can you try capture a stack trace of QEMU
> crashing 
> 
>  1. virsh start $GUEST
>  2. gdb /usr/libexec/qemu-kvm $GUEST_PID
>  3. virsh save $GUEST /tmp/$GUEST.save
> 
> And then 'thread apply all bt' when QEMU crashes.

I follow your steps, the gdb info is as following:
(gdb) c
Continuing.
[Thread 0x7fc38a3fc710 (LWP 14409) exited]
[New Thread 0x7fc38a3fc710 (LWP 14417)]
[Thread 0x7fc38a3fc710 (LWP 14417) exited]
[New Thread 0x7fc38a3fc710 (LWP 14424)]
[Thread 0x7fc38a3fc710 (LWP 14424) exited]
Detaching after fork from child process 14426.

Program received signal SIGSEGV, Segmentation fault.
reds_mig_switch (s=<value optimized out>) at reds.c:3379
3379	    migrate.port = s->port;
(gdb) thread apply all bt

Thread 4 (Thread 0x7fc3d27c3710 (LWP 14364)):
#0  0x0000003245a33a9d in sigtimedwait () from /lib64/libc.so.6
#1  0x000000000042db3f in kvm_main_loop_wait (env=0x1229a00, timeout=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1791
#2  0x000000000042e0e5 in kvm_main_loop_cpu (_env=0x1229a00) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1935
#3  ap_main_loop (_env=0x1229a00) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1982
#4  0x00000032462077e1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003245ae153d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fc38adfd710 (LWP 14365)):
#0  0x0000003245ae1b33 in epoll_wait () from /lib64/libc.so.6
#1  0x000000361b22c0c2 in red_worker_main (arg=<value optimized out>) at red_worker.c:10271
#2  0x00000032462077e1 in start_thread () from /lib64/libpthread.so.0
#3  0x0000003245ae153d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fc3d29f3940 (LWP 14341)):
#0  reds_mig_switch (s=<value optimized out>) at reds.c:3379
#1  spice_server_migrate_switch (s=<value optimized out>) at reds.c:4184
#2  0x00000000004df239 in notifier_list_notify (list=<value optimized out>) at notify.c:37
#3  0x000000000040baa0 in qemu_run_timers (timeout=1000) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:1217
#4  main_loop_wait (timeout=1000) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4458
#5  0x000000000042b2fa in kvm_main_loop () at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2165
#6  0x000000000040ef0f in main_loop (argc=<value optimized out>, argv=<value optimized out>, envp=<value optimized out>)
    at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4634
#7  main (argc=<value optimized out>, argv=<value optimized out>, envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6848
(gdb) 



I am not sure if it is what you wanted, if not, please tell me.

Comment 7 Daniel Berrangé 2011-02-23 10:33:00 UTC
This is perfect, it shows a bug in spice-server.

The 'reds_mig_switch' function is called unconditionally, whether libvirt has provided any client migrate info or not.  In save-to-disk, we obviously don't provide any migrate info. So 'reds->mig_spice' is NULL, but this is never checked, so 's->port' gets a NULL pointer crash


static void reds_mig_switch(void)
{
    RedsMigSpice *s = reds->mig_spice;
    SpiceMsgMainMigrationSwitchHost migrate;
    RedsOutItem *item;

    red_printf("");
    item = new_out_item(SPICE_MSG_MAIN_MIGRATE_SWITCH_HOST);

    migrate.port = s->port;

Comment 8 Uri Lublin 2011-02-23 23:23:08 UTC

*** This bug has been marked as a duplicate of bug 674451 ***