983415 – migration with non-shared storage with full disk copy will cause domain crash

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 983415 - migration with non-shared storage with full disk copy will cause domain crash

Summary: migration with non-shared storage with full disk copy will cause domain crash

Keywords:
Status:	CLOSED DUPLICATE of bug 916067
Alias:	None
Product:	Red Hat Enterprise Linux 6
Classification:	Red Hat
Component:	qemu-kvm
Sub Component:
Version:	6.4
Hardware:	x86_64
OS:	All
Priority:	high
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	Virtualization Maintenance
QA Contact:	Virtualization Bugs
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2013-07-11 07:37 UTC by yanbing du
Modified:	2013-07-22 21:34 UTC (History)
CC List:	12 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed:	2013-07-22 21:34:55 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
libvirtd-debug.log (187.88 KB, application/x-rar) 2013-07-11 07:37 UTC, yanbing du	no flags	Details
View All

Description yanbing du 2013-07-11 07:37:10 UTC

Created attachment 772058 [details]
libvirtd-debug.log

when do migration with --copy-storage-all via virsh, which means migration with non-shared storage with full disk copy, will cause domain crash.


Version-Release number of selected component (if applicable):
# uname -a
Linux hp-dl585g7-01.qe.lab.eng.nay.redhat.com 2.6.32-395.el6.x86_64 #1 SMP Tue Jul 2 14:34:40 EDT 2013 x86_64 x86_64 x86_64 GNU/Linux

# rpm -q libvirt
libvirt-0.10.2-19.el6.x86_64

# rpm -qa|grep qemu-kvm
qemu-kvm-rhev-tools-0.12.1.2-2.377.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.377.el6.x86_64
qemu-kvm-rhev-debuginfo-0.12.1.2-2.377.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1.
# virsh migrate --live test qemu+ssh://root@$dest_host/system --copy-storage-all --verbose
error: Unable to read from monitor: Connection reset by peer


gdb debug info:

#gdb attach $qemu-kvm-pid
...

(gdb) c
Continuing.
[New Thread 0x7fc24ebfd700 (LWP 33189)]

Program received signal SIGSEGV, Segmentation fault.
monitor_flush (mon=0x0) at /usr/src/debug/qemu-kvm-0.12.1.2/monitor.c:279
279	    buf = qstring_get_str(mon->outbuf);
(gdb) t a a bt

Thread 3 (Thread 0x7fc24ebfd700 (LWP 33189)):
#0  0x00007fc29cd0601e in __lll_lock_wait_private () from /lib64/libpthread.so.0
#1  0x00007fc29ccffecb in _L_lock_4712 () from /lib64/libpthread.so.0
#2  0x00007fc29ccffb34 in start_thread () from /lib64/libpthread.so.0
#3  0x00007fc29ad8890d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fc295357700 (LWP 33173)):
#0  0x00007fc29cd06054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007fc29cd01388 in _L_lock_854 () from /lib64/libpthread.so.0
#2  0x00007fc29cd01257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007fc29d3d3180 in post_kvm_run (kvm=<value optimized out>, env=0x7fc29fb09a60) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:922
#4  0x00007fc29d3d4728 in kvm_run (env=0x7fc29fb09a60) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1024
#5  0x00007fc29d3d4ba9 in kvm_cpu_exec (env=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:1743
#6  0x00007fc29d3d5a8d in kvm_main_loop_cpu (_env=0x7fc29fb09a60) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2004
#7  ap_main_loop (_env=0x7fc29fb09a60) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2060
#8  0x00007fc29ccff851 in start_thread () from /lib64/libpthread.so.0
#9  0x00007fc29ad8890d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fc29d31f980 (LWP 33171)):
#0  monitor_flush (mon=0x0) at /usr/src/debug/qemu-kvm-0.12.1.2/monitor.c:279
#1  0x00007fc29d43eb1a in blk_mig_save_bulked_block (mon=0x0, f=0x7fc2a00a94e0, is_async=1) at /usr/src/debug/qemu-kvm-0.12.1.2/block-migration.c:296
#2  0x00007fc29d43ed8f in block_save_live (mon=0x0, f=0x7fc2a00a94e0, stage=1, opaque=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/block-migration.c:446
#3  0x00007fc29d43935b in qemu_savevm_state_begin (mon=0x0, f=0x7fc2a00a94e0, blk_enable=<value optimized out>, shared=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/savevm.c:1524
#4  0x00007fc29d430d7f in migrate_fd_connect (s=0x7fc29fc27b00) at /usr/src/debug/qemu-kvm-0.12.1.2/migration.c:380
#5  0x00007fc29d440920 in fd_start_outgoing_migration (mon=0x7fc29fb872d0, fdname=0x7fc29fc27a43 "migrate", bandwidth_limit=4294967295, detach=1, blk=1, inc=0)
    at /usr/src/debug/qemu-kvm-0.12.1.2/migration-fd.c:115
#6  0x00007fc29d4313b5 in do_migrate (mon=0x7fc29fb872d0, qdict=<value optimized out>, ret_data=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/migration.c:124
#7  0x00007fc29d3b7e50 in monitor_call_handler (mon=0x7fc29fb872d0, cmd=0x7fc29d899de0, params=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/monitor.c:4359
#8  0x00007fc29d3b8ad4 in handle_qmp_command (parser=<value optimized out>, tokens=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/monitor.c:4982
#9  0x00007fc29d411624 in json_message_process_token (lexer=0x7fc29fb87380, token=0x7fc29fc271d0, type=JSON_OPERATOR, x=109, y=10) at /usr/src/debug/qemu-kvm-0.12.1.2/json-streamer.c:87
#10 0x00007fc29d4112c0 in json_lexer_feed_char (lexer=0x7fc29fb87380, ch=125 '}', flush=false) at /usr/src/debug/qemu-kvm-0.12.1.2/json-lexer.c:303
#11 0x00007fc29d411409 in json_lexer_feed (lexer=0x7fc29fb87380, buffer=0x7fff0c247ba0 "}\317\t", <incomplete sequence \375>, size=1) at /usr/src/debug/qemu-kvm-0.12.1.2/json-lexer.c:355
#12 0x00007fc29d3b777b in monitor_control_read (opaque=<value optimized out>, buf=<value optimized out>, size=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/monitor.c:5003
#13 0x00007fc29d434d8a in qemu_chr_be_write (chan=<value optimized out>, cond=<value optimized out>, opaque=0x7fc29faefbd0) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-char.c:191
#14 tcp_chr_read (chan=<value optimized out>, cond=<value optimized out>, opaque=0x7fc29faefbd0) at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-char.c:2349
#15 0x00007fc29ca48f0e in g_main_context_dispatch () from /lib64/libglib-2.0.so.0
#16 0x00007fc29d3b03ca in glib_select_poll (timeout=1000) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:3993
#17 main_loop_wait (timeout=1000) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4066
#18 0x00007fc29d3d2c6a in kvm_main_loop () at /usr/src/debug/qemu-kvm-0.12.1.2/qemu-kvm.c:2244
#19 0x00007fc29d3b3d48 in main_loop (argc=46, argv=<value optimized out>, envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:4260
#20 main (argc=46, argv=<value optimized out>, envp=<value optimized out>) at /usr/src/debug/qemu-kvm-0.12.1.2/vl.c:6627
(gdb) c
Continuing.
[Thread 0x7fc24ebfd700 (LWP 33189) exited]
[Thread 0x7fc295357700 (LWP 33173) exited]

Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.

--- Additional comment from Jiri Denemark on 2013-07-10 06:47:05 EDT ---

That's an unrelated qemu-kvm bug.

--- Additional comment from yanbing du on 2013-07-11 03:23:43 EDT ---

(In reply to Jiri Denemark from comment #18)
> That's an unrelated qemu-kvm bug.

OKay. So for this bug, migration for the local storage will not get the error: Unsafe migration: Migration may lead to data corruption if disks use cache != none

I will file a bug against qemu-kvm for the domain crash problem and move this bug to VERIFIED.

Comment 2 Qunfang Zhang 2013-07-11 09:21:35 UTC

We don't support migration with block migration and there is a similar bug as below.  CC'ing Paolo to have a look. 

Bug 916067  - when cancel the migration with ctrl+c during block migration(full disk copy or incremental disk copy), then migration again will cause domain destroyed

Comment 4 Ademar Reis 2013-07-22 21:34:55 UTC

As Paolo mentioned, we don't support block migration in RHEL6.5, but I'm closing it as duplicate of Bug 916067, which is open and being investigated by Paolo anyway.

*** This bug has been marked as a duplicate of bug 916067 ***

Note You need to log in before you can comment on or make changes to this bug.