Bug 1717329 - Target qemu coredump after failed to mirror from src to nbd target(data plane enable)
Summary: Target qemu coredump after failed to mirror from src to nbd target(data plane...
Keywords:
Status: CLOSED DUPLICATE of bug 1748253
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: qemu-kvm
Version: 8.1
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: rc
: ---
Assignee: Kevin Wolf
QA Contact: aihua liang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-06-05 08:35 UTC by aihua liang
Modified: 2019-09-04 03:38 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-09-04 03:36:14 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description aihua liang 2019-06-05 08:35:49 UTC
Description of problem:
  Target qemu coredump after failed to mirror from src to nbd target

Version-Release number of selected component (if applicable):
 kernel version:4.18.0-94.el8.x86_64
 qemu-kvm version:qemu-kvm-4.0.0-3.module+el8.1.0+3265+26c4ed71.x86_64

How reproducible:
 100%

Steps to Reproduce:
1.Start src guest with qemu cmds:
   /usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1' \
    -machine q35  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x1  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20190602-221944-MrlxVzia,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20190602-221944-MrlxVzia,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idn20piu  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20190602-221944-MrlxVzia,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20190602-221944-MrlxVzia,path=/var/tmp/seabios-20190602-221944-MrlxVzia,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20190602-221944-MrlxVzia,iobase=0x402 \
    -device pcie-root-port,id=pcie.0-root-port-2,slot=2,chassis=2,addr=0x2,bus=pcie.0 \
    -device qemu-xhci,id=usb1,bus=pcie.0-root-port-2,addr=0x0 \
    -device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
    -object iothread,id=iothread0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-3,addr=0x0,iothread=iothread0 \
    -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/rhel77-64-virtio.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -device pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x4,bus=pcie.0 \
    -device virtio-net-pci,mac=9a:33:34:35:36:37,id=idj01pFr,vectors=4,netdev=idMgbx8B,bus=pcie.0-root-port-4,addr=0x0  \
    -netdev tap,id=idMgbx8B,vhost=on \
    -m 7168  \
    -smp 4,maxcpus=4,cores=2,threads=1,sockets=2  \
    -cpu 'Skylake-Client',+kvm_pv_unhalt \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -monitor stdio \
    -qmp tcp:0:3000,server,nowait \

2.In dst, create an image by:
   #qemu-img create -f qcow2 /home/kvm_autotest_root/images/rhel77-64-virtio.qcow2 20G

3.Start dst guest with qemu cmds:
  /usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1' \
    -machine q35  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x1  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20190602-221944-MrlxVzia,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20190602-221944-MrlxVzia,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idn20piu  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20190602-221944-MrlxVzia,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20190602-221944-MrlxVzia,path=/var/tmp/seabios-20190602-221944-MrlxVzia,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20190602-221944-MrlxVzia,iobase=0x402 \
    -device pcie-root-port,id=pcie.0-root-port-2,slot=2,chassis=2,addr=0x2,bus=pcie.0 \
    -device qemu-xhci,id=usb1,bus=pcie.0-root-port-2,addr=0x0 \
    -device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 \
    -object iothread,id=iothread0 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-3,addr=0x0,iothread=iothread0 \
    -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/rhel77-64-virtio.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1 \
    -device pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x4,bus=pcie.0 \
    -device virtio-net-pci,mac=9a:33:34:35:36:37,id=idj01pFr,vectors=4,netdev=idMgbx8B,bus=pcie.0-root-port-4,addr=0x0  \
    -netdev tap,id=idMgbx8B,vhost=on \
    -m 7168  \
    -smp 4,maxcpus=4,cores=2,threads=1,sockets=2  \
    -cpu 'Skylake-Client',+kvm_pv_unhalt \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -monitor stdio \
    -qmp tcp:0:3000,server,nowait \
    -incoming tcp:0:5000 \

4.Expose dst image via nbd.
  { "execute": "nbd-server-start", "arguments": { "addr": { "type": "inet","data": { "host":"10.73.224.36","port":"3333"}}}}
  { "execute": "nbd-server-add", "arguments": { "device": "drive_image1","writable": true } }

5.Start mirror from src to dst
  { "execute": "drive-mirror", "arguments": { "device": "drive_image1","target":"nbd://10.73.224.36:3333/drive_image1", "sync": "full","format": "raw", "mode": "existing" } }
{"timestamp": {"seconds": 1559721809, "microseconds": 276972}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "drive_image1"}}
{"timestamp": {"seconds": 1559721809, "microseconds": 277051}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "drive_image1"}}
{"return": {}}
{"timestamp": {"seconds": 1559721809, "microseconds": 564001}, "event": "BLOCK_JOB_ERROR", "data": {"device": "drive_image1", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1559721809, "microseconds": 564048}, "event": "BLOCK_JOB_ERROR", "data": {"device": "drive_image1", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1559721809, "microseconds": 564071}, "event": "BLOCK_JOB_ERROR", "data": {"device": "drive_image1", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1559721809, "microseconds": 564093}, "event": "BLOCK_JOB_ERROR", "data": {"device": "drive_image1", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1559721809, "microseconds": 564132}, "event": "BLOCK_JOB_ERROR", "data": {"device": "drive_image1", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1559721809, "microseconds": 564183}, "event": "BLOCK_JOB_ERROR", "data": {"device": "drive_image1", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1559721809, "microseconds": 564211}, "event": "BLOCK_JOB_ERROR", "data": {"device": "drive_image1", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1559721809, "microseconds": 564263}, "event": "BLOCK_JOB_ERROR", "data": {"device": "drive_image1", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1559721809, "microseconds": 564329}, "event": "BLOCK_JOB_ERROR", "data": {"device": "drive_image1", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1559721809, "microseconds": 564368}, "event": "BLOCK_JOB_ERROR", "data": {"device": "drive_image1", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1559721809, "microseconds": 564404}, "event": "BLOCK_JOB_ERROR", "data": {"device": "drive_image1", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1559721809, "microseconds": 564425}, "event": "BLOCK_JOB_ERROR", "data": {"device": "drive_image1", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1559721809, "microseconds": 564479}, "event": "BLOCK_JOB_ERROR", "data": {"device": "drive_image1", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1559721809, "microseconds": 564502}, "event": "BLOCK_JOB_ERROR", "data": {"device": "drive_image1", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1559721809, "microseconds": 564525}, "event": "BLOCK_JOB_ERROR", "data": {"device": "drive_image1", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1559721809, "microseconds": 564565}, "event": "BLOCK_JOB_ERROR", "data": {"device": "drive_image1", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1559721809, "microseconds": 564833}, "event": "JOB_STATUS_CHANGE", "data": {"status": "aborting", "id": "drive_image1"}}
{"timestamp": {"seconds": 1559721809, "microseconds": 565166}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "drive_image1", "len": 21474836480, "offset": 0, "speed": 0, "type": "mirror", "error": "Input/output error"}}
{"timestamp": {"seconds": 1559721809, "microseconds": 565232}, "event": "JOB_STATUS_CHANGE", "data": {"status": "concluded", "id": "drive_image1"}}
{"timestamp": {"seconds": 1559721809, "microseconds": 565273}, "event": "JOB_STATUS_CHANGE", "data": {"status": "null", "id": "drive_image1"}}

Actual results:

 After step5, block mirror failed with error "Input/output error".
 And after that, the target qemu coredump with info:
  (qemu) qemu-kvm: io/channel.c:410: qio_channel_restart_read: Assertion `qemu_get_current_aio_context() == qemu_coroutine_get_aio_context(co)' failed.
live_snapshot.txt: line 36:  4800 Aborted                 (core dumped) /usr/libexec/qemu-kvm -name 'avocado-vt-vm1' -machine q35 -nodefaults -device VGA,bus=pcie.0,addr=0x1
 
  coredump info as bellow:
    (gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007fa6de789c95 in __GI_abort () at abort.c:79
#2  0x00007fa6de789b69 in __assert_fail_base (fmt=0x7fa6de8f0940 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=0x564e91249360 "qemu_get_current_aio_context() == qemu_coroutine_get_aio_context(co)", file=0x564e91249326 "io/channel.c", 
    line=410, function=<optimized out>) at assert.c:92
#3  0x00007fa6de797df6 in __GI___assert_fail (
    assertion=assertion@entry=0x564e91249360 "qemu_get_current_aio_context() == qemu_coroutine_get_aio_context(co)", 
    file=file@entry=0x564e91249326 "io/channel.c", line=line@entry=410, 
    function=function@entry=0x564e91249470 <__PRETTY_FUNCTION__.18120> "qio_channel_restart_read") at assert.c:101
#4  0x0000564e91059dd0 in qio_channel_restart_read (opaque=<optimized out>) at io/channel.c:409
#5  qio_channel_restart_read (opaque=<optimized out>) at io/channel.c:403
#6  0x0000564e910ad662 in aio_dispatch_handlers (ctx=ctx@entry=0x564e92e88bc0) at util/aio-posix.c:430
#7  0x0000564e910ae00c in aio_dispatch (ctx=0x564e92e88bc0) at util/aio-posix.c:461
#8  0x0000564e910aaaa2 in aio_ctx_dispatch (source=<optimized out>, callback=<optimized out>, user_data=<optimized out>) at util/async.c:261
#9  0x00007fa6e36125ed in g_main_dispatch (context=0x564e92f01a00) at gmain.c:3176
#10 g_main_context_dispatch (context=context@entry=0x564e92f01a00) at gmain.c:3829
#11 0x0000564e910ad1e8 in glib_pollfds_poll () at util/main-loop.c:213
#12 os_host_main_loop_wait (timeout=<optimized out>) at util/main-loop.c:236
#13 main_loop_wait (nonblocking=<optimized out>) at util/main-loop.c:512
#14 0x0000564e90ea2579 in main_loop () at vl.c:1988
#15 0x0000564e90d57078 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4637


Expected results:
 Mirror to nbd target successfully.

Additional info:
 When disable dataplane, don't hit this issue.

Comment 1 aihua liang 2019-09-04 03:36:14 UTC
In latest version: qemu-kvm-4.1.0-5.module+el8.1.0+4076+b5e41ebc.x86_64, both src and dst qemu coredump.

As the coredump of dst in bz1748253 and bz1717329 is the same, track this issue with bug 1748253, and close this bug as a dup.


Thanks,
aliang

*** This bug has been marked as a duplicate of bug 1748253 ***

Comment 2 aihua liang 2019-09-04 03:38:50 UTC
As in bz1717329, set bug's priority to high.


Note You need to log in before you can comment on or make changes to this bug.