Bug 1550335

Summary: After migration completed, reboot vm on destination host and quit qemu, sometimes qemu quit with error message "Assertion `blk_get_aio_context(d->conf.blk) == s->ctx' failed"
Product: Red Hat Enterprise Linux 7 Reporter: xianwang <xianwang>
Component: qemu-kvm-rhevAssignee: Stefan Hajnoczi <stefanha>
Status: CLOSED DUPLICATE QA Contact: aihua liang <aliang>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.5CC: aliang, chayang, coli, dgilbert, hhuang, juzhang, lvivier, michen, peterx, quintela, qzhang, stefanha, virt-maint, xuwei, yhong
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-03-13 15:31:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description xianwang 2018-03-01 03:37:44 UTC
Description of problem:
Migrate a vm with 

Version-Release number of selected component (if applicable):
Host:
3.10.0-855.el7.x86_64
qemu-kvm-rhev-2.10.0-21.el7.x86_64
seabios-bin-1.11.0-2.el7.noarch

Guest:
3.10.0-855.el7.x86_64

How reproducible:
1/10

Steps to Reproduce:
1.Boot a guest on source host with data plane and without vhost
/usr/libexec/qemu-kvm \
-name 'yhong-guest'  \
-sandbox off  \
-machine pc  \
-nodefaults  \
-vga cirrus  \
-chardev socket,id=qmp_monitor,path=/var/tmp/qmp_monitori_yhong,server,nowait \
-mon chardev=qmp_monitor,mode=control \
-device pvpanic,ioport=0x505,id=idrwzV9w  \
-chardev socket,id=serial_id_serial0,path=/var/tmp/serial-yhong,server,nowait \
-device isa-serial,chardev=serial_id_serial0  \
-chardev socket,id=seabioslog,path=/var/tmp/seabios-yhong,server,nowait \
-device isa-debugcon,chardev=seabioslog,iobase=0x402 \
-device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \
-device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \
-device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \
-device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \
-object iothread,id=iothread0 \
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x3,iothread=iothread0 \
-drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,werror=stop,rerror=stop,format=qcow2,file=/home/yhong/images/rhel75-x86_64-virtio-scsi-30G.qcow2 \
-device scsi-hd,id=image1,drive=drive_image1,scsi-id=0,lun=0,channel=0,bootindex=0 \
-drive id=drive_data0,if=none,snapshot=off,aio=threads,cache=none,werror=stop,rerror=stop,format=qcow2,file=/home/yhong/images/disk-data0-20G.qcow2 \
-device scsi-hd,id=data0,drive=drive_data0,scsi-id=0,lun=1,channel=0 \
-device virtio-net-pci,mac=9a:c8:c9:ca:cb:cc,id=id3twfht,vectors=4,netdev=idB1oiKa,bus=pci.0,addr=0x4  \
-netdev tap,id=idB1oiKa \
-m 8192  \
-smp 8,cores=4,threads=1,sockets=2  \
-cpu 'SandyBridge',+kvm_pv_unhalt \
-device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
-vnc :10  \
-rtc base=utc,clock=host,driftfix=slew  \
-boot order=cdn,once=c,menu=off,strict=off \
-enable-kvm \
-monitor stdio \

2.on destination host, launch listening mode with vhost=on
gdb -args /usr/libexec/qemu-kvm \
-name 'yhong-guest'  \
-sandbox off  \
-machine pc  \
-nodefaults  \
-vga cirrus  \
-chardev socket,id=qmp_monitor,path=/var/tmp/qmp_monitori_yhong,server,nowait \
-mon chardev=qmp_monitor,mode=control \
-device pvpanic,ioport=0x505,id=idrwzV9w  \
-chardev socket,id=serial_id_serial0,path=/var/tmp/serial-yhong,server,nowait \
-device isa-serial,chardev=serial_id_serial0  \
-chardev socket,id=seabioslog,path=/var/tmp/seabios-yhong,server,nowait \
-device isa-debugcon,chardev=seabioslog,iobase=0x402 \
-device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \
-device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \
-device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \
-device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \
-object iothread,id=iothread0 \
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x3,iothread=iothread0 \
-drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,werror=stop,rerror=stop,format=qcow2,file=/home/yhong/images/rhel75-x86_64-virtio-scsi-30G.qcow2 \
-device scsi-hd,id=image1,drive=drive_image1,scsi-id=0,lun=0,channel=0,bootindex=0 \
-drive id=drive_data0,if=none,snapshot=off,aio=threads,cache=none,werror=stop,rerror=stop,format=qcow2,file=/home/yhong/images/disk-data0-20G.qcow2 \
-device scsi-hd,id=data0,drive=drive_data0,scsi-id=0,lun=1,channel=0 \
-device virtio-net-pci,mac=9a:c8:c9:ca:cb:cc,id=id3twfht,vectors=4,netdev=idB1oiKa,bus=pci.0,addr=0x4  \
-netdev tap,id=idB1oiKa,vhost=on \
-m 8192  \
-smp 8,cores=4,threads=1,sockets=2 \
-cpu 'SandyBridge',+kvm_pv_unhalt \
-device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
-vnc :10  \
-rtc base=utc,clock=host,driftfix=slew  \
-boot order=cdn,once=c,menu=off,strict=off \
-enable-kvm \
-monitor stdio \
-incoming tcp:0:5801 \

3.on src host, scp an iso file from host to guest and then do migration to dst host
# scp ../isos/RHEL-7.5-20180221.0-Server-x86_64-dvd1.iso root.8.108:/home
(qemu) migrate -d tcp:10.66.10.208:5801

4.after file transferring completed and migration completed, reboot guest

5.on dst host, quit qemu
dst host:
(qemu) q

Actual results:
qemu quit error with some message, and sometime there is core dump generated.
(qemu) q
qemu-kvm: /builddir/build/BUILD/qemu-2.10.0/hw/scsi/virtio-scsi.c:246: virtio_scsi_ctx_check: Assertion `blk_get_aio_context(d->conf.blk) == s->ctx' failed.

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffe5fef700 (LWP 25960)]
(gdb) bt
#0  0x00007fffed88a207 in raise () at /lib64/libc.so.6
#1  0x00007fffed88b8f8 in abort () at /lib64/libc.so.6
#2  0x00007fffed883026 in __assert_fail_base () at /lib64/libc.so.6
#3  0x00007fffed8830d2 in  () at /lib64/libc.so.6
#4  0x00005555557985b7 in virtio_scsi_ctx_check (s=<optimized out>, s=<optimized out>, d=0x55555813f680)
    at /usr/src/debug/qemu-2.10.0/hw/scsi/virtio-scsi.c:246
#5  0x000055555581db96 in virtio_scsi_handle_cmd_vq (s=<optimized out>, s=<optimized out>, d=0x55555813f680)
    at /usr/src/debug/qemu-2.10.0/hw/scsi/virtio-scsi.c:246
#6  0x000055555581db96 in virtio_scsi_handle_cmd_vq (req=0x555559624f00, s=0x555558b62170)
    at /usr/src/debug/qemu-2.10.0/hw/scsi/virtio-scsi.c:559
#7  0x000055555581db96 in virtio_scsi_handle_cmd_vq (s=s@entry=0x555558b62170, vq=vq@entry=0x555558b6a100)
    at /usr/src/debug/qemu-2.10.0/hw/scsi/virtio-scsi.c:599
#8  0x000055555581e77a in virtio_scsi_data_plane_handle_cmd (vdev=<optimized out>, vq=0x555558b6a100)
    at /usr/src/debug/qemu-2.10.0/hw/scsi/virtio-scsi-dataplane.c:60
#9  0x000055555582afc6 in virtio_queue_host_notifier_aio_poll (vq=0x555558b6a100)
    at /usr/src/debug/qemu-2.10.0/hw/virtio/virtio.c:1506
#10 0x000055555582afc6 in virtio_queue_host_notifier_aio_poll (opaque=0x555558b6a168)
    at /usr/src/debug/qemu-2.10.0/hw/virtio/virtio.c:2420
#11 0x0000555555abf25e in run_poll_handlers_once (ctx=ctx@entry=0x555556d95cc0) at util/aio-posix.c:497
#12 0x0000555555abfea4 in aio_poll (max_ns=<optimized out>, ctx=0x555556d95cc0) at util/aio-posix.c:534
#13 0x0000555555abfea4 in aio_poll (blocking=true, ctx=0x555556d95cc0) at util/aio-posix.c:562
#14 0x0000555555abfea4 in aio_poll (ctx=0x555556d95cc0, blocking=blocking@entry=true) at util/aio-posix.c:602
#15 0x00005555558b2926 in iothread_run (opaque=0x555556da1260) at iothread.c:59
#16 0x00007fffedc28dd5 in start_thread () at /lib64/libpthread.so.0
#17 0x00007fffed952b3d in clone () at /lib64/libc.so.6


sometimes there will be core dump generated
(qemu) q
qemu-kvm: /builddir/build/BUILD/qemu-2.10.0/hw/scsi/virtio-scsi.c:246: virtio_scsi_ctx_check: Assertion `blk_get_aio_context(d->conf.blk) == s->ctx' failed.
guest-10052.sh: line 36: 22578 Aborted                 (core dumped) /usr/libexec/qemu-kvm -name 'yhong-guest' -sandbox of........

Expected results:
qemu quit succeeds without any error

Additional info:

Comment 2 Stefan Hajnoczi 2018-03-01 13:44:39 UTC
David Gilbert spotted that this BZ is probably a duplicate of bz#1480202.

I am working on fixes upstream and will take over this BZ.

Comment 3 xianwang 2018-03-02 08:35:27 UTC
(In reply to Stefan Hajnoczi from comment #2)
> David Gilbert spotted that this BZ is probably a duplicate of bz#1480202.
> 
> I am working on fixes upstream and will take over this BZ.

yes, I think so, this bug should a duplicate of bz#1480202, the error message of are same.

what's more, I think this bug is duplicate of an 7.3 bug bz#1363938 which is a bug about virtio-scsi, the error message of bz#1363938 is also same with this bug,so, maybe, this is a regression.

Comment 4 xianwang 2018-03-02 08:55:41 UTC
(In reply to xianwang from comment #3)
> (In reply to Stefan Hajnoczi from comment #2)
> > David Gilbert spotted that this BZ is probably a duplicate of bz#1480202.
> > 
> > I am working on fixes upstream and will take over this BZ.
> 
> yes, I think so, this bug should a duplicate of bz#1480202, the error
> message of are same.
> 
> what's more, I think this bug is duplicate of an 7.3 bug bz#1363938 which is
> a bug about virtio-scsi, the error message of bz#1363938 is also same with
> this bug,so, maybe, this is a regression.

I am sorry, I didn't check the error message carefully, this bug should just be a duplicate of bz#1480202, but not duplicate of 7.3 bug bz#1363938, because 
the former error is :
virtio_scsi_ctx_check: Assertion `blk_get_aio_context(d->conf.blk) == s->ctx'
the latter error is :
virtio_scsi_handle_cmd_req_prepare: Assertion `blk_get_aio_context(d->conf.blk) == s->ctx'.

Comment 5 Stefan Hajnoczi 2018-03-13 15:31:45 UTC

*** This bug has been marked as a duplicate of bug 1480202 ***