Created attachment 1539377 [details] gdb_debug_info_all_threads-02282019 Description of problem: After creating a snapshot for a living guest image(data plane enabled) with blockdev-create/blockdev-add while specifying the image as the backing file with 'backing-file'/'backing', qemu core dumped: # ./vm12.sh rhel76be.qcow2 QEMU 3.1.0 monitor - type 'help' for more information (qemu) (qemu) qemu-kvm: /builddir/build/BUILD/qemu-3.1.0/hw/scsi/virtio-scsi.c:246: virtio_scsi_ctx_check: Assertion `blk_get_aio_context(d->conf.blk) == s->ctx' failed. ./vm12.sh: line 28: 68870 Aborted (core dumped) /usr/libexec/qemu-kvm -name 'avocado-vt-vm1' -sandbox off -machine pseries -nodefaults -vga std -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado1,server,nowait -mon chardev=qmp_id_qmpmonitor1,mode=control -smp 8,cores=2,threads=2,sockets=2 -object iothread,id=iothread1 -device nec-usb-xhci,id=usb1,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -chardev socket,path=/tmp/qga.sock,server,nowait,id=qga0 -device virtserialport,bus=virtio-serial0.0,chardev=qga0,id=qemu-ga0,name=org.qemu.guest_agent.0 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x6,iothread=iothread1 -blockdev node-name=file0,driver=file,filename=$1 -blockdev node-name=disk0,driver=qcow2,file=file0 -device scsi-hd,drive=disk0,id=image0 -device virtio-net-pci,mac=9a:78:79:7a:7b:6c,id=id8e5D72,netdev=idrYUYaH -netdev tap,id=idrYUYaH,vhost=on -m 1024 -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -device usb-kbd,id=usb-kbd1,bus=usb1.0,port=2 -vnc :10 -rtc base=localtime,clock=host,driftfix=slew -boot menu=off,strict=off,order=cdn -enable-kvm -monitor stdio The script used to create the snapshot is as follows: # cat imagecreate_scratch.sh #!/bin/bash echo -e "{'execute':'qmp_capabilities'} {'execute':'blockdev-create','arguments':{'options': {'driver':'file','filename':'/home/ngu/inc1.qcow2','size':32212254720},'job-id':'job1'}}" | nc -U /var/tmp/avocado1 sleep 2 echo -e "{'execute':'qmp_capabilities'} {'execute':'blockdev-add','arguments':{'driver':'file','node-name':'iinc11','filename':'/home/ngu/inc1.qcow2'}}" | nc -U /var/tmp/avocado1 sleep 2 echo -e "{'execute':'qmp_capabilities'} {'execute':'blockdev-create', 'arguments': {'job-id': 'job2', 'options': {'driver': 'qcow2', 'file': 'iinc11', 'size': 32212254720,'backing-fmt':'qcow2','backing-file':'rhel76be.qcow2'}}}" | nc -U /var/tmp/avocado1 sleep 5 echo -e "{'execute':'qmp_capabilities'} {'execute':'blockdev-add','arguments':{'driver':'qcow2','node-name':'tmp','file':'iinc11','backing':'disk0'}}" | nc -U /var/tmp/avocado1 sleep 5 echo -e "{'execute':'qmp_capabilities'} {'execute':'job-dismiss','arguments':{'id':'job1'}}" | nc -U /var/tmp/avocado1 sleep 2 echo -e "{'execute':'qmp_capabilities'} {'execute':'job-dismiss','arguments':{'id':'job2'}}" | nc -U /var/tmp/avocado1 Version-Release number of selected component (if applicable): Host kernel: 4.18.0-71.el8.ppc64le qemu-kvm-3.1.0-18.module+el8+2834+fa8bb6e2.ppc64le How reproducible: 100% Steps to Reproduce: 1. Boot up a guest with a system disk(the node-name is disk0, data plane enabled) 2. Try to create a scratch disk as the snapshot of disk0 with blockdev-create/blockdev-add(as the script imagecreate_scratch.sh shows) 3. After the snapshot creation finishes, qemu core dumped occurs, or sometimes you need to do a system_reset to trigger the core dump. Actual results: In step3, qemu core dumped. Expected results: No core dump would occur. Additional info: Both system disk and data disk would cause the bug.
Created attachment 1539384 [details] gdb_debug_info_all_threads-x86-02282019 Issue reproduced on x86 platform.
Verified on qemu-kvm-3.1.0-11.module+el8+2756+0dd6db45.x86_64 and qemu-kvm-3.1.0-3.module+el8+2653+34dc2efa, also hit this issue.
Hit the same issue run auto case: block_stream.with_hot_unplug.with_data_plane qemu-kvm-rhev version:qemu-kvm-rhev-2.12.0-29.el7.x86_64 Steps: 1. Start guest with qemu cmds: /usr/libexec/qemu-kvm \ -S \ -name 'avocado-vt-vm1' \ -machine pc \ -nodefaults \ -device VGA,bus=pci.0,addr=0x2 \ -chardev socket,id=qmp_id_qmp1,path=/var/tmp/avocado_qjfkuQ/monitor-qmp1-20190522-203214-pO8ikKhP,server,nowait \ -mon chardev=qmp_id_qmp1,mode=control \ -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/avocado_qjfkuQ/monitor-catch_monitor-20190522-203214-pO8ikKhP,server,nowait \ -mon chardev=qmp_id_catch_monitor,mode=control \ -device pvpanic,ioport=0x505,id=idWcUIuL \ -chardev socket,id=serial_id_serial0,path=/var/tmp/avocado_qjfkuQ/serial-serial0-20190522-203214-pO8ikKhP,server,nowait \ -device isa-serial,chardev=serial_id_serial0 \ -chardev socket,id=seabioslog_id_20190522-203214-pO8ikKhP,path=/var/tmp/avocado_qjfkuQ/seabios-20190522-203214-pO8ikKhP,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20190522-203214-pO8ikKhP,iobase=0x402 \ -device nec-usb-xhci,id=usb1,bus=pci.0,addr=0x3 \ -object iothread,id=iothread0 \ -object iothread,id=iothread1 \ -device virtio-scsi-pci,iothread=iothread0,id=virtio_scsi_pci0,bus=pci.0,addr=0x4 \ -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/rhel77-64-virtio-scsi.qcow2 \ -device scsi-hd,id=image1,drive=drive_image1 \ -device virtio-net-pci,mac=9a:f4:f5:f6:f7:f8,id=idJqoo3m,vectors=4,netdev=idiujahB,bus=pci.0,addr=0x5 \ -netdev tap,id=idiujahB,vhost=on,vhostfd=20,fd=17 \ -m 7168 \ -smp 4,maxcpus=4,cores=2,threads=1,sockets=2 \ -cpu 'Skylake-Client',+kvm_pv_unhalt \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -vnc :0 \ -rtc base=utc,clock=host,driftfix=slew \ -boot menu=off,strict=off,order=cdn,once=c \ -enable-kvm 2. dd file in guest: (guest)# dd if=/dev/urandom of=/home/block_stream_test bs=128k count=100 # md5sum /home/block_stream_test > /home/block_stream_test.md5 3.Create live snapshot: {'execute': 'blockdev-snapshot-sync', 'arguments': {'device': u'drive_image1', 'snapshot-file': '/home/kvm_autotest_root/images/sn1', 'mode': 'absolute-paths', 'format': 'qcow2'}, 'id': 'FUnc4dl1'} 4.Do block stream: {'execute': 'block-stream', 'arguments': {'device': u'drive_image1', 'speed': 0}, 'id': 'wH5DAJ24'} 5.During block stream running, Hot unplug device drive_image1 {'execute': 'device_del', 'arguments': {'id': 'image1'}, 'id': 'kuc6shMG'} After step5, qemu hang with info: [qemu output] qemu-kvm: /builddir/build/BUILD/qemu-2.12.0/hw/scsi/virtio-scsi.c:246: virtio_scsi_ctx_check: Assertion `blk_get_aio_context(d->conf.blk) == s->ctx' failed. 20:32:46 WARNI| avocado-vt-vm1 is not alive. Can't query the register status 20:32:46 INFO | [qemu output] /tmp/aexpect_g3elZbyA/aexpect-l5_gyJ.sh: line 1: 20282 Aborted (core dumped) MALLOC_PERTURB_=1 /usr/libexec/qemu-kvm -S -name 'avocado-vt-vm1' -machine pc -nodefaults -device VGA,bus=pci.0,addr=0x2 -chardev socket,id=qmp_id_qmp1,path=/var/tmp/avocado_qjfkuQ/monitor-qmp1-20190522-203214-pO8ikKhP,server,nowait -mon chardev=qmp_id_qmp1,mode=control -chardev socket,id=qmp_ ...
Hit the same issue when run auto case: Host_RHEL.m8.u1.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.8.1.0.x86_64.io-github-autotest-qemu.block_hotplug_in_pause.data_plane.with_plug.one_pci.q35 Versions: kernel-4.18.0-112.el8.x86_64 qemu-kvm-4.0.0-6.module+el8.1.0+3736+a2aefea3 Steps: 1. create data disk qemu-img create -f qcow2 /home/kvm_autotest_root/images/storage0.qcow2 1G 2. Start guest with data plane: /usr/libexec/qemu-kvm \ -S \ -name 'avocado-vt-vm1' \ -machine q35 \ -nodefaults \ -device VGA,bus=pcie.0,addr=0x1 \ -m 14336 \ -smp 12,maxcpus=12,cores=6,threads=1,sockets=2 \ -cpu 'Opteron_G5',+kvm_pv_unhalt \ -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado_ubtw2g4_/monitor-qmpmonitor1-20190726-014326-SlnWBhGY,server,nowait \ -mon chardev=qmp_id_qmpmonitor1,mode=control \ -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/avocado_ubtw2g4_/monitor-catch_monitor-20190726-014326-SlnWBhGY,server,nowait \ -mon chardev=qmp_id_catch_monitor,mode=control \ -device pvpanic,ioport=0x505,id=idzfGHgf \ -chardev socket,nowait,id=chardev_serial0,path=/var/tmp/avocado_ubtw2g4_/serial-serial0-20190726-014326-SlnWBhGY,server \ -device isa-serial,id=serial0,chardev=chardev_serial0 \ -chardev socket,id=seabioslog_id_20190726-014326-SlnWBhGY,path=/var/tmp/avocado_ubtw2g4_/seabios-20190726-014326-SlnWBhGY,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20190726-014326-SlnWBhGY,iobase=0x402 \ -device pcie-root-port,id=pcie.0-root-port-2,slot=2,chassis=2,addr=0x2,bus=pcie.0 \ -device qemu-xhci,id=usb1,bus=pcie.0-root-port-2,addr=0x0 \ -object iothread,id=iothread0 \ -device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 \ -device virtio-scsi-pci,id=virtio_scsi_pci0,iothread=iothread0,bus=pcie.0-root-port-3,addr=0x0 \ -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/rhel810-64-virtio-scsi.qcow2 \ -device scsi-hd,id=image1,drive=drive_image1,bootindex=0 \ -device pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x4,bus=pcie.0 \ -device virtio-net-pci,mac=9a:8a:1f:08:99:53,id=idD6NL1q,netdev=idzrfhNf,bus=pcie.0-root-port-4,addr=0x0 \ -netdev tap,id=idzrfhNf,vhost=on \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -vnc :0 \ -rtc base=utc,clock=host,driftfix=slew \ -boot order=cdn,once=c,menu=off,strict=off \ -enable-kvm \ -qmp tcp:0:4444,server,nowait \ -monitor stdio \ -device pcie-root-port,id=pcie_extra_root_port_0,slot=5,chassis=5,addr=0x5,bus=pcie.0 3. boot up guest and stop it # telnet localhost 4444 {"execute":"qmp_capabilities"} {"execute": "cont", "id": "Q3cJKo4o"} after guest boot up, then stop it. {'execute': 'stop', 'id': 'FzVV6SfR'} 4. hotplug data disk {"execute": "human-monitor-command", "arguments": {"command-line": "drive_add auto id=drive_stg0,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/storage0.qcow2"}, "id": "a5gl2mJw"} {"execute": "device_add", "arguments": {"driver": "scsi-hd", "id": "stg0", "drive": "drive_stg0", "bus": "virtio_scsi_pci0.0"}, "id": "OGiWW3W5"} 5. cont guest and dd test on new hotpluged disk: {"execute": "cont", "id": "hosBKyXk"} # dd if=/dev/sdb of=/dev/null bs=1k count=1000 iflag=direct && dd if=/dev/zero of=/dev/sdb bs=1k count=1000 oflag=direct 6. unplug data disk {"execute": "device_del", "arguments": {"id": "stg0"}, "id": "Iwt4Yhmj"} After step 6, qemu core dumped. [qemu output] qemu-kvm: /builddir/build/BUILD/qemu-4.0.0/hw/scsi/virtio-scsi.c:246: virtio_scsi_ctx_check: Assertion `blk_get_aio_context(d->conf.blk) == s->ctx' failed. 01:45:38 INFO | [qemu output] /tmp/aexpect_jufL4TLP/aexpect-3rnv_f1a.sh: line 1: 20259 Aborted (core dumped) MALLOC_PERTURB_=1 /usr/libexec/qemu-kvm -S -name 'avocado-vt-vm1' -machine q35 -nodefaults -device VGA,bus=pcie.0,addr=0x1 -m 14336 -smp 12,maxcpus=12,cores=6,threads=1,sockets=2 -cpu 'Opteron_G5',+kvm_pv_unhalt -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/avocado_ubtw2g4_/monitor-qmpmonitor1-20190726-014326-SlnWBhGY,server,nowait -mon chardev=qmp_id_qmpmonitor1,mode=control -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/avocado_ubtw2g4_/monitor-catch_monitor-20190726-014326-SlnWBhGY,server,nowait -mon chardev=qmp_id_catch_monitor,mode=control -device pvpanic,ioport=0x505,id=idzfGHgf -chardev socket,nowait,id=chardev_serial0,path=/var/tmp/avocado_ubtw2g4_/serial-serial0-20190726-014326-SlnWBhGY,server -device isa-serial,id=serial0,chardev=chardev_serial0 -chardev socket,id=seabioslog_id_20190726-014326-SlnWBhGY,path=/var/tmp/avocado_ubtw2g4_/seabios-20190726-014326-SlnWBhGY,server,nowait -device isa-debugcon,chardev=seabioslog_id_20190726-014326-SlnWBhGY,iobase=0x402 -device pcie-root-port,id=pcie.0-root-port-2,slot=2,chassis=2,addr=0x2,bus=pcie.0 -device qemu-xhci,id=usb1,bus=pcie.0-root-port-2,addr=0x0 -object iothread,id=iothread0 -device pcie-root-port,id=pcie.0-root-port-3,slot=3,chassis=3,addr=0x3,bus=pcie.0 -device virtio-scsi-pci,id=virtio_scsi_pci0,iothread=iothread0,bus=pcie.0-root-port-3,addr=0x0 -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/rhel810-64-virtio-scsi.qcow2 -device scsi-hd,id=image1,drive=drive_image1,bootindex=0 -device pcie-root-port,id=pcie.0-root-port-4,slot=4,chassis=4,addr=0x4,bus=pcie.0 -device virtio-net-pci,mac=9a:8a:1f:08:99:53,id=idD6NL1q,netdev=idzrfhNf,bus=pcie.0-root-port-4,addr=0x0 -netdev tap,id=idzrfhNf,vhost=on,vhostfd=22,fd=20 -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -vnc :0 -rtc base=utc,clock=host,driftfix=slew -boot order=cdn,once=c,menu=off,strict=off -enable-kvm -device pcie-root-port,id=pcie_extra_root_port_0,slot=5,chassis=5,addr=0x5,bus=pcie.0 01:45:38 INFO | [qemu output] (Process terminated with status 134)
For reference, the original analysis for this crash is in bug 1684342. This should be fixed as of upstream commit d0ee0204f4.
Verified on qemu-kvm-4.1.0-1.module+el8.1.0+3966+4a23dca1.x86_64, the problem has been resolved, set bug's status to "Verified". Test steps: 1.Start guest with qemu cmds: /usr/libexec/qemu-kvm \ -name 'avocado-vt-vm1' \ -machine pc \ -nodefaults \ -device VGA,bus=pci.0,addr=0x2 \ -chardev socket,id=qmp_id_qmp1,path=/var/tmp/monitor-qmp1-20190522-203214-pO8ikKhP,server,nowait \ -mon chardev=qmp_id_qmp1,mode=control \ -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20190522-203214-pO8ikKhP,server,nowait \ -mon chardev=qmp_id_catch_monitor,mode=control \ -device pvpanic,ioport=0x505,id=idWcUIuL \ -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20190522-203214-pO8ikKhP,server,nowait \ -device isa-serial,chardev=serial_id_serial0 \ -chardev socket,id=seabioslog_id_20190522-203214-pO8ikKhP,path=/var/tmp/seabios-20190522-203214-pO8ikKhP,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20190522-203214-pO8ikKhP,iobase=0x402 \ -device nec-usb-xhci,id=usb1,bus=pci.0,addr=0x3 \ -object iothread,id=iothread0 \ -object iothread,id=iothread1 \ -device virtio-scsi-pci,iothread=iothread0,id=virtio_scsi_pci0,bus=pci.0,addr=0x4 \ -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/rhel810-64-virtio-scsi.qcow2 \ -device scsi-hd,id=image1,drive=drive_image1 \ -device virtio-net-pci,mac=9a:f4:f5:f6:f7:f8,id=idJqoo3m,vectors=4,netdev=idiujahB,bus=pci.0,addr=0x5 \ -netdev tap,id=idiujahB,vhost=on \ -m 7168 \ -smp 4,maxcpus=4,cores=2,threads=1,sockets=2 \ -cpu 'Skylake-Client',+kvm_pv_unhalt \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -vnc :0 \ -rtc base=utc,clock=host,driftfix=slew \ -boot menu=off,strict=off,order=cdn,once=c \ -enable-kvm \ -monitor stdio \ 2. Hotplug disk and rw on it {"execute": "human-monitor-command", "arguments": {"command-line": "drive_add auto id=drive_stg0,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/storage0.qcow2"}, "id": "a5gl2mJw"} {"execute": "device_add", "arguments": {"driver": "scsi-hd", "id": "stg0", "drive": "drive_stg0", "bus": "virtio_scsi_pci0.0"}, "id": "OGiWW3W5"} # dd if=/dev/sdb of=/dev/null bs=1k count=1000 iflag=direct && dd if=/dev/zero of=/dev/sdb bs=1k count=1000 oflag=direct 3. Unplug data disk {"execute": "device_del", "arguments": {"id": "stg0"}, "id": "Iwt4Yhmj"} After step3, data disk can be unplugged successfully. For blockdev: 1. Start guest with data disk and data plane enable. -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x6,iothread=iothread1 \ -blockdev node-name=disk1,file.driver=file,driver=qcow2,file.filename=hd1,cache.no-flush=off,cache.direct=off \ -device scsi-hd,drive=disk1,id=data1,bus=virtio_scsi_pci0.0,scsi-id=0,lun=1,channel=0,werror=stop,rerror=stop \ 2. Create an snapshot image on the data disk with qemu-img: # qemu-img create -f qcow2 scratch.img -b hd1 -F qcow2 3. Add the snapshot image with blockdev-add in qmp: {"execute":"blockdev-add","arguments":{"driver":"qcow2","node-name":"tmp","file":{"driver":"file","filename":"scratch.img"},"backing":"disk1"}} After step3, data disk hot plug successfully.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:3723