1844983 – Occurred core dump on hotunplug disk that is doing IO

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1844983 - Occurred core dump on hotunplug disk that is doing IO

Summary: Occurred core dump on hotunplug disk that is doing IO

Keywords:
Status:	CLOSED DEFERRED
Alias:	None
Product:	Red Hat Enterprise Linux 8
Classification:	Red Hat
Component:	qemu-kvm
Sub Component:
Version:	8.3
Hardware:	x86_64
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	rc
Target Release:	8.0
Assignee:	Sergio Lopez
QA Contact:	qing.wang
Docs Contact:
URL:
Whiteboard:
Duplicates (1):	1897525 (view as bug list)
Depends On:	1812399 1844343 1897525
Blocks:
TreeView+	depends on / blocked

Reported:	2020-06-08 08:22 UTC by qing.wang
Modified:	2020-11-26 01:15 UTC (History)
CC List:	8 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:	1844343
Environment:
Last Closed:	2020-11-13 12:03:31 UTC
Type:	Bug
Target Upstream Version:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Description qing.wang 2020-06-08 08:22:03 UTC

+++ This bug was initially created as a clone of Bug #1844343 +++

Description of problem:

qemu crash when hotunplug disk that is doing io.

02:45:13 INFO | Start to unplug devices "stg0" by monitor qmpmonitor1.
02:45:13 DEBUG| (monitor avocado-vt-vm1.qmpmonitor1) Sending command 'device_del' 
02:45:13 DEBUG| Send command: {'execute': 'device_del', 'arguments': {'id': 'stg0'}, 'id': 'NPwoGmu6'}
02:45:13 INFO | [qemu output] qemu-kvm: /builddir/build/BUILD/qemu-4.2.0/hw/scsi/virtio-scsi.c:250: virtio_scsi_ctx_check: Assertion `blk_get_aio_context(d->conf.blk) == s->ctx' failed.
02:45:36 WARNI| registers is not alive. Can't query the avocado-vt-vm1 status
02:45:42 DEBUG| Attempting to log into 'avocado-vt-vm1' (timeout 360s)
02:45:42 INFO | [qemu output] /tmp/aexpect_UtyR1AXR/aexpect-rly3turi.sh: line 1: 268982 Aborted                 (core dumped) MALLOC_PERTURB_=1 /usr/libexec/qemu-kvm -S -name 'avocado-vt-vm1' -sandbox on -machine q35 -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0 -nodefaults -device VGA,bus=pcie.0,addr=0x2 -m 30720 -smp 12,maxcpus=12,cores=6,threads=1,dies=1,sockets=2 -cpu 'Skylake-Server',+kvm_pv_unhalt -chardev socket,server,path=/var/tmp/avocado_6j4emk2s/monitor-qmpmonitor1-20200605-023351-cxpRnbd9,id=qmp_id_qmpmonitor1,nowait -mon chardev=qmp_id_qmpmonitor1,mode=control -chardev socket,server,path=/var/tmp/avocado_6j4emk2s/monitor-catch_monitor-20200605-023351-cxpRnbd9,id=qmp_id_catch_monitor,nowait -mon chardev=qmp_id_catch_monitor,mode=control -device pvpanic,ioport=0x505,id=id0yBSp5 -chardev socket,server,path=/var/tmp/avocado_6j4emk2s/serial-serial0-20200605-023351-cxpRnbd9,id=chardev_serial0,nowait -device isa-serial,id=serial0,chardev=chardev_serial0 -chardev socket,id=seabioslog_id_20200605-023351-cxpRnbd9,path=/var/tmp/avocado_6j4emk2s/seabios-20200605-023351-cxpRnbd9,server,nowait -device isa-debugcon,chardev=seabioslog_id_20200605-023351-cxpRnbd9,iobase=0x402 -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -object iothread,id=iothread0 -object iothread,id=iothread1 -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0,iothread=iothread0 -blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 -device scsi-hd,id=image1,drive=drive_image1,write-cache=on -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 -device virtio-net-pci,mac=9a:03:3d:56:61:b2,id=idYARjM3,netdev=idhVJiQ3,bus=pcie-root-port-3,addr=0x0 -netdev tap,id=idhVJiQ3,vhost=on,vhostfd=21,fd=6 -vnc :0 -rtc base=utc,clock=host,driftfix=slew -boot menu=off,order=cdn,once=c,strict=off -enable-kvm -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5


#0  0x00007fa9476d370f in raise () at /lib64/libc.so.6
#1  0x00007fa9476bdb25 in abort () at /lib64/libc.so.6
#2  0x00007fa9476bd9f9 in _nl_load_domain.cold.0 () at /lib64/libc.so.6
#3  0x00007fa9476cbcc6 in .annobin_assert.c_end () at /lib64/libc.so.6
#4  0x0000561e59c587c8 in virtio_scsi_ctx_check
    (d=0x561e5c347970, s=<optimized out>, s=<optimized out>)
    at /usr/src/debug/qemu-kvm-4.2.0-22.module+el8.2.1+6758+cb8d64c2.x86_64/hw/scsi/virtio-scsi.c:250
#5  0x0000561e59c587c8 in virtio_scsi_ctx_check
    (s=0x561e5d049f70, s=0x561e5d049f70, d=0x561e5c347970)
    at /usr/src/debug/qemu-kvm-4.2.0-22.module+el8.2.1+6758+cb8d64c2.x86_64/hw/scsi/virtio-scsi.c:247
#6  0x0000561e59c587c8 in virtio_scsi_handle_cmd_req_prepare
    (req=0x7fa930012ad0, s=0x561e5d049f70)
    at /usr/src/debug/qemu-kvm-4.2.0-22.module+el8.2.1+6758+cb8d64c2.x86_64/hw/scsi/virtio-scsi.c:569
#7  0x0000561e59c587c8 in virtio_scsi_handle_cmd_vq
    (s=s@entry=0x561e5d049f70, vq=vq@entry=0x7fa91c6a5140)
    at /usr/src/debug/qemu-kvm-4.2.0-22.module+el8.2.1+6758+cb8d64c2.x86_64/hw/scsi/virtio-scsi.c:612
#8  0x0000561e59c5948e in virtio_scsi_data_plane_handle_cmd
    (vdev=<optimized out>, vq=0x7fa91c6a5140)
    at /usr/src/debug/qemu-kvm-4.2.0-22.module+el8.2.1+6758+cb8d64c2.x86_64/hw/scsi/virtio-scsi---Type <RET> for more, q to quit, c to continue without paging--c
dataplane.c:60
#9  0x0000561e59c66fbe in virtio_queue_notify_aio_vq (vq=<optimized out>) at /usr/src/debug/qemu-kvm-4.2.0-22.module+el8.2.1+6758+cb8d64c2.x86_64/hw/virtio/virtio.c:2243
#10 0x0000561e59f2a046 in run_poll_handlers_once (ctx=ctx@entry=0x561e5be1bb70, timeout=timeout@entry=0x7fa93f70e658) at util/aio-posix.c:517
#11 0x0000561e59f2afba in run_poll_handlers (timeout=0x7fa93f70e658, max_ns=4000, ctx=0x561e5be1bb70) at util/aio-posix.c:562
#12 0x0000561e59f2afba in try_poll_mode (timeout=0x7fa93f70e658, ctx=0x561e5be1bb70) at util/aio-posix.c:597
#13 0x0000561e59f2afba in aio_poll (ctx=0x561e5be1bb70, blocking=blocking@entry=true) at util/aio-posix.c:639
#14 0x0000561e59d057f4 in iothread_run (opaque=0x561e5be02b60) at iothread.c:75
#15 0x0000561e59f2cd84 in qemu_thread_start (args=0x561e5be1c080) at util/qemu-thread-posix.c:519
#16 0x00007fa947a662de in start_thread () at /lib64/libpthread.so.0
#17 0x00007fa947797e83 in clone () at /lib64/libc.so.6

Version-Release number of selected component (if applicable):


How reproducible:
70% happened in automation.

Steps to Reproduce:
1.boot vm
/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35 \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 30720  \
    -smp 12,maxcpus=12,cores=6,threads=1,dies=1,sockets=2  \
    -cpu 'Skylake-Server',+kvm_pv_unhalt \
    -chardev socket,server,path=/var/tmp/avocado_6j4emk2s/monitor-qmpmonitor1-20200605-023351-cxpRnbd9,id=qmp_id_qmpmonitor1,nowait  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,server,path=/var/tmp/avocado_6j4emk2s/monitor-catch_monitor-20200605-023351-cxpRnbd9,id=qmp_id_catch_monitor,nowait  \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=id0yBSp5 \
    -chardev socket,server,path=/var/tmp/avocado_6j4emk2s/serial-serial0-20200605-023351-cxpRnbd9,id=chardev_serial0,nowait \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20200605-023351-cxpRnbd9,path=/var/tmp/avocado_6j4emk2s/seabios-20200605-023351-cxpRnbd9,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20200605-023351-cxpRnbd9,iobase=0x402 \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -object iothread,id=iothread0 \
    -object iothread,id=iothread1 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0,iothread=iothread0 \
    -blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:03:3d:56:61:b2,id=idYARjM3,netdev=idhVJiQ3,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=idhVJiQ3,vhost=on,vhostfd=21,fd=6  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5

2.hotpulg one disk
 {"execute": "blockdev-add", "arguments": {"node-name": "file_stg0", "driver": "file", "aio": "threads", "filename": "/home/kvm_autotest_root/images/storage0.qcow2", "cache": {"direct": true, "no-flush": false}}, "id": "bfCXHvne"}
 {"execute": "blockdev-add", "arguments": {"node-name": "drive_stg0", "driver": "qcow2", "cache": {"direct": true, "no-flush": false}, "file": "file_stg0"}, "id": "i2JYBkct"}
 {"execute": "device_add", "arguments": {"driver": "scsi-hd", "id": "stg0", "drive": "drive_stg0", "write-cache": "on", "bus": "virtio_scsi_pci0.0"}, "id": "ODAFuMrY"}

3.execute io test on the new disk in guest
 /home/iozone_inst/src/current/iozone -az -g 10G -y 32k -i 0 -i 1 -I -f /mnt/sdb1/iozone_test

4.hot-unplug disk

 {"execute": "device_del", "arguments": {"id": "stg0"}, "id": "O6HOCEti"}

5 remove blockdev node
 {"execute": "blockdev-del", "arguments": {"node-name": "drive_stg0"}, "id": "xVEiOfoF"}
 {"execute": "blockdev-del", "arguments": {"node-name": "file_stg0"}, "id": "wlFMohtd"}
 
6. repeat step 2-5 50 times

Actual results:
crash happened on step 

Expected results:
no crash

Additional info:
automation script:
python3 ConfigTest.py --testcase=hotplug_unplug_during_io_repeat.default.q35 --iothread_scheme=roundrobin --nr_iothreads=2 --platform=x86_64 --guestname=RHEL.8.3.0 --driveformat=virtio_scsi --nicmodel=virtio_net --imageformat=qcow2 --machines=q35 --customsparams="image_aio=threads" --clone=no

--- Additional comment from qing.wang on 2020-06-05 07:31:31 UTC ---

Version info:
host
4.18.0-193.2.1.el8_2.x86_64
qemu-kvm-core-4.2.0-22.module+el8.2.1+6758+cb8d64c2.x86_64

Test log：
http://fileshare.englab.nay.redhat.com/pub/section2/images_backup/qbugs/1844343/2020-06-05-0324/job-2020-06-05T02.33-7695316/

coredump file:
http://fileshare.englab.nay.redhat.com/pub/section2/images_backup/qbugs/1844343/core.qemu-kvm.0.a044351966c04920bece765d4fd501e7.268982.1591339513000000.lz4

--- Additional comment from qing.wang on 2020-06-05 09:24:07 UTC ---

Hit same issue on
{'kvm_version': '4.18.0-193.13.el8.x86_64', 'qemu_version': 'qemu-kvm-4.2.0-19.module+el8.3.0+6478+69f490bb.x86_64'}

Comment 1 qing.wang 2020-06-09 05:58:53 UTC

Not hit this issue on 
3.10.0-1127.el7.x86_64
qemu-kvm-common-rhev-2.12.0-44.el7_8.2.x86_64

Comment 2 John Ferlan 2020-06-09 15:35:49 UTC

If bug 1844343 is resolved in time for RHEL AV 8.2.1, then this can use (or depend on) bug 1844296 (which is the RHEL AV 8.2.1 -> RHEL 8.3.0 backport); otherwise, we'd have to make a decision whether to backport to RHEL at some point in the future as well.  Thankfully bug 1812765 (the virtio-blk one) won't have/need an extra RHEL bug since it'll be rebased anyway when the RHEL AV 8.2.1 -> RHEL 8.3.0 occurs.

Comment 7 John Ferlan 2020-11-13 11:44:46 UTC

*** Bug 1897525 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.