Description of problem: Qemu core dump when start guest with two disks using same drive Version-Release number of selected component (if applicable): kernel version:4.18.0-57.el8.x86_64 qemu-kvm version:qemu-kvm-3.1.0-2.module+el8+2606+2c716ad7.x86_64 How reproducible: 100% Steps to Reproduce: 1.Start guest, two disks using the same drive "drive_image1": /usr/libexec/qemu-kvm \ -name 'avocado-vt-vm1' \ -machine q35 \ -nodefaults \ -device VGA,bus=pcie.0,addr=0x1 \ -device pcie-root-port,id=pcie_root_port_0,slot=2,chassis=2,addr=0x2,bus=pcie.0 \ -device pcie-root-port,id=pcie_root_port_1,slot=3,chassis=3,addr=0x3,bus=pcie.0 \ -device pcie-root-port,id=pcie_root_port_2,slot=4,chassis=4,addr=0x4,bus=pcie.0 \ -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20181228-045409-0zhg5aer,server,nowait \ -mon chardev=qmp_id_qmpmonitor1,mode=control \ -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20181228-045409-0zhg5aer,server,nowait \ -mon chardev=qmp_id_catch_monitor,mode=control \ -device pvpanic,ioport=0x505,id=idjsOqD4 \ -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20181228-045409-0zhg5aer,server,nowait \ -device isa-serial,chardev=serial_id_serial0 \ -chardev socket,id=seabioslog_id_20181228-045409-0zhg5aer,path=/var/tmp/seabios-20181228-045409-0zhg5aer,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20181228-045409-0zhg5aer,iobase=0x402 \ -device pcie-root-port,id=pcie.0-root-port-5,slot=5,chassis=5,addr=0x5,bus=pcie.0 \ -device qemu-xhci,id=usb1,bus=pcie.0-root-port-5,addr=0x0 \ -device pcie-root-port,id=pcie.0-root-port-6,slot=6,chassis=6,addr=0x6,bus=pcie.0 \ -object iothread,id=iothread0 \ -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-6,addr=0x0,iothread=iothread0 \ -blockdev driver=file,node-name=driveimage1,filename=/home/kvm_autotest_root/images/rhel76-64-virtio-scsi.raw \ -blockdev node-name=drive_image1,file=driveimage1,driver=raw \ -device scsi-hd,id=image1,drive=drive_image1 \ -device pcie-root-port,id=pcie.0-root-port-7,slot=7,chassis=7,addr=0x7,bus=pcie.0 \ -device virtio-net-pci,mac=9a:0f:10:11:12:13,id=idPrzVst,vectors=4,netdev=id5uShpn,bus=pcie.0-root-port-7,addr=0x0 \ -netdev tap,id=id5uShpn,vhost=on \ -m 14336 \ -smp 10,maxcpus=10,cores=5,threads=1,sockets=2 \ -cpu 'Broadwell',+kvm_pv_unhalt \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -vnc :0 \ -rtc base=utc,clock=host,driftfix=slew \ -boot order=cdn,once=c,menu=off,strict=off \ -enable-kvm \ -monitor stdio \ -blockdev driver=file,node-name=drivedata1,filename=/home/data1.qcow2 \ -blockdev node-name=drive_data1,file=drivedata1,driver=qcow2 \ -device scsi-hd,id=data1,drive=drive_image1 \ Actual results: Qemu core dump with error msg: (qemu) qemu: qemu_mutex_unlock_impl: Operation not permitted test.txt: line 41: 22212 Aborted (core dumped) /usr/libexec/qemu-kvm -name 'avocado-vt-vm1' -machine q35 -nodefaults -device VGA,bus=pcie.0,addr=0x1 -device pcie-root-port,id=pcie_root_port_0,slot=2,chassis=2,addr=0x2,bus=pcie.0 -device pcie-root-port,id=pcie_root_port_1,slot=3,chassis=3,addr=0x3,bus=pcie.0 -device pcie-root-port,id=pcie_root_port_2,slot=4,chassis=4,addr=0x4,bus=pcie.0 -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20181228-045409-0zhg5aer,server,nowait -mon chardev=qmp_id_qmpmonitor1,mode=control -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20181228-045409-0zhg5aer,server,nowait -mon chardev=qmp_id_catch_monitor,mode=control -device pvpanic,ioport=0x505,id=idjsOqD4 -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20181228-045409-0zhg5aer,server,nowait -device isa-serial,chardev=serial_id_serial0 -chardev socket,id=seabioslog_id_20181228-045409-0zhg5aer,path=/var/tmp/seabios-20181228-045409-0zhg5aer,server,nowait -device isa-debugcon,chardev=seabioslog_id_20181228-045409-0zhg5aer,iobase=0x402 -device pcie-root-port,id=pcie.0-root-port-5,slot=5,chassis=5,addr=0x5,bus=pcie.0 -device qemu-xhci,id=usb1,bus=pcie.0-root-port-5,addr=0x0 -device pcie-root-port,id=pcie.0-root-port-6,slot=6,chassis=6,addr=0x6,bus=pcie.0 -object iothread,id=iothread0 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-6,addr=0x0,iothread=iothread0 -blockdev driver=file,node-name=driveimage1,filename=/home/kvm_autotest_root/images/rhel76-64-virtio-scsi.raw -blockdev node-name=drive_image1,file=driveimage1,driver=raw -device scsi-hd,id=image1,drive=drive_image1 -device pcie-root-port,id=pcie.0-root-port-7,slot=7,chassis=7,addr=0x7,bus=pcie.0 -device virtio-net-pci,mac=9a:0f:10:11:12:13,id=idPrzVst,vectors=4,netdev=id5uShpn,bus=pcie.0-root-port-7,addr=0x0 -netdev tap,id=id5uShpn,vhost=on -m 14336 -smp 10,maxcpus=10,cores=5,threads=1,sockets=2 -cpu 'Broadwell',+kvm_pv_unhalt -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -vnc :0 -rtc base=utc,clock=host,driftfix=slew -boot order=cdn,once=c,menu=off,strict=off -enable-kvm -monitor stdio -blockdev driver=file,node-name=drivedata1,filename=/home/data1.qcow2 -blockdev node-name=drive_data1,file=drivedata1,driver=qcow2 -device scsi-hd,id=data1,drive=drive_image1 Expected results: Guest start failed with error indications. Additional info: # gdb core.qemu-kvm.0.cac53b7105b64c4aa51274721b3d29b2.22212.1546055319000000 ... (gdb) bt full #0 0x00007fac67fc293f in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 set = {__val = {268444224, 94038308945913, 0, 18446462603027743615, 4294967295, 0, 8096, 94037222342091, 12884901888, 94037222342100, 94037242641984, 140378529752046, 1024, 140733096201440, 94037242641992, 94037218144725}} pid = <optimized out> tid = <optimized out> #1 0x00007fac67facc95 in __GI_abort () at abort.c:79 save_stage = 1 act = {__sigaction_handler = {sa_handler = 0x7fac68110639, sa_sigaction = 0x7fac68110639}, sa_mask = {__val = {0, 0, 0, 94037220303504, 65535, 0, 140378456017776, 0, 0, 0, 13516693564483341056, 0, 1, 94037242471360, 0, 0}}, sa_flags = -1088642416, sa_restorer = 0xffff} sigs = {__val = {32, 0 <repeats 15 times>}} #2 0x00005586bf25750e in error_exit (err=<optimized out>, msg=msg@entry=0x5586bf3dfd60 <__func__.19018> "qemu_mutex_unlock_impl") at util/qemu-thread-posix.c:36 #3 0x00005586bf25774a in qemu_mutex_unlock_impl (mutex=mutex@entry=0x5586c06fb2b0, file=file@entry=0x5586bf3df25f "util/async.c", line=line@entry=516) at util/qemu-thread-posix.c:96 err = <optimized out> __PRETTY_FUNCTION__ = "qemu_mutex_unlock_impl" __func__ = "qemu_mutex_unlock_impl" #4 0x00005586bf2529b9 in aio_context_release (ctx=ctx@entry=0x5586c06fb250) at util/async.c:516 #5 0x00005586bf1c8738 in blk_prw (blk=blk@entry=0x5586c2097f50, offset=offset@entry=0, buf=buf@entry=0x7ffefa350f00 "\017", bytes=bytes@entry=512, co_entry=co_entry@entry=0x5586bf1ca290 <blk_read_entry>, flags=flags@entry=0) at block/block-backend.c:1262 waited_ = <optimized out> ctx_ = 0x5586c06fb250 bs_ = <optimized out> co = <optimized out> qiov = {iov = 0x7ffefa350e30, niov = 1, nalloc = -1, size = 512} iov = {iov_base = 0x7ffefa350f00, iov_len = 512} rwco = {blk = 0x5586c2097f50, offset = 0, iobuf = 0x7ffefa350e40, ret = 2147483647, flags = 0} __PRETTY_FUNCTION__ = "blk_prw" #6 0x00005586bf1ca4db in blk_pread (count=512, buf=0x7ffefa350f00, offset=0, blk=0x5586c2097f50) at block/block-backend.c:1424 ret = <optimized out> ret = <optimized out> #7 0x00005586bf1ca4db in blk_pread_unthrottled (blk=blk@entry=0x5586c2097f50, offset=offset@entry=0, buf=buf@entry=0x7ffefa350f00 "\017", count=count@entry=512) at block/block-backend.c:1279 ret = <optimized out> #8 0x00005586bf08c7bf in guess_disk_lchs (blk=blk@entry=0x5586c2097f50, pcylinders=pcylinders@entry=0x7ffefa351144, pheads=pheads@entry=0x7ffefa351148, psectors=psectors@entry=0x7ffefa35114c) at hw/block/hd-geometry.c:71 buf = "\017\000\000\000\000\000\000\000@\000\000\000\000\000\000\000\240\373\377\377\377\377\377\377\000\000\000\000\000\000\000\000\002\000\000\000\060\000\000\000ɲ%\277\206U\000\000\000\000\000\000\000\000\000\000\300\301;\277\206U\000\000\255\275;\277\206U\000\000w\000\000\000|\000\000\000\016\000\000\000\000\000\000\000\000\213ְ.\361\224\273 \000\000\000\000\000\000\000\030\210\ah\254\177\000\000\002\000\000\000\206U\000\000\020\020a\300\206U\000\000\020\000\000\000\000\000\000\000\340\017\065\372\376\177\000\000\240\017\065\372\376\177\000\000\000\213ְ.\361\224\273\002\000\000\000\060", '\000' <repeats 11 times>, "\340\017\065\372\376\177\000\000\000\213ְ.\361\224\273\000\000\000\000\000\000\000\000\025\000\000\000\000\000\000\000"... heads = <optimized out> sectors = <optimized out> cylinders = <optimized out> p = <optimized out> nr_sects = <optimized out> --Type <RET> for more, q to quit, c to continue without paging-- nb_sectors = 41943040 #9 0x00005586bf08c91f in hd_geometry_guess (blk=0x5586c2097f50, pcyls=pcyls@entry=0x5586c11fa63c, pheads=pheads@entry=0x5586c11fa640, psecs=psecs@entry=0x5586c11fa644, ptrans=ptrans@entry=0x0) at hw/block/hd-geometry.c:136 cylinders = 0 heads = 119 secs = 124 translation = <optimized out> geo = {heads = 32684, sectors = 14, cylinders = 0} #10 0x00005586bf08c50f in blkconf_geometry (conf=conf@entry=0x5586c11fa620, ptrans=ptrans@entry=0x0, cyls_max=cyls_max@entry=65535, heads_max=heads_max@entry=255, secs_max=secs_max@entry=255, errp=errp@entry=0x7ffefa351210) at hw/block/block.c:99 __func__ = "blkconf_geometry" #11 0x00005586bf0fe869 in scsi_realize (dev=0x5586c11fa590, errp=0x7ffefa351210) at hw/scsi/scsi-disk.c:2343 s = 0x5586c11fa590 __func__ = "scsi_realize" #12 0x00005586bf105d37 in scsi_device_realize (errp=0x7ffefa351210, s=0x5586c11fa590) at hw/scsi/scsi-bus.c:54 sc = <optimized out> __func__ = "scsi_device_realize" dev = 0x5586c11fa590 __func__ = "scsi_qdev_realize" bus = 0x5586c1df6740 d = <optimized out> local_err = 0x0 __PRETTY_FUNCTION__ = "scsi_qdev_realize" #13 0x00005586bf105d37 in scsi_qdev_realize (qdev=<optimized out>, errp=0x7ffefa351270) at hw/scsi/scsi-bus.c:204 dev = 0x5586c11fa590 __func__ = "scsi_qdev_realize" bus = 0x5586c1df6740 d = <optimized out> local_err = 0x0 __PRETTY_FUNCTION__ = "scsi_qdev_realize" #14 0x00005586bf095c9c in device_set_realized (obj=<optimized out>, value=<optimized out>, errp=0x7ffefa351398) at hw/core/qdev.c:826 dev = 0x5586c11fa590 __func__ = "device_set_realized" dc = 0x5586c06a2e30 hotplug_ctrl = 0x5586c1df6530 bus = <optimized out> local_err = 0x0 unattached_parent = false unattached_count = 34 #15 0x00005586bf18e44b in property_set_bool (obj=0x5586c11fa590, v=<optimized out>, name=<optimized out>, opaque=0x5586c2096030, errp=0x7ffefa351398) at qom/object.c:1991 prop = 0x5586c2096030 value = true local_err = 0x0 #16 0x00005586bf192723 in object_property_set_qobject (obj=0x5586c11fa590, value=<optimized out>, name=0x5586bf32419d "realized", errp=0x7ffefa351398) at qom/qom-qobject.c:27 v = 0x5586c2097e70 #17 0x00005586bf190119 in object_property_set_bool (obj=0x5586c11fa590, value=<optimized out>, name=0x5586bf32419d "realized", errp=0x7ffefa351398) at qom/object.c:1249 --Type <RET> for more, q to quit, c to continue without paging-- qbool = 0x5586c2096aa0 #18 0x00005586bf0443e4 in qdev_device_add (opts=0x5586c06852e0, errp=<optimized out>) at qdev-monitor.c:641 dc = 0x5586c06a2e30 driver = 0x5586c0683470 "scsi-hd" path = <optimized out> dev = 0x5586c11fa590 bus = <optimized out> err = 0x0 __func__ = "qdev_device_add" #19 0x00005586bf046063 in device_init_func (opaque=<optimized out>, opts=<optimized out>, errp=<optimized out>) at vl.c:2300 dev = <optimized out> #20 0x00005586bf265dd2 in qemu_opts_foreach (list=<optimized out>, func=0x5586bf046050 <device_init_func>, opaque=0x0, errp=0x5586bfabb6a0 <error_fatal>) at util/qemu-option.c:1171 loc = {kind = LOC_CMDLINE, num = 2, ptr = 0x7ffefa351970, prev = 0x5586bfabb6c0 <std_loc>} opts = 0x5586c06852e0 rc = 0 __PRETTY_FUNCTION__ = "qemu_opts_foreach" #21 0x00005586bef08245 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4580 i = <optimized out> snapshot = 0 linux_boot = <optimized out> initrd_filename = 0x0 kernel_filename = 0x0 kernel_cmdline = <optimized out> boot_order = 0x5586c0683920 "cdn" boot_once = <optimized out> ds = <optimized out> opts = <optimized out> machine_opts = <optimized out> icount_opts = <optimized out> accel_opts = 0x0 olist = <optimized out> optind = 77 optarg = 0x7ffefa353548 "scsi-hd,id=data1,drive=drive_image1" loadvm = 0x0 machine_class = <optimized out> cpu_model = 0x7ffefa3533fb "Broadwell,+kvm_pv_unhalt" vga_model = 0x0 qtest_chrdev = 0x0 qtest_log = 0x0 incoming = 0x0 userconfig = <optimized out> nographic = false display_remote = <optimized out> log_mask = <optimized out> log_file = <optimized out> trace_file = <optimized out> maxram_size = 15032385536 --Type <RET> for more, q to quit, c to continue without paging-- ram_slots = 0 vmstate_dump_file = 0x0 main_loop_err = 0x0 err = 0x0 list_data_dirs = <optimized out> dir = <optimized out> bdo_queue = {sqh_first = 0x0, sqh_last = 0x7ffefa3515b0} __func__ = "main
(In reply to aihua liang from comment #0) ...... > Qemu core dump with error msg: > (qemu) qemu: qemu_mutex_unlock_impl: Operation not permitted ...... The assertion failure message from hmp output is the same as that in bz1656276, so I am wondering it's also data plane related. And the bug is not storage vm migration related, so I removed the tag and polarion test case tracker. > Coredump file in: > > 10.73.194.27:/vol/s2coredump/bug1662508/core.qemu-kvm.0. > cac53b7105b64c4aa51274721b3d29b2.22212.1546055319000000.lz4
Reproduced upstream with this simplified reproducer: $ qemu-system-x86_64 -nodefaults -display none -S -blockdev node-name=drive_image1,driver=file,filename=tmp.qcow2 -object iothread,id=iothread0 -device virtio-scsi-pci,id=virtio_scsi_pci0,iothread=iothread0 -device scsi-hd,id=image1,drive=drive_image1 -device scsi-hd,id=data1,drive=drive_image1 qemu: qemu_mutex_unlock_impl: Operation not permitted Aborted (core dumped)
We release an AioContext we didn't acquire: ### aio_context_acquire 0x55a84c1881a0 ### aio_context_release 0x55a84c1881a0 ### aio_context_acquire 0x55a84c1881a0 ### aio_context_release 0x55a84c1881a0 ### aio_context_acquire 0x55a84c1881a0 ### aio_context_release 0x55a84c1881a0 ### aio_context_acquire 0x55a84c1881a0 ### aio_context_release 0x55a84c1881a0 ### aio_context_acquire 0x55a84c197490 ### aio_context_acquire 0x55a84c197490 ### aio_context_release 0x55a84c197490 ### aio_context_release 0x55a84c197490 ### aio_context_release 0x55a84c197490 qemu: qemu_mutex_unlock_impl: Operation not permitted ### aio_context_acquire 0x55a84c197490 Aborted (core dumped) Also note the recursive acquire, which may or may not be healthy. Output is from the same obvious debugging patch as in bug 1656276.
Possible upstream patch: Subject: [PATCH 0/6] Acquire the AioContext during _realize() Message-Id: <cover.1547132561.git.berto> https://lists.nongnu.org/archive/html/qemu-devel/2019-01/msg01967.html The patch also fixes bug 1656276 for me. The two bugs are definitely related, but I'm not yet sure they're actually duplicates.
Setting ITR=8.0.0 as the patch was posted in time for 8.0.0 We still need QA_ACK tough.
Fix included in qemu-kvm-3.1.0-13.module+el8+2783+15cec5ae
Reproduced this bug with below cmd lines on qemu-kvm-3.1.0-12.module+el8+2778+279c3c9e. And retested on qemu-kvm-3.1.0-13.module+el8+2783+15cec5ae, not hit this issue, qemu will prompt conflicts as below. So set status to VERIFIED. # sh bug_1662508.sh QEMU 3.1.0 monitor - type 'help' for more information (qemu) qemu-kvm: -device scsi-hd,id=data1,drive=drive_image1: Conflicts with use by image1 as 'root', which does not allow 'write' on drive_image1 Versions: kernel-4.18.0-67.el8.x86_64 qemu-kvm-3.1.0-13.module+el8+2783+15cec5ae /usr/libexec/qemu-kvm \ -name 'avocado-vt-vm1' \ -machine q35 \ -nodefaults \ -device VGA,bus=pcie.0,addr=0x1 \ -device pcie-root-port,id=pcie_root_port_0,slot=2,chassis=2,addr=0x2,bus=pcie.0 \ -device pcie-root-port,id=pcie_root_port_1,slot=3,chassis=3,addr=0x3,bus=pcie.0 \ -device pcie-root-port,id=pcie_root_port_2,slot=4,chassis=4,addr=0x4,bus=pcie.0 \ -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20181228-045409-0zhg5aer,server,nowait \ -mon chardev=qmp_id_qmpmonitor1,mode=control \ -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20181228-045409-0zhg5aer,server,nowait \ -mon chardev=qmp_id_catch_monitor,mode=control \ -device pvpanic,ioport=0x505,id=idjsOqD4 \ -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20181228-045409-0zhg5aer,server,nowait \ -device isa-serial,chardev=serial_id_serial0 \ -chardev socket,id=seabioslog_id_20181228-045409-0zhg5aer,path=/var/tmp/seabios-20181228-045409-0zhg5aer,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20181228-045409-0zhg5aer,iobase=0x402 \ -device pcie-root-port,id=pcie.0-root-port-5,slot=5,chassis=5,addr=0x5,bus=pcie.0 \ -device qemu-xhci,id=usb1,bus=pcie.0-root-port-5,addr=0x0 \ -device pcie-root-port,id=pcie.0-root-port-6,slot=6,chassis=6,addr=0x6,bus=pcie.0 \ -object iothread,id=iothread0 \ -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie.0-root-port-6,addr=0x0,iothread=iothread0 \ -blockdev driver=file,node-name=driveimage1,filename=/home/kvm_autotest_root/images/rhel80-64-virtio-scsi-install-test-.qcow2 \ -blockdev node-name=drive_image1,file=driveimage1,driver=qcow2 \ -device scsi-hd,id=image1,drive=drive_image1 \ -device pcie-root-port,id=pcie.0-root-port-7,slot=7,chassis=7,addr=0x7,bus=pcie.0 \ -device virtio-net-pci,mac=9a:0f:10:11:12:13,id=idPrzVst,vectors=4,netdev=id5uShpn,bus=pcie.0-root-port-7,addr=0x0 \ -netdev tap,id=id5uShpn,vhost=on \ -m 14336 \ -smp 10,maxcpus=10,cores=5,threads=1,sockets=2 \ -cpu 'SandyBridge',+kvm_pv_unhalt \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -vnc :0 \ -rtc base=utc,clock=host,driftfix=slew \ -boot order=cdn,once=c,menu=off,strict=off \ -enable-kvm \ -monitor stdio \ -blockdev driver=file,node-name=drivedata1,filename=/home/kvm_autotest_root/images/data.raw \ -blockdev node-name=drive_data1,file=drivedata1,driver=raw \ -device scsi-hd,id=data1,drive=drive_image1 \
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1293