Bug 2033626

Summary: Qemu core dump when start guest with nbd node or do block jobs to nbd node
Product: Red Hat Enterprise Linux 9 Reporter: aihua liang <aliang>
Component: qemu-kvmAssignee: Hanna Czenczek <hreitz>
qemu-kvm sub component: NBD QA Contact: aihua liang <aliang>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: urgent CC: coli, hreitz, jinzhao, juzhang, kkiwi, mrezanin, timao, virt-maint
Version: 9.0Keywords: Regression, Triaged
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-6.2.0-9.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2035185 (view as bug list) Environment:
Last Closed: 2022-05-17 12:25:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1990835, 2035185    

Description aihua liang 2021-12-17 13:25:51 UTC
Description of problem:
Qemu core dump when block jobs on nbd node were blocked by firewall or server down

Version-Release number of selected component (if applicable):
 kernel version:5.14.0-30.el9.x86_64
 qemu-kvm version:qemu-kvm-6.2.0-1.el9


How reproducible:
100% by auto

Steps to Reproduce:
 1.Create data disk image.
   #qemu-img create -f qcow2 /root/avocado/data/avocado-vt/data.qcow2 2G

 2.Expose data image
   #/usr/bin/qemu-nbd    -f qcow2 -t  -p 10810   --fork --pid-file /tmp/avocado_0e1jqoko/data_nbd_server-20211217-075948-p12l.pid  /root/avocado/data/avocado-vt/data.qcow2

 3.Start guest with the exposed data disk:
   /usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35,memory-backend=mem-machine_mem \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 30720 \
    -object memory-backend-ram,size=30720M,id=mem-machine_mem  \
    -smp 10,maxcpus=10,cores=5,threads=1,dies=1,sockets=2  \
    -cpu 'Cascadelake-Server-noTSX',+kvm_pv_unhalt \
    -chardev socket,wait=off,path=/tmp/avocado_0e1jqoko/monitor-qmpmonitor1-20211217-075949-ZwO9C7pE,server=on,id=qmp_id_qmpmonitor1  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,wait=off,path=/tmp/avocado_0e1jqoko/monitor-catch_monitor-20211217-075949-ZwO9C7pE,server=on,id=qmp_id_catch_monitor  \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=id9Na7Yp \
    -chardev socket,wait=off,path=/tmp/avocado_0e1jqoko/serial-serial0-20211217-075949-ZwO9C7pE,server=on,id=chardev_serial0 \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20211217-075949-ZwO9C7pE,path=/tmp/avocado_0e1jqoko/seabios-20211217-075949-ZwO9C7pE,server=on,wait=off \
    -device isa-debugcon,chardev=seabioslog_id_20211217-075949-ZwO9C7pE,iobase=0x402 \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -object iothread,id=iothread0 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0,iothread=iothread0 \
    -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/images/rhel900-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
    -blockdev node-name=nbd_nbddata,driver=nbd,auto-read-only=on,discard=unmap,server.type=inet,server.host=dell-per440-09.lab.eng.pek2.redhat.com,server.port=10810,reconnect-delay=30,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_nbddata,driver=raw,read-only=off,cache.direct=on,cache.no-flush=off,file=nbd_nbddata \
    -device scsi-hd,id=nbddata,drive=drive_nbddata,write-cache=on \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:ed:fc:56:77:68,id=idY9yr3Q,netdev=id1qEUjK,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=id1qEUjK,vhost=on,vhostfd=22,fd=19  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5

4. Continue vm
   {'execute': 'cont', 'id': 'CE1lLVef'}

Actual results:

After step4, qemu core dump with info:
 [qemu output] qemu-kvm: ../io/channel.c:484: void qio_channel_restart_read(void *): Assertion `qemu_get_current_aio_context() == qemu_coroutine_get_aio_context(co)' failed.
 [qemu output] /tmp/aexpect_SwITdppJ/aexpect-y1zy2kcy.sh: line 1: 236656 Aborted                 (core dumped) MALLOC_PERTURB_=1 /usr/libexec/qemu-kvm -S -name 'avocado-vt-vm1' -sandbox on -machine q35,memory-backend=mem-machine_mem -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0 ...


Detailed core dump info:
  coredumpctl debug 236656
           PID: 236656 (qemu-kvm)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 6 (ABRT)
     Timestamp: Fri 2021-12-17 08:00:02 EST (8min ago)
  Command Line: /usr/libexec/qemu-kvm -S -name avocado-vt-vm1 -sandbox on -machine q35,memory-backend=mem-machine_mem -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0 -nodefaults -device VGA,bus=pcie.0,addr=0x2 -m 30720 -object memory-backend-ram,size=30720M,id=mem-machine_mem -smp 10,maxcpus=10,cores=5,threads=1,dies=1,sockets=2 -cpu Cascadelake-Server-noTSX,+kvm_pv_unhalt -chardev socket,wait=off,path=/tmp/avocado_0e1jqoko/monitor-qmpmonitor1-20211217-075949-ZwO9C7pE,server=on,id=qmp_id_qmpmonitor1 -mon chardev=qmp_id_qmpmonitor1,mode=control -chardev socket,wait=off,path=/tmp/avocado_0e1jqoko/monitor-catch_monitor-20211217-075949-ZwO9C7pE,server=on,id=qmp_id_catch_monitor -mon chardev=qmp_id_catch_monitor,mode=control -device pvpanic,ioport=0x505,id=id9Na7Yp -chardev socket,wait=off,path=/tmp/avocado_0e1jqoko/serial-serial0-20211217-075949-ZwO9C7pE,server=on,id=chardev_serial0 -device isa-serial,id=serial0,chardev=chardev_serial0 -chardev socket,id=seabioslog_id_20211217-075949-ZwO9C7pE,path=/tmp/avocado_0e1jqoko/seabios-20211217-075949-ZwO9C7pE,server=on,wait=off -device isa-debugcon,chardev=seabioslog_id_20211217-075949-ZwO9C7pE,iobase=0x402 -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -object iothread,id=iothread0 -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0,iothread=iothread0 -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/images/rhel900-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 -device scsi-hd,id=image1,drive=drive_image1,write-cache=on -blockdev node-name=nbd_nbddata,driver=nbd,auto-read-only=on,discard=unmap,server.type=inet,server.host=dell-per440-09.lab.eng.pek2.redhat.com,server.port=10810,reconnect-delay=30,cache.direct=on,cache.no-flush=off -blockdev node-name=drive_nbddata,driver=raw,read-only=off,cache.direct=on,cache.no-flush=off,file=nbd_nbddata -device scsi-hd,id=nbddata,drive=drive_nbddata,write-cache=on -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 -device virtio-net-pci,mac=9a:ed:fc:56:77:68,id=idY9yr3Q,netdev=id1qEUjK,bus=pcie-root-port-3,addr=0x0 -netdev tap,id=id1qEUjK,vhost=on,vhostfd=22,fd=19 -vnc :0 -rtc base=utc,clock=host,driftfix=slew -boot menu=off,order=cdn,once=c,strict=off -enable-kvm -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5
    Executable: /usr/libexec/qemu-kvm
 Control Group: /user.slice/user-0.slice/session-1.scope
          Unit: session-1.scope
         Slice: user-0.slice
       Session: 1
     Owner UID: 0 (root)
       Boot ID: 0107346256b84e8ebd9076ae0ca431c0
    Machine ID: 1437cd218cca45cbb2059307f6958ac6
      Hostname: dell-per440-09.lab.eng.pek2.redhat.com
       Storage: /var/lib/systemd/coredump/core.qemu-kvm.0.0107346256b84e8ebd9076ae0ca431c0.236656.1639746002000000.zst (present)
     Disk Size: 123.9M
       Message: Process 236656 (qemu-kvm) of user 0 dumped core.
                
                Found module linux-vdso.so.1 with build-id: 0417c877078abdfaf5ca0d3d449ce6699e5a520c
                Found module libpcre2-8.so.0 with build-id: a5dafbac0243bdc2e3b5ab9a07d61d4b32f86927
                Found module liblzma.so.5 with build-id: 66b7fca8b63a61d19c5ded440ee6a8bf05dcafa5
                Found module libzstd.so.1 with build-id: 415c738420cd82e182b99fc4c6e6821098e80ce4
                Found module libselinux.so.1 with build-id: c1d23d40bcf1fae271256fcbe90a46ff6dbcce20
                Found module libkmod.so.2 with build-id: 722d3027106c561018756766be8bed5215ed0f31
                Found module libuuid.so.1 with build-id: da656ae81596c151559892375d730bd35f76048c
                Found module libudev.so.1 with build-id: 7c275efc762b4b5a63016164ffd5a53962effbb1
                Found module libcrypto.so.3 with build-id: e3a618644eec4fb65c7cd059c42592a686191f72
                Found module libkeyutils.so.1 with build-id: 83c6539bd0d3140678ba836b8baa1b215efa2632
                Found module libkrb5support.so.0 with build-id: 22e9f06a5b088d401df9153948492077b4e198af
                Found module libffi.so.8 with build-id: 48e3675db4765a2e42729140922e11a10016f7ab
                Found module libpcre.so.1 with build-id: cffb947bcc416dca3cd249cdb0a1c6f614549c30
                Found module libnl-route-3.so.200 with build-id: 2f297144194e194e379c1c31dcd245220764a488
                Found module libnl-3.so.200 with build-id: 5690fe17968270ad4ee00fdafe15cf0a4db9c7ed
                Found module libdaxctl.so.1 with build-id: c46dc4ef727b75a311e76e41940451a90b9f1080
                Found module libndctl.so.6 with build-id: 8c52e205e3982b331bb9683f1a35d53bb1c11c50
                Found module libresolv.so.2 with build-id: eaebc6c6a15dcc8af04ea904eb33aeb5557fcafd
                Found module libcom_err.so.2 with build-id: deb039f83bae82e65be49dd97ac3d900a053e265
                Found module libk5crypto.so.3 with build-id: eca094d4bc539db337447b773462de7b592e785b
                Found module libkrb5.so.3 with build-id: a3fec09569ad99121e79e2d73b0d446169965d26
                Found module libgssapi_krb5.so.2 with build-id: a2daa1c10270547f24b82fcb0034e84853d35344
                Found module libcrypt.so.2 with build-id: 6ce4e5eb200e61d07398af52f8bcb316cf8466e0
                Found module libgmp.so.10 with build-id: 754b87178e725c166a68353f043ba7638e0723e1
                Found module libhogweed.so.6 with build-id: 21c32760f1f50b9d69b43e966b93fb3de8baa076
                Found module libnettle.so.8 with build-id: 3eda844d5a5df2443a281158b0d7ca6e81483b59
                Found module libtasn1.so.6 with build-id: 3d3a2f6f0d4a70919496afe25e329abd189b7882
                Found module libunistring.so.2 with build-id: 15e34cdfafa3547f9c700489b842ceb86f6fb73e
                Found module libidn2.so.0 with build-id: 958c50fc94ecb196b24f3619762e7ec3f28a5b40
                Found module libp11-kit.so.0 with build-id: b03163bf5b98313c7347e7270627a6b2dcd35060
                Found module libstdc++.so.6 with build-id: 66b742a0a51ccd8a67fdf59a31663b0cd28dfcf2
                Found module ld-linux-x86-64.so.2 with build-id: 2b7f1f415256e40ab15e5808c73eb7acf3c73ea3
                Found module libc.so.6 with build-id: 723a5d3072f3b79f3716880303439e11f7661fc5
                Found module libgcc_s.so.1 with build-id: a1a28a65f942690a11041922ba11c16f716565e2
                Found module libm.so.6 with build-id: c58412f6a7fffa4068614708cdb5b15f2ba49c0a
                Found module libaio.so.1 with build-id: 9b75d21282f8e17ddfa06aff78dae4f8dcce4106
                Found module libgmodule-2.0.so.0 with build-id: 952459b04fb217bc3e4a17b902759cf66c89a95d
                Found module libglib-2.0.so.0 with build-id: 0006a4eed3e835c7d7812e36e5d720c1a6eb8689
                Found module libslirp.so.0 with build-id: fa9d492d6d050c6f5cb906a3f087a8a63e4201e2
                Found module libibverbs.so.1 with build-id: 3e19d7349fe9ea4c79debeb8716df7000ab13159
                Found module librdmacm.so.1 with build-id: 4f6798f3f4164a4aa4d75d8a890c760e2835c495
                Found module libnuma.so.1 with build-id: a0c23354fdb39f51ec9e31aeae27a9f8ee034500
                Found module libfdt.so.1 with build-id: 31a56e0009a8824c7a09267c8205034c91cb4095
                Found module libseccomp.so.2 with build-id: 5a5b4ee3bb99ee0378bd937fdef7a7b7255d0428
                Found module libpmem.so.1 with build-id: 084e72da2642981fe75b446293655ae4a68c0663
                Found module libsasl2.so.3 with build-id: 2864d8e1600f3b46016fd62c44269e92c5ec0ca1
                Found module libgnutls.so.30 with build-id: b08009ada925278615f5592e982e0776c06e065e
                Found module libpng16.so.16 with build-id: 698f67f36c58fa443c6ba891e4a0f106553a0452
                Found module libz.so.1 with build-id: 62196acf9f75625acb09840435bb396c62829e12
                Found module liblzo2.so.2 with build-id: c42a1f1fd2f73000efcbe3d2fb6ad196170fd1cf
                Found module libsnappy.so.1 with build-id: 83553ef5fb4b9e530a3f05b68d92f320a6d53772
                Found module libpixman-1.so.0 with build-id: 4420cf633fcf726fc8cef2dd84d817083afe13e5
                Found module qemu-kvm with build-id: debf9ea821f74b76b75f78c5558eba8ca491b1b5
                Stack trace of thread 236656:
                #0  0x00007fef0ee7b7fc __pthread_kill_implementation (libc.so.6 + 0x8f7fc)
                #1  0x00007fef0ee2e676 raise (libc.so.6 + 0x42676)
                #2  0x00007fef0ee187d3 abort (libc.so.6 + 0x2c7d3)
                #3  0x00007fef0ee186fb __assert_fail_base.cold (libc.so.6 + 0x2c6fb)
                #4  0x00007fef0ee27396 __assert_fail (libc.so.6 + 0x3b396)
                #5  0x0000556d867a4791 qio_channel_restart_read (qemu-kvm + 0x760791)
                #6  0x0000556d869a147f aio_dispatch_handler (qemu-kvm + 0x95d47f)
                #7  0x0000556d869a132c aio_dispatch (qemu-kvm + 0x95d32c)
                #8  0x0000556d869bb522 aio_ctx_dispatch (qemu-kvm + 0x977522)
                #9  0x00007fef0f14df6f g_main_context_dispatch (libglib-2.0.so.0 + 0x54f6f)
                #10 0x0000556d869c81e3 main_loop_wait (qemu-kvm + 0x9841e3)
                #11 0x0000556d86647d37 qemu_main_loop (qemu-kvm + 0x603d37)
                #12 0x0000556d863a06f2 main (qemu-kvm + 0x35c6f2)
                #13 0x00007fef0ee19560 __libc_start_call_main (libc.so.6 + 0x2d560)
                #14 0x00007fef0ee1960c __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2d60c)
                #15 0x0000556d8639fdb5 _start (qemu-kvm + 0x35bdb5)
                
                Stack trace of thread 236664:
                #0  0x00007fef0ee76870 __GI___lll_lock_wait (libc.so.6 + 0x8a870)
                #1  0x00007fef0ee7cea2 __pthread_mutex_lock.5 (libc.so.6 + 0x90ea2)
                #2  0x0000556d869a521f qemu_mutex_lock_impl (qemu-kvm + 0x96121f)
                #3  0x0000556d8663c78a flatview_read_continue (qemu-kvm + 0x5f878a)
                #4  0x0000556d8663c62e flatview_read (qemu-kvm + 0x5f862e)
                #5  0x0000556d8663f621 address_space_read_full (qemu-kvm + 0x5fb621)
                #6  0x0000556d8674c695 kvm_cpu_exec (qemu-kvm + 0x708695)
                #7  0x0000556d8674e5fa kvm_vcpu_thread_fn (qemu-kvm + 0x70a5fa)
                #8  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #9  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #10 0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)
                
                Stack trace of thread 236668:
                #0  0x00007fef0eef33fb ioctl (libc.so.6 + 0x1073fb)
                #1  0x0000556d8674cd77 kvm_vcpu_ioctl (qemu-kvm + 0x708d77)
                #2  0x0000556d8674c601 kvm_cpu_exec (qemu-kvm + 0x708601)
                #3  0x0000556d8674e5fa kvm_vcpu_thread_fn (qemu-kvm + 0x70a5fa)
                #4  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #5  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)
                
                Stack trace of thread 236665:
                #0  0x00007fef0eef33fb ioctl (libc.so.6 + 0x1073fb)
                #1  0x0000556d8674cd77 kvm_vcpu_ioctl (qemu-kvm + 0x708d77)
                #2  0x0000556d8674c601 kvm_cpu_exec (qemu-kvm + 0x708601)
                #3  0x0000556d8674e5fa kvm_vcpu_thread_fn (qemu-kvm + 0x70a5fa)
                #4  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #5  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)
                
                Stack trace of thread 236670:
                #0  0x00007fef0eef33fb ioctl (libc.so.6 + 0x1073fb)
                #1  0x0000556d8674cd77 kvm_vcpu_ioctl (qemu-kvm + 0x708d77)
                #2  0x0000556d8674c601 kvm_cpu_exec (qemu-kvm + 0x708601)
                #3  0x0000556d8674e5fa kvm_vcpu_thread_fn (qemu-kvm + 0x70a5fa)
                #4  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #5  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)
                
                Stack trace of thread 236657:
                #0  0x00007fef0eef6fbd syscall (libc.so.6 + 0x10afbd)
                #1  0x0000556d869a5def qemu_event_wait (qemu-kvm + 0x961def)
                #2  0x0000556d869b1eb1 call_rcu_thread (qemu-kvm + 0x96deb1)
                #3  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #4  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #5  0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)
                
                Stack trace of thread 236667:
                #0  0x00007fef0eef33fb ioctl (libc.so.6 + 0x1073fb)
                #1  0x0000556d8674cd77 kvm_vcpu_ioctl (qemu-kvm + 0x708d77)
                #2  0x0000556d8674c601 kvm_cpu_exec (qemu-kvm + 0x708601)
                #3  0x0000556d8674e5fa kvm_vcpu_thread_fn (qemu-kvm + 0x70a5fa)
                #4  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #5  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)
                
                Stack trace of thread 236663:
                #0  0x00007fef0eef187f __poll (libc.so.6 + 0x10587f)
                #1  0x00007fef0f1a256c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa956c)
                #2  0x00007fef0f14d533 g_main_loop_run (libglib-2.0.so.0 + 0x54533)
                #3  0x0000556d867cc924 iothread_run (qemu-kvm + 0x788924)
                #4  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #5  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)
                
                Stack trace of thread 236658:
                #0  0x00007fef0eef197e ppoll (libc.so.6 + 0x10597e)
                #1  0x0000556d869a2852 fdmon_poll_wait (qemu-kvm + 0x95e852)
                #2  0x0000556d869a1fe4 aio_poll (qemu-kvm + 0x95dfe4)
                #3  0x0000556d867cc906 iothread_run (qemu-kvm + 0x788906)
                #4  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #5  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)
                
                Stack trace of thread 236671:
                #0  0x00007fef0eef33fb ioctl (libc.so.6 + 0x1073fb)
                #1  0x0000556d8674cd77 kvm_vcpu_ioctl (qemu-kvm + 0x708d77)
                #2  0x0000556d8674c601 kvm_cpu_exec (qemu-kvm + 0x708601)
                #3  0x0000556d8674e5fa kvm_vcpu_thread_fn (qemu-kvm + 0x70a5fa)
                #4  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #5  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)
                
                Stack trace of thread 236731:
                #0  0x00007fef0ee7670a __futex_abstimed_wait_common (libc.so.6 + 0x8a70a)
                #1  0x00007fef0ee81813 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0x95813)
                #2  0x0000556d869a5b0b qemu_sem_timedwait (qemu-kvm + 0x961b0b)
                #3  0x0000556d869cc6b6 worker_thread (qemu-kvm + 0x9886b6)
                #4  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #5  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)
                
                Stack trace of thread 236730:
                #0  0x00007fef0ee7670a __futex_abstimed_wait_common (libc.so.6 + 0x8a70a)
                #1  0x00007fef0ee81813 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0x95813)
                #2  0x0000556d869a5b0b qemu_sem_timedwait (qemu-kvm + 0x961b0b)
                #3  0x0000556d869cc6b6 worker_thread (qemu-kvm + 0x9886b6)
                #4  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #5  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)
                
                Stack trace of thread 236675:
                #0  0x00007fef0ee7670a __futex_abstimed_wait_common (libc.so.6 + 0x8a70a)
                #1  0x00007fef0ee78e80 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x8ce80)
                #2  0x0000556d869a56ff qemu_cond_wait_impl (qemu-kvm + 0x9616ff)
                #3  0x0000556d863dd657 vnc_worker_thread (qemu-kvm + 0x399657)
                #4  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #5  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)
                
                Stack trace of thread 236666:
                #0  0x00007fef0eef33fb ioctl (libc.so.6 + 0x1073fb)
                #1  0x0000556d8674cd77 kvm_vcpu_ioctl (qemu-kvm + 0x708d77)
                #2  0x0000556d8674c601 kvm_cpu_exec (qemu-kvm + 0x708601)
                #3  0x0000556d8674e5fa kvm_vcpu_thread_fn (qemu-kvm + 0x70a5fa)
                #4  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #5  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)
                
                Stack trace of thread 236672:
                #0  0x00007fef0eef33fb ioctl (libc.so.6 + 0x1073fb)
                #1  0x0000556d8674cd77 kvm_vcpu_ioctl (qemu-kvm + 0x708d77)
                #2  0x0000556d8674c601 kvm_cpu_exec (qemu-kvm + 0x708601)
                #3  0x0000556d8674e5fa kvm_vcpu_thread_fn (qemu-kvm + 0x70a5fa)
                #4  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #5  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)
                
                Stack trace of thread 236669:
                #0  0x00007fef0eef33fb ioctl (libc.so.6 + 0x1073fb)
                #1  0x0000556d8674cd77 kvm_vcpu_ioctl (qemu-kvm + 0x708d77)
                #2  0x0000556d8674c601 kvm_cpu_exec (qemu-kvm + 0x708601)
                #3  0x0000556d8674e5fa kvm_vcpu_thread_fn (qemu-kvm + 0x70a5fa)
                #4  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #5  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)
                
                Stack trace of thread 236673:
                #0  0x00007fef0eef33fb ioctl (libc.so.6 + 0x1073fb)
                #1  0x0000556d8674cd77 kvm_vcpu_ioctl (qemu-kvm + 0x708d77)
                #2  0x0000556d8674c601 kvm_cpu_exec (qemu-kvm + 0x708601)
                #3  0x0000556d8674e5fa kvm_vcpu_thread_fn (qemu-kvm + 0x70a5fa)
                #4  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #5  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)

Expected results:
 VM can start successfully with exposed nbd data image.


Additional info:
 Will attach core dump file later.

Comment 2 Klaus Heinrich Kiwi 2021-12-21 19:16:23 UTC
Eric, can you take this? I've classified it as high-priority as it's a crash in apparently a customer-exposed path, but let's discuss if you disagree.

Aihua Liang, can you identify if this is a new testcase, or a regression? In case of the latter, set the regression keyword and identify the last working qemu-kvm version?

Thanks

Comment 3 aihua liang 2021-12-22 02:05:16 UTC
(In reply to Klaus Heinrich Kiwi from comment #2)
> Eric, can you take this? I've classified it as high-priority as it's a crash
> in apparently a customer-exposed path, but let's discuss if you disagree.
> 
> Aihua Liang, can you identify if this is a new testcase, or a regression? In
> case of the latter, set the regression keyword and identify the last working
> qemu-kvm version?
> 
> Thanks

Hi, Klaus

  It's not a new case and its a regression bug. emu-kvm-6.1.0-8.el9 don't hit it.


BR,
Aliang

Comment 4 aihua liang 2021-12-24 05:47:56 UTC
Won't hit this issue when iothread not set.

Comment 5 Hanna Czenczek 2022-02-03 12:28:28 UTC
I believe this is the bug I described in BZ 1990835 comment 7, and which I’ve sent an RFC for (currently working on writing tests for a proper v1): https://lists.nongnu.org/archive/html/qemu-block/2022-01/msg00765.html

If you prefer, Eric, I could take this bug.  (I was about to create a new one for this issue anyway, and the duplicate detector pointed me towards this one, which is how I found it.)

Comment 6 Klaus Heinrich Kiwi 2022-02-07 14:32:03 UTC
(In reply to Hanna Reitz from comment #5)
> I believe this is the bug I described in BZ 1990835 comment 7, and which
> I’ve sent an RFC for (currently working on writing tests for a proper v1):
> https://lists.nongnu.org/archive/html/qemu-block/2022-01/msg00765.html
> 
> If you prefer, Eric, I could take this bug.  (I was about to create a new
> one for this issue anyway, and the duplicate detector pointed me towards
> this one, which is how I found it.)

Hanna, may I take you on your offer? If you have a scratch build that QE can test, please let us know. Thanks!

Comment 7 Hanna Czenczek 2022-02-07 16:51:37 UTC
As long as AArch64 isn’t needed (which seems kind of slow today), I can offer this: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=42850336

The x86-64 RPMs are here: http://download.eng.bos.redhat.com/brewroot/work/tasks/428/42850428/

(Sorry, I don’t have a repo yet, because the overall build won’t finish because of AArch64)

Could you test whether those fix the crash, Aliang?  Thanks!

Comment 8 aihua liang 2022-02-08 03:01:08 UTC
Test with qemu-kvm-6.2.0-7.el9.hreitz202202071714 by auto, don't hit this issue any more.
 (1/3) Host_RHEL.m9.u0.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.9.0.0.x86_64.io-github-autotest-qemu.blockdev_stream_remote_server_down.q35: PASS (144.42 s)
 (2/3) Host_RHEL.m9.u0.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.9.0.0.x86_64.io-github-autotest-qemu.blockdev_mirror_remote_server_down.q35: PASS (122.59 s)
 (3/3) Host_RHEL.m9.u0.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.9.0.0.x86_64.io-github-autotest-qemu.blockdev_commit_firewall.q35: PASS (321.08 s)

Comment 9 Hanna Czenczek 2022-02-08 13:38:13 UTC
Awesome, thanks!

Comment 10 Hanna Czenczek 2022-02-15 16:17:29 UTC
This fix has been merged upstream, and I’ve sent a backport for 8.6 (see BZ 2035185).  I’d love to send a backport for this BZ, too, but can’t, because the Brew build server seems to have some subscription problems.  If they persist, I suppose I’ll just send a backport without a Brew build.

Hanna

Comment 12 Yanan Fu 2022-02-21 03:25:48 UTC
QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass.

Comment 16 aihua liang 2022-02-21 10:26:08 UTC
Test with qemu-kvm-6.2.0-9.el9, not hit this issue any more.
 (1/3) Host_RHEL.m9.u0.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.9.0.0.x86_64.io-github-autotest-qemu.blockdev_stream_remote_server_down.q35: PASS (143.82 s)
 (2/3) Host_RHEL.m9.u0.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.9.0.0.x86_64.io-github-autotest-qemu.blockdev_mirror_remote_server_down.q35: PASS (119.67 s)
 (3/3) Host_RHEL.m9.u0.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.9.0.0.x86_64.io-github-autotest-qemu.blockdev_commit_firewall.q35: PASS (321.33 s)

Comment 17 aihua liang 2022-02-21 10:27:22 UTC
As comment 12 and comment 16, set bug's status to "VERIFIED".

Comment 19 errata-xmlrpc 2022-05-17 12:25:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: qemu-kvm), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2307