2033626 – Qemu core dump when start guest with nbd node or do block jobs to nbd node

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2033626 - Qemu core dump when start guest with nbd node or do block jobs to nbd node

Summary: Qemu core dump when start guest with nbd node or do block jobs to nbd node

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 9
Classification:	Red Hat
Component:	qemu-kvm
Sub Component:
Version:	9.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	urgent
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	Hanna Czenczek
QA Contact:	aihua liang
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1990835 2035185
TreeView+	depends on / blocked

Reported:	2021-12-17 13:25 UTC by aihua liang
Modified:	2022-05-17 12:30 UTC (History)
CC List:	8 users (show)
Fixed In Version:	qemu-kvm-6.2.0-9.el9
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Clones:	2035185 (view as bug list)
Environment:
Last Closed:	2022-05-17 12:25:11 UTC
Type:	Bug
Target Upstream Version:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Gitlab	redhat/centos-stream/src qemu-kvm merge_requests 74	None	None	None	2022-02-16 11:54:39 UTC
Red Hat Issue Tracker	RHELPLAN-106097	None	None	None	2021-12-17 13:30:11 UTC
Red Hat Product Errata	RHBA-2022:2307	None	None	None	2022-05-17 12:25:55 UTC

Description aihua liang 2021-12-17 13:25:51 UTC

Description of problem:
Qemu core dump when block jobs on nbd node were blocked by firewall or server down

Version-Release number of selected component (if applicable):
 kernel version:5.14.0-30.el9.x86_64
 qemu-kvm version:qemu-kvm-6.2.0-1.el9


How reproducible:
100% by auto

Steps to Reproduce:
 1.Create data disk image.
   #qemu-img create -f qcow2 /root/avocado/data/avocado-vt/data.qcow2 2G

 2.Expose data image
   #/usr/bin/qemu-nbd    -f qcow2 -t  -p 10810   --fork --pid-file /tmp/avocado_0e1jqoko/data_nbd_server-20211217-075948-p12l.pid  /root/avocado/data/avocado-vt/data.qcow2

 3.Start guest with the exposed data disk:
   /usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35,memory-backend=mem-machine_mem \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 30720 \
    -object memory-backend-ram,size=30720M,id=mem-machine_mem  \
    -smp 10,maxcpus=10,cores=5,threads=1,dies=1,sockets=2  \
    -cpu 'Cascadelake-Server-noTSX',+kvm_pv_unhalt \
    -chardev socket,wait=off,path=/tmp/avocado_0e1jqoko/monitor-qmpmonitor1-20211217-075949-ZwO9C7pE,server=on,id=qmp_id_qmpmonitor1  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,wait=off,path=/tmp/avocado_0e1jqoko/monitor-catch_monitor-20211217-075949-ZwO9C7pE,server=on,id=qmp_id_catch_monitor  \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=id9Na7Yp \
    -chardev socket,wait=off,path=/tmp/avocado_0e1jqoko/serial-serial0-20211217-075949-ZwO9C7pE,server=on,id=chardev_serial0 \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20211217-075949-ZwO9C7pE,path=/tmp/avocado_0e1jqoko/seabios-20211217-075949-ZwO9C7pE,server=on,wait=off \
    -device isa-debugcon,chardev=seabioslog_id_20211217-075949-ZwO9C7pE,iobase=0x402 \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -object iothread,id=iothread0 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0,iothread=iothread0 \
    -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/images/rhel900-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
    -blockdev node-name=nbd_nbddata,driver=nbd,auto-read-only=on,discard=unmap,server.type=inet,server.host=dell-per440-09.lab.eng.pek2.redhat.com,server.port=10810,reconnect-delay=30,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_nbddata,driver=raw,read-only=off,cache.direct=on,cache.no-flush=off,file=nbd_nbddata \
    -device scsi-hd,id=nbddata,drive=drive_nbddata,write-cache=on \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:ed:fc:56:77:68,id=idY9yr3Q,netdev=id1qEUjK,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=id1qEUjK,vhost=on,vhostfd=22,fd=19  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5

4. Continue vm
   {'execute': 'cont', 'id': 'CE1lLVef'}

Actual results:

After step4, qemu core dump with info:
 [qemu output] qemu-kvm: ../io/channel.c:484: void qio_channel_restart_read(void *): Assertion `qemu_get_current_aio_context() == qemu_coroutine_get_aio_context(co)' failed.
 [qemu output] /tmp/aexpect_SwITdppJ/aexpect-y1zy2kcy.sh: line 1: 236656 Aborted                 (core dumped) MALLOC_PERTURB_=1 /usr/libexec/qemu-kvm -S -name 'avocado-vt-vm1' -sandbox on -machine q35,memory-backend=mem-machine_mem -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0 ...


Detailed core dump info:
  coredumpctl debug 236656
           PID: 236656 (qemu-kvm)
           UID: 0 (root)
           GID: 0 (root)
        Signal: 6 (ABRT)
     Timestamp: Fri 2021-12-17 08:00:02 EST (8min ago)
  Command Line: /usr/libexec/qemu-kvm -S -name avocado-vt-vm1 -sandbox on -machine q35,memory-backend=mem-machine_mem -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0 -nodefaults -device VGA,bus=pcie.0,addr=0x2 -m 30720 -object memory-backend-ram,size=30720M,id=mem-machine_mem -smp 10,maxcpus=10,cores=5,threads=1,dies=1,sockets=2 -cpu Cascadelake-Server-noTSX,+kvm_pv_unhalt -chardev socket,wait=off,path=/tmp/avocado_0e1jqoko/monitor-qmpmonitor1-20211217-075949-ZwO9C7pE,server=on,id=qmp_id_qmpmonitor1 -mon chardev=qmp_id_qmpmonitor1,mode=control -chardev socket,wait=off,path=/tmp/avocado_0e1jqoko/monitor-catch_monitor-20211217-075949-ZwO9C7pE,server=on,id=qmp_id_catch_monitor -mon chardev=qmp_id_catch_monitor,mode=control -device pvpanic,ioport=0x505,id=id9Na7Yp -chardev socket,wait=off,path=/tmp/avocado_0e1jqoko/serial-serial0-20211217-075949-ZwO9C7pE,server=on,id=chardev_serial0 -device isa-serial,id=serial0,chardev=chardev_serial0 -chardev socket,id=seabioslog_id_20211217-075949-ZwO9C7pE,path=/tmp/avocado_0e1jqoko/seabios-20211217-075949-ZwO9C7pE,server=on,wait=off -device isa-debugcon,chardev=seabioslog_id_20211217-075949-ZwO9C7pE,iobase=0x402 -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -object iothread,id=iothread0 -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0,iothread=iothread0 -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/images/rhel900-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 -device scsi-hd,id=image1,drive=drive_image1,write-cache=on -blockdev node-name=nbd_nbddata,driver=nbd,auto-read-only=on,discard=unmap,server.type=inet,server.host=dell-per440-09.lab.eng.pek2.redhat.com,server.port=10810,reconnect-delay=30,cache.direct=on,cache.no-flush=off -blockdev node-name=drive_nbddata,driver=raw,read-only=off,cache.direct=on,cache.no-flush=off,file=nbd_nbddata -device scsi-hd,id=nbddata,drive=drive_nbddata,write-cache=on -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 -device virtio-net-pci,mac=9a:ed:fc:56:77:68,id=idY9yr3Q,netdev=id1qEUjK,bus=pcie-root-port-3,addr=0x0 -netdev tap,id=id1qEUjK,vhost=on,vhostfd=22,fd=19 -vnc :0 -rtc base=utc,clock=host,driftfix=slew -boot menu=off,order=cdn,once=c,strict=off -enable-kvm -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5
    Executable: /usr/libexec/qemu-kvm
 Control Group: /user.slice/user-0.slice/session-1.scope
          Unit: session-1.scope
         Slice: user-0.slice
       Session: 1
     Owner UID: 0 (root)
       Boot ID: 0107346256b84e8ebd9076ae0ca431c0
    Machine ID: 1437cd218cca45cbb2059307f6958ac6
      Hostname: dell-per440-09.lab.eng.pek2.redhat.com
       Storage: /var/lib/systemd/coredump/core.qemu-kvm.0.0107346256b84e8ebd9076ae0ca431c0.236656.1639746002000000.zst (present)
     Disk Size: 123.9M
       Message: Process 236656 (qemu-kvm) of user 0 dumped core.
                
                Found module linux-vdso.so.1 with build-id: 0417c877078abdfaf5ca0d3d449ce6699e5a520c
                Found module libpcre2-8.so.0 with build-id: a5dafbac0243bdc2e3b5ab9a07d61d4b32f86927
                Found module liblzma.so.5 with build-id: 66b7fca8b63a61d19c5ded440ee6a8bf05dcafa5
                Found module libzstd.so.1 with build-id: 415c738420cd82e182b99fc4c6e6821098e80ce4
                Found module libselinux.so.1 with build-id: c1d23d40bcf1fae271256fcbe90a46ff6dbcce20
                Found module libkmod.so.2 with build-id: 722d3027106c561018756766be8bed5215ed0f31
                Found module libuuid.so.1 with build-id: da656ae81596c151559892375d730bd35f76048c
                Found module libudev.so.1 with build-id: 7c275efc762b4b5a63016164ffd5a53962effbb1
                Found module libcrypto.so.3 with build-id: e3a618644eec4fb65c7cd059c42592a686191f72
                Found module libkeyutils.so.1 with build-id: 83c6539bd0d3140678ba836b8baa1b215efa2632
                Found module libkrb5support.so.0 with build-id: 22e9f06a5b088d401df9153948492077b4e198af
                Found module libffi.so.8 with build-id: 48e3675db4765a2e42729140922e11a10016f7ab
                Found module libpcre.so.1 with build-id: cffb947bcc416dca3cd249cdb0a1c6f614549c30
                Found module libnl-route-3.so.200 with build-id: 2f297144194e194e379c1c31dcd245220764a488
                Found module libnl-3.so.200 with build-id: 5690fe17968270ad4ee00fdafe15cf0a4db9c7ed
                Found module libdaxctl.so.1 with build-id: c46dc4ef727b75a311e76e41940451a90b9f1080
                Found module libndctl.so.6 with build-id: 8c52e205e3982b331bb9683f1a35d53bb1c11c50
                Found module libresolv.so.2 with build-id: eaebc6c6a15dcc8af04ea904eb33aeb5557fcafd
                Found module libcom_err.so.2 with build-id: deb039f83bae82e65be49dd97ac3d900a053e265
                Found module libk5crypto.so.3 with build-id: eca094d4bc539db337447b773462de7b592e785b
                Found module libkrb5.so.3 with build-id: a3fec09569ad99121e79e2d73b0d446169965d26
                Found module libgssapi_krb5.so.2 with build-id: a2daa1c10270547f24b82fcb0034e84853d35344
                Found module libcrypt.so.2 with build-id: 6ce4e5eb200e61d07398af52f8bcb316cf8466e0
                Found module libgmp.so.10 with build-id: 754b87178e725c166a68353f043ba7638e0723e1
                Found module libhogweed.so.6 with build-id: 21c32760f1f50b9d69b43e966b93fb3de8baa076
                Found module libnettle.so.8 with build-id: 3eda844d5a5df2443a281158b0d7ca6e81483b59
                Found module libtasn1.so.6 with build-id: 3d3a2f6f0d4a70919496afe25e329abd189b7882
                Found module libunistring.so.2 with build-id: 15e34cdfafa3547f9c700489b842ceb86f6fb73e
                Found module libidn2.so.0 with build-id: 958c50fc94ecb196b24f3619762e7ec3f28a5b40
                Found module libp11-kit.so.0 with build-id: b03163bf5b98313c7347e7270627a6b2dcd35060
                Found module libstdc++.so.6 with build-id: 66b742a0a51ccd8a67fdf59a31663b0cd28dfcf2
                Found module ld-linux-x86-64.so.2 with build-id: 2b7f1f415256e40ab15e5808c73eb7acf3c73ea3
                Found module libc.so.6 with build-id: 723a5d3072f3b79f3716880303439e11f7661fc5
                Found module libgcc_s.so.1 with build-id: a1a28a65f942690a11041922ba11c16f716565e2
                Found module libm.so.6 with build-id: c58412f6a7fffa4068614708cdb5b15f2ba49c0a
                Found module libaio.so.1 with build-id: 9b75d21282f8e17ddfa06aff78dae4f8dcce4106
                Found module libgmodule-2.0.so.0 with build-id: 952459b04fb217bc3e4a17b902759cf66c89a95d
                Found module libglib-2.0.so.0 with build-id: 0006a4eed3e835c7d7812e36e5d720c1a6eb8689
                Found module libslirp.so.0 with build-id: fa9d492d6d050c6f5cb906a3f087a8a63e4201e2
                Found module libibverbs.so.1 with build-id: 3e19d7349fe9ea4c79debeb8716df7000ab13159
                Found module librdmacm.so.1 with build-id: 4f6798f3f4164a4aa4d75d8a890c760e2835c495
                Found module libnuma.so.1 with build-id: a0c23354fdb39f51ec9e31aeae27a9f8ee034500
                Found module libfdt.so.1 with build-id: 31a56e0009a8824c7a09267c8205034c91cb4095
                Found module libseccomp.so.2 with build-id: 5a5b4ee3bb99ee0378bd937fdef7a7b7255d0428
                Found module libpmem.so.1 with build-id: 084e72da2642981fe75b446293655ae4a68c0663
                Found module libsasl2.so.3 with build-id: 2864d8e1600f3b46016fd62c44269e92c5ec0ca1
                Found module libgnutls.so.30 with build-id: b08009ada925278615f5592e982e0776c06e065e
                Found module libpng16.so.16 with build-id: 698f67f36c58fa443c6ba891e4a0f106553a0452
                Found module libz.so.1 with build-id: 62196acf9f75625acb09840435bb396c62829e12
                Found module liblzo2.so.2 with build-id: c42a1f1fd2f73000efcbe3d2fb6ad196170fd1cf
                Found module libsnappy.so.1 with build-id: 83553ef5fb4b9e530a3f05b68d92f320a6d53772
                Found module libpixman-1.so.0 with build-id: 4420cf633fcf726fc8cef2dd84d817083afe13e5
                Found module qemu-kvm with build-id: debf9ea821f74b76b75f78c5558eba8ca491b1b5
                Stack trace of thread 236656:
                #0  0x00007fef0ee7b7fc __pthread_kill_implementation (libc.so.6 + 0x8f7fc)
                #1  0x00007fef0ee2e676 raise (libc.so.6 + 0x42676)
                #2  0x00007fef0ee187d3 abort (libc.so.6 + 0x2c7d3)
                #3  0x00007fef0ee186fb __assert_fail_base.cold (libc.so.6 + 0x2c6fb)
                #4  0x00007fef0ee27396 __assert_fail (libc.so.6 + 0x3b396)
                #5  0x0000556d867a4791 qio_channel_restart_read (qemu-kvm + 0x760791)
                #6  0x0000556d869a147f aio_dispatch_handler (qemu-kvm + 0x95d47f)
                #7  0x0000556d869a132c aio_dispatch (qemu-kvm + 0x95d32c)
                #8  0x0000556d869bb522 aio_ctx_dispatch (qemu-kvm + 0x977522)
                #9  0x00007fef0f14df6f g_main_context_dispatch (libglib-2.0.so.0 + 0x54f6f)
                #10 0x0000556d869c81e3 main_loop_wait (qemu-kvm + 0x9841e3)
                #11 0x0000556d86647d37 qemu_main_loop (qemu-kvm + 0x603d37)
                #12 0x0000556d863a06f2 main (qemu-kvm + 0x35c6f2)
                #13 0x00007fef0ee19560 __libc_start_call_main (libc.so.6 + 0x2d560)
                #14 0x00007fef0ee1960c __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2d60c)
                #15 0x0000556d8639fdb5 _start (qemu-kvm + 0x35bdb5)
                
                Stack trace of thread 236664:
                #0  0x00007fef0ee76870 __GI___lll_lock_wait (libc.so.6 + 0x8a870)
                #1  0x00007fef0ee7cea2 __pthread_mutex_lock.5 (libc.so.6 + 0x90ea2)
                #2  0x0000556d869a521f qemu_mutex_lock_impl (qemu-kvm + 0x96121f)
                #3  0x0000556d8663c78a flatview_read_continue (qemu-kvm + 0x5f878a)
                #4  0x0000556d8663c62e flatview_read (qemu-kvm + 0x5f862e)
                #5  0x0000556d8663f621 address_space_read_full (qemu-kvm + 0x5fb621)
                #6  0x0000556d8674c695 kvm_cpu_exec (qemu-kvm + 0x708695)
                #7  0x0000556d8674e5fa kvm_vcpu_thread_fn (qemu-kvm + 0x70a5fa)
                #8  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #9  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #10 0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)
                
                Stack trace of thread 236668:
                #0  0x00007fef0eef33fb ioctl (libc.so.6 + 0x1073fb)
                #1  0x0000556d8674cd77 kvm_vcpu_ioctl (qemu-kvm + 0x708d77)
                #2  0x0000556d8674c601 kvm_cpu_exec (qemu-kvm + 0x708601)
                #3  0x0000556d8674e5fa kvm_vcpu_thread_fn (qemu-kvm + 0x70a5fa)
                #4  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #5  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)
                
                Stack trace of thread 236665:
                #0  0x00007fef0eef33fb ioctl (libc.so.6 + 0x1073fb)
                #1  0x0000556d8674cd77 kvm_vcpu_ioctl (qemu-kvm + 0x708d77)
                #2  0x0000556d8674c601 kvm_cpu_exec (qemu-kvm + 0x708601)
                #3  0x0000556d8674e5fa kvm_vcpu_thread_fn (qemu-kvm + 0x70a5fa)
                #4  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #5  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)
                
                Stack trace of thread 236670:
                #0  0x00007fef0eef33fb ioctl (libc.so.6 + 0x1073fb)
                #1  0x0000556d8674cd77 kvm_vcpu_ioctl (qemu-kvm + 0x708d77)
                #2  0x0000556d8674c601 kvm_cpu_exec (qemu-kvm + 0x708601)
                #3  0x0000556d8674e5fa kvm_vcpu_thread_fn (qemu-kvm + 0x70a5fa)
                #4  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #5  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)
                
                Stack trace of thread 236657:
                #0  0x00007fef0eef6fbd syscall (libc.so.6 + 0x10afbd)
                #1  0x0000556d869a5def qemu_event_wait (qemu-kvm + 0x961def)
                #2  0x0000556d869b1eb1 call_rcu_thread (qemu-kvm + 0x96deb1)
                #3  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #4  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #5  0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)
                
                Stack trace of thread 236667:
                #0  0x00007fef0eef33fb ioctl (libc.so.6 + 0x1073fb)
                #1  0x0000556d8674cd77 kvm_vcpu_ioctl (qemu-kvm + 0x708d77)
                #2  0x0000556d8674c601 kvm_cpu_exec (qemu-kvm + 0x708601)
                #3  0x0000556d8674e5fa kvm_vcpu_thread_fn (qemu-kvm + 0x70a5fa)
                #4  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #5  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)
                
                Stack trace of thread 236663:
                #0  0x00007fef0eef187f __poll (libc.so.6 + 0x10587f)
                #1  0x00007fef0f1a256c g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xa956c)
                #2  0x00007fef0f14d533 g_main_loop_run (libglib-2.0.so.0 + 0x54533)
                #3  0x0000556d867cc924 iothread_run (qemu-kvm + 0x788924)
                #4  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #5  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)
                
                Stack trace of thread 236658:
                #0  0x00007fef0eef197e ppoll (libc.so.6 + 0x10597e)
                #1  0x0000556d869a2852 fdmon_poll_wait (qemu-kvm + 0x95e852)
                #2  0x0000556d869a1fe4 aio_poll (qemu-kvm + 0x95dfe4)
                #3  0x0000556d867cc906 iothread_run (qemu-kvm + 0x788906)
                #4  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #5  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)
                
                Stack trace of thread 236671:
                #0  0x00007fef0eef33fb ioctl (libc.so.6 + 0x1073fb)
                #1  0x0000556d8674cd77 kvm_vcpu_ioctl (qemu-kvm + 0x708d77)
                #2  0x0000556d8674c601 kvm_cpu_exec (qemu-kvm + 0x708601)
                #3  0x0000556d8674e5fa kvm_vcpu_thread_fn (qemu-kvm + 0x70a5fa)
                #4  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #5  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)
                
                Stack trace of thread 236731:
                #0  0x00007fef0ee7670a __futex_abstimed_wait_common (libc.so.6 + 0x8a70a)
                #1  0x00007fef0ee81813 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0x95813)
                #2  0x0000556d869a5b0b qemu_sem_timedwait (qemu-kvm + 0x961b0b)
                #3  0x0000556d869cc6b6 worker_thread (qemu-kvm + 0x9886b6)
                #4  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #5  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)
                
                Stack trace of thread 236730:
                #0  0x00007fef0ee7670a __futex_abstimed_wait_common (libc.so.6 + 0x8a70a)
                #1  0x00007fef0ee81813 __new_sem_wait_slow64.constprop.0 (libc.so.6 + 0x95813)
                #2  0x0000556d869a5b0b qemu_sem_timedwait (qemu-kvm + 0x961b0b)
                #3  0x0000556d869cc6b6 worker_thread (qemu-kvm + 0x9886b6)
                #4  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #5  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)
                
                Stack trace of thread 236675:
                #0  0x00007fef0ee7670a __futex_abstimed_wait_common (libc.so.6 + 0x8a70a)
                #1  0x00007fef0ee78e80 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0x8ce80)
                #2  0x0000556d869a56ff qemu_cond_wait_impl (qemu-kvm + 0x9616ff)
                #3  0x0000556d863dd657 vnc_worker_thread (qemu-kvm + 0x399657)
                #4  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #5  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)
                
                Stack trace of thread 236666:
                #0  0x00007fef0eef33fb ioctl (libc.so.6 + 0x1073fb)
                #1  0x0000556d8674cd77 kvm_vcpu_ioctl (qemu-kvm + 0x708d77)
                #2  0x0000556d8674c601 kvm_cpu_exec (qemu-kvm + 0x708601)
                #3  0x0000556d8674e5fa kvm_vcpu_thread_fn (qemu-kvm + 0x70a5fa)
                #4  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #5  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)
                
                Stack trace of thread 236672:
                #0  0x00007fef0eef33fb ioctl (libc.so.6 + 0x1073fb)
                #1  0x0000556d8674cd77 kvm_vcpu_ioctl (qemu-kvm + 0x708d77)
                #2  0x0000556d8674c601 kvm_cpu_exec (qemu-kvm + 0x708601)
                #3  0x0000556d8674e5fa kvm_vcpu_thread_fn (qemu-kvm + 0x70a5fa)
                #4  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #5  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)
                
                Stack trace of thread 236669:
                #0  0x00007fef0eef33fb ioctl (libc.so.6 + 0x1073fb)
                #1  0x0000556d8674cd77 kvm_vcpu_ioctl (qemu-kvm + 0x708d77)
                #2  0x0000556d8674c601 kvm_cpu_exec (qemu-kvm + 0x708601)
                #3  0x0000556d8674e5fa kvm_vcpu_thread_fn (qemu-kvm + 0x70a5fa)
                #4  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #5  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)
                
                Stack trace of thread 236673:
                #0  0x00007fef0eef33fb ioctl (libc.so.6 + 0x1073fb)
                #1  0x0000556d8674cd77 kvm_vcpu_ioctl (qemu-kvm + 0x708d77)
                #2  0x0000556d8674c601 kvm_cpu_exec (qemu-kvm + 0x708601)
                #3  0x0000556d8674e5fa kvm_vcpu_thread_fn (qemu-kvm + 0x70a5fa)
                #4  0x0000556d869a609a qemu_thread_start (qemu-kvm + 0x96209a)
                #5  0x00007fef0ee79aaf start_thread (libc.so.6 + 0x8daaf)
                #6  0x00007fef0eefe740 __clone3 (libc.so.6 + 0x112740)

Expected results:
 VM can start successfully with exposed nbd data image.


Additional info:
 Will attach core dump file later.

Comment 2 Klaus Heinrich Kiwi 2021-12-21 19:16:23 UTC

Eric, can you take this? I've classified it as high-priority as it's a crash in apparently a customer-exposed path, but let's discuss if you disagree.

Aihua Liang, can you identify if this is a new testcase, or a regression? In case of the latter, set the regression keyword and identify the last working qemu-kvm version?

Thanks

Comment 3 aihua liang 2021-12-22 02:05:16 UTC

(In reply to Klaus Heinrich Kiwi from comment #2)
> Eric, can you take this? I've classified it as high-priority as it's a crash
> in apparently a customer-exposed path, but let's discuss if you disagree.
> 
> Aihua Liang, can you identify if this is a new testcase, or a regression? In
> case of the latter, set the regression keyword and identify the last working
> qemu-kvm version?
> 
> Thanks

Hi, Klaus

  It's not a new case and its a regression bug. emu-kvm-6.1.0-8.el9 don't hit it.


BR,
Aliang

Comment 4 aihua liang 2021-12-24 05:47:56 UTC

Won't hit this issue when iothread not set.

Comment 5 Hanna Czenczek 2022-02-03 12:28:28 UTC

I believe this is the bug I described in BZ 1990835 comment 7, and which I’ve sent an RFC for (currently working on writing tests for a proper v1): https://lists.nongnu.org/archive/html/qemu-block/2022-01/msg00765.html

If you prefer, Eric, I could take this bug.  (I was about to create a new one for this issue anyway, and the duplicate detector pointed me towards this one, which is how I found it.)

Comment 6 Klaus Heinrich Kiwi 2022-02-07 14:32:03 UTC

(In reply to Hanna Reitz from comment #5)
> I believe this is the bug I described in BZ 1990835 comment 7, and which
> I’ve sent an RFC for (currently working on writing tests for a proper v1):
> https://lists.nongnu.org/archive/html/qemu-block/2022-01/msg00765.html
> 
> If you prefer, Eric, I could take this bug.  (I was about to create a new
> one for this issue anyway, and the duplicate detector pointed me towards
> this one, which is how I found it.)

Hanna, may I take you on your offer? If you have a scratch build that QE can test, please let us know. Thanks!

Comment 7 Hanna Czenczek 2022-02-07 16:51:37 UTC

As long as AArch64 isn’t needed (which seems kind of slow today), I can offer this: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=42850336

The x86-64 RPMs are here: http://download.eng.bos.redhat.com/brewroot/work/tasks/428/42850428/

(Sorry, I don’t have a repo yet, because the overall build won’t finish because of AArch64)

Could you test whether those fix the crash, Aliang?  Thanks!

Comment 8 aihua liang 2022-02-08 03:01:08 UTC

Test with qemu-kvm-6.2.0-7.el9.hreitz202202071714 by auto, don't hit this issue any more.
 (1/3) Host_RHEL.m9.u0.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.9.0.0.x86_64.io-github-autotest-qemu.blockdev_stream_remote_server_down.q35: PASS (144.42 s)
 (2/3) Host_RHEL.m9.u0.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.9.0.0.x86_64.io-github-autotest-qemu.blockdev_mirror_remote_server_down.q35: PASS (122.59 s)
 (3/3) Host_RHEL.m9.u0.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.9.0.0.x86_64.io-github-autotest-qemu.blockdev_commit_firewall.q35: PASS (321.08 s)

Comment 9 Hanna Czenczek 2022-02-08 13:38:13 UTC

Awesome, thanks!

Comment 10 Hanna Czenczek 2022-02-15 16:17:29 UTC

This fix has been merged upstream, and I’ve sent a backport for 8.6 (see BZ 2035185).  I’d love to send a backport for this BZ, too, but can’t, because the Brew build server seems to have some subscription problems.  If they persist, I suppose I’ll just send a backport without a Brew build.

Hanna

Comment 12 Yanan Fu 2022-02-21 03:25:48 UTC

QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass.

Comment 16 aihua liang 2022-02-21 10:26:08 UTC

Test with qemu-kvm-6.2.0-9.el9, not hit this issue any more.
 (1/3) Host_RHEL.m9.u0.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.9.0.0.x86_64.io-github-autotest-qemu.blockdev_stream_remote_server_down.q35: PASS (143.82 s)
 (2/3) Host_RHEL.m9.u0.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.9.0.0.x86_64.io-github-autotest-qemu.blockdev_mirror_remote_server_down.q35: PASS (119.67 s)
 (3/3) Host_RHEL.m9.u0.qcow2.virtio_scsi.up.virtio_net.Guest.RHEL.9.0.0.x86_64.io-github-autotest-qemu.blockdev_commit_firewall.q35: PASS (321.33 s)

Comment 17 aihua liang 2022-02-21 10:27:22 UTC

As comment 12 and comment 16, set bug's status to "VERIFIED".

Comment 19 errata-xmlrpc 2022-05-17 12:25:11 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: qemu-kvm), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2307

Note You need to log in before you can comment on or make changes to this bug.