RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1897550 - qemu crashed when hotplug many disks with error virtio_scsi_data_plane_handle_ctrl: Assertion `s->ctx && s->dataplane_started' failed
Summary: qemu crashed when hotplug many disks with error virtio_scsi_data_plane_handle...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: qemu-kvm
Version: 8.4
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: rc
: 8.0
Assignee: Maxim Levitsky
QA Contact: qing.wang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-11-13 12:04 UTC by qing.wang
Modified: 2021-04-28 01:34 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-03-18 09:05:44 UTC
Type: Bug
Target Upstream Version:
Embargoed:
pm-rhel: mirror+

Attachments (Terms of Use)

Description qing.wang 2020-11-13 12:04:33 UTC 
Description of problem:

after guest boot or during boot , hotplug many disks one time (200 disks),there is no error on the qmp console, but the qemu will crash later or hang. 

(qemu) qemu-kvm: virtio_bus_set_host_notifier: unable to init event notifier: Too many open files (-24)
virtio-scsi: Failed to set host notifier (-24)
qemu-kvm: /builddir/build/BUILD/qemu-5.1.0/hw/scsi/virtio-scsi-dataplane.c:72: virtio_scsi_data_plane_handle_ctrl: Assertion `s->ctx && s->dataplane_started' failed.


Version-Release number of selected component (if applicable):
Red Hat Enterprise Linux release 8.4 Beta (Ootpa)
4.18.0-241.el8.dt1.x86_64
qemu-kvm-common-4.2.0-35.module+el8.4.0+8705+34397d87.x86_64

Red Hat Enterprise Linux release 8.3 (Ootpa)
4.18.0-240.el8.x86_64
qemu-kvm-common-5.1.0-13.module+el8.3.0+8424+e82f331d.x86_64


How reproducible:
40%

Steps to Reproduce:
1.create 200 disk image files
qemu-img create -f qcow2 /home/kvm_autotest_root/images/stg0.qcow2 128M
...
qemu-img create -f qcow2 /home/kvm_autotest_root/images/stg200.qcow2 128M

2.boot vm

/usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35,accel=kvm \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 16G  \
    -smp 10,maxcpus=10,cores=5,threads=1,dies=1,sockets=2  \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -object iothread,id=iothread0 \
    -object iothread,id=iothread1 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0,iothread=iothread0 \
    -blockdev node-name=file_image1,driver=file,aio=native,filename=/home/kvm_autotest_root/images/rhel830-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,bootindex=0,write-cache=on \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:88:7a:d3:8b:74,id=id9QVzBa,netdev=idEb8qPI,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=idEb8qPI,vhost=on  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5 \
    -device pcie-root-port,id=pcie_extra_root_port_1,addr=0x3.0x1,bus=pcie.0,chassis=6 \
    -device pcie-root-port,id=pcie_extra_root_port_2,addr=0x3.0x2,bus=pcie.0,chassis=7 \
    -device pcie-root-port,id=pcie_extra_root_port_3,addr=0x3.0x3,bus=pcie.0,chassis=8 \
    -device pcie-root-port,id=pcie_extra_root_port_4,addr=0x3.0x4,bus=pcie.0,chassis=9 \
    -device pcie-root-port,id=pcie_extra_root_port_5,addr=0x3.0x5,bus=pcie.0,chassis=10 \
    -device pcie-root-port,id=pcie_extra_root_port_6,addr=0x3.0x6,bus=pcie.0,chassis=11 \
    -device pcie-root-port,id=pcie_extra_root_port_7,addr=0x3.0x7,bus=pcie.0,chassis=12 \
    -device pcie-root-port,id=pcie_extra_root_port_8,multifunction=on,bus=pcie.0,addr=0x4,chassis=13 \
    -device pcie-root-port,id=pcie_extra_root_port_9,addr=0x4.0x1,bus=pcie.0,chassis=14 \
    -device pcie-root-port,id=pcie_extra_root_port_10,addr=0x4.0x2,bus=pcie.0,chassis=15 \
    -device pcie-root-port,id=pcie_extra_root_port_11,addr=0x4.0x3,bus=pcie.0,chassis=16 \
    -device pcie-root-port,id=pcie_extra_root_port_12,addr=0x4.0x4,bus=pcie.0,chassis=17 \
    -device pcie-root-port,id=pcie_extra_root_port_13,addr=0x4.0x5,bus=pcie.0,chassis=18 \
    -device pcie-root-port,id=pcie_extra_root_port_14,addr=0x4.0x6,bus=pcie.0,chassis=19 \
    -device pcie-root-port,id=pcie_extra_root_port_15,addr=0x4.0x7,bus=pcie.0,chassis=20 \
    -device pcie-root-port,id=pcie_extra_root_port_16,multifunction=on,bus=pcie.0,addr=0x5,chassis=21 \
    -device pcie-root-port,id=pcie_extra_root_port_17,addr=0x5.0x1,bus=pcie.0,chassis=22 \
    -device pcie-root-port,id=pcie_extra_root_port_18,addr=0x5.0x2,bus=pcie.0,chassis=23 \
    -device pcie-root-port,id=pcie_extra_root_port_19,addr=0x5.0x3,bus=pcie.0,chassis=24 \
    -device pcie-root-port,id=pcie_extra_root_port_20,addr=0x5.0x4,bus=pcie.0,chassis=25 \
    -device pcie-root-port,id=pcie_extra_root_port_21,addr=0x5.0x5,bus=pcie.0,chassis=26 \
    -device pcie-root-port,id=pcie_extra_root_port_22,addr=0x5.0x6,bus=pcie.0,chassis=27 \
    -device pcie-root-port,id=pcie_extra_root_port_23,addr=0x5.0x7,bus=pcie.0,chassis=28 \
    -device pcie-root-port,id=pcie_extra_root_port_24,multifunction=on,bus=pcie.0,addr=0x6,chassis=29 \
    -device pcie-root-port,id=pcie_extra_root_port_25,addr=0x6.0x1,bus=pcie.0,chassis=30 \
    -device pcie-root-port,id=pcie_extra_root_port_26,addr=0x6.0x2,bus=pcie.0,chassis=31 \
    -device pcie-root-port,id=pcie_extra_root_port_27,addr=0x6.0x3,bus=pcie.0,chassis=32 \
    -device pcie-root-port,id=pcie_extra_root_port_28,addr=0x6.0x4,bus=pcie.0,chassis=33 \
    -device pcie-root-port,id=pcie_extra_root_port_29,addr=0x6.0x5,bus=pcie.0,chassis=34 \
    -device pcie-root-port,id=pcie_extra_root_port_30,addr=0x6.0x6,bus=pcie.0,chassis=35 \
    -device pcie-root-port,id=pcie_extra_root_port_31,addr=0x6.0x7,bus=pcie.0,chassis=36 \
    -device pcie-root-port,id=pcie_extra_root_port_32,multifunction=on,bus=pcie.0,addr=0x7,chassis=37 \
    -device pcie-root-port,id=pcie_extra_root_port_33,addr=0x7.0x1,bus=pcie.0,chassis=38 \
    -device pcie-root-port,id=pcie_extra_root_port_34,addr=0x7.0x2,bus=pcie.0,chassis=39 \
    -device pcie-root-port,id=pcie_extra_root_port_35,addr=0x7.0x3,bus=pcie.0,chassis=40 \
    -device pcie-root-port,id=pcie_extra_root_port_36,addr=0x7.0x4,bus=pcie.0,chassis=41 \
    -device pcie-root-port,id=pcie_extra_root_port_37,addr=0x7.0x5,bus=pcie.0,chassis=42 \
    -device pcie-root-port,id=pcie_extra_root_port_38,addr=0x7.0x6,bus=pcie.0,chassis=43 \
    -device pcie-root-port,id=pcie_extra_root_port_39,addr=0x7.0x7,bus=pcie.0,chassis=44 \
    -device pcie-root-port,id=pcie_extra_root_port_40,multifunction=on,bus=pcie.0,addr=0x8,chassis=45 \
    -device pcie-root-port,id=pcie_extra_root_port_41,addr=0x8.0x1,bus=pcie.0,chassis=46 \
    -device pcie-root-port,id=pcie_extra_root_port_42,addr=0x8.0x2,bus=pcie.0,chassis=47 \
    -device pcie-root-port,id=pcie_extra_root_port_43,addr=0x8.0x3,bus=pcie.0,chassis=48 \
    -device pcie-root-port,id=pcie_extra_root_port_44,addr=0x8.0x4,bus=pcie.0,chassis=49 \
    -device pcie-root-port,id=pcie_extra_root_port_45,addr=0x8.0x5,bus=pcie.0,chassis=50 \
    -device pcie-root-port,id=pcie_extra_root_port_46,addr=0x8.0x6,bus=pcie.0,chassis=51 \
    -device pcie-root-port,id=pcie_extra_root_port_47,addr=0x8.0x7,bus=pcie.0,chassis=52 \
    -device pcie-root-port,id=pcie_extra_root_port_48,multifunction=on,bus=pcie.0,addr=0x9,chassis=53 \
    -device pcie-root-port,id=pcie_extra_root_port_49,addr=0x9.0x1,bus=pcie.0,chassis=54 \
    -device pcie-root-port,id=pcie_extra_root_port_50,addr=0x9.0x2,bus=pcie.0,chassis=55 \
    -device pcie-root-port,id=pcie_extra_root_port_51,addr=0x9.0x3,bus=pcie.0,chassis=56 \
    -device pcie-root-port,id=pcie_extra_root_port_52,addr=0x9.0x4,bus=pcie.0,chassis=57 \
    -device pcie-root-port,id=pcie_extra_root_port_53,addr=0x9.0x5,bus=pcie.0,chassis=58 \
    -device pcie-root-port,id=pcie_extra_root_port_54,addr=0x9.0x6,bus=pcie.0,chassis=59 \
    -device pcie-root-port,id=pcie_extra_root_port_55,addr=0x9.0x7,bus=pcie.0,chassis=60 \
    -device pcie-root-port,id=pcie_extra_root_port_56,multifunction=on,bus=pcie.0,addr=0xa,chassis=61 \
    -device pcie-root-port,id=pcie_extra_root_port_57,addr=0xa.0x1,bus=pcie.0,chassis=62 \
    -device pcie-root-port,id=pcie_extra_root_port_58,addr=0xa.0x2,bus=pcie.0,chassis=63 \
    -device pcie-root-port,id=pcie_extra_root_port_59,addr=0xa.0x3,bus=pcie.0,chassis=64 \
    -device pcie-root-port,id=pcie_extra_root_port_60,addr=0xa.0x4,bus=pcie.0,chassis=65 \
    -device pcie-root-port,id=pcie_extra_root_port_61,addr=0xa.0x5,bus=pcie.0,chassis=66 \
    -device pcie-root-port,id=pcie_extra_root_port_62,addr=0xa.0x6,bus=pcie.0,chassis=67 \
    -device pcie-root-port,id=pcie_extra_root_port_63,addr=0xa.0x7,bus=pcie.0,chassis=68 \
    -device pcie-root-port,id=pcie_extra_root_port_64,multifunction=on,bus=pcie.0,addr=0xb,chassis=69 \
    -device pcie-root-port,id=pcie_extra_root_port_65,addr=0xb.0x1,bus=pcie.0,chassis=70 \
    -device pcie-root-port,id=pcie_extra_root_port_66,addr=0xb.0x2,bus=pcie.0,chassis=71 \
    -device pcie-root-port,id=pcie_extra_root_port_67,addr=0xb.0x3,bus=pcie.0,chassis=72 \
    -device pcie-root-port,id=pcie_extra_root_port_68,addr=0xb.0x4,bus=pcie.0,chassis=73 \
    -device pcie-root-port,id=pcie_extra_root_port_69,addr=0xb.0x5,bus=pcie.0,chassis=74 \
    -device pcie-root-port,id=pcie_extra_root_port_70,addr=0xb.0x6,bus=pcie.0,chassis=75 \
    -device pcie-root-port,id=pcie_extra_root_port_71,addr=0xb.0x7,bus=pcie.0,chassis=76 \
    -device pcie-root-port,id=pcie_extra_root_port_72,multifunction=on,bus=pcie.0,addr=0xc,chassis=77 \
    -device pcie-root-port,id=pcie_extra_root_port_73,addr=0xc.0x1,bus=pcie.0,chassis=78 \
    -device pcie-root-port,id=pcie_extra_root_port_74,addr=0xc.0x2,bus=pcie.0,chassis=79 \
    -device pcie-root-port,id=pcie_extra_root_port_75,addr=0xc.0x3,bus=pcie.0,chassis=80 \
    -device pcie-root-port,id=pcie_extra_root_port_76,addr=0xc.0x4,bus=pcie.0,chassis=81 \
    -device pcie-root-port,id=pcie_extra_root_port_77,addr=0xc.0x5,bus=pcie.0,chassis=82 \
    -device pcie-root-port,id=pcie_extra_root_port_78,addr=0xc.0x6,bus=pcie.0,chassis=83 \
    -device pcie-root-port,id=pcie_extra_root_port_79,addr=0xc.0x7,bus=pcie.0,chassis=84 \
    -device pcie-root-port,id=pcie_extra_root_port_80,multifunction=on,bus=pcie.0,addr=0xd,chassis=85 \
    -device pcie-root-port,id=pcie_extra_root_port_81,addr=0xd.0x1,bus=pcie.0,chassis=86 \
    -device pcie-root-port,id=pcie_extra_root_port_82,addr=0xd.0x2,bus=pcie.0,chassis=87 \
    -device pcie-root-port,id=pcie_extra_root_port_83,addr=0xd.0x3,bus=pcie.0,chassis=88 \
    -device pcie-root-port,id=pcie_extra_root_port_84,addr=0xd.0x4,bus=pcie.0,chassis=89 \
    -device pcie-root-port,id=pcie_extra_root_port_85,addr=0xd.0x5,bus=pcie.0,chassis=90 \
    -device pcie-root-port,id=pcie_extra_root_port_86,addr=0xd.0x6,bus=pcie.0,chassis=91 \
    -device pcie-root-port,id=pcie_extra_root_port_87,addr=0xd.0x7,bus=pcie.0,chassis=92 \
    -device pcie-root-port,id=pcie_extra_root_port_88,multifunction=on,bus=pcie.0,addr=0xe,chassis=93 \
    -device pcie-root-port,id=pcie_extra_root_port_89,addr=0xe.0x1,bus=pcie.0,chassis=94 \
    -device pcie-root-port,id=pcie_extra_root_port_90,addr=0xe.0x2,bus=pcie.0,chassis=95 \
    -device pcie-root-port,id=pcie_extra_root_port_91,addr=0xe.0x3,bus=pcie.0,chassis=96 \
    -device pcie-root-port,id=pcie_extra_root_port_92,addr=0xe.0x4,bus=pcie.0,chassis=97 \
    -device pcie-root-port,id=pcie_extra_root_port_93,addr=0xe.0x5,bus=pcie.0,chassis=98 \
    -device pcie-root-port,id=pcie_extra_root_port_94,addr=0xe.0x6,bus=pcie.0,chassis=99 \
    -device pcie-root-port,id=pcie_extra_root_port_95,addr=0xe.0x7,bus=pcie.0,chassis=100 \
    -device pcie-root-port,id=pcie_extra_root_port_96,multifunction=on,bus=pcie.0,addr=0xf,chassis=101 \
    -device pcie-root-port,id=pcie_extra_root_port_97,addr=0xf.0x1,bus=pcie.0,chassis=102 \
    -device pcie-root-port,id=pcie_extra_root_port_98,addr=0xf.0x2,bus=pcie.0,chassis=103 \
    -device pcie-root-port,id=pcie_extra_root_port_99,addr=0xf.0x3,bus=pcie.0,chassis=104 \
    -device pcie-root-port,id=pcie_extra_root_port_100,addr=0xf.0x4,bus=pcie.0,chassis=105 \
    -device pcie-root-port,id=pcie_extra_root_port_101,addr=0xf.0x5,bus=pcie.0,chassis=106 \
    -device pcie-root-port,id=pcie_extra_root_port_102,addr=0xf.0x6,bus=pcie.0,chassis=107 \
    -device pcie-root-port,id=pcie_extra_root_port_103,addr=0xf.0x7,bus=pcie.0,chassis=108 \
    -device pcie-root-port,id=pcie_extra_root_port_104,multifunction=on,bus=pcie.0,addr=0x10,chassis=109 \
    -device pcie-root-port,id=pcie_extra_root_port_105,addr=0x10.0x1,bus=pcie.0,chassis=110 \
    -device pcie-root-port,id=pcie_extra_root_port_106,addr=0x10.0x2,bus=pcie.0,chassis=111 \
    -device pcie-root-port,id=pcie_extra_root_port_107,addr=0x10.0x3,bus=pcie.0,chassis=112 \
    -device pcie-root-port,id=pcie_extra_root_port_108,addr=0x10.0x4,bus=pcie.0,chassis=113 \
    -device pcie-root-port,id=pcie_extra_root_port_109,addr=0x10.0x5,bus=pcie.0,chassis=114 \
    -device pcie-root-port,id=pcie_extra_root_port_110,addr=0x10.0x6,bus=pcie.0,chassis=115 \
    -device pcie-root-port,id=pcie_extra_root_port_111,addr=0x10.0x7,bus=pcie.0,chassis=116 \
    -device pcie-root-port,id=pcie_extra_root_port_112,multifunction=on,bus=pcie.0,addr=0x11,chassis=117 \
    -device pcie-root-port,id=pcie_extra_root_port_113,addr=0x11.0x1,bus=pcie.0,chassis=118 \
    -device pcie-root-port,id=pcie_extra_root_port_114,addr=0x11.0x2,bus=pcie.0,chassis=119 \
    -device pcie-root-port,id=pcie_extra_root_port_115,addr=0x11.0x3,bus=pcie.0,chassis=120 \
    -device pcie-root-port,id=pcie_extra_root_port_116,addr=0x11.0x4,bus=pcie.0,chassis=121 \
    -device pcie-root-port,id=pcie_extra_root_port_117,addr=0x11.0x5,bus=pcie.0,chassis=122 \
    -device pcie-root-port,id=pcie_extra_root_port_118,addr=0x11.0x6,bus=pcie.0,chassis=123 \
    -device pcie-root-port,id=pcie_extra_root_port_119,addr=0x11.0x7,bus=pcie.0,chassis=124 \
    -device pcie-root-port,id=pcie_extra_root_port_120,multifunction=on,bus=pcie.0,addr=0x12,chassis=125 \
    -device pcie-root-port,id=pcie_extra_root_port_121,addr=0x12.0x1,bus=pcie.0,chassis=126 \
    -device pcie-root-port,id=pcie_extra_root_port_122,addr=0x12.0x2,bus=pcie.0,chassis=127 \
    -device pcie-root-port,id=pcie_extra_root_port_123,addr=0x12.0x3,bus=pcie.0,chassis=128 \
    -device pcie-root-port,id=pcie_extra_root_port_124,addr=0x12.0x4,bus=pcie.0,chassis=129 \
    -device pcie-root-port,id=pcie_extra_root_port_125,addr=0x12.0x5,bus=pcie.0,chassis=130 \
    -device pcie-root-port,id=pcie_extra_root_port_126,addr=0x12.0x6,bus=pcie.0,chassis=131 \
    -device pcie-root-port,id=pcie_extra_root_port_127,addr=0x12.0x7,bus=pcie.0,chassis=132 \
    -device pcie-root-port,id=pcie_extra_root_port_128,multifunction=on,bus=pcie.0,addr=0x13,chassis=133 \
    -device pcie-root-port,id=pcie_extra_root_port_129,addr=0x13.0x1,bus=pcie.0,chassis=134 \
    -device pcie-root-port,id=pcie_extra_root_port_130,addr=0x13.0x2,bus=pcie.0,chassis=135 \
    -device pcie-root-port,id=pcie_extra_root_port_131,addr=0x13.0x3,bus=pcie.0,chassis=136 \
    -device pcie-root-port,id=pcie_extra_root_port_132,addr=0x13.0x4,bus=pcie.0,chassis=137 \
    -device pcie-root-port,id=pcie_extra_root_port_133,addr=0x13.0x5,bus=pcie.0,chassis=138 \
    -device pcie-root-port,id=pcie_extra_root_port_134,addr=0x13.0x6,bus=pcie.0,chassis=139 \
    -device pcie-root-port,id=pcie_extra_root_port_135,addr=0x13.0x7,bus=pcie.0,chassis=140 \
    -device pcie-root-port,id=pcie_extra_root_port_136,multifunction=on,bus=pcie.0,addr=0x14,chassis=141 \
    -device pcie-root-port,id=pcie_extra_root_port_137,addr=0x14.0x1,bus=pcie.0,chassis=142 \
    -device pcie-root-port,id=pcie_extra_root_port_138,addr=0x14.0x2,bus=pcie.0,chassis=143 \
    -device pcie-root-port,id=pcie_extra_root_port_139,addr=0x14.0x3,bus=pcie.0,chassis=144 \
    -device pcie-root-port,id=pcie_extra_root_port_140,addr=0x14.0x4,bus=pcie.0,chassis=145 \
    -device pcie-root-port,id=pcie_extra_root_port_141,addr=0x14.0x5,bus=pcie.0,chassis=146 \
    -device pcie-root-port,id=pcie_extra_root_port_142,addr=0x14.0x6,bus=pcie.0,chassis=147 \
    -device pcie-root-port,id=pcie_extra_root_port_143,addr=0x14.0x7,bus=pcie.0,chassis=148 \
    -device pcie-root-port,id=pcie_extra_root_port_144,multifunction=on,bus=pcie.0,addr=0x15,chassis=149 \
    -device pcie-root-port,id=pcie_extra_root_port_145,addr=0x15.0x1,bus=pcie.0,chassis=150 \
    -device pcie-root-port,id=pcie_extra_root_port_146,addr=0x15.0x2,bus=pcie.0,chassis=151 \
    -device pcie-root-port,id=pcie_extra_root_port_147,addr=0x15.0x3,bus=pcie.0,chassis=152 \
    -device pcie-root-port,id=pcie_extra_root_port_148,addr=0x15.0x4,bus=pcie.0,chassis=153 \
    -device pcie-root-port,id=pcie_extra_root_port_149,addr=0x15.0x5,bus=pcie.0,chassis=154 \
    -device pcie-root-port,id=pcie_extra_root_port_150,addr=0x15.0x6,bus=pcie.0,chassis=155 \
    -device pcie-root-port,id=pcie_extra_root_port_151,addr=0x15.0x7,bus=pcie.0,chassis=156 \
    -device pcie-root-port,id=pcie_extra_root_port_152,multifunction=on,bus=pcie.0,addr=0x16,chassis=157 \
    -device pcie-root-port,id=pcie_extra_root_port_153,addr=0x16.0x1,bus=pcie.0,chassis=158 \
    -device pcie-root-port,id=pcie_extra_root_port_154,addr=0x16.0x2,bus=pcie.0,chassis=159 \
    -device pcie-root-port,id=pcie_extra_root_port_155,addr=0x16.0x3,bus=pcie.0,chassis=160 \
    -device pcie-root-port,id=pcie_extra_root_port_156,addr=0x16.0x4,bus=pcie.0,chassis=161 \
    -device pcie-root-port,id=pcie_extra_root_port_157,addr=0x16.0x5,bus=pcie.0,chassis=162 \
    -device pcie-root-port,id=pcie_extra_root_port_158,addr=0x16.0x6,bus=pcie.0,chassis=163 \
    -device pcie-root-port,id=pcie_extra_root_port_159,addr=0x16.0x7,bus=pcie.0,chassis=164 \
    -device pcie-root-port,id=pcie_extra_root_port_160,multifunction=on,bus=pcie.0,addr=0x17,chassis=165 \
    -device pcie-root-port,id=pcie_extra_root_port_161,addr=0x17.0x1,bus=pcie.0,chassis=166 \
    -device pcie-root-port,id=pcie_extra_root_port_162,addr=0x17.0x2,bus=pcie.0,chassis=167 \
    -device pcie-root-port,id=pcie_extra_root_port_163,addr=0x17.0x3,bus=pcie.0,chassis=168 \
    -device pcie-root-port,id=pcie_extra_root_port_164,addr=0x17.0x4,bus=pcie.0,chassis=169 \
    -device pcie-root-port,id=pcie_extra_root_port_165,addr=0x17.0x5,bus=pcie.0,chassis=170 \
    -device pcie-root-port,id=pcie_extra_root_port_166,addr=0x17.0x6,bus=pcie.0,chassis=171 \
    -device pcie-root-port,id=pcie_extra_root_port_167,addr=0x17.0x7,bus=pcie.0,chassis=172 \
    -device pcie-root-port,id=pcie_extra_root_port_168,multifunction=on,bus=pcie.0,addr=0x18,chassis=173 \
    -device pcie-root-port,id=pcie_extra_root_port_169,addr=0x18.0x1,bus=pcie.0,chassis=174 \
    -device pcie-root-port,id=pcie_extra_root_port_170,addr=0x18.0x2,bus=pcie.0,chassis=175 \
    -device pcie-root-port,id=pcie_extra_root_port_171,addr=0x18.0x3,bus=pcie.0,chassis=176 \
    -device pcie-root-port,id=pcie_extra_root_port_172,addr=0x18.0x4,bus=pcie.0,chassis=177 \
    -device pcie-root-port,id=pcie_extra_root_port_173,addr=0x18.0x5,bus=pcie.0,chassis=178 \
    -device pcie-root-port,id=pcie_extra_root_port_174,addr=0x18.0x6,bus=pcie.0,chassis=179 \
    -device pcie-root-port,id=pcie_extra_root_port_175,addr=0x18.0x7,bus=pcie.0,chassis=180 \
    -device pcie-root-port,id=pcie_extra_root_port_176,multifunction=on,bus=pcie.0,addr=0x19,chassis=181 \
    -device pcie-root-port,id=pcie_extra_root_port_177,addr=0x19.0x1,bus=pcie.0,chassis=182 \
    -device pcie-root-port,id=pcie_extra_root_port_178,addr=0x19.0x2,bus=pcie.0,chassis=183 \
    -device pcie-root-port,id=pcie_extra_root_port_179,addr=0x19.0x3,bus=pcie.0,chassis=184 \
    -device pcie-root-port,id=pcie_extra_root_port_180,addr=0x19.0x4,bus=pcie.0,chassis=185 \
    -device pcie-root-port,id=pcie_extra_root_port_181,addr=0x19.0x5,bus=pcie.0,chassis=186 \
    -device pcie-root-port,id=pcie_extra_root_port_182,addr=0x19.0x6,bus=pcie.0,chassis=187 \
    -device pcie-root-port,id=pcie_extra_root_port_183,addr=0x19.0x7,bus=pcie.0,chassis=188 \
    -device pcie-root-port,id=pcie_extra_root_port_184,multifunction=on,bus=pcie.0,addr=0x1a,chassis=189 \
    -device pcie-root-port,id=pcie_extra_root_port_185,addr=0x1a.0x1,bus=pcie.0,chassis=190 \
    -device pcie-root-port,id=pcie_extra_root_port_186,addr=0x1a.0x2,bus=pcie.0,chassis=191 \
    -device pcie-root-port,id=pcie_extra_root_port_187,addr=0x1a.0x3,bus=pcie.0,chassis=192 \
    -device pcie-root-port,id=pcie_extra_root_port_188,addr=0x1a.0x4,bus=pcie.0,chassis=193 \
    -device pcie-root-port,id=pcie_extra_root_port_189,addr=0x1a.0x5,bus=pcie.0,chassis=194 \
    -device pcie-root-port,id=pcie_extra_root_port_190,addr=0x1a.0x6,bus=pcie.0,chassis=195 \
    -device pcie-root-port,id=pcie_extra_root_port_191,addr=0x1a.0x7,bus=pcie.0,chassis=196 \
    -device pcie-root-port,id=pcie_extra_root_port_192,multifunction=on,bus=pcie.0,addr=0x1b,chassis=197 \
    -device pcie-root-port,id=pcie_extra_root_port_193,addr=0x1b.0x1,bus=pcie.0,chassis=198 \
    -device pcie-root-port,id=pcie_extra_root_port_194,addr=0x1b.0x2,bus=pcie.0,chassis=199 \
    -device pcie-root-port,id=pcie_extra_root_port_195,addr=0x1b.0x3,bus=pcie.0,chassis=200 \
    -device pcie-root-port,id=pcie_extra_root_port_196,addr=0x1b.0x4,bus=pcie.0,chassis=201 \
    -device pcie-root-port,id=pcie_extra_root_port_197,addr=0x1b.0x5,bus=pcie.0,chassis=202 \
    -device pcie-root-port,id=pcie_extra_root_port_198,addr=0x1b.0x6,bus=pcie.0,chassis=203 \
    -device pcie-root-port,id=pcie_extra_root_port_199,addr=0x1b.0x7,bus=pcie.0,chassis=204 \
    -monitor stdio \
    -chardev file,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpdbg.log,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -qmp tcp:0:5955,server,nowait  \
    -vnc :5  \
    -chardev file,path=/var/tmp/monitor-serialdbg.log,id=serial_id_serial0 \
    -device isa-serial,chardev=serial_id_serial0  \

3.after guest boot, 
NUM_LUNS=199

add_devices() {
  exec 3<>/dev/tcp/localhost/5955
  echo "$@"
  echo -e "{'execute':'qmp_capabilities'}" >&3
  read response <&3
  echo $response
  for i in $(seq 1 $NUM_LUNS) ; do
  cmd="{'execute': 'blockdev-add', 'arguments': {'node-name': 'file_stg${i}', 'driver': 'file','filename': '/home/kvm_autotest_root/images/stg${i}.qcow2'}}"
  echo "$cmd"
  echo -e "$cmd" >&3
  read response <&3
  echo "$response"
  cmd="{'execute': 'blockdev-add', 'arguments': {'node-name': 'drive_stg${i}', 'driver': 'qcow2', 'file': 'file_stg${i}'}} "
  echo "$cmd"
  echo -e "$cmd" >&3
  read response <&3
  echo "$response"
  cmd="{'execute': 'device_add', 'arguments': {'id': 'virtio_scsi_pci${i}', 'driver': 'virtio-scsi-pci', 'bus':'pcie_extra_root_port_${i}','addr':'0x0', 'iothread':'iothread1'}}"
  echo "$cmd"
  echo -e "$cmd" >&3
  read response <&3
  echo "$response"
  cmd="{'execute': 'device_add', 'arguments':{'driver':'scsi-hd', 'id':'stg${i}', 'bus':'virtio_scsi_pci${i}.0', 'drive':'drive_stg${i}'}}"
  echo "$cmd"
  echo -e "$cmd" >&3
  read response <&3
  echo "$response"
#  sleep 0.1

  done
}

add_devices

4.check disk number in guest
lsblk -d |wc -l

Actual results:
qemu crash or hang

Expected results:
guest may attach those disks 

Additional info:
Comment 1 qing.wang 2020-11-13 12:28:32 UTC 
Reference bt 
#0  0x00007fa60a1da7ff in __GI_raise (sig=sig@entry=6)
    at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007fa60a1c4c35 in __GI_abort () at abort.c:79
#2  0x00007fa60a1c4b09 in __assert_fail_base
    (fmt=0x7fa60a32acf0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x560dd8b735f0 "s->ctx && s->dataplane_started", file=0x560dd8b73578 "/builddir/build/BUILD/qemu-5.1.0/hw/scsi/virtio-scsi-dataplane.c", line=72, function=<optimized out>) at assert.c:92
#3  0x00007fa60a1d2de6 in __GI___assert_fail
    (assertion=assertion@entry=0x560dd8b735f0 "s->ctx && s->dataplane_started", file=file@entry=0x560dd8b73578 "/builddir/build/BUILD/qemu-5.1.0/hw/scsi/virtio-scsi-dataplane.c", line=line@entry=72, function=function@entry=0x560dd8b736a0 <__PRETTY_FUNCTION__.29744> "virtio_scsi_data_plane_handle_ctrl") at assert.c:101
#4  0x0000560dd87dc179 in virtio_scsi_data_plane_handle_ctrl
    (vdev=<optimized out>, vq=0x560de5c37030)
    at /usr/src/debug/qemu-kvm-5.1.0-12.module+el8.3.0+7652+b30e6901.mlevitsk202011111422.x86_64/hw/scsi/virtio-scsi-dataplane.c:72
#5  0x0000560dd87dc179 in virtio_scsi_data_plane_handle_ctrl
    (vdev=<optimized out>, vq=0x560de5c37030)
    at /usr/src/debug/qemu-kvm-5.1.0-12.module+el8.3.0+7652+b30e6901.mlevitsk202011111422.x86_64/hw/scsi/virtio-scsi-dataplane.c:65
#6  0x0000560dd87e9b8e in virtio_queue_notify_aio_vq (vq=vq@entry=0x560de5c37030)
    at /usr/src/debug/qemu-kvm-5.1.0-12.module+el8.3.0+7652+b30e6901.mlevitsk202011111422.x86_64/hw/virtio/virtio.c:2325
--Type <RET> for more, q to quit, c to continue without paging--
#7  0x0000560dd87e9be9 in virtio_queue_notify_aio_vq (vq=0x560de5c37030)
    at /usr/src/debug/qemu-kvm-5.1.0-12.module+el8.3.0+7652+b30e6901.mlevitsk202011111422.x86_64/hw/virtio/virtio.c:3530
#8  0x0000560dd87e9be9 in virtio_queue_host_notifier_aio_read (n=n@entry=0x560de5c370a8)
    at /usr/src/debug/qemu-kvm-5.1.0-12.module+el8.3.0+7652+b30e6901.mlevitsk202011111422.x86_64/hw/virtio/virtio.c:3530
#9  0x0000560dd87ecf7d in virtio_queue_aio_set_host_notifier_handler
    (vq=0x560de5c37030, ctx=0x560dd9b90a00, handle_output=handle_output@entry=0x0)
    at /usr/src/debug/qemu-kvm-5.1.0-12.module+el8.3.0+7652+b30e6901.mlevitsk202011111422.x86_64/hw/virtio/virtio.c:3576
#10 0x0000560dd87dbe53 in virtio_scsi_dataplane_stop_bh (opaque=0x560dddcf71d0)
    at /usr/src/debug/qemu-kvm-5.1.0-12.module+el8.3.0+7652+b30e6901.mlevitsk202011111422.x86_64/hw/scsi/virtio-scsi-dataplane.c:117
#11 0x0000560dd8ad12af in aio_wait_bh (opaque=0x7fa1c3ffe360)
    at /usr/src/debug/qemu-kvm-5.1.0-12.module+el8.3.0+7652+b30e6901.mlevitsk202011111422.x86_64/util/aio-wait.c:55
#12 0x0000560dd8ad16ad in aio_bh_call (bh=0x7fa1bd515c40)
    at /usr/src/debug/qemu-kvm-5.1.0-12.module+el8.3.0+7652+b30e6901.mlevitsk202011111422.x86_64/util/async.c:164
#13 0x0000560dd8ad16ad in aio_bh_poll (ctx=ctx@entry=0x560dd9b90a00)
    at /usr/src/debug/qemu-kvm-5.1.0-12.module+el8.3.0+7652+b30e6901.mlevitsk202011111422.x86_64/util/async.c:164
#14 0x0000560dd8ac0e82 in aio_poll (ctx=0x560dd9b90a00, blocking=blocking@entry=true)
Comment 2 qing.wang 2020-11-13 12:31:20 UTC 
it enable add timeout in each plug like as sleep 0.3

{'execute': 'blockdev-add', 'arguments': {'node-name': 'file_stg198', 'driver': 'file','filename': '/home/kvm_autotest_root/images/stg198.qcow2'}}
{'execute': 'blockdev-add', 'arguments': {'node-name': 'drive_stg198', 'driver': 'qcow2', 'file': 'file_stg198'}} 
{"error": {"class": "GenericError", "desc": "Could not open '/home/kvm_autotest_root/images/stg198.qcow2': Too many open files"}}

So the question is how many disks can be added?
Comment 9 Maxim Levitsky 2020-12-10 16:30:55 UTC 
Just reproduced this on latest upstream qemu.

qemu-system-x86_64: ../src/hw/scsi/virtio-scsi-dataplane.c:72: virtio_scsi_data_plane_handle_ctrl: Assertion `s->ctx && s->dataplane_started' failed.

Best regards,
   Maxim Levitsky
Comment 10 Maxim Levitsky 2020-12-17 10:47:05 UTC 
I finished analyzing this problem.

The root cause of this is running out of
open file descriptors, which are limited by corresponding ulimit.
On my system it is 1024.

Each virtio device when running on anything but very old KVM
uses a single irqfd and ioeventfd per virtio queue.

Each virtio-scsi, even with a single data vq (by default) has at least 3 extra vqs
used for control, events, etc, which makes it 4 vqs per device.

In total for 200 virtio devices we need 200 * 4 * 2 = 1600 open file descriptors 
which is over the default ulimit on my machine at least.

Now if the creation of an eventfd/irqfd fails, the failback path that has 
a bunch of bugs that cause qemu either to crash with an assert as was reported in 
this bug report or it can hang with a deadlock.
I fixed a bunch of such issues and I am sending an RFC with the fixes, 
but the deadlock is nothing but a simple issue to fix.

The deadlock happens because 

1. vCPU thread gets the doorbell notification (which otherwise should go 
   directly to iothread via ioeventfd)

2. it takes BQL (big qemu lock)
3. it calls into virtio-scsi code which takes AIO (iothread) lock before it 
   touches the stuff that is shared with the iothread.

On the other hand the iothread does the following

1. it takes AIO lock (it is taken most of the time the IO thread is active)
2. it processes IO that was just completed, and needs to signal an MSI
   which due to lack of irqfd, goes through a MMIO write which
   tries to take the BQL

This leads to AB-BA deadlock.

I will eventually work on eliminating the AIO lock taking in this code path 
as was suggested to me by Paulo, which together with my fixes to the
fallback path might make qemu work in fallback mode,  but I am not sure that
the fallback mode is worth it.

For now I think it is best to just increase the open file limit, 
since without irqfd/ioeventfd there will be a very significant performance 
hit anyway even if qemu were coping with the failure and activated the fallback
path.

One thing that can be done on qemu level is to check the ulimit vs number of 
virtio devices and kill the VM earlier before the guest starts 
enabling the devices.

Or this can also be easily done on libvirt level, 
by adjusting the ulimit by the number of iothread enabled devices 
times number of queues each device has.

Finally note that qemu does warn about the root cause of the problem.

qemu-system-x86_64: virtio_bus_set_host_notifier: unable to init event notifier: Too many open files (-24)
virtio-scsi: Failed to set host notifier (-24)
qemu-system-x86_64: ../src/hw/scsi/virtio-scsi-dataplane.c:59: virtio_scsi_data_plane_handle_cmd: Assertion `s->ctx && s->dataplane_started' failed
Comment 11 Maxim Levitsky 2020-12-17 10:50:52 UTC 
Adding few random people from libvirt to hear your opinion.

Does libvirt adjust open file limit based on number of
virtio devices?
Comment 12 Maxim Levitsky 2020-12-17 15:02:58 UTC 
I posted upstream few random hacks I made during debug of this bug
with title

"RFC: few random hacks to improve eventfd fallback path"
Comment 13 Peter Krempa 2020-12-17 16:02:32 UTC 
We don't do it automatically, but users wishing to use massive VMs can adjust the limit in /etc/libvirt/qemu.conf by setting 'max_files'.
Comment 14 Maxim Levitsky 2020-12-17 17:19:13 UTC 
Maybe we should?

For say 32 core VM, and say 15 disks, number of open files will be (32 + 3) * 2 * 15  = 1050
which is already above the default 1024 limit I have on my fedora 32.

Of course such VM does to some extent qualify as massive...

Thanks for the quick reply though!
Comment 15 Maxim Levitsky 2021-03-18 09:05:44 UTC 
Due to all feedback received I am closing this bug.

I do still think that libvirt should be able to increase the 
max open file limit automatically for large VMs though.
Note You need to log in before you can comment on or make changes to this bug.