Description of problem: Version-Release number of selected component (if applicable): [root@dell-per730-52 ~]# uname -r 5.14.0-230.mergert.el9.x86_64 [root@dell-per730-52 ~]# rpm -qa|grep openvs kernel-kernel-networking-openvswitch-common-2.0-279.noarch kernel-kernel-networking-openvswitch-perf-1.0-371.noarch openvswitch-selinux-extra-policy-1.0-31.el9fdp.noarch openvswitch2.17-2.17.0-55.el9fdp.x86_64 How reproducible: Steps to Reproduce: Run ovs dpdk pvp 1queue viommu case 1. Build ovs dpdk pvp topo [root@dell-per730-52 ~]# ovs-vsctl show d44eedd4-336a-4f9a-9b2e-ea4e8c83bd10 Bridge ovsbr0 datapath_type: netdev Port dpdk1 Interface dpdk1 type: dpdk options: {dpdk-devargs="0000:07:00.1", n_rxq="1", n_rxq_desc="1024", n_txq_desc="1024"} Port vhost0 Interface vhost0 type: dpdkvhostuserclient options: {vhost-server-path="/tmp/vhostuser/vhost0"} Port vhost1 Interface vhost1 type: dpdkvhostuserclient options: {vhost-server-path="/tmp/vhostuser/vhost1"} Port ovsbr0 Interface ovsbr0 type: internal Port dpdk0 Interface dpdk0 type: dpdk options: {dpdk-devargs="0000:07:00.0", n_rxq="1", n_rxq_desc="1024", n_txq_desc="1024"} ovs_version: "2.17.4" [root@dell-per730-52 ~]# ovs-vsctl get Open_vSwitch . other_config {dpdk-init="true", dpdk-lcore-mask="0x1", dpdk-socket-mem="4096", pmd-cpu-mask="40000004000000", userspace-tso-enable="false", vhost-iommu-support="true"} 2. start guest with following xml <domain type='kvm'> <name>g1</name> <memory unit='KiB'>8388608</memory> <currentMemory unit='KiB'>8388608</currentMemory> <memoryBacking> <hugepages> <page size='1048576' unit='KiB'/> </hugepages> <locked/> <access mode='shared'/> </memoryBacking> <vcpu placement='static'>3</vcpu> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='30'/> <vcpupin vcpu='2' cpuset='4'/> <emulatorpin cpuset='0,28'/> </cputune> <numatune> <memory mode='strict' nodeset='0'/> </numatune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='q35'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pmu state='off'/> <vmport state='off'/> <ioapic driver='qemu'/> </features> <cpu mode='host-passthrough' check='none'> <feature policy='require' name='tsc-deadline'/> <numa> <cell id='0' cpus='0-2' memory='8388608' unit='KiB' memAccess='shared'/> </numa> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/g1.qcow2'/> <backingStore/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </disk> <controller type='usb' index='0' model='none'> <alias name='usb'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x10'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x11'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0x8'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0x9'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xa'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xb'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <controller type='sata' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <interface type='bridge'> <mac address='52:54:00:01:02:03'/> <source bridge='virbr0'/> <model type='virtio'/> </interface> <interface type='vhostuser'> <mac address='00:de:ad:00:00:01'/> <source type='unix' path='/tmp/vhostuser/vhost0' mode='server'/> <model type='virtio'/> <driver name='vhost' queues='1' rx_queue_size='1024' tx_queue_size='1024' iommu='on' ats='on'> <host mrg_rxbuf='off'/> </driver> <address type='pci' domain='0x0000' bus='0x3' slot='0x00' function='0x0'/> </interface> <interface type='vhostuser'> <mac address='00:de:ad:00:00:02'/> <source type='unix' path='/tmp/vhostuser/vhost1' mode='server'/> <model type='virtio'/> <driver name='vhost' queues='1' rx_queue_size='1024' tx_queue_size='1024' iommu='on' ats='on'> <host mrg_rxbuf='off'/> </driver> <address type='pci' domain='0x0000' bus='0x4' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/1'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/1'> <source path='/dev/pts/1'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <input type='mouse' bus='ps2'> <alias name='input0'/> </input> <input type='keyboard' bus='ps2'> <alias name='input1'/> </input> <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='cirrus' vram='16384' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </memballoon> <iommu model='intel'> <driver intremap='on' caching_mode='on' iotlb='on'/> </iommu> </devices> <seclabel type='dynamic' model='selinux' relabel='yes'/> </domain> Actual results: Start guest failed. So the 1 queue viommu case run failed. Expected results: The 1 queue viommu case can work well. Additional info: https://beaker.engineering.redhat.com/jobs/7424884
libvirt log: [root@dell-per730-52 ~]# cat /var/log/libvirt/qemu/g1.log 2023-01-13 13:10:44.059+0000: starting up libvirt version: 8.10.0, package: 2.el9 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2022-12-06-04:27:41, ), qemu version: 7.2.0qemu-kvm-7.2.0-2.el9, kernel: 5.14.0-230.mergert.el9.x86_64, hostname: dell-per730-52.rhts.eng.pek2.redhat.com LC_ALL=C \ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \ HOME=/var/lib/libvirt/qemu/domain-1-g1 \ XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-g1/.local/share \ XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-g1/.cache \ XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-g1/.config \ /usr/libexec/qemu-kvm \ -name guest=g1,debug-threads=on \ -S \ -object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-1-g1/master-key.aes"}' \ -machine pc-q35-rhel9.2.0,usb=off,vmport=off,kernel_irqchip=split,dump-guest-core=off \ -accel kvm \ -cpu host,migratable=on,tsc-deadline=on,pmu=off \ -m 8192 \ -overcommit mem-lock=on \ -smp 3,sockets=3,cores=1,threads=1 \ -object '{"qom-type":"thread-context","id":"tc-ram-node0","node-affinity":[0]}' \ -object '{"qom-type":"memory-backend-file","id":"ram-node0","mem-path":"/dev/hugepages/libvirt/qemu/1-g1","share":true,"prealloc":true,"size":8589934592,"host-nodes":[0],"policy":"bind","prealloc-context":"tc-ram-node0"}' \ -numa node,nodeid=0,cpus=0-2,memdev=ram-node0 \ -uuid 36095c8c-6865-41b1-9c0d-c3a6731ac6aa \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=32,server=on,wait=off \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=utc,driftfix=slew \ -global kvm-pit.lost_tick_policy=delay \ -no-hpet \ -no-shutdown \ -global ICH9-LPC.disable_s3=1 \ -global ICH9-LPC.disable_s4=1 \ -boot strict=on \ -device '{"driver":"intel-iommu","id":"iommu0","intremap":"on","caching-mode":true,"device-iotlb":true}' \ -device '{"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","addr":"0x2"}' \ -device '{"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x3"}' \ -device '{"driver":"pcie-root-port","port":8,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x4"}' \ -device '{"driver":"pcie-root-port","port":9,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x5"}' \ -device '{"driver":"pcie-root-port","port":10,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x6"}' \ -device '{"driver":"pcie-root-port","port":11,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x7"}' \ -blockdev '{"driver":"file","filename":"/var/lib/libvirt/images/g1.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":null}' \ -device '{"driver":"virtio-blk-pci","bus":"pci.1","addr":"0x0","drive":"libvirt-1-format","id":"virtio-disk0","bootindex":1}' \ -netdev '{"type":"tap","fd":"33","vhost":true,"vhostfd":"35","id":"hostnet0"}' \ -device '{"driver":"virtio-net-pci","netdev":"hostnet0","id":"net0","mac":"52:54:00:01:02:03","bus":"pci.2","addr":"0x0"}' \ -chardev socket,id=charnet1,path=/tmp/vhostuser/vhost0,server=on \ -netdev '{"type":"vhost-user","chardev":"charnet1","id":"hostnet1"}' \ -device '{"driver":"virtio-net-pci","iommu_platform":true,"ats":true,"mrg_rxbuf":false,"rx_queue_size":1024,"tx_queue_size":1024,"netdev":"hostnet1","id":"net1","mac":"00:de:ad:00:00:01","bus":"pci.3","addr":"0x0"}' \ -chardev socket,id=charnet2,path=/tmp/vhostuser/vhost1,server=on \ -netdev '{"type":"vhost-user","chardev":"charnet2","id":"hostnet2"}' \ -device '{"driver":"virtio-net-pci","iommu_platform":true,"ats":true,"mrg_rxbuf":false,"rx_queue_size":1024,"tx_queue_size":1024,"netdev":"hostnet2","id":"net2","mac":"00:de:ad:00:00:02","bus":"pci.4","addr":"0x0"}' \ -chardev pty,id=charserial0 \ -device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0}' \ -audiodev '{"id":"audio1","driver":"none"}' \ -vnc 0.0.0.0:0,audiodev=audio1 \ -device '{"driver":"cirrus-vga","id":"video0","bus":"pci.5","addr":"0x0"}' \ -device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.6","addr":"0x0"}' \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on 2023-01-13T13:10:44.130006Z qemu-kvm: -chardev socket,id=charnet1,path=/tmp/vhostuser/vhost0,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhostuser/vhost0,server=on 2023-01-13T13:10:44.644338Z qemu-kvm: -chardev socket,id=charnet2,path=/tmp/vhostuser/vhost1,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhostuser/vhost1,server=on char device redirected to /dev/pts/2 (label charserial0) 2023-01-13T13:10:47.444502Z qemu-kvm: -device {"driver":"cirrus-vga","id":"video0","bus":"pci.5","addr":"0x0"}: warning: 'cirrus-vga' is deprecated, please use a different VGA card instead 2023-01-13T13:10:57.093391Z qemu-kvm: Received unexpected msg type. Expected 22 received 40 2023-01-13T13:10:57.093414Z qemu-kvm: Fail to update device iotlb 2023-01-13T13:10:57.093463Z qemu-kvm: Received unexpected msg type. Expected 40 received 22 2023-01-13T13:10:57.104339Z qemu-kvm: Received unexpected msg type. Expected 22 received 11 2023-01-13T13:10:57.104349Z qemu-kvm: Fail to update device iotlb 2023-01-13T13:10:57.104583Z qemu-kvm: Received unexpected msg type. Expected 11 received 22 2023-01-13T13:10:57.104592Z qemu-kvm: vhost VQ 1 ring restore failed: -71: Protocol error (71) 2023-01-13T13:10:57.108169Z qemu-kvm: unable to start vhost net: 71: falling back on userspace virtio 2023-01-13T13:10:57.208006Z qemu-kvm: Received unexpected msg type. Expected 22 received 40 2023-01-13T13:10:57.208020Z qemu-kvm: Fail to update device iotlb 2023-01-13T13:10:57.208071Z qemu-kvm: Received unexpected msg type. Expected 40 received 22 2023-01-13T13:10:57.208321Z qemu-kvm: Received unexpected msg type. Expected 22 received 11 2023-01-13T13:10:57.208330Z qemu-kvm: Fail to update device iotlb 2023-01-13T13:10:57.208538Z qemu-kvm: Received unexpected msg type. Expected 11 received 22 2023-01-13T13:10:57.208546Z qemu-kvm: vhost VQ 1 ring restore failed: -71: Protocol error (71) 2023-01-13T13:10:57.208607Z qemu-kvm: Received unexpected msg type. Expected 22 received 11 2023-01-13T13:10:57.208615Z qemu-kvm: Fail to update device iotlb 2023-01-13T13:10:57.208641Z qemu-kvm: Received unexpected msg type. Expected 11 received 22 2023-01-13T13:10:57.208649Z qemu-kvm: vhost VQ 0 ring restore failed: -71: Protocol error (71) 2023-01-13T13:10:57.212170Z qemu-kvm: unable to start vhost net: 71: falling back on userspace virtio 2023-01-13T13:12:26.201845Z qemu-kvm: Received unexpected msg type. Expected 22 received 40 2023-01-13T13:12:26.201867Z qemu-kvm: Fail to update device iotlb 2023-01-13T13:12:26.201916Z qemu-kvm: Received unexpected msg type. Expected 40 received 22 2023-01-13T13:12:26.202204Z qemu-kvm: Received unexpected msg type. Expected 22 received 11 2023-01-13T13:12:26.202213Z qemu-kvm: Fail to update device iotlb 2023-01-13T13:12:26.202445Z qemu-kvm: Received unexpected msg type. Expected 11 received 22 2023-01-13T13:12:26.202454Z qemu-kvm: vhost VQ 1 ring restore failed: -71: Protocol error (71)
I met the same issue. https://beaker.engineering.redhat.com/recipes/13233855#task154894511 BTW- 2q works well.
The rhel9.2 ctc2 also has this issue. https://beaker.engineering.redhat.com/jobs/7442474 https://beaker-archive.hosts.prod.psi.bos.redhat.com/beaker-logs/2023/01/74424/7442474/13247732/155100364/i40e_25.html
mlx5: https://beaker.engineering.redhat.com/jobs/7442362 https://beaker-archive.hosts.prod.psi.bos.redhat.com/beaker-logs/2023/01/74423/7442362/13247467/155097963/mlx5_100.html
This bug is a duplicate of Bz2155173. The issue is on QEMU side, not in OVS. Closing as duplicate. *** This bug has been marked as a duplicate of bug 2155173 ***