Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 2160718

Summary: The guest start failed when running ovs dpdk pvp 1 queue viommu case on kernel 5.14.0-230.mergert.el9.x86_64
Product: Red Hat Enterprise Linux Fast Datapath Reporter: liting <tli>
Component: openvswitch2.17Assignee: Maxime Coquelin <maxime.coquelin>
Status: CLOSED DUPLICATE QA Contact: ovs-qe
Severity: medium Docs Contact:
Priority: high    
Version: FDP 22.LCC: ctrautma, fleitner, jhsiao, jiqiu, ralongi
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-01-19 20:01:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description liting 2023-01-13 13:09:31 UTC
Description of problem:


Version-Release number of selected component (if applicable):
[root@dell-per730-52 ~]# uname -r
5.14.0-230.mergert.el9.x86_64
[root@dell-per730-52 ~]# rpm -qa|grep openvs
kernel-kernel-networking-openvswitch-common-2.0-279.noarch
kernel-kernel-networking-openvswitch-perf-1.0-371.noarch
openvswitch-selinux-extra-policy-1.0-31.el9fdp.noarch
openvswitch2.17-2.17.0-55.el9fdp.x86_64


How reproducible:


Steps to Reproduce:
Run ovs dpdk pvp 1queue viommu case
1. Build ovs dpdk pvp topo
[root@dell-per730-52 ~]# ovs-vsctl show
d44eedd4-336a-4f9a-9b2e-ea4e8c83bd10
    Bridge ovsbr0
        datapath_type: netdev
        Port dpdk1
            Interface dpdk1
                type: dpdk
                options: {dpdk-devargs="0000:07:00.1", n_rxq="1", n_rxq_desc="1024", n_txq_desc="1024"}
        Port vhost0
            Interface vhost0
                type: dpdkvhostuserclient
                options: {vhost-server-path="/tmp/vhostuser/vhost0"}
        Port vhost1
            Interface vhost1
                type: dpdkvhostuserclient
                options: {vhost-server-path="/tmp/vhostuser/vhost1"}
        Port ovsbr0
            Interface ovsbr0
                type: internal
        Port dpdk0
            Interface dpdk0
                type: dpdk
                options: {dpdk-devargs="0000:07:00.0", n_rxq="1", n_rxq_desc="1024", n_txq_desc="1024"}
    ovs_version: "2.17.4"
[root@dell-per730-52 ~]# ovs-vsctl get Open_vSwitch . other_config
{dpdk-init="true", dpdk-lcore-mask="0x1", dpdk-socket-mem="4096", pmd-cpu-mask="40000004000000", userspace-tso-enable="false", vhost-iommu-support="true"}

2. start guest with following xml
<domain type='kvm'>
  <name>g1</name>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB'/>
    </hugepages>
    <locked/>
    <access mode='shared'/>
  </memoryBacking>
  <vcpu placement='static'>3</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='30'/>
    <vcpupin vcpu='2' cpuset='4'/>
    <emulatorpin cpuset='0,28'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='q35'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pmu state='off'/>
    <vmport state='off'/>
    <ioapic driver='qemu'/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <feature policy='require' name='tsc-deadline'/>
    <numa>
      <cell id='0' cpus='0-2' memory='8388608' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/g1.qcow2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='none'>
      <alias name='usb'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x8'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x9'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xa'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xb'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:01:02:03'/>
      <source bridge='virbr0'/>
      <model type='virtio'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='00:de:ad:00:00:01'/>
      <source type='unix' path='/tmp/vhostuser/vhost0' mode='server'/>
      <model type='virtio'/>
      <driver name='vhost' queues='1' rx_queue_size='1024' tx_queue_size='1024' iommu='on' ats='on'>
        <host mrg_rxbuf='off'/>
      </driver>
      <address type='pci' domain='0x0000' bus='0x3' slot='0x00' function='0x0'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='00:de:ad:00:00:02'/>
      <source type='unix' path='/tmp/vhostuser/vhost1' mode='server'/>
      <model type='virtio'/>
      <driver name='vhost' queues='1' rx_queue_size='1024' tx_queue_size='1024' iommu='on' ats='on'>
        <host mrg_rxbuf='off'/>
      </driver>
      <address type='pci' domain='0x0000' bus='0x4' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/1'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/1'>
      <source path='/dev/pts/1'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </memballoon>
    <iommu model='intel'>
      <driver intremap='on' caching_mode='on' iotlb='on'/>
    </iommu>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'/>
</domain>


Actual results:
Start guest failed. So the 1 queue viommu case run failed.

Expected results:
The 1 queue viommu case can work well.

Additional info:
https://beaker.engineering.redhat.com/jobs/7424884

Comment 1 liting 2023-01-13 13:16:18 UTC
libvirt log:
[root@dell-per730-52 ~]# cat /var/log/libvirt/qemu/g1.log
2023-01-13 13:10:44.059+0000: starting up libvirt version: 8.10.0, package: 2.el9 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2022-12-06-04:27:41, ), qemu version: 7.2.0qemu-kvm-7.2.0-2.el9, kernel: 5.14.0-230.mergert.el9.x86_64, hostname: dell-per730-52.rhts.eng.pek2.redhat.com
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \
HOME=/var/lib/libvirt/qemu/domain-1-g1 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-g1/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-g1/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-g1/.config \
/usr/libexec/qemu-kvm \
-name guest=g1,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-1-g1/master-key.aes"}' \
-machine pc-q35-rhel9.2.0,usb=off,vmport=off,kernel_irqchip=split,dump-guest-core=off \
-accel kvm \
-cpu host,migratable=on,tsc-deadline=on,pmu=off \
-m 8192 \
-overcommit mem-lock=on \
-smp 3,sockets=3,cores=1,threads=1 \
-object '{"qom-type":"thread-context","id":"tc-ram-node0","node-affinity":[0]}' \
-object '{"qom-type":"memory-backend-file","id":"ram-node0","mem-path":"/dev/hugepages/libvirt/qemu/1-g1","share":true,"prealloc":true,"size":8589934592,"host-nodes":[0],"policy":"bind","prealloc-context":"tc-ram-node0"}' \
-numa node,nodeid=0,cpus=0-2,memdev=ram-node0 \
-uuid 36095c8c-6865-41b1-9c0d-c3a6731ac6aa \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=32,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device '{"driver":"intel-iommu","id":"iommu0","intremap":"on","caching-mode":true,"device-iotlb":true}' \
-device '{"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","addr":"0x2"}' \
-device '{"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x3"}' \
-device '{"driver":"pcie-root-port","port":8,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x4"}' \
-device '{"driver":"pcie-root-port","port":9,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x5"}' \
-device '{"driver":"pcie-root-port","port":10,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x6"}' \
-device '{"driver":"pcie-root-port","port":11,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x7"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/images/g1.qcow2","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":null}' \
-device '{"driver":"virtio-blk-pci","bus":"pci.1","addr":"0x0","drive":"libvirt-1-format","id":"virtio-disk0","bootindex":1}' \
-netdev '{"type":"tap","fd":"33","vhost":true,"vhostfd":"35","id":"hostnet0"}' \
-device '{"driver":"virtio-net-pci","netdev":"hostnet0","id":"net0","mac":"52:54:00:01:02:03","bus":"pci.2","addr":"0x0"}' \
-chardev socket,id=charnet1,path=/tmp/vhostuser/vhost0,server=on \
-netdev '{"type":"vhost-user","chardev":"charnet1","id":"hostnet1"}' \
-device '{"driver":"virtio-net-pci","iommu_platform":true,"ats":true,"mrg_rxbuf":false,"rx_queue_size":1024,"tx_queue_size":1024,"netdev":"hostnet1","id":"net1","mac":"00:de:ad:00:00:01","bus":"pci.3","addr":"0x0"}' \
-chardev socket,id=charnet2,path=/tmp/vhostuser/vhost1,server=on \
-netdev '{"type":"vhost-user","chardev":"charnet2","id":"hostnet2"}' \
-device '{"driver":"virtio-net-pci","iommu_platform":true,"ats":true,"mrg_rxbuf":false,"rx_queue_size":1024,"tx_queue_size":1024,"netdev":"hostnet2","id":"net2","mac":"00:de:ad:00:00:02","bus":"pci.4","addr":"0x0"}' \
-chardev pty,id=charserial0 \
-device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-vnc 0.0.0.0:0,audiodev=audio1 \
-device '{"driver":"cirrus-vga","id":"video0","bus":"pci.5","addr":"0x0"}' \
-device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.6","addr":"0x0"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2023-01-13T13:10:44.130006Z qemu-kvm: -chardev socket,id=charnet1,path=/tmp/vhostuser/vhost0,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhostuser/vhost0,server=on
2023-01-13T13:10:44.644338Z qemu-kvm: -chardev socket,id=charnet2,path=/tmp/vhostuser/vhost1,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhostuser/vhost1,server=on
char device redirected to /dev/pts/2 (label charserial0)
2023-01-13T13:10:47.444502Z qemu-kvm: -device {"driver":"cirrus-vga","id":"video0","bus":"pci.5","addr":"0x0"}: warning: 'cirrus-vga' is deprecated, please use a different VGA card instead
2023-01-13T13:10:57.093391Z qemu-kvm: Received unexpected msg type. Expected 22 received 40
2023-01-13T13:10:57.093414Z qemu-kvm: Fail to update device iotlb
2023-01-13T13:10:57.093463Z qemu-kvm: Received unexpected msg type. Expected 40 received 22
2023-01-13T13:10:57.104339Z qemu-kvm: Received unexpected msg type. Expected 22 received 11
2023-01-13T13:10:57.104349Z qemu-kvm: Fail to update device iotlb
2023-01-13T13:10:57.104583Z qemu-kvm: Received unexpected msg type. Expected 11 received 22
2023-01-13T13:10:57.104592Z qemu-kvm: vhost VQ 1 ring restore failed: -71: Protocol error (71)
2023-01-13T13:10:57.108169Z qemu-kvm: unable to start vhost net: 71: falling back on userspace virtio
2023-01-13T13:10:57.208006Z qemu-kvm: Received unexpected msg type. Expected 22 received 40
2023-01-13T13:10:57.208020Z qemu-kvm: Fail to update device iotlb
2023-01-13T13:10:57.208071Z qemu-kvm: Received unexpected msg type. Expected 40 received 22
2023-01-13T13:10:57.208321Z qemu-kvm: Received unexpected msg type. Expected 22 received 11
2023-01-13T13:10:57.208330Z qemu-kvm: Fail to update device iotlb
2023-01-13T13:10:57.208538Z qemu-kvm: Received unexpected msg type. Expected 11 received 22
2023-01-13T13:10:57.208546Z qemu-kvm: vhost VQ 1 ring restore failed: -71: Protocol error (71)
2023-01-13T13:10:57.208607Z qemu-kvm: Received unexpected msg type. Expected 22 received 11
2023-01-13T13:10:57.208615Z qemu-kvm: Fail to update device iotlb
2023-01-13T13:10:57.208641Z qemu-kvm: Received unexpected msg type. Expected 11 received 22
2023-01-13T13:10:57.208649Z qemu-kvm: vhost VQ 0 ring restore failed: -71: Protocol error (71)
2023-01-13T13:10:57.212170Z qemu-kvm: unable to start vhost net: 71: falling back on userspace virtio
2023-01-13T13:12:26.201845Z qemu-kvm: Received unexpected msg type. Expected 22 received 40
2023-01-13T13:12:26.201867Z qemu-kvm: Fail to update device iotlb
2023-01-13T13:12:26.201916Z qemu-kvm: Received unexpected msg type. Expected 40 received 22
2023-01-13T13:12:26.202204Z qemu-kvm: Received unexpected msg type. Expected 22 received 11
2023-01-13T13:12:26.202213Z qemu-kvm: Fail to update device iotlb
2023-01-13T13:12:26.202445Z qemu-kvm: Received unexpected msg type. Expected 11 received 22
2023-01-13T13:12:26.202454Z qemu-kvm: vhost VQ 1 ring restore failed: -71: Protocol error (71)

Comment 2 Jiying Qiu 2023-01-18 02:11:05 UTC
I met the same issue. 
https://beaker.engineering.redhat.com/recipes/13233855#task154894511 

BTW- 2q works well.

Comment 6 Maxime Coquelin 2023-01-19 20:01:58 UTC
This bug is a duplicate of Bz2155173.
The issue is on QEMU side, not in OVS.

Closing as duplicate.

*** This bug has been marked as a duplicate of bug 2155173 ***