Bug 2129007

Description of problem:
start vhostuserserver port failed with iommu_platform=true is not supported by the device.

Version-Release number of selected component (if applicable):
kernel version: 5.14.0-70.26.1.rt21.98.el9_0.x86_64
ovs version: openvswitch2.17-2.17.0-43.el9fdp
qemu version: qemu-kvm-6.2.0-11.el9_0.5.x86_64
libvirt version: libvirt-8.0.0-8.1.el9_0.x86_64

How reproducible: 100%


Steps to Reproduce:
1. enable iommu on cmdline as below:
# cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-70.26.1.rt21.98.el9_0.x86_64 root=/dev/mapper/rhel_hp--dl388g10--03-root ro enforcing=1 kpti selinux=1 resume=/dev/mapper/rhel_hp--dl388g10--03-swap rd.lvm.lv=rhel_hp-dl388g10-03/root rd.lvm.lv=rhel_hp-dl388g10-03/swap console=ttyS0,115200n81 skew_tick=1 isolcpus=managed_irq,domain,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39 intel_pstate=disable nosoftlockup tsc=nowatchdog nohz=on nohz_full=1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39 rcu_nocbs=1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39 irqaffinity=0,20 crashkernel=1G-4G:384M,4G-16G:512M,16G-64G:1G,64G-128G:2G,128G-:4G mitigations=off nohz=on default_hugepagesz=1G hugepagesz=1G hugepages=48 intel_iommu=on iommu=pt modprobe.blacklist=qedi modprobe.blacklist=qedf modprobe.blacklist=qedr

2. start openvswitch and enable vhost-iommu-support and create vhostuserclient port.
# ovs-vsctl list Open_vSwitch .
_uuid               : ba631ab8-b621-46d2-8e29-6332a0eae59c
bridges             : [d1d81053-ece8-4c09-83ca-8d5fd3e3b204]
cur_cfg             : 6
datapath_types      : [netdev, system]
datapaths           : {}
db_version          : []
dpdk_initialized    : true
dpdk_version        : "DPDK 21.11.2"
external_ids        : {}
iface_types         : [bareudp, dpdk, dpdkvhostuser, dpdkvhostuserclient, erspan, geneve, gre, gtpu, internal, ip6erspan, ip6gre, lisp, patch, stt, system, tap, vxlan]
manager_options     : []
next_cfg            : 6
other_config        : {dpdk-init="true", dpdk-lcore-mask="0x1", dpdk-socket-mem="1024", vhost-iommu-support="true"}
ovs_version         : []
ssl                 : []
statistics          : {}
system_type         : []
system_version      : []
# ovs-vsctl add-port ovsbr0 vhost0 -- set interface vhost0 type=dpdkvhostuserclient options:vhost-server-path=/tmp/vhost0 options:n_rxq=2 mtu_request=9120
# ovs-vsctl show
ba631ab8-b621-46d2-8e29-6332a0eae59c
    Bridge ovsbr0
        datapath_type: netdev
        Port ovsbr0
            Interface ovsbr0
                type: internal
        Port vhost0
            Interface vhost0
                type: dpdkvhostuserclient
                options: {n_rxq="2", vhost-server-path="/tmp/vhost0"}

3. create guest XML and add vhostuserserver port.
# cat g1.xml 
<domain type="kvm">
  <name>guest30032</name>
  <uuid>37425e76-af6a-44a6-aba0-73434afe34c0</uuid>
  <memory unit="KiB">8388608</memory>
  <currentMemory unit="KiB">8388608</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size="1048576" unit="KiB" />
    </hugepages>
    <access mode="shared" />
  </memoryBacking>
  <vcpu placement="static">3</vcpu>
  <cputune>
    <vcpupin vcpu="0" cpuset="1" />
    <vcpupin vcpu="1" cpuset="2" />
    <vcpupin vcpu="2" cpuset="3" />
    <vcpusched vcpus="0" scheduler="fifo" priority="1" />
    <vcpusched vcpus="1" scheduler="fifo" priority="1" />
    <vcpusched vcpus="2" scheduler="fifo" priority="1" />
    <emulatorpin cpuset="0" />
  </cputune>
  <numatune>
    <memory mode="strict" nodeset="0" />
    <memnode cellid="0" mode="strict" nodeset="0" />
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch="x86_64" machine="q35">hvm</type>
    <boot dev="hd" />
  </os>
  <features>
    <acpi />
    <pmu state="off" />
    <vmport state="off" />
    <ioapic driver="qemu" />
  </features>
  <cpu mode="host-passthrough" check="none">
    <feature policy="require" name="tsc-deadline" />
    <numa>
      <cell id="0" cpus="0-2" memory="8388608" unit="KiB" memAccess="shared" />
    </numa>
  </cpu>
  <clock offset="utc">
    <timer name="rtc" tickpolicy="catchup" />
    <timer name="pit" tickpolicy="delay" />
    <timer name="hpet" present="no" />
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled="no" />
    <suspend-to-disk enabled="no" />
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2" />
      <source file="/home/guest30032.qcow2" />
      <target dev="vda" bus="virtio" />
      <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0" />
    </disk>
    <controller type="usb" index="0" model="none" />
    <controller type="pci" index="0" model="pcie-root" />
    <controller type="pci" index="1" model="pcie-root-port">
      <model name="pcie-root-port" />
      <target chassis="1" port="0x10" />
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" />
    </controller>
    <controller type="pci" index="2" model="pcie-root-port">
      <model name="pcie-root-port" />
      <target chassis="2" port="0x11" />
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0" />
    </controller>
    <controller type="pci" index="3" model="pcie-root-port">
      <model name="pcie-root-port" />
      <target chassis="3" port="0x8" />
      <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0" />
    </controller>
    <controller type="pci" index="4" model="pcie-root-port">
      <model name="pcie-root-port" />
      <target chassis="4" port="0x9" />
      <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0" />
    </controller>
    <controller type="pci" index="5" model="pcie-root-port">
      <model name="pcie-root-port" />
      <target chassis="5" port="0xa" />
      <address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x0" />
    </controller>
    <controller type="pci" index="6" model="pcie-root-port">
      <model name="pcie-root-port" />
      <target chassis="6" port="0xb" />
      <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x0" />
    </controller>
    <controller type="sata" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2" />
    </controller>
    <interface type="vhostuser">
      <mac address="2a:10:e8:20:35:34" />
      <source type="unix" path="/tmp/vhost0" mode="server" />
      <model type="virtio" />
      <driver name="vhost" iommu='on' ats='on' rx_queue_size="1024" tx_queue_size="1024" queues="2" />
      <address type="pci" domain="0x0000" bus="0x03" slot="0" function="0x0" />
    </interface>
    <interface type="bridge">
      <mac address="00:de:ad:01:01:01" />
      <source bridge="virbr0" />
      <model type="virtio" />
      <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0" />
    </interface>
    <serial type="pty">
      <target type="isa-serial" port="0">
        <model name="isa-serial" />
      </target>
    </serial>
    <console type="pty">
      <target type="serial" port="0" />
    </console>
    <input type="mouse" bus="ps2" />
    <input type="keyboard" bus="ps2" />
    <graphics type="vnc" port="-1" autoport="yes" listen="0.0.0.0">
      <listen type="address" address="0.0.0.0" />
    </graphics>
    <video>
      <model type="cirrus" vram="16384" heads="1" primary="yes" />
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0" />
    </video>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0" />
    </memballoon>
    <panic model="isa" />
    <iommu model="intel">
      <driver intremap="on" caching_mode="on" iotlb="on" />
    </iommu>
  </devices>
  <seclabel type="dynamic" model="selinux" relabel="yes" />
</domain>

4. start guest
# virsh create g1.xml

Actual results:
check qemu log and got error message as below:
022-09-22 09:17:08.471+0000: 31488: info : libvirt version: 8.0.0, package: 8.1.el9_0 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2022-04-27-07:09:25, )
2022-09-22 09:17:08.471+0000: 31488: info : hostname: hp-dl388g10-03.rhts.eng.pek2.redhat.com
2022-09-22 09:17:08.471+0000: 31488: info : virObjectUnref:380 : OBJECT_UNREF: obj=0x7f9d2c0281d0
2022-09-22T09:17:08.575101Z qemu-kvm: -chardev socket,id=charnet0,path=/tmp/vhost0,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost0,server=on
char device redirected to /dev/pts/0 (label charserial0)
2022-09-22T09:17:11.239476Z qemu-kvm: -device virtio-net-pci,iommu_platform=on,ats=on,mq=on,vectors=6,rx_queue_size=1024,tx_queue_size=1024,netdev=hostnet0,id=net0,mac=2a:10:e8:20:35:34,bus=pci.3,addr=0x0: iommu_platform=true is not supported by the device
2022-09-22 09:17:11.247+0000: shutting down, reason=failed


Expected results:
1. start guest as well.

Additional info:
if I delete viommu='on' and ats='on' in g1.xml. guest can start as well.
Here is the vhostuserserver port which can start as well.
    <interface type="vhostuser">
      <mac address="2a:10:e8:20:35:34" />
      <source type="unix" path="/tmp/vhost0" mode="server" />
      <model type="virtio" />
      <driver name="vhost" rx_queue_size="1024" tx_queue_size="1024" queues="2" />
      <address type="pci" domain="0x0000" bus="0x03" slot="0" function="0x0" />
    </interface>

In bug1551508 it was just highlighted that vhostuser no longer supports iommu and only supports iommu on the vhostuserclient port. Not sure about the expected behavior of enabling iommu for the vhostuserserver port here. And the note of the point is only on FDP 22.I test can hit this issue.

Another things need to highlighted, this issue occurred when use openvswitch2.17-2.17.0-43.el9fdp but can't hit on openvswitch2.17-2.17.0-32.4.el9fdp/openvswitch2.17-2.17.0-32.3.el9fdp.x86_64