This bug was initially created as a copy of Bug #2129007 I am copying this bug because: This issue main occurred on openvswitch2.17-2.17.0-43.el9fdp Description of problem: start vhostuserserver port failed with iommu_platform=true is not supported by the device. Version-Release number of selected component (if applicable): kernel version: 5.14.0-70.26.1.rt21.98.el9_0.x86_64 ovs version: openvswitch2.17-2.17.0-43.el9fdp qemu version: qemu-kvm-6.2.0-11.el9_0.5.x86_64 libvirt version: libvirt-8.0.0-8.1.el9_0.x86_64 How reproducible: 100% Steps to Reproduce: 1. enable iommu on cmdline as below: # cat /proc/cmdline BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-70.26.1.rt21.98.el9_0.x86_64 root=/dev/mapper/rhel_hp--dl388g10--03-root ro enforcing=1 kpti selinux=1 resume=/dev/mapper/rhel_hp--dl388g10--03-swap rd.lvm.lv=rhel_hp-dl388g10-03/root rd.lvm.lv=rhel_hp-dl388g10-03/swap console=ttyS0,115200n81 skew_tick=1 isolcpus=managed_irq,domain,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39 intel_pstate=disable nosoftlockup tsc=nowatchdog nohz=on nohz_full=1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39 rcu_nocbs=1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39 irqaffinity=0,20 crashkernel=1G-4G:384M,4G-16G:512M,16G-64G:1G,64G-128G:2G,128G-:4G mitigations=off nohz=on default_hugepagesz=1G hugepagesz=1G hugepages=48 intel_iommu=on iommu=pt modprobe.blacklist=qedi modprobe.blacklist=qedf modprobe.blacklist=qedr 2. start openvswitch and enable vhost-iommu-support and create vhostuserclient port. # ovs-vsctl list Open_vSwitch . _uuid : ba631ab8-b621-46d2-8e29-6332a0eae59c bridges : [d1d81053-ece8-4c09-83ca-8d5fd3e3b204] cur_cfg : 6 datapath_types : [netdev, system] datapaths : {} db_version : [] dpdk_initialized : true dpdk_version : "DPDK 21.11.2" external_ids : {} iface_types : [bareudp, dpdk, dpdkvhostuser, dpdkvhostuserclient, erspan, geneve, gre, gtpu, internal, ip6erspan, ip6gre, lisp, patch, stt, system, tap, vxlan] manager_options : [] next_cfg : 6 other_config : {dpdk-init="true", dpdk-lcore-mask="0x1", dpdk-socket-mem="1024", vhost-iommu-support="true"} ovs_version : [] ssl : [] statistics : {} system_type : [] system_version : [] # ovs-vsctl add-port ovsbr0 vhost0 -- set interface vhost0 type=dpdkvhostuserclient options:vhost-server-path=/tmp/vhost0 options:n_rxq=2 mtu_request=9120 # ovs-vsctl show ba631ab8-b621-46d2-8e29-6332a0eae59c Bridge ovsbr0 datapath_type: netdev Port ovsbr0 Interface ovsbr0 type: internal Port vhost0 Interface vhost0 type: dpdkvhostuserclient options: {n_rxq="2", vhost-server-path="/tmp/vhost0"} 3. create guest XML and add vhostuserserver port. # cat g1.xml <domain type="kvm"> <name>guest30032</name> <uuid>37425e76-af6a-44a6-aba0-73434afe34c0</uuid> <memory unit="KiB">8388608</memory> <currentMemory unit="KiB">8388608</currentMemory> <memoryBacking> <hugepages> <page size="1048576" unit="KiB" /> </hugepages> <access mode="shared" /> </memoryBacking> <vcpu placement="static">3</vcpu> <cputune> <vcpupin vcpu="0" cpuset="1" /> <vcpupin vcpu="1" cpuset="2" /> <vcpupin vcpu="2" cpuset="3" /> <vcpusched vcpus="0" scheduler="fifo" priority="1" /> <vcpusched vcpus="1" scheduler="fifo" priority="1" /> <vcpusched vcpus="2" scheduler="fifo" priority="1" /> <emulatorpin cpuset="0" /> </cputune> <numatune> <memory mode="strict" nodeset="0" /> <memnode cellid="0" mode="strict" nodeset="0" /> </numatune> <resource> <partition>/machine</partition> </resource> <os> <type arch="x86_64" machine="q35">hvm</type> <boot dev="hd" /> </os> <features> <acpi /> <pmu state="off" /> <vmport state="off" /> <ioapic driver="qemu" /> </features> <cpu mode="host-passthrough" check="none"> <feature policy="require" name="tsc-deadline" /> <numa> <cell id="0" cpus="0-2" memory="8388608" unit="KiB" memAccess="shared" /> </numa> </cpu> <clock offset="utc"> <timer name="rtc" tickpolicy="catchup" /> <timer name="pit" tickpolicy="delay" /> <timer name="hpet" present="no" /> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled="no" /> <suspend-to-disk enabled="no" /> </pm> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type="file" device="disk"> <driver name="qemu" type="qcow2" /> <source file="/home/guest30032.qcow2" /> <target dev="vda" bus="virtio" /> <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0" /> </disk> <controller type="usb" index="0" model="none" /> <controller type="pci" index="0" model="pcie-root" /> <controller type="pci" index="1" model="pcie-root-port"> <model name="pcie-root-port" /> <target chassis="1" port="0x10" /> <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" /> </controller> <controller type="pci" index="2" model="pcie-root-port"> <model name="pcie-root-port" /> <target chassis="2" port="0x11" /> <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0" /> </controller> <controller type="pci" index="3" model="pcie-root-port"> <model name="pcie-root-port" /> <target chassis="3" port="0x8" /> <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0" /> </controller> <controller type="pci" index="4" model="pcie-root-port"> <model name="pcie-root-port" /> <target chassis="4" port="0x9" /> <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0" /> </controller> <controller type="pci" index="5" model="pcie-root-port"> <model name="pcie-root-port" /> <target chassis="5" port="0xa" /> <address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x0" /> </controller> <controller type="pci" index="6" model="pcie-root-port"> <model name="pcie-root-port" /> <target chassis="6" port="0xb" /> <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x0" /> </controller> <controller type="sata" index="0"> <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2" /> </controller> <interface type="vhostuser"> <mac address="2a:10:e8:20:35:34" /> <source type="unix" path="/tmp/vhost0" mode="server" /> <model type="virtio" /> <driver name="vhost" iommu='on' ats='on' rx_queue_size="1024" tx_queue_size="1024" queues="2" /> <address type="pci" domain="0x0000" bus="0x03" slot="0" function="0x0" /> </interface> <interface type="bridge"> <mac address="00:de:ad:01:01:01" /> <source bridge="virbr0" /> <model type="virtio" /> <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0" /> </interface> <serial type="pty"> <target type="isa-serial" port="0"> <model name="isa-serial" /> </target> </serial> <console type="pty"> <target type="serial" port="0" /> </console> <input type="mouse" bus="ps2" /> <input type="keyboard" bus="ps2" /> <graphics type="vnc" port="-1" autoport="yes" listen="0.0.0.0"> <listen type="address" address="0.0.0.0" /> </graphics> <video> <model type="cirrus" vram="16384" heads="1" primary="yes" /> <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0" /> </video> <memballoon model="virtio"> <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0" /> </memballoon> <panic model="isa" /> <iommu model="intel"> <driver intremap="on" caching_mode="on" iotlb="on" /> </iommu> </devices> <seclabel type="dynamic" model="selinux" relabel="yes" /> </domain> 4. start guest # virsh create g1.xml 5. check selinux context as below: # ll -Z /tmp/vhost0 srwxrwxr-x. 1 qemu hugetlbfs system_u:object_r:svirt_tmp_t:s0 0 Sep 23 09:41 /tmp/vhost0 # ll -Z /home/guest30032.qcow2 -rw-r--r--. 1 qemu hugetlbfs system_u:object_r:svirt_image_t:s0:c38,c965 2885877760 Sep 23 08:50 /home/guest30032.qcow2 Actual results: check qemu log and got error message as below: 022-09-22 09:17:08.471+0000: 31488: info : libvirt version: 8.0.0, package: 8.1.el9_0 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2022-04-27-07:09:25, ) 2022-09-22 09:17:08.471+0000: 31488: info : hostname: hp-dl388g10-03.rhts.eng.pek2.redhat.com 2022-09-22 09:17:08.471+0000: 31488: info : virObjectUnref:380 : OBJECT_UNREF: obj=0x7f9d2c0281d0 2022-09-22T09:17:08.575101Z qemu-kvm: -chardev socket,id=charnet0,path=/tmp/vhost0,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost0,server=on char device redirected to /dev/pts/0 (label charserial0) 2022-09-22T09:17:11.239476Z qemu-kvm: -device virtio-net-pci,iommu_platform=on,ats=on,mq=on,vectors=6,rx_queue_size=1024,tx_queue_size=1024,netdev=hostnet0,id=net0,mac=2a:10:e8:20:35:34,bus=pci.3,addr=0x0: iommu_platform=true is not supported by the device 2022-09-22 09:17:11.247+0000: shutting down, reason=failed Or vhostuserserver still wait connecting. 2022-09-23T13:41:22.081903Z qemu-kvm: -chardev socket,id=charnet0,path=/tmp/vhost0,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost0,server=on Expected results: 1. start guest as well. Additional info: if I delete viommu='on' and ats='on' in g1.xml. guest can start as well. Here is the vhostuserserver port which can start as well. <interface type="vhostuser"> <mac address="2a:10:e8:20:35:34" /> <source type="unix" path="/tmp/vhost0" mode="server" /> <model type="virtio" /> <driver name="vhost" rx_queue_size="1024" tx_queue_size="1024" queues="2" /> <address type="pci" domain="0x0000" bus="0x03" slot="0" function="0x0" /> </interface> In bug1551508 it was just highlighted that vhostuser no longer supports iommu and only supports iommu on the vhostuserclient port. Not sure about the expected behavior of enabling iommu for the vhostuserserver port here. And the note of the point is only on FDP 22.I test can hit this issue. Another things need to highlighted, this issue occurred when use openvswitch2.17-2.17.0-43.el9fdp but can't hit on openvswitch2.17-2.17.0-32.4.el9fdp/openvswitch2.17-2.17.0-32.3.el9fdp.x86_64 capture ovs log as below: 2022-09-26T07:46:19.840Z|00001|dpdk|INFO|VHOST_CONFIG: /tmp/vhost0: connected 2022-09-26T07:46:19.841Z|00002|dpdk|INFO|VHOST_CONFIG: new device, handle is 0, path is /tmp/vhost0 2022-09-26T07:46:19.999Z|00001|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_GET_FEATURES 2022-09-26T07:46:19.999Z|00002|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_GET_PROTOCOL_FEATURES 2022-09-26T07:46:19.999Z|00003|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_PROTOCOL_FEATURES 2022-09-26T07:46:19.999Z|00004|dpdk|INFO|VHOST_CONFIG: negotiated Vhost-user protocol features: 0xcbf 2022-09-26T07:46:19.999Z|00005|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_GET_QUEUE_NUM 2022-09-26T07:46:19.999Z|00006|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_SLAVE_REQ_FD 2022-09-26T07:46:19.999Z|00007|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_OWNER 2022-09-26T07:46:19.999Z|00008|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_GET_FEATURES 2022-09-26T07:46:19.999Z|00009|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL 2022-09-26T07:46:20.003Z|00010|dpdk|INFO|VHOST_CONFIG: vring call idx:0 file:107 2022-09-26T07:46:20.003Z|00011|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL 2022-09-26T07:46:20.003Z|00012|dpdk|INFO|VHOST_CONFIG: vring call idx:1 file:108 2022-09-26T07:46:20.003Z|00013|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_GET_FEATURES 2022-09-26T07:46:20.003Z|00014|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_GET_PROTOCOL_FEATURES 2022-09-26T07:46:20.003Z|00015|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_PROTOCOL_FEATURES 2022-09-26T07:46:20.003Z|00016|dpdk|INFO|VHOST_CONFIG: negotiated Vhost-user protocol features: 0xcbf 2022-09-26T07:46:20.003Z|00017|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_GET_FEATURES 2022-09-26T07:46:20.003Z|00018|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL 2022-09-26T07:46:20.003Z|00019|dpdk|INFO|VHOST_CONFIG: vring call idx:2 file:109 2022-09-26T07:46:20.003Z|00020|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL 2022-09-26T07:46:20.003Z|00021|dpdk|INFO|VHOST_CONFIG: vring call idx:3 file:110 2022-09-26T07:46:35.232Z|00022|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE 2022-09-26T07:46:35.232Z|00023|dpdk|INFO|VHOST_CONFIG: set queue enable: 1 to qp idx: 0 2022-09-26T07:46:35.232Z|00024|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE 2022-09-26T07:46:35.232Z|00025|dpdk|INFO|VHOST_CONFIG: set queue enable: 1 to qp idx: 1 2022-09-26T07:46:35.232Z|00026|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE 2022-09-26T07:46:35.232Z|00027|dpdk|INFO|VHOST_CONFIG: set queue enable: 0 to qp idx: 2 2022-09-26T07:46:35.232Z|00028|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE 2022-09-26T07:46:35.232Z|00029|dpdk|INFO|VHOST_CONFIG: set queue enable: 0 to qp idx: 3 2022-09-26T07:46:35.232Z|00030|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE 2022-09-26T07:46:35.232Z|00031|dpdk|INFO|VHOST_CONFIG: set queue enable: 1 to qp idx: 0 2022-09-26T07:46:35.232Z|00032|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE 2022-09-26T07:46:35.232Z|00033|dpdk|INFO|VHOST_CONFIG: set queue enable: 1 to qp idx: 1 2022-09-26T07:46:35.232Z|00034|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE 2022-09-26T07:46:35.232Z|00035|dpdk|INFO|VHOST_CONFIG: set queue enable: 0 to qp idx: 2 2022-09-26T07:46:35.232Z|00036|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE 2022-09-26T07:46:35.232Z|00037|dpdk|INFO|VHOST_CONFIG: set queue enable: 0 to qp idx: 3 2022-09-26T07:46:35.435Z|00038|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_FEATURES 2022-09-26T07:46:35.435Z|00039|dpdk|INFO|VHOST_CONFIG: negotiated Virtio features: 0x37060a782 2022-09-26T07:46:35.435Z|00040|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_MEM_TABLE 2022-09-26T07:46:35.819Z|00041|dpdk|INFO|VHOST_CONFIG: guest memory region size: 0x80000000 guest physical addr: 0x0 guest virtual addr: 0x7f2280000000 host virtual addr: 0x7f2880000000 mmap addr : 0x7f2880000000 mmap size : 0x80000000 mmap align: 0x40000000 mmap off : 0x0 2022-09-26T07:46:36.973Z|00042|dpdk|INFO|VHOST_CONFIG: guest memory region size: 0x180000000 guest physical addr: 0x100000000 guest virtual addr: 0x7f2300000000 host virtual addr: 0x7f2700000000 mmap addr : 0x7f2680000000 mmap size : 0x200000000 mmap align: 0x40000000 mmap off : 0x80000000 2022-09-26T07:46:36.973Z|00043|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM 2022-09-26T07:46:36.973Z|00044|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE 2022-09-26T07:46:36.973Z|00045|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR 2022-09-26T07:46:36.973Z|00046|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK 2022-09-26T07:46:36.973Z|00047|dpdk|INFO|VHOST_CONFIG: vring kick idx:0 file:113 2022-09-26T07:46:36.973Z|00049|dpdk|INFO|VHOST_CONFIG: vring call idx:0 file:114 2022-09-26T07:46:36.973Z|00050|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM 2022-09-26T07:46:36.973Z|00051|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE 2022-09-26T07:46:36.973Z|00052|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR 2022-09-26T07:46:36.973Z|00053|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK 2022-09-26T07:46:36.973Z|00054|dpdk|INFO|VHOST_CONFIG: vring kick idx:1 file:107 2022-09-26T07:46:36.973Z|00055|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL 2022-09-26T07:46:36.973Z|00056|dpdk|INFO|VHOST_CONFIG: vring call idx:1 file:115 2022-09-26T07:46:36.973Z|00057|dpdk|INFO|VHOST_CONFIG: reallocated virtqueue on node 1 2022-09-26T07:46:36.977Z|00058|dpdk|INFO|VHOST_CONFIG: reallocated device on node 1 2022-09-26T07:46:36.977Z|00059|dpdk|INFO|VHOST_CONFIG: reallocated virtqueue on node 1
*** Bug 2129007 has been marked as a duplicate of this bug. ***
Test on openvswitch2.17-2.17.0-44.el9fdp and didn't hit this issue again. beaker job: https://beaker.engineering.redhat.com/jobs/7065531