Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 2129374

Summary:	vhostuserserver port can't start with "iommu=on" & "ats=on"
Product:	Red Hat Enterprise Linux Fast Datapath	Reporter:	mhou <mhou>
Component:	openvswitch2.17	Assignee:	Maxime Coquelin <maxime.coquelin>
Status:	CLOSED CURRENTRELEASE	QA Contact:	mhou <mhou>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	FDP 22.I	CC:	ctrautma, fleitner, jhsiao, kzhang, maxime.coquelin, ralongi, yalzhang, yanghliu
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	openvswitch2.17-2.17.0-44.el9fdp	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-10-02 01:55:54 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description mhou 2022-09-23 13:57:51 UTC

This bug was initially created as a copy of Bug #2129007

I am copying this bug because: 

This issue main occurred on openvswitch2.17-2.17.0-43.el9fdp

Description of problem:
start vhostuserserver port failed with iommu_platform=true is not supported by the device.

Version-Release number of selected component (if applicable):
kernel version: 5.14.0-70.26.1.rt21.98.el9_0.x86_64
ovs version: openvswitch2.17-2.17.0-43.el9fdp
qemu version: qemu-kvm-6.2.0-11.el9_0.5.x86_64
libvirt version: libvirt-8.0.0-8.1.el9_0.x86_64

How reproducible: 100%


Steps to Reproduce:
1. enable iommu on cmdline as below:
# cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-70.26.1.rt21.98.el9_0.x86_64 root=/dev/mapper/rhel_hp--dl388g10--03-root ro enforcing=1 kpti selinux=1 resume=/dev/mapper/rhel_hp--dl388g10--03-swap rd.lvm.lv=rhel_hp-dl388g10-03/root rd.lvm.lv=rhel_hp-dl388g10-03/swap console=ttyS0,115200n81 skew_tick=1 isolcpus=managed_irq,domain,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39 intel_pstate=disable nosoftlockup tsc=nowatchdog nohz=on nohz_full=1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39 rcu_nocbs=1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39 irqaffinity=0,20 crashkernel=1G-4G:384M,4G-16G:512M,16G-64G:1G,64G-128G:2G,128G-:4G mitigations=off nohz=on default_hugepagesz=1G hugepagesz=1G hugepages=48 intel_iommu=on iommu=pt modprobe.blacklist=qedi modprobe.blacklist=qedf modprobe.blacklist=qedr

2. start openvswitch and enable vhost-iommu-support and create vhostuserclient port.
# ovs-vsctl list Open_vSwitch .
_uuid               : ba631ab8-b621-46d2-8e29-6332a0eae59c
bridges             : [d1d81053-ece8-4c09-83ca-8d5fd3e3b204]
cur_cfg             : 6
datapath_types      : [netdev, system]
datapaths           : {}
db_version          : []
dpdk_initialized    : true
dpdk_version        : "DPDK 21.11.2"
external_ids        : {}
iface_types         : [bareudp, dpdk, dpdkvhostuser, dpdkvhostuserclient, erspan, geneve, gre, gtpu, internal, ip6erspan, ip6gre, lisp, patch, stt, system, tap, vxlan]
manager_options     : []
next_cfg            : 6
other_config        : {dpdk-init="true", dpdk-lcore-mask="0x1", dpdk-socket-mem="1024", vhost-iommu-support="true"}
ovs_version         : []
ssl                 : []
statistics          : {}
system_type         : []
system_version      : []
# ovs-vsctl add-port ovsbr0 vhost0 -- set interface vhost0 type=dpdkvhostuserclient options:vhost-server-path=/tmp/vhost0 options:n_rxq=2 mtu_request=9120
# ovs-vsctl show
ba631ab8-b621-46d2-8e29-6332a0eae59c
    Bridge ovsbr0
        datapath_type: netdev
        Port ovsbr0
            Interface ovsbr0
                type: internal
        Port vhost0
            Interface vhost0
                type: dpdkvhostuserclient
                options: {n_rxq="2", vhost-server-path="/tmp/vhost0"}

3. create guest XML and add vhostuserserver port.
# cat g1.xml 
<domain type="kvm">
  <name>guest30032</name>
  <uuid>37425e76-af6a-44a6-aba0-73434afe34c0</uuid>
  <memory unit="KiB">8388608</memory>
  <currentMemory unit="KiB">8388608</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size="1048576" unit="KiB" />
    </hugepages>
    <access mode="shared" />
  </memoryBacking>
  <vcpu placement="static">3</vcpu>
  <cputune>
    <vcpupin vcpu="0" cpuset="1" />
    <vcpupin vcpu="1" cpuset="2" />
    <vcpupin vcpu="2" cpuset="3" />
    <vcpusched vcpus="0" scheduler="fifo" priority="1" />
    <vcpusched vcpus="1" scheduler="fifo" priority="1" />
    <vcpusched vcpus="2" scheduler="fifo" priority="1" />
    <emulatorpin cpuset="0" />
  </cputune>
  <numatune>
    <memory mode="strict" nodeset="0" />
    <memnode cellid="0" mode="strict" nodeset="0" />
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch="x86_64" machine="q35">hvm</type>
    <boot dev="hd" />
  </os>
  <features>
    <acpi />
    <pmu state="off" />
    <vmport state="off" />
    <ioapic driver="qemu" />
  </features>
  <cpu mode="host-passthrough" check="none">
    <feature policy="require" name="tsc-deadline" />
    <numa>
      <cell id="0" cpus="0-2" memory="8388608" unit="KiB" memAccess="shared" />
    </numa>
  </cpu>
  <clock offset="utc">
    <timer name="rtc" tickpolicy="catchup" />
    <timer name="pit" tickpolicy="delay" />
    <timer name="hpet" present="no" />
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled="no" />
    <suspend-to-disk enabled="no" />
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2" />
      <source file="/home/guest30032.qcow2" />
      <target dev="vda" bus="virtio" />
      <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0" />
    </disk>
    <controller type="usb" index="0" model="none" />
    <controller type="pci" index="0" model="pcie-root" />
    <controller type="pci" index="1" model="pcie-root-port">
      <model name="pcie-root-port" />
      <target chassis="1" port="0x10" />
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" />
    </controller>
    <controller type="pci" index="2" model="pcie-root-port">
      <model name="pcie-root-port" />
      <target chassis="2" port="0x11" />
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0" />
    </controller>
    <controller type="pci" index="3" model="pcie-root-port">
      <model name="pcie-root-port" />
      <target chassis="3" port="0x8" />
      <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0" />
    </controller>
    <controller type="pci" index="4" model="pcie-root-port">
      <model name="pcie-root-port" />
      <target chassis="4" port="0x9" />
      <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0" />
    </controller>
    <controller type="pci" index="5" model="pcie-root-port">
      <model name="pcie-root-port" />
      <target chassis="5" port="0xa" />
      <address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x0" />
    </controller>
    <controller type="pci" index="6" model="pcie-root-port">
      <model name="pcie-root-port" />
      <target chassis="6" port="0xb" />
      <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x0" />
    </controller>
    <controller type="sata" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2" />
    </controller>
    <interface type="vhostuser">
      <mac address="2a:10:e8:20:35:34" />
      <source type="unix" path="/tmp/vhost0" mode="server" />
      <model type="virtio" />
      <driver name="vhost" iommu='on' ats='on' rx_queue_size="1024" tx_queue_size="1024" queues="2" />
      <address type="pci" domain="0x0000" bus="0x03" slot="0" function="0x0" />
    </interface>
    <interface type="bridge">
      <mac address="00:de:ad:01:01:01" />
      <source bridge="virbr0" />
      <model type="virtio" />
      <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0" />
    </interface>
    <serial type="pty">
      <target type="isa-serial" port="0">
        <model name="isa-serial" />
      </target>
    </serial>
    <console type="pty">
      <target type="serial" port="0" />
    </console>
    <input type="mouse" bus="ps2" />
    <input type="keyboard" bus="ps2" />
    <graphics type="vnc" port="-1" autoport="yes" listen="0.0.0.0">
      <listen type="address" address="0.0.0.0" />
    </graphics>
    <video>
      <model type="cirrus" vram="16384" heads="1" primary="yes" />
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0" />
    </video>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0" />
    </memballoon>
    <panic model="isa" />
    <iommu model="intel">
      <driver intremap="on" caching_mode="on" iotlb="on" />
    </iommu>
  </devices>
  <seclabel type="dynamic" model="selinux" relabel="yes" />
</domain>

4. start guest
# virsh create g1.xml

5. check selinux context as below:
# ll -Z /tmp/vhost0 
srwxrwxr-x. 1 qemu hugetlbfs system_u:object_r:svirt_tmp_t:s0 0 Sep 23 09:41 /tmp/vhost0
# ll -Z /home/guest30032.qcow2 
-rw-r--r--. 1 qemu hugetlbfs system_u:object_r:svirt_image_t:s0:c38,c965 2885877760 Sep 23 08:50 /home/guest30032.qcow2

Actual results:
check qemu log and got error message as below:
022-09-22 09:17:08.471+0000: 31488: info : libvirt version: 8.0.0, package: 8.1.el9_0 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2022-04-27-07:09:25, )
2022-09-22 09:17:08.471+0000: 31488: info : hostname: hp-dl388g10-03.rhts.eng.pek2.redhat.com
2022-09-22 09:17:08.471+0000: 31488: info : virObjectUnref:380 : OBJECT_UNREF: obj=0x7f9d2c0281d0
2022-09-22T09:17:08.575101Z qemu-kvm: -chardev socket,id=charnet0,path=/tmp/vhost0,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost0,server=on
char device redirected to /dev/pts/0 (label charserial0)
2022-09-22T09:17:11.239476Z qemu-kvm: -device virtio-net-pci,iommu_platform=on,ats=on,mq=on,vectors=6,rx_queue_size=1024,tx_queue_size=1024,netdev=hostnet0,id=net0,mac=2a:10:e8:20:35:34,bus=pci.3,addr=0x0: iommu_platform=true is not supported by the device
2022-09-22 09:17:11.247+0000: shutting down, reason=failed

Or vhostuserserver still wait connecting.
2022-09-23T13:41:22.081903Z qemu-kvm: -chardev socket,id=charnet0,path=/tmp/vhost0,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost0,server=on



Expected results:
1. start guest as well.

Additional info:
if I delete viommu='on' and ats='on' in g1.xml. guest can start as well.
Here is the vhostuserserver port which can start as well.
    <interface type="vhostuser">
      <mac address="2a:10:e8:20:35:34" />
      <source type="unix" path="/tmp/vhost0" mode="server" />
      <model type="virtio" />
      <driver name="vhost" rx_queue_size="1024" tx_queue_size="1024" queues="2" />
      <address type="pci" domain="0x0000" bus="0x03" slot="0" function="0x0" />
    </interface>

In bug1551508 it was just highlighted that vhostuser no longer supports iommu and only supports iommu on the vhostuserclient port. Not sure about the expected behavior of enabling iommu for the vhostuserserver port here. And the note of the point is only on FDP 22.I test can hit this issue.

Another things need to highlighted, this issue occurred when use openvswitch2.17-2.17.0-43.el9fdp but can't hit on openvswitch2.17-2.17.0-32.4.el9fdp/openvswitch2.17-2.17.0-32.3.el9fdp.x86_64

capture ovs log as below:
2022-09-26T07:46:19.840Z|00001|dpdk|INFO|VHOST_CONFIG: /tmp/vhost0: connected
2022-09-26T07:46:19.841Z|00002|dpdk|INFO|VHOST_CONFIG: new device, handle is 0, path is /tmp/vhost0
2022-09-26T07:46:19.999Z|00001|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
2022-09-26T07:46:19.999Z|00002|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_GET_PROTOCOL_FEATURES
2022-09-26T07:46:19.999Z|00003|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_PROTOCOL_FEATURES
2022-09-26T07:46:19.999Z|00004|dpdk|INFO|VHOST_CONFIG: negotiated Vhost-user protocol features: 0xcbf
2022-09-26T07:46:19.999Z|00005|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_GET_QUEUE_NUM
2022-09-26T07:46:19.999Z|00006|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_SLAVE_REQ_FD
2022-09-26T07:46:19.999Z|00007|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_OWNER
2022-09-26T07:46:19.999Z|00008|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
2022-09-26T07:46:19.999Z|00009|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
2022-09-26T07:46:20.003Z|00010|dpdk|INFO|VHOST_CONFIG: vring call idx:0 file:107
2022-09-26T07:46:20.003Z|00011|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
2022-09-26T07:46:20.003Z|00012|dpdk|INFO|VHOST_CONFIG: vring call idx:1 file:108
2022-09-26T07:46:20.003Z|00013|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
2022-09-26T07:46:20.003Z|00014|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_GET_PROTOCOL_FEATURES
2022-09-26T07:46:20.003Z|00015|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_PROTOCOL_FEATURES
2022-09-26T07:46:20.003Z|00016|dpdk|INFO|VHOST_CONFIG: negotiated Vhost-user protocol features: 0xcbf
2022-09-26T07:46:20.003Z|00017|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
2022-09-26T07:46:20.003Z|00018|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
2022-09-26T07:46:20.003Z|00019|dpdk|INFO|VHOST_CONFIG: vring call idx:2 file:109
2022-09-26T07:46:20.003Z|00020|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
2022-09-26T07:46:20.003Z|00021|dpdk|INFO|VHOST_CONFIG: vring call idx:3 file:110
2022-09-26T07:46:35.232Z|00022|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
2022-09-26T07:46:35.232Z|00023|dpdk|INFO|VHOST_CONFIG: set queue enable: 1 to qp idx: 0
2022-09-26T07:46:35.232Z|00024|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
2022-09-26T07:46:35.232Z|00025|dpdk|INFO|VHOST_CONFIG: set queue enable: 1 to qp idx: 1
2022-09-26T07:46:35.232Z|00026|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
2022-09-26T07:46:35.232Z|00027|dpdk|INFO|VHOST_CONFIG: set queue enable: 0 to qp idx: 2
2022-09-26T07:46:35.232Z|00028|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
2022-09-26T07:46:35.232Z|00029|dpdk|INFO|VHOST_CONFIG: set queue enable: 0 to qp idx: 3
2022-09-26T07:46:35.232Z|00030|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
2022-09-26T07:46:35.232Z|00031|dpdk|INFO|VHOST_CONFIG: set queue enable: 1 to qp idx: 0
2022-09-26T07:46:35.232Z|00032|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
2022-09-26T07:46:35.232Z|00033|dpdk|INFO|VHOST_CONFIG: set queue enable: 1 to qp idx: 1
2022-09-26T07:46:35.232Z|00034|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
2022-09-26T07:46:35.232Z|00035|dpdk|INFO|VHOST_CONFIG: set queue enable: 0 to qp idx: 2
2022-09-26T07:46:35.232Z|00036|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
2022-09-26T07:46:35.232Z|00037|dpdk|INFO|VHOST_CONFIG: set queue enable: 0 to qp idx: 3
2022-09-26T07:46:35.435Z|00038|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_FEATURES
2022-09-26T07:46:35.435Z|00039|dpdk|INFO|VHOST_CONFIG: negotiated Virtio features: 0x37060a782
2022-09-26T07:46:35.435Z|00040|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_MEM_TABLE
2022-09-26T07:46:35.819Z|00041|dpdk|INFO|VHOST_CONFIG: guest memory region size: 0x80000000
         guest physical addr: 0x0
         guest virtual  addr: 0x7f2280000000
         host  virtual  addr: 0x7f2880000000
         mmap addr : 0x7f2880000000
         mmap size : 0x80000000
         mmap align: 0x40000000
         mmap off  : 0x0
2022-09-26T07:46:36.973Z|00042|dpdk|INFO|VHOST_CONFIG: guest memory region size: 0x180000000
         guest physical addr: 0x100000000
         guest virtual  addr: 0x7f2300000000
         host  virtual  addr: 0x7f2700000000
         mmap addr : 0x7f2680000000
         mmap size : 0x200000000
         mmap align: 0x40000000
         mmap off  : 0x80000000
2022-09-26T07:46:36.973Z|00043|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
2022-09-26T07:46:36.973Z|00044|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
2022-09-26T07:46:36.973Z|00045|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
2022-09-26T07:46:36.973Z|00046|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
2022-09-26T07:46:36.973Z|00047|dpdk|INFO|VHOST_CONFIG: vring kick idx:0 file:113
2022-09-26T07:46:36.973Z|00049|dpdk|INFO|VHOST_CONFIG: vring call idx:0 file:114
2022-09-26T07:46:36.973Z|00050|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
2022-09-26T07:46:36.973Z|00051|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
2022-09-26T07:46:36.973Z|00052|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
2022-09-26T07:46:36.973Z|00053|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
2022-09-26T07:46:36.973Z|00054|dpdk|INFO|VHOST_CONFIG: vring kick idx:1 file:107
2022-09-26T07:46:36.973Z|00055|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
2022-09-26T07:46:36.973Z|00056|dpdk|INFO|VHOST_CONFIG: vring call idx:1 file:115
2022-09-26T07:46:36.973Z|00057|dpdk|INFO|VHOST_CONFIG: reallocated virtqueue on node 1
2022-09-26T07:46:36.977Z|00058|dpdk|INFO|VHOST_CONFIG: reallocated device on node 1
2022-09-26T07:46:36.977Z|00059|dpdk|INFO|VHOST_CONFIG: reallocated virtqueue on node 1

Comment 2 mhou 2022-09-23 13:59:38 UTC

*** Bug 2129007 has been marked as a duplicate of this bug. ***

Comment 5 mhou 2022-10-02 01:55:54 UTC

Test on openvswitch2.17-2.17.0-44.el9fdp and didn't hit this issue again.

beaker job: https://beaker.engineering.redhat.com/jobs/7065531