Created attachment 1554156 [details] xml for vm Description of problem: Boot vm as vhost user client mode, with iommu='on' ats='on', dpdk's testpmd can not receive packets. Version-Release number of selected component (if applicable): 3.10.0-1037.el7.x86_64 openvswitch2.11-2.11.0-5.el7fdp.x86_64 dpdk-18.11-4.el7_6.x86_64 qemu-kvm-rhev-2.12.0-25.el7.x86_64 libvirt-4.5.0-10.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1. Boot ovs, refer to[1] 2. Boot VM with vhostuser client mode, set "iommu='on' ats='on'", full xml will be attached. <interface type='vhostuser'> <mac address='18:66:da:5f:dd:02'/> <source type='unix' path='/var/run/openvswitch/vhost-user0' mode='client'/> <model type='virtio'/> <driver name='vhost' rx_queue_size='1024' iommu='on' ats='on'/> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </interface> 3. In guest, load vfio and start testpmd # modprobe vfio # modprobe vfio-pci # /usr/bin/testpmd \ -l 1,2,3 -n 4 -d \ /usr/lib64/librte_pmd_virtio.so \ -w 0000:00:06.0 -w 0000:00:07.0 \ -- --nb-cores=2 -i \ --disable-rss \ --rxd=512 --txd=512 \ --rxq=1 --txq=1 4. Start MoonGen in another host, testpmd can not receive packets. Actual results: dpdk's testpmd can not receive packets. Expected results: dpdk's testpmd can receive packets. Additional info: 1. Tested without iommu='on' ats='on', testpmd works well. 2. Tested with qemu as vhost user server mode, testpmd works well. 3. Tested with openvswitch2.11-2.11.0-1, testpmd can't receive packets neither. 4. There is a similar bug in https://bugzilla.redhat.com/show_bug.cgi?id=1532956. References: [1] # ovs-vsctl show 7259d18d-97a8-4d67-b572-3b399f9982f5 Bridge "ovsbr1" Port "ovsbr1" Interface "ovsbr1" type: internal Port "vhost-user2" Interface "vhost-user2" type: dpdkvhostuser Port "dpdk2" Interface "dpdk2" type: dpdk options: {dpdk-devargs="0000:5f:00.0", n_rxq="1", n_txq="1"} Bridge "ovsbr0" Port "dpdk1" Interface "dpdk1" type: dpdk options: {dpdk-devargs="0000:5e:00.1", n_rxq="1", n_txq="1"} Port "dpdk0" Interface "dpdk0" type: dpdk options: {dpdk-devargs="0000:5e:00.0", n_rxq="1", n_txq="1"} Port "vhost-user1" Interface "vhost-user1" type: dpdkvhostuser Port "ovsbr0" Interface "ovsbr0" type: internal Port "vhost-user0" Interface "vhost-user0" type: dpdkvhostuser
Not sure if qemu as vhost user client mode is deprecated or not, but in rhel8 test testpmd can get packets in this mode. Versions: openvswitch2.11-2.11.0-4.el8fdp.x86_64 qemu-kvm-3.1.0-20.module+el8+2888+cdc893a8.x86_64
Hi, are you running qemu as a user that is a member of hugetlbfs group (see bz#1544948)?
Also which card are you running on? We have some open issues with a vendor or two on viommu mode not working at all.
This bug looks like same with: Bug 1551508 - Request for doc update about dpdkvhostuser port is not supported with vIOMMU.
(In reply to Timothy Redaelli from comment #3) > Hi, > > are you running qemu as a user that is a member of hugetlbfs group (see > bz#1544948)? Thanks for reply. No, we run qemu as root by default. But after I added root to hugetlbfs group and updated conf file like below, sockets seem still in root group. # cat /etc/libvirt/qemu.conf group = "hugetlbfs" # cat /etc/sysconfig/openvswitch OVS_USER_ID="openvswitch:hugetlbfs" # ll /var/run/openvswitch/vhost-* srwxr-xr-x 1 root root 0 Apr 11 00:36 /var/run/openvswitch/vhost-* If I missed something, please correct me, thanks! (In reply to Christian Trautman from comment #4) > Also which card are you running on? We have some open issues with a vendor > or two on viommu mode not working at all. I am using ixgbe, we tested it with our first priority, please see: # dpdk-devbind --status Network devices using DPDK-compatible driver ============================================ 0000:5e:00.0 'Ethernet Controller 10-Gigabit X540-AT2 1528' drv=vfio-pci unused=ixgbe 0000:5e:00.1 'Ethernet Controller 10-Gigabit X540-AT2 1528' drv=vfio-pci unused=ixgbe 0000:5f:00.0 'Ethernet Controller 10-Gigabit X540-AT2 1528' drv=vfio-pci unused=ixgbe
(In reply to Pei Zhang from comment #5) > This bug looks like same with: > > Bug 1551508 - Request for doc update about dpdkvhostuser port is not > supported with vIOMMU. Thanks for info! If IOMMU is not supported with vhost-user ports from OVS 2.10, can we suppose that rhel8 support this scenario with no meaning? If so, I will delete this case from our test plan, thanks! Best regards, Sitong Liu
(In reply to Pei Zhang from comment #5) > This bug looks like same with: > > Bug 1551508 - Request for doc update about dpdkvhostuser port is not > supported with vIOMMU. Yes, it looks to be the same issue as reported there. See https://bugzilla.redhat.com/show_bug.cgi?id=1551508#c4 for further explanation as to why it's not supported. In summary, OVS "vhostuser" type ports are deprecated. vIOMMU=on with ovs:vhostuser + qemu:client is not a valid test. vIOMMU=on with ovs:vhostuserclient + qemu:server is a valid test. The docs were updated as part of #1551508, so closing as NOTABUG.