Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
The FDP team is no longer accepting new bugs in Bugzilla. Please report your issues under FDP project in Jira. Thanks.

Bug 1698396

Summary: Boot vm with viommu as vhost user client mode, dpdk's testpmd can not receive packets
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Sitong Liu <siliu>
Component: openvswitch2.11Assignee: Open vSwitch development team <ovs-team>
Status: CLOSED NOTABUG QA Contact: Sitong Liu <siliu>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: FDP 19.ACC: atragler, chayang, ctrautma, fhallal, jhsiao, juzhang, ktraynor, ovs-qe, pezhang, qding, ralongi, tredaelli
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-04-18 11:27:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
xml for vm none

Description Sitong Liu 2019-04-10 10:04:11 UTC
Created attachment 1554156 [details]
xml for vm

Description of problem:
Boot vm as vhost user client mode, with iommu='on' ats='on', dpdk's testpmd can not receive packets.

Version-Release number of selected component (if applicable):
3.10.0-1037.el7.x86_64
openvswitch2.11-2.11.0-5.el7fdp.x86_64
dpdk-18.11-4.el7_6.x86_64
qemu-kvm-rhev-2.12.0-25.el7.x86_64
libvirt-4.5.0-10.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Boot ovs, refer to[1]

2. Boot VM with vhostuser client mode, set "iommu='on' ats='on'", full xml will be attached.

   <interface type='vhostuser'>
     <mac address='18:66:da:5f:dd:02'/>
     <source type='unix' path='/var/run/openvswitch/vhost-user0' mode='client'/>
     <model type='virtio'/>
     <driver name='vhost' rx_queue_size='1024' iommu='on' ats='on'/>
     <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
   </interface>
  
3. In guest, load vfio and start testpmd
# modprobe vfio
# modprobe vfio-pci 

# /usr/bin/testpmd \
-l 1,2,3 -n 4 -d \
/usr/lib64/librte_pmd_virtio.so \
-w 0000:00:06.0 -w 0000:00:07.0 \
-- --nb-cores=2 -i \
--disable-rss \
--rxd=512 --txd=512 \
--rxq=1 --txq=1


4. Start MoonGen in another host, testpmd can not receive packets.

Actual results:
dpdk's testpmd can not receive packets.

Expected results:
dpdk's testpmd can receive packets.

Additional info:
1. Tested without iommu='on' ats='on', testpmd works well.
2. Tested with qemu as vhost user server mode, testpmd works well.
3. Tested with openvswitch2.11-2.11.0-1, testpmd can't receive packets neither.
4. There is a similar bug in https://bugzilla.redhat.com/show_bug.cgi?id=1532956.

References:
[1]
# ovs-vsctl show
7259d18d-97a8-4d67-b572-3b399f9982f5
    Bridge "ovsbr1"
        Port "ovsbr1"
            Interface "ovsbr1"
                type: internal
        Port "vhost-user2"
            Interface "vhost-user2"
                type: dpdkvhostuser
        Port "dpdk2"
            Interface "dpdk2"
                type: dpdk
                options: {dpdk-devargs="0000:5f:00.0", n_rxq="1", n_txq="1"}
    Bridge "ovsbr0"
        Port "dpdk1"
            Interface "dpdk1"
                type: dpdk
                options: {dpdk-devargs="0000:5e:00.1", n_rxq="1", n_txq="1"}
        Port "dpdk0"
            Interface "dpdk0"
                type: dpdk
                options: {dpdk-devargs="0000:5e:00.0", n_rxq="1", n_txq="1"}
        Port "vhost-user1"
            Interface "vhost-user1"
                type: dpdkvhostuser
        Port "ovsbr0"
            Interface "ovsbr0"
                type: internal
        Port "vhost-user0"
            Interface "vhost-user0"
                type: dpdkvhostuser

Comment 2 Sitong Liu 2019-04-10 10:36:52 UTC
Not sure if qemu as vhost user client mode is deprecated or not,
but in rhel8 test testpmd can get packets in this mode.

Versions:
openvswitch2.11-2.11.0-4.el8fdp.x86_64
qemu-kvm-3.1.0-20.module+el8+2888+cdc893a8.x86_64

Comment 3 Timothy Redaelli 2019-04-10 13:27:21 UTC
Hi,

are you running qemu as a user that is a member of hugetlbfs group (see bz#1544948)?

Comment 4 Christian Trautman 2019-04-10 19:58:37 UTC
Also which card are you running on?  We have some open issues with a vendor or two on viommu mode not working at all.

Comment 5 Pei Zhang 2019-04-11 02:00:44 UTC
This bug looks like same with:

Bug 1551508 - Request for doc update about dpdkvhostuser port is not supported with vIOMMU.

Comment 6 Sitong Liu 2019-04-11 05:54:09 UTC
(In reply to Timothy Redaelli from comment #3)
> Hi,
> 
> are you running qemu as a user that is a member of hugetlbfs group (see
> bz#1544948)?

Thanks for reply.

No, we run qemu as root by default.

But after I added root to hugetlbfs group and updated conf file like below,
sockets seem still in root group.

# cat /etc/libvirt/qemu.conf
group = "hugetlbfs"

# cat /etc/sysconfig/openvswitch
OVS_USER_ID="openvswitch:hugetlbfs"

# ll /var/run/openvswitch/vhost-*
srwxr-xr-x 1 root root 0 Apr 11 00:36 /var/run/openvswitch/vhost-*

If I missed something, please correct me, thanks!

(In reply to Christian Trautman from comment #4)
> Also which card are you running on?  We have some open issues with a vendor
> or two on viommu mode not working at all.

I am using ixgbe, we tested it with our first priority, please see:

# dpdk-devbind --status

Network devices using DPDK-compatible driver
============================================
0000:5e:00.0 'Ethernet Controller 10-Gigabit X540-AT2 1528' drv=vfio-pci unused=ixgbe
0000:5e:00.1 'Ethernet Controller 10-Gigabit X540-AT2 1528' drv=vfio-pci unused=ixgbe
0000:5f:00.0 'Ethernet Controller 10-Gigabit X540-AT2 1528' drv=vfio-pci unused=ixgbe

Comment 7 Sitong Liu 2019-04-11 06:00:29 UTC
(In reply to Pei Zhang from comment #5)
> This bug looks like same with:
> 
> Bug 1551508 - Request for doc update about dpdkvhostuser port is not
> supported with vIOMMU.

Thanks for info!

If IOMMU is not supported with vhost-user ports from OVS 2.10, can we suppose
that rhel8 support this scenario with no meaning?

If so, I will delete this case from our test plan, thanks!

Best regards,
Sitong Liu

Comment 8 Kevin Traynor 2019-04-18 11:27:09 UTC
(In reply to Pei Zhang from comment #5)
> This bug looks like same with:
> 
> Bug 1551508 - Request for doc update about dpdkvhostuser port is not
> supported with vIOMMU.

Yes, it looks to be the same issue as reported there. See https://bugzilla.redhat.com/show_bug.cgi?id=1551508#c4 for further explanation as to why it's not supported.

In summary, 

OVS "vhostuser" type ports are deprecated.

vIOMMU=on with ovs:vhostuser + qemu:client is not a valid test.
vIOMMU=on with ovs:vhostuserclient + qemu:server is a valid test.

The docs were updated as part of #1551508, so closing as NOTABUG.