Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be unavailable on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1566996 - [ovs]geneve over nfp nic got very low udp_stream throughput
Summary: [ovs]geneve over nfp nic got very low udp_stream throughput
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: openvswitch
Version: 7.5
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: Pablo Cascon (Netronome)
QA Contact: ovs-qe
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-13 09:36 UTC by LiLiang
Modified: 2018-04-20 07:43 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-04-20 07:43:50 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description LiLiang 2018-04-13 09:36:45 UTC
Description of problem:
geneve over nfp nic got very low udp_stream throughput

Version-Release number of selected component (if applicable):
[root@hp-dl380g9-01 topo]# uname -r
3.10.0-860.el7.x86_64
[root@hp-dl380g9-01 topo]# ethtool -i ens2np0
driver: nfp
version: 3.10.0-860.el7.x86_64 SMP mod_u
firmware-version: 0.0.3.5 0.22 nic-2.0.4 nic
expansion-rom-version: 
bus-info: 0000:0b:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
[root@hp-dl380g9-01 topo]# lspci -s 0000:0b:00.0
0b:00.0 Ethernet controller: Netronome Systems, Inc. Device 4000


How reproducible:
always


Steps to Reproduce:
1.systemA as client
        #create geneve on nfp NIC
        ip addr add 193.1.1.1/24 dev $nic_test
        ip link add $tun type geneve id 193.1.1.1 remote 193.1.1.2
        ip link set $tun up

        #add geneve to ovs bridge
        ovs-vsctl add-port $ovs_br $tun

        #create vlan if on ovs bridge 
        ovs-vsctl add-port $ovs_br vlan$vid tag=$vid -- set interface vlan$vid type=internal
        ip link set vlan$vid up
        
        #config mtu
        ip link set mtu $((1500-50-4)) dev vlan$vid
        ip link set mtu $((1500-50-4)) dev $tun
        ovs-vsctl show 
        
        #config ip
        ip addr add 194.1.1.1/24 dev vlan$vid
        ip addr add $ip6addr_test/64 dev vlan$vid

2.systemB as server
Do the same config with systemA
        #create geneve
        ip addr add 193.1.1.2/24 dev $nic_test
        ip link add $tun type geneve id 193.1.1.2 remote 193.1.1.1
        ip link set $tun up

        #add geneve to ovs bridge
        ovs-vsctl add-port $ovs_br $tun

        #create vlan if on ovs bridge 
        ovs-vsctl add-port $ovs_br vlan$vid tag=$vid -- set interface vlan$vid type=internal
        ip link set vlan$vid up
        
        #config mtu
        ip link set mtu $((1500-50-4)) dev vlan$vid
        ip link set mtu $((1500-50-4)) dev $tun
        ovs-vsctl show 
        
        #config ip
        ip addr add 194.1.1.2/24 dev vlan$vid
        ip addr add $ip6addr_test/64 dev vlan$vid

3.on systemA, run:
#netperf -t udp_stream -h 194.1.1.2 -- -m 10000

got very low throughput:

Socket  Message  Elapsed      Messages                
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec

212992   10000   10.00      557953      0    4463.60
212992           10.00        1963             15.70


Actual results:


Expected results:


Additional info:
I confirm thsi issue is relevant to nfp NIC, don't relevant to NIC on systemB,.
Because i run the same test between systemB and another systemC succeeded.

if mtu=1500, this issue occur.
if mtu=9000, this issue don't occur.

Comment 4 Brendan Galloway 2018-04-18 07:05:43 UTC
Hello

We have been attempting to reproduce this issue in our own lab, but require some additional information.

Is the NFP installed on systemA or systemB?
What are the expected netperf results?
What parameters were used to start netserver on systemB?

Comment 5 LiLiang 2018-04-18 07:29:59 UTC
(In reply to Brendan Galloway from comment #4)
> Hello
> 
> We have been attempting to reproduce this issue in our own lab, but require
> some additional information.
> 
> Is the NFP installed on systemA or systemB?
systemA

> What are the expected netperf results?
I don't know, but 15kbps is too low. This seems don't normal.

> What parameters were used to start netserver on systemB?
just "netserver"

Comment 6 jakub.kicinski 2018-04-18 17:33:47 UTC
Would you be able to track what happens to the dropped packets? Can you look at the interface stats for the NFP and see if the packets are getting out of the system at all?  

Is the CPU utilization on any core high on the systems?

Comment 7 Brendan Galloway 2018-04-18 20:40:08 UTC
We have not been able to reproduce this on our labs.  With NFP as server or client we see 3-4Gbps sent and received.

Was SystemB also the server when testing with systemC?  Your ticket indicated 4Gbps is sent, but 15Mbps received, so the issue with the packets being received.
Is there a switch between the two systems, or are they connected back to back?
Can you post the output of ethtool -S $nic_test for both systems before and after the test.

Comment 8 LiLiang 2018-04-20 07:43:50 UTC
I can't reproduce this issue now. Close it. If it occur again, let me reopen it with detail debug info.


Note You need to log in before you can comment on or make changes to this bug.