RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1566996 - [ovs]geneve over nfp nic got very low udp_stream throughput
Summary: [ovs]geneve over nfp nic got very low udp_stream throughput
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: openvswitch
Version: 7.5
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: Pablo Cascon (Netronome)
QA Contact: ovs-qe
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-13 09:36 UTC by LiLiang
Modified: 2018-04-20 07:43 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-04-20 07:43:50 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description LiLiang 2018-04-13 09:36:45 UTC 
Description of problem:
geneve over nfp nic got very low udp_stream throughput

Version-Release number of selected component (if applicable):
[root@hp-dl380g9-01 topo]# uname -r
3.10.0-860.el7.x86_64
[root@hp-dl380g9-01 topo]# ethtool -i ens2np0
driver: nfp
version: 3.10.0-860.el7.x86_64 SMP mod_u
firmware-version: 0.0.3.5 0.22 nic-2.0.4 nic
expansion-rom-version: 
bus-info: 0000:0b:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
[root@hp-dl380g9-01 topo]# lspci -s 0000:0b:00.0
0b:00.0 Ethernet controller: Netronome Systems, Inc. Device 4000


How reproducible:
always


Steps to Reproduce:
1.systemA as client
        #create geneve on nfp NIC
        ip addr add 193.1.1.1/24 dev $nic_test
        ip link add $tun type geneve id 193.1.1.1 remote 193.1.1.2
        ip link set $tun up

        #add geneve to ovs bridge
        ovs-vsctl add-port $ovs_br $tun

        #create vlan if on ovs bridge 
        ovs-vsctl add-port $ovs_br vlan$vid tag=$vid -- set interface vlan$vid type=internal
        ip link set vlan$vid up
        
        #config mtu
        ip link set mtu $((1500-50-4)) dev vlan$vid
        ip link set mtu $((1500-50-4)) dev $tun
        ovs-vsctl show 
        
        #config ip
        ip addr add 194.1.1.1/24 dev vlan$vid
        ip addr add $ip6addr_test/64 dev vlan$vid

2.systemB as server
Do the same config with systemA
        #create geneve
        ip addr add 193.1.1.2/24 dev $nic_test
        ip link add $tun type geneve id 193.1.1.2 remote 193.1.1.1
        ip link set $tun up

        #add geneve to ovs bridge
        ovs-vsctl add-port $ovs_br $tun

        #create vlan if on ovs bridge 
        ovs-vsctl add-port $ovs_br vlan$vid tag=$vid -- set interface vlan$vid type=internal
        ip link set vlan$vid up
        
        #config mtu
        ip link set mtu $((1500-50-4)) dev vlan$vid
        ip link set mtu $((1500-50-4)) dev $tun
        ovs-vsctl show 
        
        #config ip
        ip addr add 194.1.1.2/24 dev vlan$vid
        ip addr add $ip6addr_test/64 dev vlan$vid

3.on systemA, run:
#netperf -t udp_stream -h 194.1.1.2 -- -m 10000

got very low throughput:

Socket  Message  Elapsed      Messages                
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec

212992   10000   10.00      557953      0    4463.60
212992           10.00        1963             15.70


Actual results:


Expected results:


Additional info:
I confirm thsi issue is relevant to nfp NIC, don't relevant to NIC on systemB,.
Because i run the same test between systemB and another systemC succeeded.

if mtu=1500, this issue occur.
if mtu=9000, this issue don't occur.
Comment 4 Brendan Galloway 2018-04-18 07:05:43 UTC 
Hello

We have been attempting to reproduce this issue in our own lab, but require some additional information.

Is the NFP installed on systemA or systemB?
What are the expected netperf results?
What parameters were used to start netserver on systemB?
Comment 5 LiLiang 2018-04-18 07:29:59 UTC 
(In reply to Brendan Galloway from comment #4)
> Hello
> 
> We have been attempting to reproduce this issue in our own lab, but require
> some additional information.
> 
> Is the NFP installed on systemA or systemB?
systemA

> What are the expected netperf results?
I don't know, but 15kbps is too low. This seems don't normal.

> What parameters were used to start netserver on systemB?
just "netserver"
Comment 6 jakub.kicinski 2018-04-18 17:33:47 UTC 
Would you be able to track what happens to the dropped packets? Can you look at the interface stats for the NFP and see if the packets are getting out of the system at all?  

Is the CPU utilization on any core high on the systems?
Comment 7 Brendan Galloway 2018-04-18 20:40:08 UTC 
We have not been able to reproduce this on our labs.  With NFP as server or client we see 3-4Gbps sent and received.

Was SystemB also the server when testing with systemC?  Your ticket indicated 4Gbps is sent, but 15Mbps received, so the issue with the packets being received.
Is there a switch between the two systems, or are they connected back to back?
Can you post the output of ethtool -S $nic_test for both systems before and after the test.
Comment 8 LiLiang 2018-04-20 07:43:50 UTC 
I can't reproduce this issue now. Close it. If it occur again, let me reopen it with detail debug info.
Note You need to log in before you can comment on or make changes to this bug.