Bug 860035

Summary: Reduced bandwidth gigabit NIC, rx packet loss, buffer overruns, terrible iperf and ethtool stats
Product: Red Hat Enterprise Linux 5 Reporter: that_IT_guy <mjklein>
Component: kernelAssignee: Nikolay Aleksandrov <naleksan>
Status: CLOSED NOTABUG QA Contact: Red Hat Kernel QE team <kernel-qe>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 5.8CC: agospoda, nhorman, peterm
Target Milestone: rc   
Target Release: ---   
Hardware: i686   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-09-28 18:42:02 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description that_IT_guy 2012-09-24 17:00:41 UTC
User-Agent:       Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:13.0) Gecko/20100101 Firefox/13.0
Build Identifier: 

I currently have two Dell Precision T3500 systems that both have less than acceptable bandwidth performance on gigabit NIC devices for both onboard and add-in PCI cards.
iperf tests on identical T3500 systems result in an average of 800-900Mbits/sec (Bandwidth).  However, on two of these systems the results average 300-400Mbits/sec (Bandwidth).
These results are persistent through i386 and x86_64 distributions as well as persisting through addin PCI NIC cards.

Reproducible: Always

Steps to Reproduce:
1.iperf -c "waiting server"
2.ethool -S eth0

Actual Results:  
Client connecting to "waiting server", TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 131.199.238.46 port 32982 connected with 131.199.238.5 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec    542 MBytes    454 Mbits/sec

ethtool -S eth0
NIC statistics:
     rx_octets: 22201666579
     rx_fragments: 0
     rx_ucast_packets: 30753404
     rx_mcast_packets: 62036
     rx_bcast_packets: 2505861
     rx_fcs_errors: 0
     rx_align_errors: 0
     rx_xon_pause_rcvd: 0
     rx_xoff_pause_rcvd: 0
     rx_mac_ctrl_rcvd: 0
     rx_xoff_entered: 0
     rx_frame_too_long_errors: 0
     rx_jabbers: 0
     rx_undersize_packets: 0
     rx_in_length_errors: 0
     rx_out_length_errors: 0
     rx_64_or_less_octet_packets: 0
     rx_65_to_127_octet_packets: 0
     rx_128_to_255_octet_packets: 0
     rx_256_to_511_octet_packets: 0
     rx_512_to_1023_octet_packets: 0
     rx_1024_to_1522_octet_packets: 0
     rx_1523_to_2047_octet_packets: 0
     rx_2048_to_4095_octet_packets: 0
     rx_4096_to_8191_octet_packets: 0
     rx_8192_to_9022_octet_packets: 0
     tx_octets: 40059885241
     tx_collisions: 0
     tx_xon_sent: 0
     tx_xoff_sent: 0
     tx_flow_control: 0
     tx_mac_errors: 0
     tx_single_collisions: 0
     tx_mult_collisions: 0
     tx_deferred: 0
     tx_excessive_collisions: 0
     tx_late_collisions: 0
     tx_collide_2times: 0
     tx_collide_3times: 0
     tx_collide_4times: 0
     tx_collide_5times: 0
     tx_collide_6times: 0
     tx_collide_7times: 0
     tx_collide_8times: 0
     tx_collide_9times: 0
     tx_collide_10times: 0
     tx_collide_11times: 0
     tx_collide_12times: 0
     tx_collide_13times: 0
     tx_collide_14times: 0
     tx_collide_15times: 0
     tx_ucast_packets: 39928439
     tx_mcast_packets: 36118
     tx_bcast_packets: 36128
     tx_carrier_sense_errors: 0
     tx_discards: 0
     tx_errors: 0
     dma_writeq_full: 0
     dma_write_prioq_full: 0
     rxbds_empty: 0
     rx_discards: 87118
     rx_errors: 0
     rx_threshold_hit: 0
     dma_readq_full: 0
     dma_read_prioq_full: 0
     tx_comp_queue_full: 0
     ring_set_send_prod_index: 0
     ring_status_update: 0
     nic_irqs: 0
     nic_avoided_irqs: 0
     nic_tx_threshold_hit: 0


Expected Results:  
Client connecting to "waiting server", TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 131.199.238.233 port 49032 connected with 131.199.238.5 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.05 GBytes    904 Mbits/sec


ethtool -S eth0
NIC statistics:
     rx_octets: 2490
     rx_fragments: 0
     rx_ucast_packets: 16
     rx_mcast_packets: 0
     rx_bcast_packets: 6
     rx_fcs_errors: 0
     rx_align_errors: 0
     rx_xon_pause_rcvd: 0
     rx_xoff_pause_rcvd: 0
     rx_mac_ctrl_rcvd: 0
     rx_xoff_entered: 0
     rx_frame_too_long_errors: 0
     rx_jabbers: 0
     rx_undersize_packets: 0
     rx_in_length_errors: 0
     rx_out_length_errors: 0
     rx_64_or_less_octet_packets: 0
     rx_65_to_127_octet_packets: 0
     rx_128_to_255_octet_packets: 0
     rx_256_to_511_octet_packets: 0
     rx_512_to_1023_octet_packets: 0
     rx_1024_to_1522_octet_packets: 0
     rx_1523_to_2047_octet_packets: 0
     rx_2048_to_4095_octet_packets: 0
     rx_4096_to_8191_octet_packets: 0
     rx_8192_to_9022_octet_packets: 0
     tx_octets: 10349
     tx_collisions: 0
     tx_xon_sent: 0
     tx_xoff_sent: 0
     tx_flow_control: 0
     tx_mac_errors: 0
     tx_single_collisions: 0
     tx_mult_collisions: 0
     tx_deferred: 0
     tx_excessive_collisions: 0
     tx_late_collisions: 0
     tx_collide_2times: 0
     tx_collide_3times: 0
     tx_collide_4times: 0
     tx_collide_5times: 0
     tx_collide_6times: 0
     tx_collide_7times: 0
     tx_collide_8times: 0
     tx_collide_9times: 0
     tx_collide_10times: 0
     tx_collide_11times: 0
     tx_collide_12times: 0
     tx_collide_13times: 0
     tx_collide_14times: 0
     tx_collide_15times: 0
     tx_ucast_packets: 10
     tx_mcast_packets: 33
     tx_bcast_packets: 2
     tx_carrier_sense_errors: 0
     tx_discards: 0
     tx_errors: 0
     dma_writeq_full: 0
     dma_write_prioq_full: 0
     rxbds_empty: 0
     rx_discards: 0
     rx_errors: 0
     rx_threshold_hit: 0
     dma_readq_full: 0
     dma_read_prioq_full: 0
     tx_comp_queue_full: 0
     ring_set_send_prod_index: 0
     ring_status_update: 0
     nic_irqs: 0
     nic_avoided_irqs: 0
     nic_tx_threshold_hit: 0


I have installed an add-in Dlink DGE-530T on the 64bit 5.8 installation and obtained very similar results.  I have also gone from a 64bit installation to a 32bit 5.8 installation and both onboard (BCM5761) and add-in NICs exhibited the same performance issues regardless of OS or active NIC.
The only difference between the add-in NIC and the onboard NIC that I have seen is that the onboard experiences rx packet loss in high numbers where as the dlink NIC experiences TCP buffer overruns.
Reducing the net.core.optmem_max from 10240 to 5120 in the case of the Dlink NIC appears to have resolved the buffer overruns but has not addressed the bandwidth performance issue.
Modifying the stack settings of the onboard BCM5761 however did not make any noticeable difference to the packet loss or overall bandwidth performance.

Comment 1 Jay Fenlason 2012-09-24 17:22:15 UTC
Why do you think that this is a problem with ethtool, and not the kernel?

Do you only see these problems with RHEL-5, or do they occur with RHEL-6 / Fedora as well?

Comment 2 that_IT_guy 2012-09-24 17:32:25 UTC
Honestly, I wasn't sure which component to categorize it under (first bug report, so forgive my noobishness).  I chose the ethtool category simply based on that it is network related.  I haven't had the chance to test it on Fedora fedora is not allowed on my corporate network, and RHEL6 has not been put into production in my environment as of yet.
The one RHEL6 system that I do have is on a different hardware platform (T3400, BCM5754) and does not seem to exhibit any of these issues as far as bandwidth performance, packet loss, or buffer overruns.

Comment 3 that_IT_guy 2012-09-25 19:51:52 UTC
Stumbled upon something interesting today on one of the two problem boxes.. Had my rx ring size scaled up to 500 for a day just to check results.. and had framing errors out the wazoo.. Reduced my rx ring back to 200 and no more framing errors. yay.. What I found most interesting is that at this stage this system (tg3 bcm5761 64bit RHEL5.3) is not exhibiting any packet loss, framing errors, or buffer overruns on the receiving end and even though I am no longer seeing any of the above the NIC is doing something strange.  If I bring the interface down and back up either using /etc/init.d/network restart or ifconfig ifup/down eth0 and then test with iperf I hit an average of around 830Mbit/sec up to about a 10GB transfer overall.  At the point that I hit the 10GB mark my performance halves.

My Ring Test
Ring parameters for eth0:
Pre-set maximums:
RX:             511
RX Mini:        0
RX Jumbo:       0
TX:             511
Current hardware settings:
RX:             500
RX Mini:        0
RX Jumbo:       0
TX:             511

Back to 200
Ring parameters for eth0:
Pre-set maximums:
RX:             511
RX Mini:        0
RX Jumbo:       0
TX:             511
Current hardware settings:
RX:             200
RX Mini:        0
RX Jumbo:       0
TX:             511

iperf results immediately after eth0 down/up (this was run 5 times)
Client connecting to server, TCP port 5001
TCP window size:   256 KByte (WARNING: requested 2.00 MByte)
------------------------------------------------------------
[  3] local 0.0.0.0 port 34168 connected with 0.0.0.0 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 2.0 sec    202 MBytes    847 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  2.0- 4.0 sec    202 MBytes    846 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  4.0- 6.0 sec    202 MBytes    848 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  6.0- 8.0 sec    202 MBytes    849 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  8.0-10.0 sec    202 MBytes    848 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3] 10.0-12.0 sec    203 MBytes    851 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3] 12.0-14.0 sec    201 MBytes    843 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3] 14.0-16.0 sec    197 MBytes    827 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3] 16.0-18.0 sec    200 MBytes    837 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3] 18.0-20.0 sec    199 MBytes    835 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-20.0 sec  1.96 GBytes    843 Mbits/sec

ifconfig stats of eth0 minus identifying info
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3955787 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7666484 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:281073766 (268.0 MiB)  TX bytes:11631517756 (10.8 GiB)
          Interrupt:177

iperf results after client has transferred 10GB from previous iperf tests
Client connecting to server, TCP port 5001
TCP window size:   256 KByte (WARNING: requested 2.00 MByte)
------------------------------------------------------------
[  3] local 0.0.0.0 port 37186 connected with 0.0.0.0 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 2.0 sec  87.8 MBytes    368 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  2.0- 4.0 sec  94.0 MBytes    394 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  4.0- 6.0 sec  99.9 MBytes    419 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  6.0- 8.0 sec    101 MBytes    425 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  8.0-10.0 sec    101 MBytes    422 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3] 10.0-12.0 sec    101 MBytes    423 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3] 12.0-14.0 sec  93.5 MBytes    392 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3] 14.0-16.0 sec  99.2 MBytes    416 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3] 16.0-18.0 sec    102 MBytes    428 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3] 18.0-20.0 sec    101 MBytes    425 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-20.0 sec    981 MBytes    411 Mbits/sec

The above output is something I was seeing as an initial symptom on both of these systems and apparently slipped my mind.  Until today I hadn't realized that the bandwidth was halving after the client received a particular amount of data.

Comment 5 that_IT_guy 2012-09-28 17:21:47 UTC
NOT a RHEL BUG.  Found to be faulty PCI BUS.  Report can be closed.