Bug 728364

Summary: e1000e network really really slow
Product: Red Hat Enterprise Linux 6 Reporter: Need Real Name <lsof>
Component: kernelAssignee: Andy Gospodarek <agospoda>
Status: CLOSED NOTABUG QA Contact: Red Hat Kernel QE team <kernel-qe>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.1CC: arozansk, lsof, peterm
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-08-08 02:33:54 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
lspci -v none

Description Need Real Name 2011-08-04 20:24:00 UTC
Description of problem:
I get amazingly slow network speeds since upgrading from 5.6 to 6.1

Version-Release number of selected component (if applicable):
2.6.32-131.6.1.el6.x86_64

Problem machine:
# wget http://cachefly.cachefly.net/100mb.test
 1% [>                           ] 1,662,304    142K/s  eta 9m 26s

Gentoo box (same switch):
51% [==============>             ] 54,042,576  9.08M/s  eta 6s    
  
# ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: off
large-receive-offload: off

# ethtool eth0
Settings for eth0:
	Supported ports: [ TP ]
	Supported link modes:   10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Full 
	Supports auto-negotiation: Yes
	Advertised link modes:  Not reported
	Advertised pause frame use: No
	Advertised auto-negotiation: No
	Speed: 100Mb/s
	Duplex: Full
	Port: Twisted Pair
	PHYAD: 2
	Transceiver: internal
	Auto-negotiation: off
	MDI-X: off
	Supports Wake-on: pumbag
	Wake-on: g
	Current message level: 0x00000001 (1)
	Link detected: yes

Comment 1 Need Real Name 2011-08-04 20:24:31 UTC
Created attachment 516780 [details]
lspci -v

Comment 3 Andy Gospodarek 2011-08-05 15:18:34 UTC
That definitely seems odd.  

Is this the:

00:19.0 Ethernet controller: Intel Corporation 82578DM Gigabit Network Connection

or

06:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection

Can you paste the output of 'ethtool -S eth0' before and after your test download is started.  I'm curious if the driver is reporting any sort of error or drop and that manifests itself with a slow transfer rate.

Comment 4 Need Real Name 2011-08-05 16:06:55 UTC
# ethtool -i eth0
driver: e1000e
version: 1.2.20-k2
firmware-version: 0.12-2
bus-info: 0000:00:19.0

# ethtool -S eth0
NIC statistics:
     rx_packets: 5115771
     tx_packets: 8486888
     rx_bytes: 1162613050
     tx_bytes: 11252325421
     rx_broadcast: 306798
     tx_broadcast: 330
     rx_multicast: 160632
     tx_multicast: 906
     rx_errors: 42560
     tx_errors: 0
     tx_dropped: 0
     multicast: 160632
     collisions: 0
     rx_length_errors: 0
     rx_over_errors: 0
     rx_crc_errors: 42560
     rx_frame_errors: 0
     rx_no_buffer_count: 0
     rx_missed_errors: 0
     tx_aborted_errors: 0
     tx_carrier_errors: 0
     tx_fifo_errors: 0
     tx_heartbeat_errors: 0
     tx_window_errors: 0
     tx_abort_late_coll: 0
     tx_deferred_ok: 0
     tx_single_coll_ok: 0
     tx_multi_coll_ok: 0
     tx_timeout_count: 0
     tx_restart_queue: 0
     rx_long_length_errors: 0
     rx_short_length_errors: 0
     rx_align_errors: 0
     tx_tcp_seg_good: 0
     tx_tcp_seg_failed: 0
     rx_flow_control_xon: 0
     rx_flow_control_xoff: 0
     tx_flow_control_xon: 0
     tx_flow_control_xoff: 0
     rx_long_byte_count: 1162613050
     rx_csum_offload_good: 4604822
     rx_csum_offload_errors: 7
     rx_header_split: 0
     alloc_rx_buff_failed: 0
     tx_smbus: 0
     rx_smbus: 0
     dropped_smbus: 0
     rx_dma_failed: 0
     tx_dma_failed: 0

Comment 5 Need Real Name 2011-08-05 16:34:41 UTC
Doesn't look good:
     rx_crc_errors: 42560

I will get the cable checked. The switch reports no errors.

Comment 6 Andy Gospodarek 2011-08-05 16:38:54 UTC
(In reply to comment #4)
> # ethtool -i eth0
> driver: e1000e
> version: 1.2.20-k2
> firmware-version: 0.12-2
> bus-info: 0000:00:19.0

Thanks, looks like this is the 82578DM.

> # ethtool -S eth0
> NIC statistics:
>      rx_packets: 5115771
>      tx_packets: 8486888
>      rx_bytes: 1162613050
>      tx_bytes: 11252325421
>      rx_broadcast: 306798
>      tx_broadcast: 330
>      rx_multicast: 160632
>      tx_multicast: 906
>      rx_errors: 42560
>      tx_errors: 0
>      tx_dropped: 0
>      multicast: 160632
>      collisions: 0
>      rx_length_errors: 0
>      rx_over_errors: 0
>      rx_crc_errors: 42560

This is a bit odd.  I would not expect to see many rx_crc_errors during normal operation.

I noticed in comment #0 that auto-negotiation is disabled.  Is the performance any different or are there reduced rx_crc_errors when auto-negotiation is enabled?

Comment 7 Andy Gospodarek 2011-08-05 16:39:16 UTC
(In reply to comment #5)
> Doesn't look good:
>      rx_crc_errors: 42560
> 
> I will get the cable checked. The switch reports no errors.

Agreed. :-)

Comment 8 Need Real Name 2011-08-05 17:25:38 UTC
# netstat -s
Ip:
    1343573 total packets received
    0 forwarded
    0 incoming packets discarded
    1229107 incoming packets delivered
    1202994 requests sent out
    40 dropped because of missing route
Icmp:
    729 ICMP messages received
    0 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 85
        timeout in transit: 592
        echo requests: 4
        echo replies: 48
    1361 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 314
        echo request: 2
        echo replies: 4
IcmpMsg:
        InType0: 48
        InType3: 85
        InType8: 4
        InType11: 592
        OutType0: 4
        OutType3: 314
        OutType8: 2
        OutType69: 1041
Tcp:
    5322 active connections openings
    70 passive connection openings
    5168 failed connection attempts
    3 connection resets received
    9 connections established
    1227421 segments received
    1197491 segments send out
    3088 segments retransmited
    0 bad segments received.
    5311 resets sent
Udp:
    954 packets received
    3 packets to unknown port received.
    0 packet receive errors
    1072 packets sent
UdpLite:
TcpExt:
    2 resets received for embryonic SYN_RECV sockets
    1 packets pruned from receive queue because of socket buffer overrun
    87 TCP sockets finished time wait in fast timer
    2 packets rejects in established connections because of timestamp
    17914 delayed acks sent
    14 delayed acks further delayed because of locked socket
    Quick ack mode was activated 3282 times
    69 packets directly queued to recvmsg prequeue.
    55024 packets directly received from backlog
    294 packets directly received from prequeue
    631368 packets header predicted
    43 packets header predicted and directly queued to user
    164602 acknowledgments not containing data received
    232210 predicted acknowledgments
    1294 times recovered from packet loss due to SACK data
    14 congestion windows recovered after partial ack
    1019 TCP data loss events
    TCPLostRetransmit: 34
    218 timeouts after SACK recovery
    13 timeouts in loss state
    1886 fast retransmits
    162 forward retransmits
    290 retransmits in slow start
    426 other TCP timeouts
    45 sack retransmits failed
    152 packets collapsed in receive queue due to low socket buffer
    3293 DSACKs sent for old packets
    12 DSACKs sent for out of order packets
    61 DSACKs received
    14 connections reset due to unexpected data
    1 connections reset due to early user close
    1 connections aborted due to timeout
    TCPDSACKIgnoredOld: 18
    TCPDSACKIgnoredNoUndo: 23
    TCPSpuriousRTOs: 1
    TCPSackShifted: 674
    TCPSackMerged: 1956
    TCPSackShiftFallback: 5024
IpExt:
    InMcastPkts: 2297
    OutMcastPkts: 64
    InBcastPkts: 4447
    InOctets: 2518825255
    OutOctets: 2035507350
    InMcastOctets: 124467
    OutMcastOctets: 15614
    InBcastOctets: 725550

Comment 9 Need Real Name 2011-08-07 15:26:12 UTC
The title of this bug should be "when auto-negotiate goes very, very wrong".

Sorry for the noise :)

Comment 10 Andy Gospodarek 2011-08-08 02:33:54 UTC
From comment #9, it sounds like this can be closed.

If it looks like there are more problems with the driver and auto-negotiate, feel free to reopen this.