Bug 1627816 - 1Gb/s ethernet port is configured as 10Mb/s port on many different laptops, network-scripts work well
Summary: 1Gb/s ethernet port is configured as 10Mb/s port on many different laptops, n...
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 30
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: https://fedoraproject.org/wiki/Common...
: 1718586 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-09-11 14:38 UTC by Kamil Páral
Modified: 2020-05-26 18:23 UTC (History)
56 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-26 18:23:36 UTC
Type: Bug


Attachments (Terms of Use)
journal after cable connect (8.80 KB, text/plain)
2018-09-11 14:38 UTC, Kamil Páral
no flags Details
journal after running ethtool -r (3.42 KB, text/plain)
2018-09-11 14:39 UTC, Kamil Páral
no flags Details
NM trace log (267.76 KB, text/plain)
2018-09-14 16:03 UTC, Kamil Páral
no flags Details
lspci and lspci -vvv output (26.21 KB, text/plain)
2019-10-23 10:58 UTC, Marko Karg
no flags Details

Description Kamil Páral 2018-09-11 14:38:10 UTC
Description of problem:
I have a Thinkpad T480s laptop and it has a 1Gb/s ethernet port. If I boot the laptop with ethernet cable attached, it works with 1Gb/s speeds. But once I detach and re-attach the cable, it suddenly works with just 10Mb/s. This is nicely visible in gnome control center, the initial "Connected - 1000Mb/s" gets replaced by "Connected - 10Mb/s". This only affects the download speed, I can still upload with 1Gb/s speeds. The same problem happens when I boot the laptop with the cable unplugged, and plug it in after boot.

I have debugged this problem for a long time and finally traced it down to NetworkManager (or kernel, but NM is definitely affecting it somehow). Here's a summary:
* This is not a single hardware failure, because it occurs on a colleague's T480s as well (so it probably affects this whole laptop family).
* This is not a hardware/firmware fault of this laptop family, because this problem doesn't happen on Windows 10 (always runs at 1Gb/s).
* This problem occurs in all kernel version available in Fedora 29.
* This problem occurs when I boot the Fedora 28(!) Workstation Live, so it's not brand new.
* This problem occurs even with selinux in permissive mode, so it's probably not selinux related.
* This problem doesn't occur when NetworkManager service is stopped and I use ifup/ifdown scripts to manage my ethernet port connection. This made me report this against NM.
* This problem can be fixed while NM is running when I execute "ethtool -r device".


So it seems that NM somehow breaks speed negotiation. If the connection is made outside of NM (during boot or using ifup), the negotiation works fine.

My network card is:
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (4) I219-LM [8086:15d7] (rev 21)

I also have a network card on a thunderbolt dock, and everything works fine there. The problem affects just that internal laptop port displayed above.

When I connect the cable, I see these messages in the journal:

Sep 11 15:34:58 phoenix NetworkManager[9872]: <info>  [1536672898.9113] device (enp0s31f6): carrier: link connected
Sep 11 15:34:58 phoenix kernel: e1000e: enp0s31f6 NIC Link is Up 10 Mbps Full Duplex, Flow Control: Rx/Tx
Sep 11 15:34:58 phoenix kernel: e1000e 0000:00:1f.6 enp0s31f6: 10/100 speed: disabling TSO
Sep 11 15:34:58 phoenix NetworkManager[9872]: <info>  [1536672898.9115] device (enp0s31f6): state change: unavailable -> disconnected (reason 'carrier-changed', sys-iface-state: 'managed')
Sep 11 15:34:58 phoenix NetworkManager[9872]: <info>  [1536672898.9124] policy: auto-activating connection 'enp0s31f6' (803b0104-53b1-3c32-9688-a1d51ac083b8)
...

And ethtool says:
$ sudo ethtool enp0s31f6 | grep Speed
	Speed: 10Mb/s

When I run "ethtool -r enp0s31f6", I see this in the journal:

Sep 11 15:35:31 phoenix NetworkManager[9872]: <info>  [1536672931.8034] device (enp0s31f6): carrier: link connected
Sep 11 15:35:31 phoenix kernel: e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Sep 11 15:35:31 phoenix NetworkManager[9872]: <info>  [1536672931.8037] device (enp0s31f6): DHCPv4 lease renewal requested
Sep 11 15:35:31 phoenix NetworkManager[9872]: <info>  [1536672931.8200] dhcp4 (enp0s31f6): canceled DHCP transaction, DHCP client pid 12266
Sep 11 15:35:31 phoenix NetworkManager[9872]: <info>  [1536672931.8201] dhcp4 (enp0s31f6): state changed bound -> done
Sep 11 15:35:31 phoenix NetworkManager[9872]: <info>  [1536672931.8204] dhcp4 (enp0s31f6): activation: beginning transaction (timeout in 45 seconds)
...

And ethtool says:
$ sudo ethtool enp0s31f6 | grep Speed
	Speed: 1000Mb/s


Do you need any more information to help you fix this? Thanks.


Version-Release number of selected component (if applicable):
kernel-4.18.5-300.fc29.x86_64
NetworkManager-1.12.2-2.fc29.x86_64
network-scripts-10.01-1.fc29.x86_64
ethtool-4.17-2.fc29.x86_64


How reproducible:
always

Steps to Reproduce:
1. get a T480s
2. plug in a cable (and disable wifi), reboot
3. see that your download and upload speed is 1Gb/s (use iperf3 with a different machine, or just speedtest.net)
4. disconnect the cable, wait 10 seconds (or check "ip a" until IP addresses for your ethernet device are gone)
5. reconnect the cable
6. see that your download speed is 10Mb/s, but your upload speed still 1Gb/s
7. run "sudo ethtool -r device", wait some seconds
8. see that your download and upload speed is again 1Gb/s
- and/or alternatively -
9. repeat steps 2-6
10. stop NetworkManager.service
11. run "sudo ifdown device"
12. disconnect the cable, wait 10 seconds
13. reconnect the cable
14. run "sudo ifup device", wait some seconds
15. see that your download and upload speed is again 1Gb/s

Actual results:
I get a 10Mb/s download speed if I don't know how to fix it

Expected results:
I get 1Gb/s download speed

Comment 1 Kamil Páral 2018-09-11 14:38:49 UTC
Created attachment 1482384 [details]
journal after cable connect

Comment 2 Kamil Páral 2018-09-11 14:39:09 UTC
Created attachment 1482386 [details]
journal after running ethtool -r

Comment 3 Kamil Páral 2018-09-11 14:40:22 UTC
I'm going to mark this as a common bug, because quite a few owners of this laptop could be surprised.

Comment 4 Dan Williams 2018-09-12 13:46:00 UTC
FWIW, I cannot reproduce this on F27 with 4.17.12-100.fc27.x86_64 and NetworkManager-1.8.8-1.fc27.x86_64.  My T480s always negotiates 1000Mb/s on cable plug/unplug while NM is active.

Comment 5 Thomas Haller 2018-09-12 14:08:21 UTC
could you provide a full logfile with level=TRACE ? See https://cgit.freedesktop.org/NetworkManager/NetworkManager/tree/contrib/fedora/rpm/NetworkManager.conf for hints about logs.

Comment 6 Kamil Páral 2018-09-14 15:40:17 UTC
(In reply to Dan Williams from comment #4)
> FWIW, I cannot reproduce this on F27 with 4.17.12-100.fc27.x86_64 and
> NetworkManager-1.8.8-1.fc27.x86_64.  My T480s always negotiates 1000Mb/s on
> cable plug/unplug while NM is active.

Confirmed. This is working well if I boot F27 Workstation Live (containing kernel-4.13.9-300.fc27.x86_64 and NetworkManager-1.8.4-2.fc27.x86_64). At least we have a reference point at when it worked.

I'll work on getting the logs.

Comment 7 Kamil Páral 2018-09-14 16:03:39 UTC
Created attachment 1483369 [details]
NM trace log

Here's the log. I disabled wifi, restarted the service, and connected the cable, nothing else. The interface in question is enp0s31f6, and here's an interesting line:

Sep 14 17:51:46 phoenix NetworkManager[4180]: <debug> [1536940306.3754] device[0x559d05be2520] (enp0s31f6): speed is now 10 Mb/s

Comment 8 Thomas Haller 2018-09-14 18:29:57 UTC
NetworkManager has support for settings ethernet.auto-negotiate, ethernet.speed, and ethernet.duplex.

It seems you didn't specify these parameters in the connection profile, hence, NetworkManager would be expected to not touch these settings.
NetworkManager configures this via ethtool ioctl "ETHTOOL_SSET", and in comment 7 no logging indicates that NetworkManager would set that. As one would expect.


In the logfile there is

<info>  [1536940306.3752] device (enp0s31f6): carrier: link connected
<trace> [1536940306.3753] ethtool[2]: ETHTOOL_GSET, enp0s31f6: success
<debug> [1536940306.3754] device[0x559d05be2520] (enp0s31f6): speed is now 10 Mb/s

This is not something done by NetworkManager. It just reports the set speed.

Before that, NetworkManager does very little with the device:
  - it sets ipv6 addrgen mode none
  - it sets some sysctl relevant for IPv6
  - it deletes a fq_codel qdisc
  - it sets the device IFF_UP

none of this seems relevant to me.



With
  $ grep -r -i ethtool /etc/sysconfig/
does that find anything relevant? (ignoring the matches in "network-functions" and "ifup-eth" files)

Comment 9 Kamil Páral 2018-09-18 08:11:31 UTC
(In reply to Thomas Haller from comment #8)
> It seems you didn't specify these parameters in the connection profile,

I'd like to point out that I'm running a clean F29 installation and I haven't configured anything manually, I'm running with defaults. I also see the same behavior when I test this using F28 Workstation Live image (guaranteed to be using NM defaults).

> With
>   $ grep -r -i ethtool /etc/sysconfig/
> does that find anything relevant? (ignoring the matches in
> "network-functions" and "ifup-eth" files)

No, the only matches are in ifup-eth and network-functions.

Comment 10 Beniamino Galvani 2018-09-18 10:08:04 UTC
This sounds very similar to bug:

https://bugzilla.redhat.com/show_bug.cgi?id=1038156#c15

Comment 11 Beniamino Galvani 2018-09-27 16:40:41 UTC
As Thomas said it would be strange if this was related to
NetworkManager, as it seems more a physical link negotiation issue.

Have you tried if the issue also happens when connected to a different
switch? Also, could you please try to disable eee with the following
command:

 ethtool --set-eee enp0s31f6 eee off

and see if it makes any difference? Thanks.

Comment 12 Kamil Páral 2018-10-01 11:01:23 UTC
(In reply to Beniamino Galvani from comment #11)
> As Thomas said it would be strange if this was related to
> NetworkManager, as it seems more a physical link negotiation issue.

Yeah, however stopping NM service fixes the issue.

> Have you tried if the issue also happens when connected to a different
> switch? 

I tested at home and at work, behaves the same.

> Also, could you please try to disable eee with the following
> command:
> 
>  ethtool --set-eee enp0s31f6 eee off
> 
> and see if it makes any difference? Thanks.

Before running the command, the status was:

$ sudo ethtool --show-eee enp0s31f6
EEE Settings for enp0s31f6:
	EEE status: enabled - inactive
	Tx LPI: 17 (us)
	Supported EEE link modes:  100baseT/Full 
	                           1000baseT/Full 
	Advertised EEE link modes:  100baseT/Full 
	                            1000baseT/Full 
	Link partner advertised EEE link modes:  Not reported

After running it:
	EEE status: disabled

And it did not fix the problem.


An additional interesting fact: If I remove the cable and then plug it back again fast enough (under a second or two, before GNOME/NM has time to react and update the GUI), the connection is "uninterrupted" and the speed stays 1Gb/s. Only when I wait a few more seconds, so that GNOME/NM shows "cable disconnected" and re-connect it again, then I get 10Mb/s.

Comment 13 Kamil Páral 2018-10-01 11:41:06 UTC
On my F29 system I downgraded to NetworkManager-1:1.8.4-2.fc27 (the F27 Live version that worked OK for me), and it did not fix the problem. I also downgraded to kernel-4.13.9-300.fc27 (again, F27 Live version), and kept NM in the latest fc29 version, and it fixed the problem!

So, this is definitely somehow related to kernel, because downgrading the kernel fixes the issue. However, stopping the NM service fixes the issue as well. How to explain this?

Comment 14 Beniamino Galvani 2018-10-01 12:53:40 UTC
(In reply to Kamil Páral from comment #13)
> On my F29 system I downgraded to NetworkManager-1:1.8.4-2.fc27 (the F27 Live
> version that worked OK for me), and it did not fix the problem. I also
> downgraded to kernel-4.13.9-300.fc27 (again, F27 Live version), and kept NM
> in the latest fc29 version, and it fixed the problem!
> 
> So, this is definitely somehow related to kernel, because downgrading the
> kernel fixes the issue. However, stopping the NM service fixes the issue as
> well. How to explain this?

Perhaps the bug is triggered by NM when it applies some settings to the interface, like bringing it up at the wrong moment. Do you have any chance to do a bisection or try an intermediate kernel between 4.13.9 and 4.18.5?

Comment 15 Kamil Páral 2018-10-02 08:52:12 UTC
The change occurred somewhere between these Koji builds:
kernel-4.15.0-1.fc28    - works fine
kernel-4.15.3-300.fc27  - broken

Is that sufficient to identify the problem?

Comment 16 Beniamino Galvani 2018-10-02 09:48:09 UTC
It seems to me that those two kernels have the same e1000e driver:

 $ diff -Naur kernel-4.15.fc28/linux-4.15.0-1.fc28.x86_64/drivers/net/ethernet/intel/e1000e/ kernel-4.15.fc27/linux-4.15.3-300.fc27.x86_64/drivers/net/ethernet/intel/e1000e/
 $

so I am clueless. I'm reassigning the bug to kernel for investigation.

Comment 17 Laura Abbott 2018-10-02 14:30:11 UTC
4.15 is old, please test on the latest kernel

Comment 18 Thomas Haller 2018-10-03 08:21:52 UTC
(In reply to Laura Abbott from comment #17)
> 4.15 is old, please test on the latest kernel

Laura, it seems in comment 0 a later kernel (kernel-4.18.5-300.fc29.x86_64) was already found to have the same issue.

Kamil merely bisected the issue to above 4.15 kernel.

Comment 19 John Eckersberg 2018-10-26 19:08:06 UTC
I have this exact same behavior on my T460s, with kernel-4.18.16-200.fc28.x86_64.  I will dig a bit and see what I can find.

Comment 20 John Eckersberg 2018-10-26 20:46:50 UTC
Things I've tried so far...

Updated BIOS to latest (from https://support.lenovo.com/us/en/downloads/ds112118):

$ sudo dmidecode | grep 'BIOS Revision'
        BIOS Revision: 1.39

Built latest upstream kernel:

$ uname -r
4.19.0

The problem seems to be a bit different now.  I get 1000 Mbps on boot, then twice in a row I get 10 Mbps, after that it is consistently 1000 Mbps:

$ dmesg -w | grep e1000e
[    3.104444] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[    3.104445] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    3.104653] e1000e 0000:00:1f.6: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[    3.201608] e1000e 0000:00:1f.6 0000:00:1f.6 (uninitialized): registered PHC clock
[    3.284880] e1000e 0000:00:1f.6 eth0: (PCI Express:2.5GT/s:Width x1) c8:5b:76:7e:7a:c6
[    3.284885] e1000e 0000:00:1f.6 eth0: Intel(R) PRO/1000 Network Connection
[    3.284969] e1000e 0000:00:1f.6 eth0: MAC: 12, PHY: 12, PBA No: 1000FF-0FF
[    3.286604] e1000e 0000:00:1f.6 enp0s31f6: renamed from eth0
[   35.195560] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[   76.660652] e1000e: enp0s31f6 NIC Link is Down
[   93.282669] e1000e: enp0s31f6 NIC Link is Up 10 Mbps Full Duplex, Flow Control: None
[   93.282674] e1000e 0000:00:1f.6 enp0s31f6: 10/100 speed: disabling TSO
[  737.448473] e1000e: enp0s31f6 NIC Link is Down
[  745.562407] e1000e: enp0s31f6 NIC Link is Up 10 Mbps Full Duplex, Flow Control: None
[  745.562411] e1000e 0000:00:1f.6 enp0s31f6: 10/100 speed: disabling TSO
[  806.904369] e1000e: enp0s31f6 NIC Link is Down
[  814.088757] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[  859.921403] e1000e: enp0s31f6 NIC Link is Down
[  867.063791] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[  921.268931] e1000e: enp0s31f6 NIC Link is Down
[  924.592915] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[  995.766034] e1000e: enp0s31f6 NIC Link is Down
[  999.102321] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[ 1000.545721] e1000e: enp0s31f6 NIC Link is Down
[ 1005.122067] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[ 1007.768207] e1000e: enp0s31f6 NIC Link is Down
[ 1013.988160] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[ 1017.053427] e1000e: enp0s31f6 NIC Link is Down
[ 1022.172265] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[ 1024.567449] e1000e: enp0s31f6 NIC Link is Down
[ 1029.606174] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None


Next I will run the same test with the same updated BIOS but the older kernel (kernel-4.18.16-200.fc28.x86_64)

Comment 21 John Eckersberg 2018-10-26 21:10:24 UTC
Pretty similar pattern with kernel-4.18.16-200.fc28.x86_64:

$ dmesg -w | grep e1000e
[    1.741793] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[    1.741794] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    1.741981] e1000e 0000:00:1f.6: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[    1.834695] e1000e 0000:00:1f.6 0000:00:1f.6 (uninitialized): registered PHC clock
[    1.913896] e1000e 0000:00:1f.6 eth0: (PCI Express:2.5GT/s:Width x1) c8:5b:76:7e:7a:c6
[    1.913898] e1000e 0000:00:1f.6 eth0: Intel(R) PRO/1000 Network Connection
[    1.913973] e1000e 0000:00:1f.6 eth0: MAC: 12, PHY: 12, PBA No: 1000FF-0FF
[    1.914891] e1000e 0000:00:1f.6 enp0s31f6: renamed from eth0
[   32.561503] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[   66.700267] e1000e: enp0s31f6 NIC Link is Down
[   70.731540] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[   73.693298] e1000e: enp0s31f6 NIC Link is Down
[   77.613970] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[   80.638428] e1000e: enp0s31f6 NIC Link is Down
[   84.989762] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[   87.815556] e1000e: enp0s31f6 NIC Link is Down
[   95.227889] e1000e: enp0s31f6 NIC Link is Up 10 Mbps Full Duplex, Flow Control: None
[   95.227893] e1000e 0000:00:1f.6 enp0s31f6: 10/100 speed: disabling TSO
[   98.596870] e1000e: enp0s31f6 NIC Link is Down
[  106.356123] e1000e: enp0s31f6 NIC Link is Up 10 Mbps Full Duplex, Flow Control: None
[  106.356126] e1000e 0000:00:1f.6 enp0s31f6: 10/100 speed: disabling TSO
[  109.607069] e1000e: enp0s31f6 NIC Link is Down
[  114.362889] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[  118.920055] e1000e: enp0s31f6 NIC Link is Down
[  126.134775] e1000e: enp0s31f6 NIC Link is Up 10 Mbps Full Duplex, Flow Control: None
[  126.134779] e1000e 0000:00:1f.6 enp0s31f6: 10/100 speed: disabling TSO
[  128.998360] e1000e: enp0s31f6 NIC Link is Down
[  133.835132] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[  136.090335] e1000e: enp0s31f6 NIC Link is Down
[  140.576624] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[  143.460126] e1000e: enp0s31f6 NIC Link is Down
[  148.998810] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[  151.685566] e1000e: enp0s31f6 NIC Link is Down
[  157.043840] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[  160.032760] e1000e: enp0s31f6 NIC Link is Down
[  164.642052] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[  167.014840] e1000e: enp0s31f6 NIC Link is Down
[  170.913777] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[  174.440027] e1000e: enp0s31f6 NIC Link is Down
[  178.324896] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[  181.932049] e1000e: enp0s31f6 NIC Link is Down
[  187.479069] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[  190.271259] e1000e: enp0s31f6 NIC Link is Down
[  193.743551] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[  195.502304] e1000e: enp0s31f6 NIC Link is Down
[  200.861489] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[  203.323081] e1000e: enp0s31f6 NIC Link is Down
[  206.649749] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[  259.983925] e1000e: enp0s31f6 NIC Link is Down
[  272.626332] e1000e: enp0s31f6 NIC Link is Up 10 Mbps Full Duplex, Flow Control: None
[  272.626341] e1000e 0000:00:1f.6 enp0s31f6: 10/100 speed: disabling TSO
[  280.980884] e1000e: enp0s31f6 NIC Link is Down
[  293.881833] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[  310.377810] e1000e: enp0s31f6 NIC Link is Down
[  323.145610] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None


I think the BIOS update definitely helped; before it was consistently 10 Mbps when unplugging/plugging after system boot.  Now it mostly works.

I'll see if I can reproduce the results from comment 15 on the 4.15.x kernels, if so I can start bisecting between commits to find the culprit.

Comment 22 Luiz Agostinho 2018-11-05 16:57:24 UTC
Hi guys,

On my notebook I tested all the options and the problem continued, the connection problem was only solved when I performed a hibernation.

My NB: Dell Latitude E5470
Network: Ethernet controller: Intel Corporation Ethernet Connection I219-LM (rev 21)
Kernel: 4.18.16-300.fc29.x86_64
Module e1000e: 3.2.6-k

Comment 23 Dean Jenkins 2018-11-08 10:04:03 UTC
Copied here my 2 comments from (and fixed my typos):
https://bugzilla.redhat.com/show_bug.cgi?id=1038156

Hi, I am observing my eno1 network device using I219-LM rev10 running at 10Mbps instead of the expected 1000Mbps.

My laptop is a new Dell Precision 7530.
Distro: Fedora 28 KDE
Kernel: 4.18.16-200.fc28.x86_64
Module e1000e: 3.2.6-k

When the eno1 network cable is connected to my router before powering-up my laptop then eno1 successfully gets set to 1000Mbps.

However, when the network cable is connected after logging into Fedora then eno1 gets set to 10Mbps.

I can successfully manually adjust eno1 to 1000Mbps by running the command:
sudo ethtool -s eno1 speed 1000 duplex full

I am wondering whether this issue is due to Network Manager (KDE) rather than the e1000e kernel driver ?

Below is a snippet from dmesg showing 10Mbps being unexpectedly selected.

The scenario is:
1. Boot-up laptop with eno1 connected to the router (DHCP used).
2. dmesg shows e1000e kernel driver using 1000Mbps
3. Physically remove the eno1 network cable
4. Wait around 10 seconds for Network Manager to realise the Internet has dropped
5. Reinsert the eno1 network cable
6. dmesg shows e1000e kernel driver using 10Mbps (expected this to be 1000Mbps)
7. use "sudo ethtool -s eno1 speed 1000 duplex full" to set 1000Mps

dmesg | grep e1000e
[    1.897192] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[    1.899561] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    1.901813] e1000e 0000:00:1f.6: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[    1.993509] e1000e 0000:00:1f.6 0000:00:1f.6 (uninitialized): registered PHC clock
[    2.076141] e1000e 0000:00:1f.6 eth0: (PCI Express:2.5GT/s:Width x1) e4:b9:7a:18:7d:5c
[    2.076145] e1000e 0000:00:1f.6 eth0: Intel(R) PRO/1000 Network Connection
[    2.076204] e1000e 0000:00:1f.6 eth0: MAC: 13, PHY: 12, PBA No: FFFFFF-0FF
[    2.076861] e1000e 0000:00:1f.6 eno1: renamed from eth0
[   37.092569] e1000e: eno1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
<removed the eno1 Network cable here>
[ 4815.754914] e1000e: eno1 NIC Link is Down
<reinserted the eno1 Network cable here>
[ 4829.980307] e1000e: eno1 NIC Link is Up 10 Mbps Full Duplex, Flow Control: Rx/Tx
[ 4829.980311] e1000e 0000:00:1f.6 eno1: 10/100 speed: disabling TSO
<ran "sudo ethtool -s eno1 speed 1000 duplex full" to set 1000Mbps"
[ 5208.476975] e1000e: eno1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx

Comment 24 Dean Jenkins 2018-11-08 10:48:14 UTC
Here is a list of the mainline kernel commits for e1000e for the various Linux releases (ignored the stable kernel updates):

git log --oneline v4.14..v4.15 ./drivers/net/ethernet/intel/e1000e

4110e02eb45e e1000e: Fix e1000_check_for_copper_link_ich8lan return value.
c0f4b163a03e e1000e: fix the use of magic numbers for buffer overrun issue
26566eae8051 ethernet/intel: Convert timers to use timer_setup()
377b62736c01 e1000e: Be drop monitor friendly
48072ae1ec7a e1000e: apply burst mode settings only on default
b10effb92e27 e1000e: fix buffer overrun while the I219 is processing DMA transactions
4aea7a5c5e94 e1000e: Avoid receiver overrun interrupt bursts
19110cfbb34d e1000e: Separate signaling for link check/link up
d3509f8bc7b0 e1000e: Fix return value test
65a29da1f5fd e1000e: Fix wrong comment related to link detection
c4c40e51f9c3 e1000e: Fix error path in link detection
4a9c07ed71c2 drivers: net: e1000e: use setup_timer() helper.


git log --oneline v4.15..v4.16 ./drivers/net/ethernet/intel/e1000e

e2710dbf0dc1 e1000e: Fix link check race condition
3016e0a0c912 Revert "e1000e: Separate signaling for link check/link up"
aea3fca005fb e1000e: allocate ring descriptors with dma_zalloc_coherent
4e7dc08e57c9 e1000e: Fix check_for_link return value with autoneg off
116f4a640b31 e1000e: Avoid missed interrupts following ICR read
361a954e6a72 e1000e: Fix queue interrupt re-raising in Other interrupt
1f0ea19722ef Partial revert "e1000e: Avoid receiver overrun interrupt bursts"
745d0bd3af99 e1000e: Remove Other from EIAC
8299b006d743 e1000e: Alert the user that C-states will be disabled by enabling jumbo frames
b701cacdbcfb e1000e: Set HTHRESH when PTHRESH is used
a0ce093180f2 Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net


git log --oneline v4.16..v4.17 ./drivers/net/ethernet/intel/e1000e

ae06c70b1358 intel: add SPDX identifiers to all the Intel drivers
03fe2debbb27 Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net


git log --oneline v4.17..v4.18 ./drivers/net/ethernet/intel/e1000e

6396bb221514 treewide: kzalloc() -> kcalloc()
6da2ec56059c treewide: kmalloc() -> kmalloc_array()
fff200caf6f9 e1000e: Ignore TSYNCRXCTL when getting I219 clock attributes
51dce24bcdbd net: intel: Cleanup the copyright/license headers


git log --oneline v4.18..v4.19 ./drivers/net/ethernet/intel/e1000e
(no new commits)

Comment 25 Kamil Páral 2018-11-08 12:26:04 UTC
I identified the problem to occur between 4.15.0 and 4.15.3 in comment 15, for Thinkpad T480s. Can you verify that it's the same set of kernels that affects you too?

Here are the Koji links:
https://koji.fedoraproject.org/koji/buildinfo?buildID=1021748
https://koji.fedoraproject.org/koji/buildinfo?buildID=1044175

Comment 26 Dean Jenkins 2018-11-08 17:53:26 UTC
I am using Fedora 28 KDE on a Dell Precision 7530 laptop using a I219-LM rev10.
Kernel: 4.18.16-200.fc28.x86_64

I have created a workaround to mitigate against eno1 from using 10Mbps by modifying
/etc/sysconfig/network-scripts/ifcfg-eno1

I changed the following line in ifcfg-eno1 from:
ETHTOOL_OPTS="autoneg on"
to
ETHTOOL_OPTS="speed 1000 duplex full"

I tested the change by removing the eno1 Network cable for roughly 10 seconds and then reconnected the Network cable.

dmesg | grep e1000e

[33274.105579] e1000e: eno1 NIC Link is Down
[33303.610993] e1000e: eno1 NIC Link is Up 10 Mbps Full Duplex, Flow Control: Rx/Tx
[33303.610997] e1000e 0000:00:1f.6 eno1: 10/100 speed: disabling TSO
[33308.883575] e1000e: eno1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx

Interestingly, eno1 comes back up at 10Mbps and then 5 seconds later is now updated to 1000Mbps by ifcfg-eno1 (is performed automatically).

So I am guessing there is some issue with the auto negotiation procedure which causes 10Mbps to be selected. This issue is repeatable for me.

Comment 27 abyss.7 2018-11-21 12:48:22 UTC
I have a very similar problem with a card which is inside Thunderbolt Dock.

Laptop: Dell Latitude E7480
Dock: Dell TB16
OS: Fedora 29
Kernels: both 4.18.18 and 4.19.2

The speed is always set to 10Mbs without respect to when I plug the cable or connect dock station.

Tested the same cable with builtin network card - it's always set to 1000Mbs.

The trick with manual setting the speed with `ethtool` works.

I can collect more logs if needed.

Comment 28 Doron Fediuck 2018-11-28 13:45:10 UTC
Same issue with a new X1 (6 generation) and Ethernet extension-
OS: Fedora 29
Kernel: 4.19.3-300.fc29.x86_64

This is what I'm seeing when plugging (get 10mb), unplugging and plugging again (get 1gb):
[48642.335101] e1000e: enp0s31f6 NIC Link is Up 10 Mbps Full Duplex, Flow Control: Rx/Tx 
[48642.335110] e1000e 0000:00:1f.6 enp0s31f6: 10/100 speed: disabling TSO 
[48642.335537] IPv6: ADDRCONF(NETDEV_CHANGE): enp0s31f6: link becomes ready 
[48645.287642] thunderbolt 0000:07:00.0: stopping RX ring 0 
[48645.287654] thunderbolt 0000:07:00.0: disabling interrupt at register 0x38200 bit 12 (0x1001 -> 0x1) 
[48645.287673] thunderbolt 0000:07:00.0: stopping TX ring 0 
[48645.287682] thunderbolt 0000:07:00.0: disabling interrupt at register 0x38200 bit 0 (0x1 -> 0x0) 
[48645.287692] thunderbolt 0000:07:00.0: control channel stopped 
[48670.813355] IPv6: ADDRCONF(NETDEV_UP): wlp2s0: link is not ready 
[48710.295968] e1000e: enp0s31f6 NIC Link is Down 
[48725.200294] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx

Comment 29 Jeremy Cline 2018-12-03 17:33:07 UTC
We apologize for the inconvenience.  There is a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 29 kernel bugs.
 
Fedora 29 has now been rebased to 4.19.5-300.fc29.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.
 
If you experience different issues, please open a new bug report for those.

Comment 30 Kamil Páral 2018-12-04 09:27:02 UTC
With the host of people affected, I think it's quite clear this is still an issue.

Comment 31 Jeremy Cline 2018-12-04 15:48:22 UTC
I don't see anything obvious in v4.15 to v4.15.3. It'd be good if everyone else here confirmed Kamil's findings from comment #15. I don't have the hardware to reproduce this so someone will need to do a git bisect. A guide is at https://docs.fedoraproject.org/en-US/quick-docs/kernel/troubleshooting/index.html#_bisecting_the_kernel.

Comment 32 Kamil Páral 2019-01-18 12:38:25 UTC
I tried to compile the vanilla 4.15 kernel on my F29 using the instructions (jumping through several hoops in the process), in order to start bisecting, but I can't compile it:

  CC       /home/kparal/devel/linux-stable/tools/objtool/pager.o
pager.c: In function ‘pager_preexec’:
pager.c:36:12: error: passing argument 2 to restrict-qualified parameter aliases with argument 4 [-Werror=restrict]
  select(1, &in, NULL, &in, NULL);
            ^~~        ~~~
cc1: all warnings being treated as errors

I tried to work around it (no familiarity with C build process), but failed.

Comment 33 Jeremy Cline 2019-01-21 10:09:33 UTC
Commit ad343a98e74e ("tools/lib/subcmd/pager.c: do not alias select() params") should fix that warning and you'll need commit b3348dab4e79 ("objtool, perf: Fix GCC 8 -Wrestrict error"). It looks like there are a couple of build issues with F29 toolchains, though, so it's probably easiest to use the F27 toolchain.

Comment 34 Kamil Páral 2019-01-21 11:15:30 UTC
I tried F27 toolchain as well (in a mock), but that exited immediately failing to find stdarg.h (or similar). I had gcc installed and the file was present, I didn't know what to do with that. I also tried F28 toolchain, and that behaved the same as F29. I might try F29 again with the commits you mention. I tried to disable "warnings as errors" gcc behavior, but couldn't figure out how.

Comment 35 Kamil Páral 2019-01-21 12:03:41 UTC
So unfortunately commit ad343a98e74e arrived too late, around the time of 4.15.3 kernel. And I need to bisect 4.15 and 4.15.3. The commit is also not referenced in any 4.15.x branch, just master branch. Commit b3348dab4e79 is not present in repo at all. I'm using the linux-stable git, because I need to use x.y.z tags, and Torvald's git doesn't contain those. So it seems I'm not able to compile and bisect those on F29.

I'll try F27 buildroot once again.

Comment 36 Jeremy Cline 2019-01-21 12:44:47 UTC
Oops, sorry, b3348dab4e79 was the reference to my cherry-pick. The upstream commit is 854e55ad289e. If you build with pre-gcc8 you probably won't need that, though.

You can add a second remote with "git remote add -f torvalds git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git" so you'll have the upstream commits and tags. Then "git bisect start", "git bisect good v4.15", "git bisect bad v4.15.3", and then "git apply fixup.patch" where fixup.patch has those two commits. It should leave them modified as you bisect without making git-bisect upset.

Comment 37 Kamil Páral 2019-01-22 11:33:01 UTC
I managed to bisect the commits using F27 toolchaing. Here's the bad commit:


commit 4e45815fcd38e0a335f9be45336fd95011f6275b (HEAD, refs/bisect/bad)
Author: Tomas Winkler <tomas.winkler@intel.com>
Date:   Tue Jan 2 12:01:41 2018 +0200

    mei: me: allow runtime pm for platform with D0i3
    
    commit cc365dcf0e56271bedf3de95f88922abe248e951 upstream.
    
    >From the pci power documentation:
    "The driver itself should not call pm_runtime_allow(), though. Instead,
    it should let user space or some platform-specific code do that (user space
    can do it via sysfs as stated above)..."
    
    However, the S0ix residency cannot be reached without MEI device getting
    into low power state. Hence, for mei devices that support D0i3, it's better
    to make runtime power management mandatory and not rely on the system
    integration such as udev rules.
    This policy cannot be applied globally as some older platforms
    were found to have broken power management.
    
    Cc: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
    Signed-off-by: Tomas Winkler <tomas.winkler@intel.com>
    Reviewed-by: Alexander Usyskin <alexander.usyskin@intel.com>
    Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

diff --git a/drivers/misc/mei/pci-me.c b/drivers/misc/mei/pci-me.c
index f4f17552c9b8..4a0ccda4d04b 100644
--- a/drivers/misc/mei/pci-me.c
+++ b/drivers/misc/mei/pci-me.c
@@ -238,8 +238,11 @@ static int mei_me_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
         */
        mei_me_set_pm_domain(dev);
 
-       if (mei_pg_is_enabled(dev))
+       if (mei_pg_is_enabled(dev)) {
                pm_runtime_put_noidle(&pdev->dev);
+               if (hw->d0i3_supported)
+                       pm_runtime_allow(&pdev->dev);
+       }
 
        dev_dbg(&pdev->dev, "initialization successful.\n");
 

This commit causes my T480s ethernet to connect as 10Mb/s instead of 1000Mb/s.

Jeremy, can you communicate this to the upstream kernel maintainers?

Additionally, you could also provide koji scratch builds for cc365dcf0 and the previous good commit (76ee8f3d7) to get confirmation from other people with other laptops that this is also the broken commit for them, if you think that would help to confirm this.

Comment 38 Jeremy Cline 2019-01-22 13:23:26 UTC
Kamil, excellent. I've started a v4.20.3 build with that commit reverted - if people can compare this to the standard v4.20.3 build in F29 that would be great: https://koji.fedoraproject.org/koji/taskinfo?taskID=32188526. I'll email upstream once I double-check the build since I happen to have access to a T480s this afternoon.

Comment 39 Kamil Páral 2019-01-22 16:43:03 UTC
I can confirm your new scratch build fixes the problem for me.

Comment 40 Tomas Winkler 2019-01-23 10:50:28 UTC
Can you please provide the fw version? 
cat /sys/class/mei/mei0/fw_ver

Thanks
Tomas

Comment 41 Kamil Páral 2019-01-23 11:24:07 UTC
My Thinkpad T480s:
$ cat /sys/class/mei/mei0/fw_ver
0:11.8.55.3510
0:11.8.55.3510
0:11.8.50.3460

Comment 42 Tomas Winkler 2019-01-23 12:01:27 UTC
Thanks, do you know if your machine is labeled with vPro enabled?
If you are not sure you can compile
linux/samples/mei/mei-amt-version.c 
cd 

It needs a little fix
-me->fd = open("/dev/mei0", O_RDWR);
+me->fd = open("/dev/mei0", O_RDWR);

cd samples/mei/
make 
sudo ./mei-amt-version 

It should provide the vPro version if it is enabled. 
or it will print something like 
Error: IOCTL_MEI_CONNECT_CLIENT receive message. err=-1

Comment 43 Kamil Páral 2019-01-23 15:28:15 UTC
$ sudo ./mei-amt-version 
Intel AMT: ENABLED
Flash:	11.8.55
Netstack:	11.8.55
AMTApps:	11.8.55
AMT:	11.8.55
Sku:	8
VendorID:	8086
Build Number:	3510
Recovery Version:	11.8.55
Recovery Build Num:	3510
Legacy Mode:	False

Comment 44 Tomas Winkler 2019-01-24 08:20:16 UTC
Thanks, for the report, at the first glance look like a HW / FW issue, I forwarding all the info to our network and mei engineering.

Comment 45 Jani Nikula 2019-03-21 15:57:20 UTC
ThinkPad T470p with I219-LM here hitting the same issue (though not on Fedora or Red Hat).

Reloading e1000e module with 'modprobe -r e1000e; modprobe e1000e' seems to workaround the issue:

$ sudo dmesg |grep e1000e
[    1.241912] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[    1.241913] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    1.242908] e1000e 0000:00:1f.6: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[    1.437393] e1000e 0000:00:1f.6 0000:00:1f.6 (uninitialized): registered PHC clock
[    1.524237] e1000e 0000:00:1f.6 eth0: (PCI Express:2.5GT/s:Width x1) 8c:16:45:29:ab:ef
[    1.524240] e1000e 0000:00:1f.6 eth0: Intel(R) PRO/1000 Network Connection
[    1.524322] e1000e 0000:00:1f.6 eth0: MAC: 12, PHY: 12, PBA No: FFFFFF-0FF
[    1.525439] e1000e 0000:00:1f.6 enp0s31f6: renamed from eth0
[  156.243207] e1000e: enp0s31f6 NIC Link is Up 10 Mbps Full Duplex, Flow Control: Rx/Tx
[  156.243209] e1000e 0000:00:1f.6 enp0s31f6: 10/100 speed: disabling TSO
[  357.220811] e1000e 0000:00:1f.6 enp0s31f6: removed PHC
[  357.321953] e1000e: enp0s31f6 NIC Link is Down
[  368.713214] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[  368.713215] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[  368.713434] e1000e 0000:00:1f.6: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[  368.805884] e1000e 0000:00:1f.6 0000:00:1f.6 (uninitialized): registered PHC clock
[  368.892445] e1000e 0000:00:1f.6 eth0: (PCI Express:2.5GT/s:Width x1) 8c:16:45:29:ab:ef
[  368.892448] e1000e 0000:00:1f.6 eth0: Intel(R) PRO/1000 Network Connection
[  368.892567] e1000e 0000:00:1f.6 eth0: MAC: 12, PHY: 12, PBA No: FFFFFF-0FF
[  368.896857] e1000e 0000:00:1f.6 enp0s31f6: renamed from eth0
[  375.316818] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx

$ sudo ./mei-amt-version 
Intel AMT: ENABLED
Flash:	11.8.50
Netstack:	11.8.50
AMTApps:	11.8.50
AMT:	11.8.50
Sku:	16392
VendorID:	8086
Build Number:	3425
Recovery Version:	11.8.50
Recovery Build Num:	3425
Legacy Mode:	False

$ sudo lspci -vvnns 00:1f.6
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (5) I219-LM [8086:15e3] (rev 31)
	Subsystem: Lenovo Ethernet Connection (5) I219-LM [17aa:505d]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 127
	Region 0: Memory at f1300000 (32-bit, non-prefetchable) [size=128K]
	Capabilities: [c8] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee002d8  Data: 0000
	Capabilities: [e0] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: e1000e
	Kernel modules: e1000e

Comment 46 afox@redhat.com 2019-06-11 09:17:04 UTC
I'm also seeing this issue on the Lenovo P50 with I219-LM on Fedora 30. 

# cat /sys/class/mei/mei0/fw_ver
0:11.8.50.3448
0:11.8.50.3448
0:11.8.50.3448

# lspci -vvnns 00:1f.6
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (2) I219-LM [8086:15b7] (rev 31)
	Subsystem: Lenovo Device [17aa:2233]
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 143
	Region 0: Memory at b4900000 (32-bit, non-prefetchable) [size=128K]
	Capabilities: [c8] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee00538  Data: 0000
	Capabilities: [e0] PCI Advanced Features
		AFCap: TP+ FLR+
		AFCtrl: FLR-
		AFStatus: TP-
	Kernel driver in use: e1000e
	Kernel modules: e1000e

Comment 47 Øystein Bedin 2019-06-13 04:59:50 UTC
Just to join the party, I'm seeing the same issue as well - Lenovo T470s, fresh install of Fedora 30. I am using the NIC connector on the docking station. After removing the laptop from the dock and re-docking, the NIC gets negotiated to the correct 1Gbps speed.

# cat /etc/fedora-release 
Fedora release 30 (Thirty)

# uname -r
5.1.8-300.fc30.x86_64

#  dmesg | grep e1000e
[    4.481271] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[    4.481273] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[    4.481529] e1000e 0000:00:1f.6: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[    4.678382] e1000e 0000:00:1f.6 0000:00:1f.6 (uninitialized): registered PHC clock
[    4.756227] e1000e 0000:00:1f.6 eth0: (PCI Express:2.5GT/s:Width x1) 54:e1:ad:6c:26:de
[    4.756229] e1000e 0000:00:1f.6 eth0: Intel(R) PRO/1000 Network Connection
[    4.756309] e1000e 0000:00:1f.6 eth0: MAC: 12, PHY: 12, PBA No: 1000FF-0FF
[    4.757727] e1000e 0000:00:1f.6 enp0s31f6: renamed from eth0
[ 1626.332442] e1000e: enp0s31f6 NIC Link is Up 10 Mbps Full Duplex, Flow Control: None
[ 1626.332446] e1000e 0000:00:1f.6 enp0s31f6: 10/100 speed: disabling TSO
[ 7762.896008] e1000e: enp0s31f6 NIC Link is Down
[ 7773.056655] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None

Comment 48 Matthew Levi 2019-06-26 20:04:55 UTC
One more joining to the club... really tough issue.

Laptop: Dell Inspiron 15 7000 Gaming

It works perfectly on Kernel 4.18.16-300.fc29.x86_64, but anything beyond that makes my 1Gb/s connection to 10Mpb/s... I upgraded to Fedora 30, which uses Kernels +5, and it was a real nightmare in terms of connection. I had to revert it to Fedora 29. It looks like I'll be stuck on this version until this issue gets solved.

Uhmm re-plugging or reseting network from command line didn't help me...

Do we have any estimate from the Fedora team?

Thank you so much for your help and efforts towards this network issue.

Comment 49 Matthew Levi 2019-06-26 20:13:00 UTC
*** Bug 1718586 has been marked as a duplicate of this bug. ***

Comment 50 Raymond Sit 2019-06-27 03:16:23 UTC
I have been tracking this issue since F28 and it is affecting multiple of my platforms, all which use the Intel 219 Chipset. The logs below are from one of the affected machines, a Panasonic CF-20mk2 where it is still occuring under FC30 with kernel 5.1.12-300.

I was previously working around it by doing ip li set dev <dev-name> down; ip li set dev <dev-name> up but with the latest upgrade to 5.1.12-300 this no longer resolves the issue.

I have attached dmesg output which captures the e1000 detection of a hardware hang and the subsequent kernel oops:

[  374.871083] e1000e: enp0s31f6 NIC Link is Up 10 Mbps Full Duplex, Flow Control: Rx/Tx
[  374.871089] e1000e 0000:00:1f.6 enp0s31f6: 10/100 speed: disabling TSO
[  393.115085] e1000e 0000:00:1f.6 enp0s31f6: Detected Hardware Unit Hang:
                 TDH                  <38>
                 TDT                  <3b>
                 next_to_use          <3b>
                 next_to_clean        <38>
               buffer_info[next_to_clean]:
                 time_stamp           <100012540>
                 next_to_watch        <38>
                 jiffies              <100016a40>
                 next_to_watch.status <0>
               MAC Status             <80043>
               PHY Status             <796d>
               PHY 1000BASE-T Status  <0>
               PHY Extended Status    <3000>
               PCI Status             <10>
[  395.099104] e1000e 0000:00:1f.6 enp0s31f6: Detected Hardware Unit Hang:
                 TDH                  <38>
                 TDT                  <3b>
                 next_to_use          <3b>
                 next_to_clean        <38>
               buffer_info[next_to_clean]:
                 time_stamp           <100012540>
                 next_to_watch        <38>
                 jiffies              <100017200>
                 next_to_watch.status <0>
               MAC Status             <80043>
               PHY Status             <796d>
               PHY 1000BASE-T Status  <0>
               PHY Extended Status    <3000>
               PCI Status             <10>
[  397.083075] e1000e 0000:00:1f.6 enp0s31f6: Detected Hardware Unit Hang:
                 TDH                  <38>
                 TDT                  <3b>
                 next_to_use          <3b>
                 next_to_clean        <38>
               buffer_info[next_to_clean]:
                 time_stamp           <100012540>
                 next_to_watch        <38>
                 jiffies              <1000179c0>
                 next_to_watch.status <0>
               MAC Status             <80043>
               PHY Status             <796d>
               PHY 1000BASE-T Status  <0>
               PHY Extended Status    <3000>
               PCI Status             <10>
[  397.146811] ------------[ cut here ]------------
[  397.146812] NETDEV WATCHDOG: enp0s31f6 (e1000e): transmit queue 0 timed out
[  397.146830] WARNING: CPU: 3 PID: 0 at net/sched/sch_generic.c:460 dev_watchdog+0x1e9/0x1f0
[  397.146830] Modules linked in: vfio_pci vfio_virqfd vfio_iommu_type1 vfio vhost_net vhost tap uas usb_storage vfat fat xt_CHECKSUM ipt_MASQUERADE ipt_REJECT nf_reject_ipv4 tun xt_conntrack ip6table_mangle ip6table_nat iptable_mangle iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter veth bridge stp llc snd_soc_skl snd_soc_hdac_hda snd_hda_ext_core snd_soc_skl_ipc snd_soc_sst_ipc snd_soc_sst_dsp snd_soc_acpi_intel_match snd_soc_acpi snd_soc_core snd_hda_codec_hdmi iTCO_wdt mei_wdt iTCO_vendor_support mei_hdcp snd_hda_codec_realtek snd_hda_codec_generic snd_compress intel_rapl ac97_bus ledtrig_audio snd_pcm_dmaengine x86_pkg_temp_thermal intel_powerclamp snd_hda_intel coretemp snd_hda_codec kvm_intel sunrpc kvm snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm irqbypass e1000e intel_cstate snd_timer intel_uncore intel_rapl_perf snd uvcvideo i2c_i801 soundcore r8152 ipu3_cio2 videobuf2_vmalloc
[  397.146867]  v4l2_fwnode joydev cp210x videobuf2_dma_sg mii videobuf2_memops videobuf2_v4l2 videobuf2_common mei_me videodev ecdh_generic idma64 mei rfkill hid_sensor_gyro_3d hid_sensor_incl_3d hid_sensor_als intel_xhci_usb_role_switch hid_sensor_accel_3d intel_lpss_pci hid_sensor_rotation media hid_sensor_magn_3d intel_lpss intel_pch_thermal hid_sensor_trigger hid_sensor_iio_common roles industrialio_triggered_buffer kfifo_buf industrialio processor_thermal_device intel_soc_dts_iosf int3403_thermal int340x_thermal_zone int3406_thermal panasonic_laptop int3400_thermal acpi_pad intel_vbtn acpi_thermal_rel sparse_keymap pcc_cpufreq ip_tables overlay zram dm_verity reed_solomon squashfs zstd_decompress dm_crypt wacom hid_sensor_hub intel_ishtp_hid hid_multitouch i915 crct10dif_pclmul crc32_pclmul i2c_algo_bit crc32c_intel drm_kms_helper ghash_clmulni_intel drm serio_raw intel_ish_ipc intel_ishtp video loop
[  397.146887] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 5.1.12-300.fc30.x86_64 #1
[  397.146888] Hardware name: Panasonic Corporation CF-20-2/CF20-2, BIOS V2.00L12 02/20/2018
[  397.146889] RIP: 0010:dev_watchdog+0x1e9/0x1f0
[  397.146890] Code: 48 63 55 e0 eb 93 4c 89 ef c6 05 32 43 ad 00 01 e8 fc 71 fb ff 44 89 e1 4c 89 ee 48 c7 c7 50 79 18 90 48 89 c2 e8 c6 1d 87 ff <0f> 0b eb bf 0f 1f 00 0f 1f 44 00 00 48 c7 47 08 00 00 00 00 48 c7
[  397.146891] RSP: 0018:ffff9e0645b83e88 EFLAGS: 00010282
[  397.146891] RAX: 0000000000000000 RBX: ffff9e062c9c3a00 RCX: 0000000000000006
[  397.146892] RDX: 0000000000000007 RSI: 0000000000000086 RDI: ffff9e0645b968c0
[  397.146892] RBP: ffff9e062f2fc480 R08: 0000000000000001 R09: 00000000000004b8
[  397.146893] R10: ffffffff909d803c R11: 0000000000000003 R12: 0000000000000000
[  397.146893] R13: ffff9e062f2fc000 R14: 0000000000000000 R15: ffff9e0645b83ed8
[  397.146894] FS:  0000000000000000(0000) GS:ffff9e0645b80000(0000) knlGS:0000000000000000
[  397.146895] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  397.146895] CR2: 00007effd7d3a000 CR3: 000000043a20e003 CR4: 00000000003626e0
[  397.146896] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  397.146896] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  397.146896] Call Trace:
[  397.146898]  <IRQ>
[  397.146901]  ? qdisc_put_unlocked+0x30/0x30
[  397.146904]  call_timer_fn+0x2b/0x130
[  397.146905]  run_timer_softirq+0x3bd/0x440
[  397.146907]  ? timerqueue_add+0x56/0x90
[  397.146908]  ? enqueue_hrtimer+0x36/0x90
[  397.146910]  __do_softirq+0xed/0x30e
[  397.146912]  irq_exit+0xf1/0x100
[  397.146913]  smp_apic_timer_interrupt+0x76/0x140
[  397.146914]  apic_timer_interrupt+0xf/0x20
[  397.146915]  </IRQ>
[  397.146918] RIP: 0010:cpuidle_enter_state+0xc4/0x450
[  397.146919] Code: e8 21 de 94 ff 80 7c 24 0f 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 61 03 00 00 31 ff e8 e3 ad 9a ff fb 66 0f 1f 44 00 00 <45> 85 e4 0f 88 8c 02 00 00 49 63 cc 4c 2b 6c 24 10 48 8d 04 49 48
[  397.146919] RSP: 0018:ffffb0bc01973e88 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[  397.146920] RAX: ffff9e0645ba0f40 RBX: ffffffff902d6f20 RCX: 000000000000001f
[  397.146920] RDX: 0000000000000000 RSI: 000000004f9a1c05 RDI: 0000000000000000
[  397.146921] RBP: ffff9e0645babc00 R08: 0000005c77cb5bc6 R09: 000000007fffffff
[  397.146921] R10: ffff9e0645b9fdc4 R11: ffff9e0645b9fda4 R12: 0000000000000004
[  397.146922] R13: 0000005c77cb5bc6 R14: 0000000000000004 R15: ffff9e0643621fc0
[  397.146924]  ? cpuidle_enter_state+0x9f/0x450
[  397.146926]  do_idle+0x1dd/0x260
[  397.146927]  cpu_startup_entry+0x19/0x20
[  397.146929]  start_secondary+0x17d/0x1d0
[  397.146931]  secondary_startup_64+0xa4/0xb0
[  397.146933] ---[ end trace 87e294d67cdc77e1 ]---
[  397.146939] e1000e 0000:00:1f.6 enp0s31f6: Reset adapter unexpectedly
[  422.686095] e1000e: enp0s31f6 NIC Link is Up 10 Mbps Full Duplex, Flow Control: Rx/Tx
[  422.686099] e1000e 0000:00:1f.6 enp0s31f6: 10/100 speed: disabling TSO
[  439.131186] e1000e 0000:00:1f.6 enp0s31f6: Detected Hardware Unit Hang:
                 TDH                  <1>
                 TDT                  <2>
                 next_to_use          <2>
                 next_to_clean        <1>
               buffer_info[next_to_clean]:
                 time_stamp           <10001dec0>
                 next_to_watch        <1>
                 jiffies              <100021e00>
                 next_to_watch.status <0>
               MAC Status             <80043>
               PHY Status             <796d>
               PHY 1000BASE-T Status  <0>
               PHY Extended Status    <3000>
               PCI Status             <10>
[  440.154894] e1000e 0000:00:1f.6 enp0s31f6: Reset adapter unexpectedly

ethtool -i is as per below:

driver: e1000e
version: 3.2.6-k
firmware version: 0.1-4
expansion-rom version: 
bus-info: 0000:00:1f.6
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

Let me know if there is any other information you might need to resolve it - thanks again.

Comment 51 Raymond Sit 2019-08-06 04:55:57 UTC
Hello, just wondering if there are any plans to fix this for Fedora 31? It is still an issue right now for us in Fedora 30.

Or will Intel be doing anything about it in Firmware?

Comment 52 Tomas Winkler 2019-08-06 09:08:52 UTC
This looks duplicated it was resolved, here https://bugzilla.redhat.com/show_bug.cgi?id=1689436

Comment 53 Matt McHenry 2019-08-20 14:25:05 UTC
bug 1689436 is not public.  can you provide some high-level detail (e.g. commit that fixed it, release it'll first appear in, etc.)?  Thanks!

Comment 54 Jeremy Cline 2019-08-20 15:13:06 UTC
The upstream commit is def4ec6dce39 ("e1000e: PCIm function state support"), it looks to be in v5.3-rc1 and above. You can test it out by grabbing the Rawhide kernels, or wait for stable Fedora releases to be rebased (I'd guess in around 6-8 weeks?).

Comment 55 Kamil Páral 2019-08-21 12:00:56 UTC
I have used Fedora-Workstation-Live-x86_64-31-20190820.n.4.iso image containing kernel-5.3.0-0.rc5.git0.1.fc31 on Thinkpad T480s, and everything seems to work well now, download and upload speeds are around 1Gb/s. It seems that the kernel fix worked.

Comment 56 Marko Karg 2019-10-23 09:54:22 UTC
Unfortunately that's not true for me:

[ 1767.426961] e1000e: enp0s31f6 NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
[ 1767.426973] e1000e 0000:00:1f.6 enp0s31f6: 10/100 speed: disabling TSO
[ 2027.519998] e1000e: enp0s31f6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[ 2086.912216] e1000e: enp0s31f6 NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
[ 2086.912224] e1000e 0000:00:1f.6 enp0s31f6: 10/100 speed: disabling TSO
[ 2215.931643] e1000e: enp0s31f6 NIC Link is Down
[ 2220.031573] e1000e: enp0s31f6 NIC Link is Up 100 Mbps Full Duplex, Flow Control: None

with the latest f30 kernel:

x1:~$ uname -r
5.3.6-200.fc30.x86_64

This is from a  lenovo x1 gen6. Happy to provide more info if needed. just let me know

Comment 57 Vitaly Lifshits 2019-10-23 10:00:50 UTC
What is the flow you ran?
Can you add lspci output?
Does blacklisting mei module solves it?

Comment 58 Marko Karg 2019-10-23 10:58:09 UTC
Not sure what you mean by flow, but I've attached the lspci output. 
Will blacklist the mei module now and see if that helps.

Comment 59 Marko Karg 2019-10-23 10:58:45 UTC
Created attachment 1628392 [details]
lspci and lspci -vvv output

Comment 60 Tomas Winkler 2019-10-23 11:10:32 UTC
The commit below should resolve the wrong interaction with mei, so I guess this should be resolved.
You don't need to disable the mei driver just disable runtime_pm to validate it.

commit def4ec6dce393e2136b62a05712f35a7fa5f5e56
Author: Vitaly Lifshits <vitaly.lifshits@intel.com>
Date:   Tue Jun 25 17:39:11 2019 +0300

    e1000e: PCIm function state support

There are other fixes to wrong speed detection, check out this one:

commit dee23594d587386e9fda76732aa5f5a487709510
Author: Kai-Heng Feng <kai.heng.feng@canonical.com>
Date:   Mon Jul 15 20:25:55 2019 +0800

    e1000e: Make speed detection on hotplugging cable more reliable

Comment 61 Marko Karg 2019-10-23 11:14:39 UTC
Yep, disabling mei had no effect at all, still the same problem:

[  356.169156] e1000e: enp0s31f6 NIC Link is Up 100 Mbps Full Duplex, Flow Control: None
[  356.169161] e1000e 0000:00:1f.6 enp0s31f6: 10/100 speed: disabling TSO

Comment 62 Vitaly Lifshits 2019-10-23 11:31:09 UTC
Please try applying the following patches:
https://patchwork.ozlabs.org/patch/1175300/
https://patchwork.ozlabs.org/patch/1175302/

Comment 63 Ben Cotton 2019-10-31 18:44:52 UTC
This message is a reminder that Fedora 29 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 29 on 2019-11-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '29'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 29 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 64 Justin M. Forbes 2020-03-03 16:35:50 UTC
*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There are a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 30 kernel bugs.

Fedora 30 has now been rebased to 5.5.7-100.fc30.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you have moved on to Fedora 31, and are still experiencing this issue, please change the version to Fedora 31.

If you experience different issues, please open a new bug report for those.

Comment 65 Ben Cotton 2020-04-30 20:23:32 UTC
This message is a reminder that Fedora 30 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 30 on 2020-05-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '30'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 30 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 66 Ben Cotton 2020-05-26 18:23:36 UTC
Fedora 30 changed to end-of-life (EOL) status on 2020-05-26. Fedora 30 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.