Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Created attachment 1151252[details]
NetworkManager logs, nm-online strace, systemd-analyze plots
Description of problem:
NetworkManager.service never reaches its 'startup complete' state and
thus NetworkManager-wait-online.service (nm-online -s -timeout=30)
always times out and fails IFF I set MTU=9000 on ens1f0 (ixgbe driver).
Version-Release number of selected component (if applicable):
(Full disclosure: I see this on a CentOS 7.2 system. So feel free to ignore
this bug report.)
NetworkManager-1.0.6-29.el7_2.x86_64
kernel-3.10.0-327.13.1.el7.x86_64
initscripts-9.49.30-1.el7_2.2.x86_64
How reproducible:
Always, if I use "MTU=9000" in /etc/sysconfig/network-scripts/ifcfg-ens1f0.
It works fine if I comment his line (=> default MTU=1500).
Steps to Reproduce:
1. Set "MTU=9000" in /etc/sysconfig/network-scripts/ifcfg-ens1f0
2. Reboot
3. "systemctl --failed" will show NetworkManager-wait-online.service as a
failed service.
Actual results:
NetworkManager-wait-online.service times out after 30s and fails during
startup.
Reason: "nm-online -s --timeout=30" is not able to connect to NetworkManager
because NM doesn't reach "startup complete" state. ("-s" : --wait-for-startup)
Please notice that despite of this fact all the network devices are
actually configured correctly - including the MTU=9000 on ens1f0!
Expected results:
NetworkManager-wait-online.service (nm-online) finishes successfully after
a reasonable number of seconds (as it does with MTU=1500).
Additional info:
# grep 'complete' NetworkManager_MTU1500_info.txt
Apr 26 16:20:34 smtcfc0157 NetworkManager[1331]: <info> startup complete
# grep 'complete' NetworkManager_MTU9000_info.txt
#
Setting MTU=9000 seems to trigger a DHCPv4 renewal:
$ grep ens1f0 NetworkManager-dispatcher_MTU9000.txt |cut -d: -f4-
------------ Action ID 0x7f352c0031e0 'up' Interface ens1f0 Environment ------------
DEVICE_IP_IFACE=ens1f0
DEVICE_IFACE=ens1f0
CONNECTION_FILENAME=/etc/sysconfig/network-scripts/ifcfg-ens1f0
Dispatching action 'up' for ens1f0
Dispatch 'up' on ens1f0 complete
------------ Action ID 0x7f352c003170 'dhcp4-change' Interface ens1f0 Environment ------------
DEVICE_IP_IFACE=ens1f0
DEVICE_IFACE=ens1f0
CONNECTION_FILENAME=/etc/sysconfig/network-scripts/ifcfg-ens1f0
Dispatching action 'dhcp4-change' for ens1f0
Dispatch 'dhcp4-change' on ens1f0 complete
Compare this to the MTU=1500 case:
$ grep ens1f0 NetworkManager-dispatcher_MTU1500.txt |cut -d: -f4-
------------ Action ID 0x7f344c0031e0 'up' Interface ens1f0 Environment ------------
DEVICE_IP_IFACE=ens1f0
DEVICE_IFACE=ens1f0
CONNECTION_FILENAME=/etc/sysconfig/network-scripts/ifcfg-ens1f0
Dispatching action 'up' for ens1f0
Dispatch 'up' on ens1f0 complete
$ ethtool -i ens1f0
driver: ixgbe
version: 4.0.1-k-rh7.2
firmware-version: 0x80000868
bus-info: 0000:06:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
# nmcli d s
DEVICE TYPE STATE CONNECTION
eno1 ethernet connected team1-slave
eno2 ethernet connected team1-slave
eno3 ethernet connected team1-slave
eno4 ethernet connected team1-slave
ens1f0 ethernet connected private
team1 team connected team1
ens1f1 ethernet unavailable --
ens5f0 ethernet unavailable --
ens5f1 ethernet unavailable --
ib0 infiniband unmanaged --
ib1 infiniband unmanaged --
lo loopback unmanaged --
# nmcli c s
NAME UUID TYPE DEVICE
System ens1f1 3ba7a201-5d77-d373-4bef-c46ac05ad53e 802-3-ethernet --
System ens5f1 d6a47d27-79f0-63fc-251b-e991514a87a6 802-3-ethernet --
team1-slave 24871ea9-4411-efbd-924f-49cd9fbda6e2 802-3-ethernet eno3
team1-slave abf4c85b-57cc-4484-4fa9-b4a71689c359 802-3-ethernet eno1
team1 4293abb7-d898-84ff-dae6-bffba04cbee9 team team1
System ens5f0 c7ca5207-4897-488b-a379-6ba658e133cf 802-3-ethernet --
private 0720bdf0-87bd-7885-f805-bbeef9d40ecb 802-3-ethernet ens1f0
team1-slave b186f945-cc80-911d-668c-b51be8596980 802-3-ethernet eno2
team1-slave 8e777a66-a032-83ef-59c9-77e69b94ede4 802-3-ethernet eno4
Increasing the NetworkManager-wait-online.service timeout does not help
as it will still time out. The boot process will just take more time.
Disabling NetworkManager-wait-online.service does not help as it is pulled
in by network.service anyway.
Setting MTU=9000 via DHCP doesn't help either.
I've attached a some log files (info and debug) from NetworkManager
and NetworkManager-dispatcher, a strace from NetworkManager-wait-online's
nm-online and a systemd-analyze plot.
Comment 2Beniamino Galvani
2016-05-02 15:31:33 UTC
Created attachment 1152996[details]
[PATCH] device: remove pending dhcp actions also in IP_DONE state
Untested fix.
Comment 3Francesco Giudici
2016-05-10 13:31:14 UTC
LGTM
Comment 4Beniamino Galvani
2016-05-10 16:14:01 UTC
[root ~]# systemctl --failed --all
UNIT LOAD ACTIVE SUB DESCRIPTION
● NetworkManager-wait-online.service loaded failed failed Network Manager Wait Online
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
1 loaded units listed.
To show all installed unit files use 'systemctl list-unit-files'.
in 1.0.6 and no fail in 1.4.0-11
was verified with:
01:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://rhn.redhat.com/errata/RHSA-2016-2581.html
Created attachment 1151252 [details] NetworkManager logs, nm-online strace, systemd-analyze plots Description of problem: NetworkManager.service never reaches its 'startup complete' state and thus NetworkManager-wait-online.service (nm-online -s -timeout=30) always times out and fails IFF I set MTU=9000 on ens1f0 (ixgbe driver). Version-Release number of selected component (if applicable): (Full disclosure: I see this on a CentOS 7.2 system. So feel free to ignore this bug report.) NetworkManager-1.0.6-29.el7_2.x86_64 kernel-3.10.0-327.13.1.el7.x86_64 initscripts-9.49.30-1.el7_2.2.x86_64 How reproducible: Always, if I use "MTU=9000" in /etc/sysconfig/network-scripts/ifcfg-ens1f0. It works fine if I comment his line (=> default MTU=1500). Steps to Reproduce: 1. Set "MTU=9000" in /etc/sysconfig/network-scripts/ifcfg-ens1f0 2. Reboot 3. "systemctl --failed" will show NetworkManager-wait-online.service as a failed service. Actual results: NetworkManager-wait-online.service times out after 30s and fails during startup. Reason: "nm-online -s --timeout=30" is not able to connect to NetworkManager because NM doesn't reach "startup complete" state. ("-s" : --wait-for-startup) Please notice that despite of this fact all the network devices are actually configured correctly - including the MTU=9000 on ens1f0! Expected results: NetworkManager-wait-online.service (nm-online) finishes successfully after a reasonable number of seconds (as it does with MTU=1500). Additional info: # grep 'complete' NetworkManager_MTU1500_info.txt Apr 26 16:20:34 smtcfc0157 NetworkManager[1331]: <info> startup complete # grep 'complete' NetworkManager_MTU9000_info.txt # Setting MTU=9000 seems to trigger a DHCPv4 renewal: $ grep ens1f0 NetworkManager-dispatcher_MTU9000.txt |cut -d: -f4- ------------ Action ID 0x7f352c0031e0 'up' Interface ens1f0 Environment ------------ DEVICE_IP_IFACE=ens1f0 DEVICE_IFACE=ens1f0 CONNECTION_FILENAME=/etc/sysconfig/network-scripts/ifcfg-ens1f0 Dispatching action 'up' for ens1f0 Dispatch 'up' on ens1f0 complete ------------ Action ID 0x7f352c003170 'dhcp4-change' Interface ens1f0 Environment ------------ DEVICE_IP_IFACE=ens1f0 DEVICE_IFACE=ens1f0 CONNECTION_FILENAME=/etc/sysconfig/network-scripts/ifcfg-ens1f0 Dispatching action 'dhcp4-change' for ens1f0 Dispatch 'dhcp4-change' on ens1f0 complete Compare this to the MTU=1500 case: $ grep ens1f0 NetworkManager-dispatcher_MTU1500.txt |cut -d: -f4- ------------ Action ID 0x7f344c0031e0 'up' Interface ens1f0 Environment ------------ DEVICE_IP_IFACE=ens1f0 DEVICE_IFACE=ens1f0 CONNECTION_FILENAME=/etc/sysconfig/network-scripts/ifcfg-ens1f0 Dispatching action 'up' for ens1f0 Dispatch 'up' on ens1f0 complete $ ethtool -i ens1f0 driver: ixgbe version: 4.0.1-k-rh7.2 firmware-version: 0x80000868 bus-info: 0000:06:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: no # nmcli d s DEVICE TYPE STATE CONNECTION eno1 ethernet connected team1-slave eno2 ethernet connected team1-slave eno3 ethernet connected team1-slave eno4 ethernet connected team1-slave ens1f0 ethernet connected private team1 team connected team1 ens1f1 ethernet unavailable -- ens5f0 ethernet unavailable -- ens5f1 ethernet unavailable -- ib0 infiniband unmanaged -- ib1 infiniband unmanaged -- lo loopback unmanaged -- # nmcli c s NAME UUID TYPE DEVICE System ens1f1 3ba7a201-5d77-d373-4bef-c46ac05ad53e 802-3-ethernet -- System ens5f1 d6a47d27-79f0-63fc-251b-e991514a87a6 802-3-ethernet -- team1-slave 24871ea9-4411-efbd-924f-49cd9fbda6e2 802-3-ethernet eno3 team1-slave abf4c85b-57cc-4484-4fa9-b4a71689c359 802-3-ethernet eno1 team1 4293abb7-d898-84ff-dae6-bffba04cbee9 team team1 System ens5f0 c7ca5207-4897-488b-a379-6ba658e133cf 802-3-ethernet -- private 0720bdf0-87bd-7885-f805-bbeef9d40ecb 802-3-ethernet ens1f0 team1-slave b186f945-cc80-911d-668c-b51be8596980 802-3-ethernet eno2 team1-slave 8e777a66-a032-83ef-59c9-77e69b94ede4 802-3-ethernet eno4 Increasing the NetworkManager-wait-online.service timeout does not help as it will still time out. The boot process will just take more time. Disabling NetworkManager-wait-online.service does not help as it is pulled in by network.service anyway. Setting MTU=9000 via DHCP doesn't help either. I've attached a some log files (info and debug) from NetworkManager and NetworkManager-dispatcher, a strace from NetworkManager-wait-online's nm-online and a systemd-analyze plot.