Bug 1433891 - Problems with GUI/tui reconifiguration of NIC having static config from initramfs if there is another dhcp-configured NIC present. [NEEDINFO]
Summary: Problems with GUI/tui reconifiguration of NIC having static config from initr...
Keywords:
Status: NEW
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: anaconda
Version: 7.3
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Anaconda Maintenance Team
QA Contact: Release Test Team
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-03-20 10:12 UTC by Radek Vykydal
Modified: 2019-03-02 00:55 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:
rvykydal: needinfo? (lnykryn)


Attachments (Terms of Use)
Logs from RHEL-7.4-20170315.n.1 - anaconda.log syslog and ifcfg.log merged into one file (160.39 KB, text/plain)
2017-03-20 10:12 UTC, Radek Vykydal
no flags Details
Logs from RHEL-7.4-20170315.n.1 - anaconda.log syslog and ifcfg.log merged into one file (145.39 KB, text/plain)
2017-03-20 10:15 UTC, Radek Vykydal
no flags Details
merged logs for reproducer run on 7.2 (107.39 KB, text/plain)
2017-03-20 10:19 UTC, Radek Vykydal
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1439220 None None None Never
Red Hat Bugzilla 1443587 None None None Never
Red Hat Bugzilla 1444887 None None None Never

Internal Links: 1439220 1443587 1444887

Description Radek Vykydal 2017-03-20 10:12:55 UTC
Created attachment 1264731 [details]
Logs from RHEL-7.4-20170315.n.1 - anaconda.log syslog and ifcfg.log merged into one file

Description of problem:

If there are two devices configured in initramfs via installer boot options with static and dhcp configuration, say
ip=10.34.102.233::10.34.102.254:255.255.255.0::ens3:none ip=ens9:dhcp, NM fails to match the static configuration with ifcfg file created for ens3 in initramfs

from the attached /tmp/syslog:

09:03:20,977 DEBUG NetworkManager:<debug> [1490000600.9329] Connection 'ens3' differs from candidate 'ens3' in ipv4.gateway
09:03:20,979 DEBUG NetworkManager:<debug> [1490000600.9329] manager: (ens3): generated connection 'ens3'

and therefore generates another in-memory connection for ens3. Anaconda is assuming there is only one connection for given ethernet device. When editing ens3 configuration in GUI or tui, it is using the first connection it founds for the device. In 7.3 the ordering of the connections was such that we picked the ifcfg file connection. Now in 7.4 the in-memory connection goes first and is used for editing in GUI/TUI. The connection does not have the gateway and editing it leads to another ifcfg-ens3-1 being created (instead of updating ifcfg-ens3).


Version-Release number of selected component (if applicable):

RHEL-7.4-20170315.n.1
RHEL 7.3 has the same NM matching problem but the correct connection is edited in tui/GUI probably due to a different ordering of results in ListConnections().


How reproducible:
Always


Steps to Reproduce: 
1. Start an installation (text or gui) with 2 NICs, both activated in intiramfs, one manual, one dhcp, eg:
ip=10.34.102.233::10.34.102.254:255.255.255.0::ens3:none ip=ens9:dhcp
2. In TUI or GUI try to edit configuration for the statically configured NIC (ens3)


Actual results:

There is no gateway configured in the edited connection as it is in-memory connection for ens3 created by NM. After saving the connection, additional ifcfg-ens3-1 config file is created for ens3.

Expected results:

Connection from ifcfg-ens3 (created in initramfs) is edited in GUI/tui and the file is updated accordingly.


Additional info:

Note that the connection matching with the same static configuration works if only the statically configured iface is activated/configured in initramfs, ie when removing ip=ens9:dhcp from installer boot options in step 1.

We might try harder not to assume there is only one connection for each device in installer, but I'd like to hear from NM people if the matching failiure is something expected and if it could be fixed in NM.
The same happens also in 7.3, in 7.2 inv4.addresses differs in addition to ipv4.gateway.

I am attaching merged anaconda.log, NM log and ifcfg.log containing content of ifcfg files and dump of settings of existing connections. (Just a note, for the 7.4 nightly I was using a patch for adding the info into ifcfg.log after configuring a device).

Comment 2 Radek Vykydal 2017-03-20 10:15:45 UTC
Created attachment 1264732 [details]
Logs from RHEL-7.4-20170315.n.1 - anaconda.log syslog and ifcfg.log merged into one file

These are the logs for working case - only one NIC activated (with the same static configuration) -> NM is able to match the connection:

09:06:32,609 INFO NetworkManager:<info>  [1490000792.5467] manager: (ens3): found matching connection 'ens3'

Comment 3 Radek Vykydal 2017-03-20 10:19:17 UTC
Created attachment 1264733 [details]
merged logs for reproducer run on 7.2

When reproducing on 7.2 NM fails to match not only ipv4.gateway but also ipv4.addresses:

09:24:26,347      INFO NetworkManager: <debug> [1490001866.347365] [NetworkManagerUtils.c:2164] nm_utils_match_connection(): Connection 'ens3' differs from candidate 'ens3' in ipv4.gateway, ipv4.addresses
09:24:26,349      INFO NetworkManager: <debug> [1490001866.349796] [nm-manager.c:1605] get_existing_connection(): (ens3): generated connection 'ens3'

Comment 4 Radek Vykydal 2017-03-20 10:21:39 UTC
(In reply to Radek Vykydal from comment #0)

> We might try harder not to assume there is only one connection for each
> device in installer, but I'd like to hear from NM people if the matching
> failiure is something expected and if it could be fixed in NM.
> The same happens also in 7.3, in 7.2 inv4.addresses differs in addition to
> ipv4.gateway.

Lubomir, please see the Description, do you think it is something that can be fixed in NM?

Comment 6 Radek Vykydal 2017-04-20 08:29:15 UTC
Thomas, do you think the failing matching in this case is something that could be fixed in NM?

Comment 8 Thomas Haller 2017-04-24 17:34:05 UTC
when NM starts, it generates an in-memory connection based on the device's configuration.

The generated candidate has no IPv4 gateway set. That is because the gateway is detected based on the presence of a default route, but ens3 has no default route.

Not sure what to do about that. Maybe the heuristic about whether a connection matches could be relaxed...



It seems to me the approach of configuring the device externally and expecting NM to assume a connection is flawed.

Comment 9 Thomas Haller 2017-04-24 17:44:00 UTC
not sure this is something that should be fixed in NM (or how to fix it)...

I would argue, anaconda should activate the on-disk connection after NM picks up the external configuration. But currently, that would mean to bring down the device first, which is destructive. NM should have a way to gracefully activate a connection (without destroying what is configured already)... but that is a missing feature as well.

Comment 10 Thomas Haller 2017-04-26 08:54:39 UTC
Both devices should have a default-route. One is defined statically via the gateway, and the other is obtained via DHCP.

It seems that the component that sets up networking before NetworkManager is unable/unwilling to configure a default-route on both devices. It could do so either by using different metrics, or by using `ip route append`.

Then, when NetworkManager starts, it sees that the ifcfg file with static configuration should have a gateway, but the device hasn't a default route. Hence, it determines that the ifcfg is not a right match.


It's not clear that NM does anything wrong.


Note that if you add two default routes with the same metric -- ip route append, you run into another bug (missing feature) of NetworkManager where NM doesn't properly handle such overlapping routes (rh 1337855). NM handles always uses different metrics for same routes.

Comment 11 Radek Vykydal 2017-04-26 11:43:10 UTC
Thanks for looking into it, I'll look what we can do with dracut/initramfs.

Comment 12 Radek Vykydal 2017-04-27 12:45:01 UTC
(In reply to Thomas Haller from comment #10)
> Both devices should have a default-route. One is defined statically via the
> gateway, and the other is obtained via DHCP.
> 
> It seems that the component that sets up networking before NetworkManager is
> unable/unwilling to configure a default-route on both devices. It could do
> so either by using different metrics, or by using `ip route append`.
> 
> Then, when NetworkManager starts, it sees that the ifcfg file with static
> configuration should have a gateway, but the device hasn't a default route.
> Hence, it determines that the ifcfg is not a right match.

Yes, dracut, when activating a device is replacing the default route instead of adding another one.
Lukas, do you think it would make sense to add the default route (with increasing/deceasing metrics so that the last added route is preferred to be more consistent with previous behavior) similarly as NM seems to be doing?

(BTW adding the default route for the device manually in initramfs shell in pre-pivot indeed fixed the matching problem in my tests.)


Note You need to log in before you can comment on or make changes to this bug.