Bug 1727904 - Network configuration based on boot option is not passed to installed system. [NEEDINFO]
Summary: Network configuration based on boot option is not passed to installed system.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: NetworkManager
Version: 31
Hardware: s390x
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Lubomir Rintel
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: RejectedBlocker AcceptedFreezeExcepti...
: 1728781 (view as bug list)
Depends On: 1753975
Blocks: ZedoraTracker F31BetaFreezeException
TreeView+ depends on / blocked
 
Reported: 2019-07-08 13:25 UTC by Radek Vykydal
Modified: 2019-09-23 13:46 UTC (History)
25 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-09-21 00:02:32 UTC
Type: Bug
dan: needinfo? (lkundrak)


Attachments (Terms of Use)
journal from the installation (1.93 MB, text/plain)
2019-07-08 13:25 UTC, Radek Vykydal
no flags Details
Log related to comment #7 (193.88 KB, text/plain)
2019-08-02 11:44 UTC, Radek Vykydal
no flags Details
/var/log/anaconda/syslog (348.36 KB, text/plain)
2019-08-28 12:31 UTC, Dan Horák
no flags Details


Links
System ID Private Priority Status Summary Last Updated
IBM Linux Technology Center 181373 0 None None None 2019-09-08 14:59:44 UTC

Description Radek Vykydal 2019-07-08 13:25:07 UTC
Created attachment 1588361 [details]
journal from the installation

Description of problem:

Network configuration from dracut boot options (ip=) is not passed to installed system.

Version-Release number of selected component (if applicable):

Fedora-Rawhide-20190707.n.0
NetworkManager-1.20.0-0.2.fc31

How reproducible:

Always

Steps to Reproduce:

1. Start installation configuring a network device with ip= option, eg static configuration
"ip=10.43.136.233::10.43.136.254:255.255.255.0:myhostname:ens3:none nameserver=10.43.136.2"

2. The configuration is applied in initramfs and during installation.

3. After reboot, default connection using dhcp is applied on ens3 by NM as there is no ifcfg file for ens3.

Actual results:

Installed system does not have ens3 configured, is using default dhcp auto connection for the device.

Expected results:

Installed system is using the static configuration defined by installer boot options for ens3 (from ifcfg file passed to the system by the installer)

Additional info:

It is caused by changes from integration of NetworkManager in initramfs. Ifcfg files for devices configured in initramfs are no more created and passed to installer during switch root.

Comment 1 Radek Vykydal 2019-07-09 09:42:10 UTC
Lubomir, is there a way to 'migrate' the connection created in initramfs (eg 'ens3', whose config file is in "/run/NetworkManager/system-connections/ens3.nmconnection" after the switch root) to be backed by ifcfg file (/etc/sysconfig/network-scripts/ifcfg-ens3) anaconda will use for configuration of the device and pass to the installed system?

We need to get to the initial installer state where the active connection for ens3 is the one which is in respective ifcfg file (ifcfg-ens3) containing the configuration which was applied in initramfs.
Formerly (before the NM integration in dracut) - to achieve this - NM matched the active ens3 connection created by dracut with the ifcfg file created in initramfs (now it tries the keyfile?). If it didn't succeed in matching (which is fragile) and created another connection for the device we even reactivated the device with the ifcfg connection to ensure our initial installer state.

Comment 2 Radek Vykydal 2019-07-11 06:56:53 UTC
*** Bug 1728781 has been marked as a duplicate of this bug. ***

Comment 3 Radek Vykydal 2019-07-30 13:13:24 UTC
(In reply to Radek Vykydal from comment #1)
> Lubomir, is there a way to 'migrate' the connection created in initramfs (eg
> 'ens3', whose config file is in
> "/run/NetworkManager/system-connections/ens3.nmconnection" after the switch
> root) to be backed by ifcfg file (/etc/sysconfig/network-scripts/ifcfg-ens3)
> anaconda will use for configuration of the device and pass to the installed
> system?
> 
> We need to get to the initial installer state where the active connection
> for ens3 is the one which is in respective ifcfg file (ifcfg-ens3)
> containing the configuration which was applied in initramfs.
> Formerly (before the NM integration in dracut) - to achieve this - NM
> matched the active ens3 connection created by dracut with the ifcfg file
> created in initramfs (now it tries the keyfile?). If it didn't succeed in
> matching (which is fragile) and created another connection for the device we
> even reactivated the device with the ifcfg connection to ensure our initial
> installer state.

Lubo, Thomas, any idea?

Comment 4 Thomas Haller 2019-07-30 14:38:28 UTC
> Lubomir, is there a way to 'migrate' the connection created in initramfs 

Theoretically, this would be possible to implement.
In practice, it's cumbersome (really ugly, with many corner cases) and sounds
like a bad idea to do.

It seems better to drop anaconda's reliance on ifcfg file.

I didn't fully understand why that is required. Was anaconda calling `ifup` on that
ifcfg file? Could you explain that again, and are there alternatives? Can you explain
a example scenario where this comes into play?

Comment 5 Radek Vykydal 2019-07-30 15:20:26 UTC
The scenario / use case is network device configuration via installer (ie dracut) boot options, as in a duplicate - bug 1728781, the options: [2]).
We need the ifcfg file because it is the persistent configuration passed to the installed system.

So we need a way to transform the initramfs configuration into ifcfg file that will be copied to the installed system.
Formerly (before NM in initramfs), the ifcfg files were created by dracut, imported to installer root (by import state service), anaconda made sure the connection for the device is the one corresponding to the ifcfg file (as I describe in comment #3, we were using libnm to reactivate the device with the connection) [1], and at the end of installation the ifcfg file is copied to the installed system.

[1] So that the correct connection / ifcfg file is modified in case there will be further device configuration update in anaconda, eg in GUI which runs nm-c-e on the connection, or TUI which modifies the connection, or via kickstart %pre section which applies kickstart by modifying the connection via NM API.

[2]
16:43:23,139 NOTICE kernel:Kernel command line: repo=http://download.eng.bos.redhat.com/fedmsg/dumpdata/secondary/Fedora-Rawhide-20190707.n.0/Server/s390x/os/ ro ramdisk_size=40000 cio_ignore=all,!condev rd.dasd=0.0.3228 rd.dasd=0.0.3428 rd.dasd=0.0.3128 rd.dasd=0.0.3528 rd.dasd=0.0.3028 rd.dasd=0.0.3628 rd.dasd=0.0.3328 rd.dasd=0.0.3728 rd.znet=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portno=0 ip=10.16.105.199::10.16.111.254:21:rtt8.s390.bos.redhat.com:enca00:none nameserver=10.11.5.19 BOOT_IMAGE=0

Comment 6 Thomas Haller 2019-08-01 10:50:54 UTC
> So we need a way to transform the initramfs configuration into ifcfg file that will be copied to the installed system.

Ah, so this should actually work.

I said, there is no direct D-Bus API to move connection profiles from file to file (between settings plugin).

With one exception: moving in-memory connections to disk and back.

Already previously NetworkManager understood in-memory connections. Sometimes it would generate in-memory connection, and there was API like

  nmcli connection add save no ...
  nmcli connection --temporary ...

(yes, this is inconsistent).

See also the underlying D-Bus API Update2 [1] and AddConnection2 [2].

[1] https://developer.gnome.org/NetworkManager/unstable/gdbus-org.freedesktop.NetworkManager.Settings.Connection.html#gdbus-method-org-freedesktop-NetworkManager-Settings-Connection.Update2
[2] https://developer.gnome.org/NetworkManager/unstable/gdbus-org.freedesktop.NetworkManager.Settings.html#gdbus-method-org-freedesktop-NetworkManager-Settings.AddConnection2


Before 1.20, in-memory connections were really just "in-memory". With upcoming NetworkManager 1.20, also in-memory connections are stored to file-system: under /run/NetworkManager/system-connections. There are two benefits of this: (1) in-memory connections survive service restart and (2) you can now create in-memory connections by placing a keyfile only. And /run is also the place where the initrd-generator places the generated profiles.

Check also `nmcli -f UUID,NAME,FILENAME connection show`


So, if you call Update2() with flag NM_SETTINGS_UPDATE2_FLAG_TO_DISK [3], then NetworkManager will store the profile to a new file and remove the file from /run. This works by asking the settings plugins (see "main.plugins" in NetworkManager.conf, which on Fedora/RHEL defaults to "ifcfg-rh,keyfile") to store the file to disk.

[3] https://developer.gnome.org/libnm/unstable/libnm-nm-dbus-interface.html#NMSettingsUpdate2Flags



So, I think all that Anaconda needs it to call Update2 with the TO-DISK flag. If course, that has no negative effect, if the profile happens to be on disk already.

While there is no D-Bus API to explicitly move a connection profile to disk, this automatic moving with TO-DISK and IN-MEMORY flags should suffice here.


Note that you have no control over the filename that NetworkManager chooses for the new ifcfg-file. As always, that name is derived from the connection-id, and in case that file already exists (with unrelated content), then a new filename is picked by appending the UUID (and numbers).



How does that sound?

Comment 7 Radek Vykydal 2019-08-02 11:42:23 UTC
Thank you for the hints, I think we could go in this direction but I am still not able to dump into ifcfg file.

When I first hit the issue a few weeks ago I tried to persist the connection using Save() method:
https://github.com/rvykydal/anaconda/commit/06badb532b3c0d0d626e9b176ec1dccd7b7692c4
which didn't work for me (actually maybe the problem was just that a keyfile was used instead of ifcfg to persist the connection but I am not sure).

Now using Update2
https://github.com/rvykydal/anaconda/commit/802bf2bae40582c9d3e36cf47d1a6a131d9ae7e1
it seems keyfile plugin is used to save the connection into a keyfile instead of ifcfg file:

Aug 02 10:56:57 localhost NetworkManager[1742]: <info>  [1564743417.7945] settings-connection[8be31720d3388662,44609ac0-e9a0-402c-a128-a254ef2ebbdd]: write: successfully updated (keyfile: update /etc/NetworkManager/system-connections/ens3.nmconnection (44609ac0-e9a0-402c-a128-a254ef2ebbdd,"ens3") and rename from "/run/NetworkManager/system-connections/ens3.nmconnection")

I'll attach more logging in following comment.

Also it seems that when I hit this issue for the first time few weeks ago, there was default auto connection created for the device configured in initramfs (Wired connection 3) while now it does not seem to be the case, there is only ens3 connection for ens3 device (which makes things easier for anaconda I think). So back then I tried to persist the connection here (single connection for the device found):

https://github.com/rvykydal/anaconda/commit/06badb532b3c0d0d626e9b176ec1dccd7b7692c4#diff-2205c02576c19630fc4397ea631d23c7R399

while now it happens here (multiple connections for the device found):

https://github.com/rvykydal/anaconda/commit/802bf2bae40582c9d3e36cf47d1a6a131d9ae7e1#diff-2205c02576c19630fc4397ea631d23c7R389

But that is something I should be able to handle, I was just curious if there has been any recent change explaining the difference.

Comment 8 Radek Vykydal 2019-08-02 11:44:10 UTC
Created attachment 1600005 [details]
Log related to comment #7

Comment 9 Thomas Haller 2019-08-02 11:56:48 UTC
hi Radek. You were testing "1.20.0-0.4.fc31.1". This one is an earlier version of pre-1.20.

Please try instead "NetworkManager-1.20.0-0.5.fc31" (or newer).

Sorry for not being clear about that.

NetworkManager-1.20.0-0.5.fc31 is only a week old and exactly the upstream "1.20-rc1" release. The final 1.20.0 release will be very similar to that, and happen in a few days (and also hit Rawhide quickly).

There is a big difference between 1.20.0-0.4 and 1.20.0-0.5.

Comment 10 Thomas Haller 2019-08-02 12:33:20 UTC
> Also it seems that when I hit this issue for the first time few weeks ago, there was default auto connection created for the device configured in initramfs (Wired connection 3)

This might have been a mundane bug [1]. That was already fixed ealier "1:1.20.0-0.4".

[1] https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=c6106672861f9a188469f7e490cc38af60943a10



Also, I said you should use Update2() API with the TO-DISK flag.
The older "Save()" is exactly the same as calling Update2() with NM_SETTINGS_UPDATE2_FLAG_TO_DISK.

"Update2()" is a super-set of the older "Save()", "Update()", and "UpdateUnsaved()" functions (Update2() can fully replace them, depending on the arguments you pass it).

Comment 11 Radek Vykydal 2019-08-02 13:24:43 UTC
(In reply to Thomas Haller from comment #9)

> 
> There is a big difference between 1.20.0-0.4 and 1.20.0-0.5.

Yes, this seems to be working for me (all the APIs that can save the in-memory connection created in initramfs, eg connection.commit_changes(), connection.save() we've been using). When the build hits rawhide I'll run our networking kickstart tests suite on it. Meanwhile I am going to check what else needs to be adapted. We will probably have to do some more adjustments related to ifcfg file not being created in dracut but I believe we can adapt to the change.

Also (unrelated) I've noticed there is NM_SETTINGS_UPDATE2_FLAG_BLOCK_AUTOCONNECT flag thanks to which we could get rid of some hacks around setting 'autoconnect' via ONBOOT in ifcfg file.

Thank you.

Comment 12 Thomas Haller 2019-08-02 13:31:55 UTC
> Also (unrelated) I've noticed there is NM_SETTINGS_UPDATE2_FLAG_BLOCK_AUTOCONNECT flag thanks to which we could get rid of some hacks around setting 'autoconnect' via ONBOOT in ifcfg file.

NM_SETTINGS_UPDATE2_FLAG_BLOCK_AUTOCONNECT exists since NM 1.12.0.

New in 1.20.0 will be NM_SETTINGS_ADD_CONNECTION2_FLAG_BLOCK_AUTOCONNECT, which serves a similar purpose during AddConnection2().

Comment 13 Dan Horák 2019-08-05 08:20:55 UTC
I have a related question - how can one debug the network connection in dracut's initrd with NM configuring the network when tools like "ip" are not present? "cat /proc/net/*" isn't much user friendly :-) Network mis-configuration is one the most common issues when installing Fedora or RHEL on s390x.

Comment 14 Thomas Haller 2019-08-05 11:38:54 UTC
Lubomir, do you about comment 13 ?

Comment 15 Ben Cotton 2019-08-13 16:52:14 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to '31'.

Comment 16 Ben Cotton 2019-08-13 18:33:32 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to 31.

Comment 17 Fedora Blocker Bugs Application 2019-08-26 08:54:25 UTC
Proposed as a Freeze Exception for 31-beta by Fedora user sharkcz using the blocker tracking app because:

 Proposing as freeze exception as it makes a newly installed s390x system unusable, because "network" is the primary installation method and we don't block on non-primary arches. But the bug itself affects all arches when network setup is specified on the kernel parameters line before starting the installation. So it might be considered as a blocker.

Comment 18 Julen Landa Alustiza 2019-08-26 12:48:24 UTC
AFAIK aarch64 server is blocking deliverable and the recommended installation method is net boot, so proposing it as Beta Blocker

Additionaly, we have this beta criteria:
Direct kernel boot
It must be possible to install by booting the installation kernel directly (including via PXE) and correctly specifying a remote source for the installer itself.

Comment 19 Adam Williamson 2019-08-26 16:25:13 UTC
So, in blocker meeting discussion we figured it should be possible to work around this by simply duplicating the static network config in the kickstart (for a kickstart-driven install) or the installer UI (for an interactive install). Can someone confirm this? Thanks!

Comment 20 Geoffrey Marr 2019-08-26 18:07:53 UTC
Discussed during the 2019-08-26 blocker review meeting: [1]

The decision to classify this bug as a "RejectedBlocker" and an "AcceptedFreezeException" was made as while this is a regression from previous releases and will be a problem for some users, we deem that it won't be an issue in most cases as DHCP discovery is most common, and for cases where manual network config is needed, can be worked around by adding kickstart or UI configuration.

[1] https://meetbot.fedoraproject.org/fedora-blocker-review/2019-08-26/f31-blocker-review.2019-08-26-16.00.txt

Comment 21 Radek Vykydal 2019-08-28 10:31:45 UTC
https://github.com/rhinstaller/anaconda/pull/2078/

Comment 22 Dan Horák 2019-08-28 11:38:16 UTC
I tried on Monday with Fedora-31-20190826.n.0 (means with a new enough anaconda) and still didn't get the network up. Based on your comment I checked the system and indeed it already contains the ifcfg file for the configured interface, but unfortunately without the s390x specific options (SUBCHANNELS, OPTIONS, NETTYPE, LAYER2).

Comment 23 Radek Vykydal 2019-08-28 12:11:33 UTC
(In reply to Dan Horák from comment #22)
> I tried on Monday with Fedora-31-20190826.n.0 (means with a new enough
> anaconda) and still didn't get the network up. Based on your comment I
> checked the system and indeed it already contains the ifcfg file for the
> configured interface, but unfortunately without the s390x specific options
> (SUBCHANNELS, OPTIONS, NETTYPE, LAYER2).

The fix on anaconda side is dumping the connection created in initramfs into ifcfg file. The connection itself is created by NM so reassigning for debugging.

Dan, could you please attach /tmp/syslog from the installation environment? (Or /var/log/anaconda/journal.log from installed system).

Comment 24 Dan Horák 2019-08-28 12:31:50 UTC
Created attachment 1609000 [details]
/var/log/anaconda/syslog

Comment 25 Dan Horák 2019-08-28 12:40:30 UTC
Another thing I noticed is that rd.znet= is kept on the kernel parameter line in the installed system.

Comment 26 Jan Stodola 2019-08-28 13:07:51 UTC
The rd.znet parameter is copied to the installed system since ~2014 (https://github.com/rhinstaller/anaconda/commit/7c707f20e846649e58483ac841edbfe9b3ec9caf) as a solution for bug 1087502. This behavior is present in RHEL-7 and RHEL-8.

Comment 27 Dan Horák 2019-08-28 13:25:39 UTC
(In reply to Jan Stodola from comment #26)
> The rd.znet parameter is copied to the installed system since ~2014
> (https://github.com/rhinstaller/anaconda/commit/
> 7c707f20e846649e58483ac841edbfe9b3ec9caf) as a solution for bug 1087502.
> This behavior is present in RHEL-7 and RHEL-8.

Interesting, I haven't noticed :-)

Comment 28 Dan Horák 2019-09-04 14:12:25 UTC
NetworkManager maintainers, please help :-) There is still a missing bit for s390x rendering F-31 unusable.

Comment 29 Adam Williamson 2019-09-12 19:00:07 UTC
Setting this back to ASSIGNED as it seems more work is needed on NetworkManager side. Also adding CommonBugs so we document this.

Comment 30 Fedora Update System 2019-09-17 00:36:55 UTC
FEDORA-2019-2cabc3f936 has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2019-2cabc3f936

Comment 31 Adam Williamson 2019-09-17 01:55:18 UTC
anaconda maintainers, please stop marking anaconda updates as fixing this, since we're waiting on NM...

Comment 32 Fedora Update System 2019-09-18 01:59:56 UTC
anaconda-31.22.4-1.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-2cabc3f936

Comment 33 Radek Vykydal 2019-09-18 13:29:55 UTC
We were trying to remove this BZ from the bodhi update https://bodhi.fedoraproject.org/updates/FEDORA-2019-2cabc3f936 , but:
https://github.com/fedora-infra/bodhi/issues/3508

Comment 34 Dan Horák 2019-09-20 10:43:33 UTC
I think I know what the problem is. The nm-initrd-generator tool doesn't put the s390 specific details from the rd.znet parameter into the in-memory representation of the connection in NM so it can't be dumped later into the ifcfg file. It would be the same with wireless if dracut allowed setting SSID, password and other details on the command line. Now I need to verify the hypothesis.

Comment 35 Fedora Update System 2019-09-21 00:02:32 UTC
anaconda-31.22.4-1.fc31 has been pushed to the Fedora 31 stable repository. If problems still persist, please make note of it in this bug report.

Comment 36 Radek Vykydal 2019-09-23 13:46:07 UTC
Okay, let's track the znet args issue in bug 1753975.


Note You need to log in before you can comment on or make changes to this bug.