Bug 1727904

Lubomir, is there a way to 'migrate' the connection created in initramfs (eg 'ens3', whose config file is in "/run/NetworkManager/system-connections/ens3.nmconnection" after the switch root) to be backed by ifcfg file (/etc/sysconfig/network-scripts/ifcfg-ens3) anaconda will use for configuration of the device and pass to the installed system?

We need to get to the initial installer state where the active connection for ens3 is the one which is in respective ifcfg file (ifcfg-ens3) containing the configuration which was applied in initramfs.
Formerly (before the NM integration in dracut) - to achieve this - NM matched the active ens3 connection created by dracut with the ifcfg file created in initramfs (now it tries the keyfile?). If it didn't succeed in matching (which is fragile) and created another connection for the device we even reactivated the device with the ifcfg connection to ensure our initial installer state.

*** Bug 1728781 has been marked as a duplicate of this bug. ***

(In reply to Radek Vykydal from comment #1)
> Lubomir, is there a way to 'migrate' the connection created in initramfs (eg
> 'ens3', whose config file is in
> "/run/NetworkManager/system-connections/ens3.nmconnection" after the switch
> root) to be backed by ifcfg file (/etc/sysconfig/network-scripts/ifcfg-ens3)
> anaconda will use for configuration of the device and pass to the installed
> system?
> 
> We need to get to the initial installer state where the active connection
> for ens3 is the one which is in respective ifcfg file (ifcfg-ens3)
> containing the configuration which was applied in initramfs.
> Formerly (before the NM integration in dracut) - to achieve this - NM
> matched the active ens3 connection created by dracut with the ifcfg file
> created in initramfs (now it tries the keyfile?). If it didn't succeed in
> matching (which is fragile) and created another connection for the device we
> even reactivated the device with the ifcfg connection to ensure our initial
> installer state.

Lubo, Thomas, any idea?

> Lubomir, is there a way to 'migrate' the connection created in initramfs 

Theoretically, this would be possible to implement.
In practice, it's cumbersome (really ugly, with many corner cases) and sounds
like a bad idea to do.

It seems better to drop anaconda's reliance on ifcfg file.

I didn't fully understand why that is required. Was anaconda calling `ifup` on that
ifcfg file? Could you explain that again, and are there alternatives? Can you explain
a example scenario where this comes into play?

The scenario / use case is network device configuration via installer (ie dracut) boot options, as in a duplicate - bug 1728781, the options: [2]).
We need the ifcfg file because it is the persistent configuration passed to the installed system.

So we need a way to transform the initramfs configuration into ifcfg file that will be copied to the installed system.
Formerly (before NM in initramfs), the ifcfg files were created by dracut, imported to installer root (by import state service), anaconda made sure the connection for the device is the one corresponding to the ifcfg file (as I describe in comment #3, we were using libnm to reactivate the device with the connection) [1], and at the end of installation the ifcfg file is copied to the installed system.

[1] So that the correct connection / ifcfg file is modified in case there will be further device configuration update in anaconda, eg in GUI which runs nm-c-e on the connection, or TUI which modifies the connection, or via kickstart %pre section which applies kickstart by modifying the connection via NM API.

[2]
16:43:23,139 NOTICE kernel:Kernel command line: repo=http://download.eng.bos.redhat.com/fedmsg/dumpdata/secondary/Fedora-Rawhide-20190707.n.0/Server/s390x/os/ ro ramdisk_size=40000 cio_ignore=all,!condev rd.dasd=0.0.3228 rd.dasd=0.0.3428 rd.dasd=0.0.3128 rd.dasd=0.0.3528 rd.dasd=0.0.3028 rd.dasd=0.0.3628 rd.dasd=0.0.3328 rd.dasd=0.0.3728 rd.znet=qeth,0.0.0a00,0.0.0a01,0.0.0a02,layer2=1,portno=0 ip=10.16.105.199::10.16.111.254:21:rtt8.s390.bos.redhat.com:enca00:none nameserver=10.11.5.19 BOOT_IMAGE=0

> So we need a way to transform the initramfs configuration into ifcfg file that will be copied to the installed system.

Ah, so this should actually work.

I said, there is no direct D-Bus API to move connection profiles from file to file (between settings plugin).

With one exception: moving in-memory connections to disk and back.

Already previously NetworkManager understood in-memory connections. Sometimes it would generate in-memory connection, and there was API like

  nmcli connection add save no ...
  nmcli connection --temporary ...

(yes, this is inconsistent).

See also the underlying D-Bus API Update2 [1] and AddConnection2 [2].

[1] https://developer.gnome.org/NetworkManager/unstable/gdbus-org.freedesktop.NetworkManager.Settings.Connection.html#gdbus-method-org-freedesktop-NetworkManager-Settings-Connection.Update2
[2] https://developer.gnome.org/NetworkManager/unstable/gdbus-org.freedesktop.NetworkManager.Settings.html#gdbus-method-org-freedesktop-NetworkManager-Settings.AddConnection2


Before 1.20, in-memory connections were really just "in-memory". With upcoming NetworkManager 1.20, also in-memory connections are stored to file-system: under /run/NetworkManager/system-connections. There are two benefits of this: (1) in-memory connections survive service restart and (2) you can now create in-memory connections by placing a keyfile only. And /run is also the place where the initrd-generator places the generated profiles.

Check also `nmcli -f UUID,NAME,FILENAME connection show`


So, if you call Update2() with flag NM_SETTINGS_UPDATE2_FLAG_TO_DISK [3], then NetworkManager will store the profile to a new file and remove the file from /run. This works by asking the settings plugins (see "main.plugins" in NetworkManager.conf, which on Fedora/RHEL defaults to "ifcfg-rh,keyfile") to store the file to disk.

[3] https://developer.gnome.org/libnm/unstable/libnm-nm-dbus-interface.html#NMSettingsUpdate2Flags



So, I think all that Anaconda needs it to call Update2 with the TO-DISK flag. If course, that has no negative effect, if the profile happens to be on disk already.

While there is no D-Bus API to explicitly move a connection profile to disk, this automatic moving with TO-DISK and IN-MEMORY flags should suffice here.


Note that you have no control over the filename that NetworkManager chooses for the new ifcfg-file. As always, that name is derived from the connection-id, and in case that file already exists (with unrelated content), then a new filename is picked by appending the UUID (and numbers).



How does that sound?

Thank you for the hints, I think we could go in this direction but I am still not able to dump into ifcfg file.

When I first hit the issue a few weeks ago I tried to persist the connection using Save() method:
https://github.com/rvykydal/anaconda/commit/06badb532b3c0d0d626e9b176ec1dccd7b7692c4
which didn't work for me (actually maybe the problem was just that a keyfile was used instead of ifcfg to persist the connection but I am not sure).

Now using Update2
https://github.com/rvykydal/anaconda/commit/802bf2bae40582c9d3e36cf47d1a6a131d9ae7e1
it seems keyfile plugin is used to save the connection into a keyfile instead of ifcfg file:

Aug 02 10:56:57 localhost NetworkManager[1742]: <info>  [1564743417.7945] settings-connection[8be31720d3388662,44609ac0-e9a0-402c-a128-a254ef2ebbdd]: write: successfully updated (keyfile: update /etc/NetworkManager/system-connections/ens3.nmconnection (44609ac0-e9a0-402c-a128-a254ef2ebbdd,"ens3") and rename from "/run/NetworkManager/system-connections/ens3.nmconnection")

I'll attach more logging in following comment.

Also it seems that when I hit this issue for the first time few weeks ago, there was default auto connection created for the device configured in initramfs (Wired connection 3) while now it does not seem to be the case, there is only ens3 connection for ens3 device (which makes things easier for anaconda I think). So back then I tried to persist the connection here (single connection for the device found):

https://github.com/rvykydal/anaconda/commit/06badb532b3c0d0d626e9b176ec1dccd7b7692c4#diff-2205c02576c19630fc4397ea631d23c7R399

while now it happens here (multiple connections for the device found):

https://github.com/rvykydal/anaconda/commit/802bf2bae40582c9d3e36cf47d1a6a131d9ae7e1#diff-2205c02576c19630fc4397ea631d23c7R389

But that is something I should be able to handle, I was just curious if there has been any recent change explaining the difference.

Created attachment 1600005 [details]
Log related to comment #7

hi Radek. You were testing "1.20.0-0.4.fc31.1". This one is an earlier version of pre-1.20.

Please try instead "NetworkManager-1.20.0-0.5.fc31" (or newer).

Sorry for not being clear about that.

NetworkManager-1.20.0-0.5.fc31 is only a week old and exactly the upstream "1.20-rc1" release. The final 1.20.0 release will be very similar to that, and happen in a few days (and also hit Rawhide quickly).

There is a big difference between 1.20.0-0.4 and 1.20.0-0.5.

> Also it seems that when I hit this issue for the first time few weeks ago, there was default auto connection created for the device configured in initramfs (Wired connection 3)

This might have been a mundane bug [1]. That was already fixed ealier "1:1.20.0-0.4".

[1] https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=c6106672861f9a188469f7e490cc38af60943a10



Also, I said you should use Update2() API with the TO-DISK flag.
The older "Save()" is exactly the same as calling Update2() with NM_SETTINGS_UPDATE2_FLAG_TO_DISK.

"Update2()" is a super-set of the older "Save()", "Update()", and "UpdateUnsaved()" functions (Update2() can fully replace them, depending on the arguments you pass it).

(In reply to Thomas Haller from comment #9)

> 
> There is a big difference between 1.20.0-0.4 and 1.20.0-0.5.

Yes, this seems to be working for me (all the APIs that can save the in-memory connection created in initramfs, eg connection.commit_changes(), connection.save() we've been using). When the build hits rawhide I'll run our networking kickstart tests suite on it. Meanwhile I am going to check what else needs to be adapted. We will probably have to do some more adjustments related to ifcfg file not being created in dracut but I believe we can adapt to the change.

Also (unrelated) I've noticed there is NM_SETTINGS_UPDATE2_FLAG_BLOCK_AUTOCONNECT flag thanks to which we could get rid of some hacks around setting 'autoconnect' via ONBOOT in ifcfg file.

Thank you.

> Also (unrelated) I've noticed there is NM_SETTINGS_UPDATE2_FLAG_BLOCK_AUTOCONNECT flag thanks to which we could get rid of some hacks around setting 'autoconnect' via ONBOOT in ifcfg file.

NM_SETTINGS_UPDATE2_FLAG_BLOCK_AUTOCONNECT exists since NM 1.12.0.

New in 1.20.0 will be NM_SETTINGS_ADD_CONNECTION2_FLAG_BLOCK_AUTOCONNECT, which serves a similar purpose during AddConnection2().

I have a related question - how can one debug the network connection in dracut's initrd with NM configuring the network when tools like "ip" are not present? "cat /proc/net/*" isn't much user friendly :-) Network mis-configuration is one the most common issues when installing Fedora or RHEL on s390x.

Lubomir, do you about comment 13 ?

This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to '31'.

This bug appears to have been reported against 'rawhide' during the Fedora 31 development cycle.
Changing version to 31.

Proposed as a Freeze Exception for 31-beta by Fedora user sharkcz using the blocker tracking app because:

 Proposing as freeze exception as it makes a newly installed s390x system unusable, because "network" is the primary installation method and we don't block on non-primary arches. But the bug itself affects all arches when network setup is specified on the kernel parameters line before starting the installation. So it might be considered as a blocker.

AFAIK aarch64 server is blocking deliverable and the recommended installation method is net boot, so proposing it as Beta Blocker

Additionaly, we have this beta criteria:
Direct kernel boot
It must be possible to install by booting the installation kernel directly (including via PXE) and correctly specifying a remote source for the installer itself.

So, in blocker meeting discussion we figured it should be possible to work around this by simply duplicating the static network config in the kickstart (for a kickstart-driven install) or the installer UI (for an interactive install). Can someone confirm this? Thanks!

Discussed during the 2019-08-26 blocker review meeting: [1]

The decision to classify this bug as a "RejectedBlocker" and an "AcceptedFreezeException" was made as while this is a regression from previous releases and will be a problem for some users, we deem that it won't be an issue in most cases as DHCP discovery is most common, and for cases where manual network config is needed, can be worked around by adding kickstart or UI configuration.

[1] https://meetbot.fedoraproject.org/fedora-blocker-review/2019-08-26/f31-blocker-review.2019-08-26-16.00.txt

https://github.com/rhinstaller/anaconda/pull/2078/

I tried on Monday with Fedora-31-20190826.n.0 (means with a new enough anaconda) and still didn't get the network up. Based on your comment I checked the system and indeed it already contains the ifcfg file for the configured interface, but unfortunately without the s390x specific options (SUBCHANNELS, OPTIONS, NETTYPE, LAYER2).

(In reply to Dan Horák from comment #22)
> I tried on Monday with Fedora-31-20190826.n.0 (means with a new enough
> anaconda) and still didn't get the network up. Based on your comment I
> checked the system and indeed it already contains the ifcfg file for the
> configured interface, but unfortunately without the s390x specific options
> (SUBCHANNELS, OPTIONS, NETTYPE, LAYER2).

The fix on anaconda side is dumping the connection created in initramfs into ifcfg file. The connection itself is created by NM so reassigning for debugging.

Dan, could you please attach /tmp/syslog from the installation environment? (Or /var/log/anaconda/journal.log from installed system).

Created attachment 1609000 [details]
/var/log/anaconda/syslog

Another thing I noticed is that rd.znet= is kept on the kernel parameter line in the installed system.

The rd.znet parameter is copied to the installed system since ~2014 (https://github.com/rhinstaller/anaconda/commit/7c707f20e846649e58483ac841edbfe9b3ec9caf) as a solution for bug 1087502. This behavior is present in RHEL-7 and RHEL-8.

(In reply to Jan Stodola from comment #26)
> The rd.znet parameter is copied to the installed system since ~2014
> (https://github.com/rhinstaller/anaconda/commit/
> 7c707f20e846649e58483ac841edbfe9b3ec9caf) as a solution for bug 1087502.
> This behavior is present in RHEL-7 and RHEL-8.

Interesting, I haven't noticed :-)

NetworkManager maintainers, please help :-) There is still a missing bit for s390x rendering F-31 unusable.

Setting this back to ASSIGNED as it seems more work is needed on NetworkManager side. Also adding CommonBugs so we document this.

FEDORA-2019-2cabc3f936 has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2019-2cabc3f936

anaconda maintainers, please stop marking anaconda updates as fixing this, since we're waiting on NM...

anaconda-31.22.4-1.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-2cabc3f936

We were trying to remove this BZ from the bodhi update https://bodhi.fedoraproject.org/updates/FEDORA-2019-2cabc3f936 , but:
https://github.com/fedora-infra/bodhi/issues/3508

I think I know what the problem is. The nm-initrd-generator tool doesn't put the s390 specific details from the rd.znet parameter into the in-memory representation of the connection in NM so it can't be dumped later into the ifcfg file. It would be the same with wireless if dracut allowed setting SSID, password and other details on the command line. Now I need to verify the hypothesis.

anaconda-31.22.4-1.fc31 has been pushed to the Fedora 31 stable repository. If problems still persist, please make note of it in this bug report.

Okay, let's track the znet args issue in bug 1753975.

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days