Bug 1714610

+++ This bug was initially created as a clone of Bug #1705054 +++

Description of problem:

After the upgrade from Fedora 29 to Fedora 30 the security-settings on the network-interfaces were missing.


How reproducible:
Every time (already had the issue 6 times).


Steps to Reproduce:
1. configure 802.1x security on any network interface
2. upgrade to Fedora 30

Actual results:
Security settings are off and had to be re-configured.

Expected results:
Security settings still exists.

--- Additional comment from Matthias Summer on 2019-05-01 16:36:14 CEST ---

Short update - the 802.1x Security is disabled after every reboot.

--- Additional comment from Matthias Summer on 2019-05-08 18:43:35 CEST ---

We use 802.1x with TLS as Authentication (CAcerfificate/Private key and password). These settings get lost after every reboot. This also happens to a network-profile.

When useing 802.1x for wifi it is the same. Thill a reboot the connection is available. New wifi-connections without 802.1x are still available after a restart.

--- Additional comment from Beniamino Galvani on 2019-05-09 08:37:40 CEST ---

Can you please paste the content of the /etc/sysconfig/network-scripts/ifcfg-${connection_name} file, with sensitive data redacted?

--- Additional comment from Beniamino Galvani on 2019-05-09 08:47:33 CEST ---

Also, do you see any warnings in the output of 'journalctl -u NetworkManager -b' about invalid connections?

--- Additional comment from Matthias Summer on 2019-05-09 09:07:45 CEST ---

The content of the etc/sysconfig/network-scripts/ifcfg-Wired_connection_1-10 is


HWADDR=xx:xx:xx:xx:xx:xx
TYPE=Ethernet
KEY_MGMT=IEEE8021X
IEEE_8021X_EAP_METHODS=TLS
IEEE_8021X_IDENTITY=HOSTNAME.FQDN
IEEE_8021X_CA_CERT=/home/matthias/Documents/network-certs/root_ca2.pem
IEEE_8021X_PRIVATE_KEY=/home/matthias/Documents/network-certs/host.p12
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME="Wired connection 1"
UUID=5c4d0f64-47d8-37de-ac6c-548e0c8bb037
ONBOOT=yes
AUTOCONNECT_PRIORITY=-999


In the /etc/sysconfig/network-scripts there is a ifcfg-Wired_connection_1-X file for every time connecting to that network:

-rw-r--r--. 1 root root   598 May 17  2018 ifcfg-Wired_connection_1
-rw-r--r--. 1 root root   532 May  1 09:35 ifcfg-Wired_connection_1-1
-rw-r--r--. 1 root root   611 May  9 08:20 ifcfg-Wired_connection_1-10
-rw-r--r--. 1 root root   616 May  1 16:28 ifcfg-Wired_connection_1-2
-rw-r--r--. 1 root root   611 May  1 16:37 ifcfg-Wired_connection_1-3
-rw-r--r--. 1 root root   626 May  1 16:42 ifcfg-Wired_connection_1-4
-rw-r--r--. 1 root root   611 May  2 09:29 ifcfg-Wired_connection_1-5
-rw-r--r--. 1 root root   578 May  3 08:41 ifcfg-Wired_connection_1-6
-rw-r--r--. 1 root root   611 May  3 08:49 ifcfg-Wired_connection_1-7
-rw-r--r--. 1 root root   611 May  7 13:55 ifcfg-Wired_connection_1-8
-rw-r--r--. 1 root root   611 May  8 08:39 ifcfg-Wired_connection_1-9



The warnings from 'journalctl -u NetworkManager -b' are:

May 09 08:18:48 XXXXXXX NetworkManager[1322]: <warn>  [1557382728.3648] ifcfg-rh: loading "/etc/sysconfig/network-scripts/ifcfg-Wired_connection_1-9" fails: Missing certificate for EAP method 'tls'.
May 09 08:18:48 XXXXXXX NetworkManager[1322]: <warn>  [1557382728.3655] ifcfg-rh: loading "/etc/sysconfig/network-scripts/ifcfg-Wired_connection_1-8" fails: Missing certificate for EAP method 'tls'.
May 09 08:18:48 XXXXXXX NetworkManager[1322]: <warn>  [1557382728.3662] ifcfg-rh: loading "/etc/sysconfig/network-scripts/ifcfg-Wired_connection_1-7" fails: Missing certificate for EAP method 'tls'.
May 09 08:18:48 XXXXXXX NetworkManager[1322]: <warn>  [1557382728.3681] ifcfg-rh: loading "/etc/sysconfig/network-scripts/ifcfg-Wired_connection_1-6" fails: Missing certificate for EAP method 'tls'.
May 09 08:18:48 XXXXXXX NetworkManager[1322]: <warn>  [1557382728.3686] ifcfg-rh: loading "/etc/sysconfig/network-scripts/ifcfg-Wired_connection_1-5" fails: Missing certificate for EAP method 'tls'.
May 09 08:18:48 XXXXXXX NetworkManager[1322]: <warn>  [1557382728.3689] ifcfg-rh: loading "/etc/sysconfig/network-scripts/ifcfg-enp0s25" fails: Missing certificate for EAP method 'tls'.
May 09 08:18:48 XXXXXXX NetworkManager[1322]: <warn>  [1557382728.3693] ifcfg-rh: loading "/etc/sysconfig/network-scripts/ifcfg-Wired_connection_1-4" fails: Missing certificate for EAP method 'tls'.
May 09 08:18:48 XXXXXXX NetworkManager[1322]: <warn>  [1557382728.3696] ifcfg-rh: loading "/etc/sysconfig/network-scripts/ifcfg-Wired_connection_1-3" fails: Missing certificate for EAP method 'tls'.
May 09 08:18:48 XXXXXXX NetworkManager[1322]: <warn>  [1557382728.3720] ifcfg-rh: loading "/etc/sysconfig/network-scripts/ifcfg-Wired_connection_1-2" fails: Missing certificate for EAP method 'tls'.
May 09 08:18:48 XXXXXXX NetworkManager[1322]: <warn>  [1557382728.3724] ifcfg-rh: loading "/etc/sysconfig/network-scripts/ifcfg-intern.cube.ch-1" fails: Missing certificate for EAP method 'tls'.
May 09 08:18:48 XXXXXXX NetworkManager[1322]: <warn>  [1557382728.3740] ifcfg-rh: loading "/etc/sysconfig/network-scripts/ifcfg-Wired_connection_1-1" fails: Missing certificate for EAP method 'tls'.
May 09 08:18:48 XXXXXXX NetworkManager[1322]: <warn>  [1557382728.3770] ifcfg-rh: loading "/etc/sysconfig/network-scripts/ifcfg-Wired_connection_1" fails: Missing certificate for EAP method 'tls'.
May 09 08:18:48 XXXXXXX NetworkManager[1322]: <warn>  [1557382728.3794] ifcfg-rh: loading "/etc/sysconfig/network-scripts/ifcfg-intern.cube.ch" fails: Missing certificate for EAP method 'tls'.

--- Additional comment from Beniamino Galvani on 2019-05-09 09:31:31 CEST ---

How are you creating the connection? With nmcli, nm-connection-editor, gnome control-center ?

--- Additional comment from Matthias Summer on 2019-05-09 09:36:55 CEST ---

We use the gnome-control-center.

--- Additional comment from Beniamino Galvani on 2019-05-09 15:21:05 CEST ---

Does the following command create a working connection?

nmcli connection add \
        type ethernet \
        ifname '*' \
        ethernet.mac-address xx:xx:xx:xx:xx:xx \
        802-1x.eap tls \
        802-1x.identity HOSTNAME.FQDN \
        802-1x.ca-cert /home/matthias/Documents/network-certs/root_ca2.pem \
        802-1x.client-cert xxxxxxxxx \
        802-1x.private-key /home/matthias/Documents/network-certs/host.p12

I suspect the issue could be in gnome control-center.

--- Additional comment from Matthias Summer on 2019-05-09 16:07:46 CEST ---

Yes, it creates a working connection. But after a reboot the created connection is not available.

When I executed the command a second time after the reboot a new ifcfg-ethernet-1 file was created under /etc/sysconfig/network-scripts and the connection was working again.

--- Additional comment from Beniamino Galvani on 2019-05-09 16:32:00 CEST ---

I can't reproduce the problem. Please set level=TRACE in the [logging] section of /etc/NetworkManager/NetworkManager.conf, restart NM, reproduce the problem by adding the connection through nmcli, reboot, and then attach the output of 'journalctl -u NetworkManager -b -1; journalctl -u NetworkManager -b'. Thanks.

--- Additional comment from Matthias Summer on 2019-05-10 10:27 CEST ---

You will find the output in the attached file.

--- Additional comment from Beniamino Galvani on 2019-05-10 11:37:08 CEST ---

Ok, I can reproduce the problem using control-center. If I add connection with EAP-TLS authentication and specify a p12 private key file, the client certificate is left empty and the connection fails validation. Perhaps NM should reject that connection. nmcli seems to work instead.

--- Additional comment from  on 2019-05-10 12:16:03 CEST ---

@Beniamino Galvani

I have the very same problem. Can add a profile (with certificates) with NetworkManaager (gnome) or nm-connection-editor and connect, but after reboot profile is lost. 

What do you mean that "nmcli seems to work"? And if this would be a work around, HOW does it work?

--- Additional comment from Beniamino Galvani on 2019-05-10 14:32:36 CEST ---

(In reply to dirk from comment #13)
> @Beniamino Galvani
> 
> I have the very same problem. Can add a profile (with certificates) with
> NetworkManaager (gnome) or nm-connection-editor and connect, but after
> reboot profile is lost. 
> 
> What do you mean that "nmcli seems to work"? And if this would be a work
> around, HOW does it work?

See comment 8. Does the following work for you?

nmcli connection add \
        type ethernet \
        ifname eth0 \
        con-name test+ \
        802-1x.eap tls \
        802-1x.identity id \
        802-1x.ca-cert ca.pem \
        802-1x.private-key client.p12

As a quick test, you can 'systemct restart NetworkManager' and check whether the connection is still there with 'nmcli connection'.

--- Additional comment from Christoph Sievers on 2019-05-12 13:56:50 CEST ---

a previously working ifcfg- and keys- File stops working after updating from f29 (nm 1.14?) to f30 (nm 1.16?).

using ifcfg-rh (default for fedora) - when NetworkManager tries to read a formerly working configuration from ifcfg-blablanetwork:

Mai 11 19:24:40 lblabla NetworkManager[805]: <warn>  [1557595480.0829] ifcfg-rh: loading "/etc/sysconfig/network-scripts/ifcfg-blablanetwork" fails: Missing certificate for EAP method 'tls'.

investigating:

set NetworkManager.conf plugins= (to prevent ifcfg-rh from loading) and using nm-connection-editor I can create a working EAP-TLS configuration that survices a restart.

IMHO this is a ifcfg-rh problem since writing the ifcfg- file works but reading does not. Could this be something to ask thaller?

--- Additional comment from James Hewitt on 2019-05-13 08:55:36 CEST ---

I think I identified the cause here:

https://gitlab.freedesktop.org/NetworkManager/NetworkManager/issues/173 (thaller has seen it)

--- Additional comment from James Hewitt on 2019-05-13 09:01:52 CEST ---

This should only affect people using pkcs12 user certs.

The code detect a p12 file as the private key, and then doesn't save the cert config as the cert is included with the key. As a workaround, we can extract the key and use a separate file, to make it save both settings, then the load works.

openssl pkcs12 -in mywirelesscert.p12 -nocerts -out mywirelesscert-key.pem

Now use the p12 for the user cert in the configuration, and the pem for the private key, and the connection will survive reloads.

--- Additional comment from James Hewitt on 2019-05-13 11:47:14 CEST ---

An easier workaround than splitting just the key out, which can still leave a problem with loading the p12 file as nm isn't great at saving the password for the p12 cert, is to just convert the p12 to pem entirely:

openssl pkcs12 -in mywirelesscert.p12 -out mywirelesscert.pem

And use that instead. The cert in the pem isn't password protected in the same way as in p12 so it works better with nm.

--- Additional comment from Christoph Sievers on 2019-05-13 12:43:49 CEST ---

but also leaves the private key unprotected so...

--- Additional comment from James Hewitt on 2019-05-13 15:47:11 CEST ---

The private key by default will still be protected. For an unprotected version, you would also need -nodes.

--- Additional comment from  on 2019-05-13 17:36:22 CEST ---

I can confirm this bug, I have been having it since when I clean installed fedora 30 (so no possible upgrade-related implications for me), and I have the same log messages about networkmanager in journalctl; I have this bug in both wifi and ethernet networks which use 802.1x authentication with cert file and password-protected private key (university network) both by configuring connections with gnome-control-center and by using eduroam's python configuration script (which basically automatically creates the same configuration in my place).

--- Additional comment from  on 2019-05-15 13:43:23 CEST ---

@Christoph Sievers

For me your described workaround does NOT work.

--- Additional comment from Christoph Sievers on 2019-05-15 13:48:54 CEST ---

@dirk

does your NetworkManager.conf look like this?

[main]
#plugins=ifcfg-rh,ibft
plugins=

also: Things are not going to reappear that way because that stops it from using ifcfg-rh which reads/writes stuff to sysconfig/network-scripts so you would have to create a new connection afert restarting NetworkManager with above config.

--- Additional comment from Thomas Haller on 2019-05-15 21:32:34 CEST ---

Beniamino opened a merge-request, which is currently on review: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/merge_requests/148

@Christoph, as the bug is in ifcfg-rh plugin, disabling the ifcfg-rh plugin (and only use "keyfile" setting plugin) does indeed work around the issue. But this work around is a bit drastic, as you don't use ifcfg-rh files anymore... Independent of this issue that may be a good idea.