Bug 2092800

Summary: After upgrade openvpn from 2.5.6-1.fc37 to 2.5.7-1.fc37 version NM cant connect to working network.
Product: [Fedora] Fedora Reporter: Mikhail <mikhail.v.gavrilov>
Component: NetworkManager-openvpnAssignee: Lubomir Rintel <lkundrak>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: bgalvani, choeger, code, dazo, dcbw, dirk, francesco.giudici, hello, huzaifas, klember, lkundrak, luca.giuzzi, mhjacks, opensource, s.i.v.892, steve, tdawson, thaller, thomas.moschny, vbenes
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-06-12 02:05:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Mikhail 2022-06-02 09:34:37 UTC
Description of problem:
After upgrade openvpn from 2.5.6-1.fc37 to 2.5.7-1.fc37 version NM cant connect to working network.


2.5.7-1.fc37 logs:
Jun 02 11:55:02 mikhail-laptop NetworkManager[1004]: <info>  [1654152902.0197] vpn[0x560b7e264680,72caf94c-8746-427b-939f-f8f015ca802b,"tensor_dev_yar"]: starting openvpn
Jun 02 11:55:02 mikhail-laptop NetworkManager[1004]: <info>  [1654152902.0202] audit: op="connection-activate" uuid="72caf94c-8746-427b-939f-f8f015ca802b" name="tensor_dev_yar" pid=1921 uid=1000 result="success"
Jun 02 11:55:02 mikhail-laptop nm-openvpn[36098]: --cipher is not set. Previous OpenVPN version defaulted to BF-CBC as fallback when cipher negotiation failed in this case. If you need this fallback please add '--data-ciphers-fallback BF-CBC' to your configuration and/or add BF-CBC to --data-ciphers.
Jun 02 11:55:02 mikhail-laptop nm-openvpn[36098]: OpenVPN 2.5.6 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Mar 16 2022
Jun 02 11:55:02 mikhail-laptop nm-openvpn[36098]: library versions: OpenSSL 3.0.2 15 Mar 2022, LZO 2.10
Jun 02 11:55:02 mikhail-laptop nm-openvpn[36098]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Jun 02 11:55:02 mikhail-laptop nm-openvpn[36098]: TCP/UDP: Preserving recently used remote address: [AF_INET]91.232.93.189:502
Jun 02 11:55:02 mikhail-laptop nm-openvpn[36098]: UDP link local: (not bound)
Jun 02 11:55:02 mikhail-laptop nm-openvpn[36098]: UDP link remote: [AF_INET]91.232.93.189:502
Jun 02 11:55:02 mikhail-laptop nm-openvpn[36098]: NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
Jun 02 11:55:02 mikhail-laptop nm-openvpn[36098]: WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1541', remote='link-mtu 1549'
Jun 02 11:55:02 mikhail-laptop nm-openvpn[36098]: WARNING: 'auth' is used inconsistently, local='auth SHA1', remote='auth [null-digest]'
Jun 02 11:55:02 mikhail-laptop nm-openvpn[36098]: [server] Peer Connection Initiated with [AF_INET]91.232.93.189:502
Jun 02 11:55:08 mikhail-laptop nm-openvpn[36098]: AUTH: Received control message: AUTH_FAILED,Неверный логин или пароль
Jun 02 11:55:08 mikhail-laptop nm-openvpn[36098]: SIGUSR1[soft,auth-failure] received, process restarting
Jun 02 11:55:21 mikhail-laptop nm-openvpn[36098]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Jun 02 11:55:21 mikhail-laptop nm-openvpn[36098]: TCP/UDP: Preserving recently used remote address: [AF_INET]91.232.93.189:502
Jun 02 11:55:21 mikhail-laptop nm-openvpn[36098]: UDP link local: (not bound)
Jun 02 11:55:21 mikhail-laptop nm-openvpn[36098]: UDP link remote: [AF_INET]91.232.93.189:502
Jun 02 11:55:21 mikhail-laptop nm-openvpn[36098]: WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1541', remote='link-mtu 1549'
Jun 02 11:55:21 mikhail-laptop nm-openvpn[36098]: WARNING: 'auth' is used inconsistently, local='auth SHA1', remote='auth [null-digest]'
Jun 02 11:55:21 mikhail-laptop nm-openvpn[36098]: [server] Peer Connection Initiated with [AF_INET]91.232.93.189:502
Jun 02 11:55:22 mikhail-laptop nm-openvpn[36098]: AUTH: Received control message: AUTH_FAILED,CRV1:R,E:eeb4efc3ec001a9c2af099cd8df32513fe:cmVnaW9uXG12LmdhdnJpbG92:Введите код:
Jun 02 11:55:22 mikhail-laptop nm-openvpn[36098]: SIGUSR1[soft,auth-failure] received, process restarting
Jun 02 11:55:39 mikhail-laptop nm-openvpn[36098]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Jun 02 11:55:39 mikhail-laptop nm-openvpn[36098]: TCP/UDP: Preserving recently used remote address: [AF_INET]91.232.93.189:502
Jun 02 11:55:39 mikhail-laptop nm-openvpn[36098]: UDP link local: (not bound)
Jun 02 11:55:39 mikhail-laptop nm-openvpn[36098]: UDP link remote: [AF_INET]91.232.93.189:502
Jun 02 11:55:39 mikhail-laptop nm-openvpn[36098]: WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1541', remote='link-mtu 1549'
Jun 02 11:55:39 mikhail-laptop nm-openvpn[36098]: WARNING: 'auth' is used inconsistently, local='auth SHA1', remote='auth [null-digest]'
Jun 02 11:55:39 mikhail-laptop nm-openvpn[36098]: [server] Peer Connection Initiated with [AF_INET]91.232.93.189:502
Jun 02 11:55:40 mikhail-laptop nm-openvpn[36098]: TUN/TAP device tun0 opened
Jun 02 11:55:40 mikhail-laptop nm-openvpn[36098]: /usr/libexec/nm-openvpn-service-openvpn-helper --debug 0 36091 --bus-name org.freedesktop.NetworkManager.openvpn.Connection_7 --tun -- tun0 1500 1552 10.206.129.103 255.255.240.0 init
Jun 02 11:55:40 mikhail-laptop NetworkManager[1004]: <info>  [1654152940.4073] manager: (tun0): new Tun device (/org/freedesktop/NetworkManager/Devices/4)
Jun 02 11:55:40 mikhail-laptop nm-openvpn[36098]: GID set to nm-openvpn
Jun 02 11:55:40 mikhail-laptop nm-openvpn[36098]: UID set to nm-openvpn
Jun 02 11:55:40 mikhail-laptop nm-openvpn[36098]: Initialization Sequence Completed
Jun 02 11:55:40 mikhail-laptop audit[979]: NETFILTER_CFG table=firewalld:5 family=1 entries=5 op=nft_register_rule pid=979 subj=system_u:system_r:firewalld_t:s0 comm="firewalld"
Jun 02 11:55:40 mikhail-laptop NetworkManager[1004]: <info>  [1654152940.4233] device (tun0): state change: unmanaged -> unavailable (reason 'connection-assumed', sys-iface-state: 'external')
Jun 02 11:55:40 mikhail-laptop NetworkManager[1004]: <info>  [1654152940.4248] device (tun0): state change: unavailable -> disconnected (reason 'connection-assumed', sys-iface-state: 'external')
Jun 02 11:55:40 mikhail-laptop NetworkManager[1004]: <info>  [1654152940.4254] device (tun0): Activation: starting connection 'tun0' (41a131d7-b3a2-420a-ab43-1a0d364c3032)
Jun 02 11:55:40 mikhail-laptop NetworkManager[1004]: <info>  [1654152940.4270] device (tun0): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'external')
Jun 02 11:55:40 mikhail-laptop NetworkManager[1004]: <info>  [1654152940.4272] device (tun0): state change: prepare -> config (reason 'none', sys-iface-state: 'external')
Jun 02 11:55:40 mikhail-laptop NetworkManager[1004]: <info>  [1654152940.4273] device (tun0): state change: config -> ip-config (reason 'none', sys-iface-state: 'external')
Jun 02 11:55:40 mikhail-laptop NetworkManager[1004]: <info>  [1654152940.4275] device (tun0): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'external')
Jun 02 11:55:40 mikhail-laptop systemd[1]: Starting NetworkManager-dispatcher.service - Network Manager Script Dispatcher Service...
Jun 02 11:55:40 mikhail-laptop systemd[1]: Started NetworkManager-dispatcher.service - Network Manager Script Dispatcher Service.
Jun 02 11:55:40 mikhail-laptop audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager-dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jun 02 11:55:40 mikhail-laptop NetworkManager[1004]: <info>  [1654152940.4352] policy: set 'tensor_dev_yar' (tun0) as default for IPv4 routing and DNS
Jun 02 11:55:40 mikhail-laptop systemd-resolved[866]: wlp5s0: Bus client set default route setting: no
Jun 02 11:55:40 mikhail-laptop NetworkManager[1004]: <info>  [1654152940.4365] device (tun0): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'external')
Jun 02 11:55:40 mikhail-laptop NetworkManager[1004]: <info>  [1654152940.4368] device (tun0): state change: secondaries -> activated (reason 'none', sys-iface-state: 'external')
Jun 02 11:55:40 mikhail-laptop NetworkManager[1004]: <info>  [1654152940.4371] device (tun0): Activation: successful, device activated.
Jun 02 11:55:40 mikhail-laptop systemd-resolved[866]: wlp5s0: Bus client reset DNS server list.
Jun 02 11:55:40 mikhail-laptop systemd-resolved[866]: tun0: Bus client set default route setting: yes
Jun 02 11:55:40 mikhail-laptop systemd-resolved[866]: tun0: Bus client set DNS server list to: 10.76.4.153
Jun 02 11:55:40 mikhail-laptop systemd[1]: iscsi.service: Unit cannot be reloaded because it is inactive.
Jun 02 11:55:40 mikhail-laptop systemd[1]: iscsi.service: Unit cannot be reloaded because it is inactive.
Jun 02 11:55:50 mikhail-laptop systemd[1]: NetworkManager-dispatcher.service: Deactivated successfully.
Jun 02 11:55:50 mikhail-laptop audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager-dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'


2.5.7-1.fc37 logs:
Jun 02 11:49:40 mikhail-laptop NetworkManager[1004]: <info>  [1654152580.5316] vpn[0x560b7e264900,72caf94c-8746-427b-939f-f8f015ca802b,"tensor_dev_yar"]: starting openvpn
Jun 02 11:49:40 mikhail-laptop NetworkManager[1004]: <info>  [1654152580.5319] audit: op="connection-activate" uuid="72caf94c-8746-427b-939f-f8f015ca802b" name="tensor_dev_yar" pid=1921 uid=1000 result="success"
Jun 02 11:49:40 mikhail-laptop nm-openvpn[35170]: --cipher is not set. Previous OpenVPN version defaulted to BF-CBC as fallback when cipher negotiation failed in this case. If you need this fallback please add '--data-ciphers-fallback BF-CBC' to your configuration and/or add BF-CBC to --data-ciphers.
Jun 02 11:49:40 mikhail-laptop nm-openvpn[35170]: OpenVPN 2.5.7 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on May 31 2022
Jun 02 11:49:40 mikhail-laptop nm-openvpn[35170]: library versions: OpenSSL 3.0.2 15 Mar 2022, LZO 2.10
Jun 02 11:49:40 mikhail-laptop nm-openvpn[35170]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Jun 02 11:49:40 mikhail-laptop nm-openvpn[35170]: Cipher BF-CBC not supported
Jun 02 11:49:40 mikhail-laptop nm-openvpn[35170]: Exiting due to fatal error
Jun 02 11:49:40 mikhail-laptop NetworkManager[1004]: <warn>  [1654152580.8438] vpn[0x560b7e264900,72caf94c-8746-427b-939f-f8f015ca802b,"tensor_dev_yar"]: dbus: failure: connect-failed (1)
Jun 02 11:49:40 mikhail-laptop NetworkManager[1004]: <warn>  [1654152580.8441] vpn[0x560b7e264900,72caf94c-8746-427b-939f-f8f015ca802b,"tensor_dev_yar"]: dbus: failure: connect-failed (1)


In the log I see the message "Cipher BF-CBC not supported" before in openvpn 2.5.6-1.fc37 I didn't see this message.
It may not be worth restricting Ciphers for clients software, the clients cannot influence to server admins, which leads to the inability to use Linux Fedora for work.

Comment 1 dirk 2022-06-02 13:57:05 UTC
I can confirm this. Version 2.5.7 does not work for me, but downgrading was again fine.

NOTE: I just realize: I am on F36 (not Rawhide)!!!

Comment 2 Anthony Rabbito 2022-06-02 14:26:52 UTC
I also was able to reproduce the error "Cipher BF-CBC not supported" while using my Yubikey as a PKCS#11 token. As noted in openVPN release notes https://github.com/OpenVPN/openvpn/blob/release/2.5/Changes.rst#new-features I added "providers legacy default" to my configuration and was able to start the vpn. Not sure yet why that's required yet.

Comment 3 dirk 2022-06-02 15:21:24 UTC
@Anthony Rabbito

"I added "providers legacy default" to my configuration"

How did you do it?

Comment 4 David Sommerseth 2022-06-02 19:08:23 UTC
"providers legacy default" are required to re-enable deprecated and legacy cipher algorithms in OpenSSL 3.0.  Prior openvpn-2.5 releases used the older OpenSSL 1.1 API while v2.5.7 have backported a few OpenSSL 3 API compatibility patches.

Now, these issues appears because your configuration files, private keys or certificates are using deprecated or really legacy crypto algorithms.  Many of them (like BF-CBC, SHA1, RSA keys shorter than 2048 bits, to mention a few) are really NOT recommended to use.  OpenSSL 3 by default disables a lot of them.

Do consider the "providers legacy default" approach to be a temporarily workaround until you get your configurations, key files and certificates upgraded to a more reasonable security level.

Comment 5 David Sommerseth 2022-06-02 19:18:08 UTC
(In reply to dirk from comment #3)
> @Anthony Rabbito
> 
> "I added "providers legacy default" to my configuration"
> 
> How did you do it?

I suspect a plain-text configuration file was used instead of NetworkManager-openvpn.  And to be honest, I don't think NetworkManager-openvpn should support the --providers option for the time being.

Yes, there will be some turmoil due to the new requirements in OpenSSL 3.  But solving the real issues (weak setups) is by far better than working around them through some options.

Comment 6 Mikhail 2022-06-02 19:30:22 UTC
> I suspect a plain-text configuration file was used instead of NetworkManager-openvpn.

The ovpn file was provided to me by my employer. Re-importing the ovpn file with the new openvpn package did not help to solve this issue.

Comment 7 Anthony Rabbito 2022-06-02 20:10:39 UTC
> I suspect a plain-text configuration file was used instead of NetworkManager-openvpn.  And to be honest, I don't think >NetworkManager-openvpn should support the --providers option for the time being.


Yes this is a plain-text configuration as I cannot use NetworkManager with a pkcs11 token. See here https://gitlab.gnome.org/GNOME/NetworkManager-openvpn/-/issues/78 for more info

Comment 8 Thomas Moschny 2022-06-03 00:29:09 UTC
This problem is now also present in F36 (this could be considered a violation of the stable releases updates policy).

As nm-openvpn doesn't seem to support the `--providers` option, I used a brute-force workaround:

$ mv /usr/sbin/openvpn{,.orig}
$ echo -e '#!/bin/sh\nexec openvpn.orig --providers legacy default "$@"' > /usr/sbin/openvpn
$ chmod +x /usr/sbin/openvpn
$ /sbin/restorecon -v /usr/sbin/openvpn.orig /usr/sbin/openvpn

Note: This is a hack, and it will also not survive the next RPM update. Use at your own risk!

Comment 9 David Sommerseth 2022-06-03 10:19:13 UTC
Upstream has sent a patch to the mailing list to be considered: 

I've put together a few quick test builds

F36: https://koji.fedoraproject.org/koji/taskinfo?taskID=87830798
Rawhide: https://koji.fedoraproject.org/koji/taskinfo?taskID=87830820

If some of you can do a quick test of those build to see if that helps, I can get a 2.5.7-2 prepared later this weekend.

Comment 10 David Sommerseth 2022-06-03 10:19:54 UTC
Upstream patch: https://patchwork.openvpn.net/patch/2504/

Comment 11 Martin Jackson 2022-06-06 01:13:41 UTC
Thanks for the update!

I have tested the koji builds, and they work well on the client side.

I have two Fedora 36 openvpn clients and am also running the server on Fedora 36; I still had to add "providers legacy default" line to my server config to allow it to start, even with the new Koji builds. (I did not have to modify the client configs with the new Koji builds.)  This might be considered a separate issue from the one described in the bug title, but it seems they might stem from the same cause.

Comment 12 Luca Giuzzi 2022-06-06 15:06:32 UTC
I confirm that the koji builds fix the problem of connecting with an external vpn (not under my control) for me as well.

Comment 13 Vladimir Benes 2022-06-10 11:53:56 UTC
(In reply to David Sommerseth from comment #9)
> Upstream has sent a patch to the mailing list to be considered: 
> 
> I've put together a few quick test builds
> 
> F36: https://koji.fedoraproject.org/koji/taskinfo?taskID=87830798
> Rawhide: https://koji.fedoraproject.org/koji/taskinfo?taskID=87830820
> 
> If some of you can do a quick test of those build to see if that helps, I
> can get a 2.5.7-2 prepared later this weekend.

This works for me too in NetworkManager CI tests where I use a straightforward server config under c9s. 
https://gitlab.freedesktop.org/NetworkManager/NetworkManager-ci/-/blob/main/nmci/ctx.py#L690

Could you please push the new version to EPEL as well? Thank you so much!
Vladimir

Comment 14 Fedora Update System 2022-06-10 20:07:48 UTC
FEDORA-2022-8ca0f56650 has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-8ca0f56650

Comment 15 David Sommerseth 2022-06-10 20:14:16 UTC
The update I've pushed out via Fedora Bodhi now (FEDORA-2022-8ca0f56650) carries an updated upstream patch for better handling deprecated and unavailable ciphers. This patch is now included officially for a coming OpenVPN 2.5.8 release coming later.

In addition this update also removes BF-CBC from the --data-ciphers list in the openvpn-server@.service unit file.

The change to the unit file will result in OpenVPN server running on Fedora 36 and newer will no longer support BF-CBC out-of-the-box.  This will now require an explicit configuration change to re-enable this.  I concluded this was the best approach, since OpenSSL 3.0 by default does no longer support this cipher without re-enabling legacy ciphers.

Please test this update and give the update positive karma on successful testing.  Since the 2.5.7-1 release caused so much issues, I decided to require more +1s to label it as stable.

Comment 16 David Sommerseth 2022-06-10 20:16:43 UTC
(In reply to Vladimir Benes from comment #13)
[...snip...]
>
> Could you please push the new version to EPEL as well? Thank you so much!

I presume you mean EPEL-9.  I can do that once I am certain the updated F36 build headed for updates-testing works as expected.

Comment 17 Fedora Update System 2022-06-11 01:31:40 UTC
FEDORA-2022-8ca0f56650 has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-8ca0f56650`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-8ca0f56650

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 18 Fedora Update System 2022-06-12 02:05:18 UTC
FEDORA-2022-8ca0f56650 has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 19 Fedora Update System 2022-06-14 16:06:23 UTC
FEDORA-EPEL-2022-dd161a4d75 has been submitted as an update to Fedora EPEL 9. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-dd161a4d75

Comment 20 David Sommerseth 2022-06-14 16:10:07 UTC
@vbenes : I have pushed out an EPEL-9 build via Fedora Updates now.  Please test.

Comment 21 Fedora Update System 2022-06-15 02:00:56 UTC
FEDORA-EPEL-2022-dd161a4d75 has been pushed to the Fedora EPEL 9 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-dd161a4d75

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 22 Mikhail 2022-07-04 11:22:32 UTC
(In reply to Fedora Update System from comment #17)
> FEDORA-2022-8ca0f56650 has been pushed to the Fedora 36 testing repository.
> Soon you'll be able to install the update with the following command:
> `sudo dnf upgrade --enablerepo=updates-testing
> --advisory=FEDORA-2022-8ca0f56650`
> You can provide feedback for this update here:
> https://bodhi.fedoraproject.org/updates/FEDORA-2022-8ca0f56650
> 
> See also https://fedoraproject.org/wiki/QA:Updates_Testing for more
> information on how to test updates.

This issue still not fixed in Rawhide.

Comment 23 David Sommerseth 2022-07-04 12:32:25 UTC
(In reply to Mikhail from comment #22)
[...snip...]
> 
> This issue still not fixed in Rawhide.

Sorry about that!  I've kicked off a rawhide build as well.  https://koji.fedoraproject.org/koji/taskinfo?taskID=89074902

Comment 24 Fedora Update System 2022-07-16 00:38:43 UTC
FEDORA-EPEL-2022-dd161a4d75 has been pushed to the Fedora EPEL 9 stable repository.
If problem still persists, please make note of it in this bug report.