Bug 2087181 - OpenVPN Connection stopped working after upgrade to F36
Summary: OpenVPN Connection stopped working after upgrade to F36
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: openvpn
Version: 36
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: David Sommerseth
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 2068425 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-05-17 14:08 UTC by Cajus Pollmeier
Modified: 2022-06-09 12:24 UTC (History)
7 users (show)

Fixed In Version: openvpn-2.5.7-1.fc36 openvpn-2.5.7-1.el9 openvpn-2.5.7-1.fc35
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-06-02 01:33:30 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Cajus Pollmeier 2022-05-17 14:08:24 UTC
Description of problem:

I've just updated my workstation to F36 and noticed that I'm unable to establish any (Open)VPN connections now.

Alghough I'm more or less sure that this is some kind of OpenVPN / OpenSSL issue, here's the side note that I'm using the KDE Plasma Networkmanager stuff to establish the connection.

I've traced that down to the details in "Actual results".


Version-Release number of selected component (if applicable):

openssl-3.0.2-5.fc36.x86_64
openvpn-2.5.6-1.fc36.x86_64
NetworkManager-openvpn-1.8.18-1.fc36.x86_64
plasma-nm-openvpn-5.24.5-1.fc36.x86_64


How reproducible:

Always.


Steps to Reproduce:
1. Create a OpenVPN with p12 "cert / key with passphrase" combo.
2. Try to connect.
3.

Actual results:

Here's what I spotted with journalctl / strace ordered by time.

Log

Mai 17 15:47:05 NetworkManager[1485]: <info>  [1652795225.3362] vpn[0x55efc07b65a0,f8de1bcd-8b05-4153-8ac5-1962e34be1e7,"FOOBAR"]: starting openvpn
Mai 17 15:47:05 NetworkManager[1485]: <info>  [1652795225.3364] audit: op="connection-activate" uuid="f8de1bcd-8b05-4153-8ac5-1962e34be1e7" name="FOOBAR" pid=2215 uid=1011 result="success"

Communication with OpenVPN management

>INFO:OpenVPN Management Interface Version 3 -- type 'help' for more info
>PASSWORD:Need 'Private Key' password
password "Private Key" "foobarbaz"
SUCCESS: 'Private Key' password entered, but not yet verified

Captured ltrace

[pid 58152] PKCS12_parse(0x55b53de75dd0, 0x7ffd1ba5fe50, 0x7ffd1ba5fe10, 0x7ffd1ba5fe18 <unfinished ...>
[pid 58144] <... g_free resumed> )               = 0
[pid 58152] __vsnprintf_chk(0x7ffd1ba5ceb0, 0x2800, 1, 0x2800) = 137
[pid 58152] __vsnprintf_chk(0x7ffd1ba5ceb0, 0x2800, 1, 0x2800) = 137
[pid 58152] __vsnprintf_chk(0x7ffd1ba5ceb0, 0x2800, 1, 0x2800) = 137
[pid 58152] __vsnprintf_chk(0x7ffd1ba5ceb0, 0x2800, 1, 0x2800) = 137
[pid 58152] __vsnprintf_chk(0x7ffd1ba5d2d0, 0x2800, 1, 0x2800) = 137
[pid 58152] <... PKCS12_parse resumed> )         = 0
[pid 58152] ERR_peek_error(7, 0x55b53ddae010, 0x55b53de7ac70, 0x55b066b4720a) = 0x11800071
[pid 58152] __errno_location()                   = 0x7ffad504f698
[pid 58152] malloc(8200)                         = 0x55b53de90920
[pid 58152] malloc(8200)                         = 0x55b53de92930
[pid 58152] __vsnprintf_chk(0x55b53de90928, 8192, 1, 8192) = 44
[pid 58152] __vsnprintf_chk(0x55b53de92938, 8192, 1, -1) = 44
[pid 58152] time(nil)                            = 1652789722
[pid 58152] malloc(8200)                         = 0x55b53de94940
[pid 58152] __vsnprintf_chk(0x55b53de94948, 8192, 1, -1) = 44
[pid 58152] strlen(">PASSWORD:Verification Failed: 'Private Key'") = 44


Continued communication

>PASSWORD:Verification Failed: 'Private Key'


Continued log

Mai 17 15:47:06 SIGUSR1[soft,private-key-password-failure] received, process restarting

So it doesn't like the key. It gets the correct password, and seems to use it on the correct key file. The cert/key combo and the passwords work with "openssl pkcs12 -info -in ~/.cert/cert.p12".


Expected results:

Just works.


Additional info:

If I can provide any more information, please let me know. Thanks!

Comment 1 David Sommerseth 2022-05-18 09:28:42 UTC
Are you able to recreate your OpenVPN configuration and run openvpn on the command line, preferably with --verb 4.  This will give us more valuable input.

There is an upstream project discussion going on now on adding OpenSSL 3 API support into a v2.5.7 release (backported from the coming v2.6 release), but we need to be sure we're fixing the real problem first.

Comment 2 David Sommerseth 2022-05-18 10:21:20 UTC
A fresh build of the latest release/2.5 (to become v2.5.7) development branch can be found here: https://koji.fedoraproject.org/koji/taskinfo?taskID=87198883

Please test this and see if this resolves your issue, it contains several backports for improving the OpenSSL 3.0 support.

Comment 3 Cajus Pollmeier 2022-05-18 18:52:10 UTC
Hi David,

thanks for the quick response. I'm just back from family stuff and tried the latest and greatest build at first. It is a bit more verbose and says:

Mai 18 20:46:53 nm-openvpn[5689]: OpenSSL: error:11800071:PKCS12 routines::mac verify failure
Mai 18 20:46:53 nm-openvpn[5689]: OpenSSL: error:0308010C:digital envelope routines::unsupported
Mai 18 20:46:53 nm-openvpn[5689]: Decoding PKCS12 failed. Probably wrong password or unsupported/legacy encryption

Mai 18 20:40:10 nm-openvpn[5071]: SIGUSR1[soft,private-key-password-failure] received, process restarting

So it doesn't seem to work. Just requested a new certificate, and for the new one the "error:0308010C:digital envelope routines::unsupported" is gone. But it fails, too:

Mai 18 20:48:17 nm-openvpn[5689]: OpenSSL: error:11800071:PKCS12 routines::mac verify failure
Mai 18 20:48:17 nm-openvpn[5689]: OpenSSL: error:11800071:PKCS12 routines::mac verify failure
Mai 18 20:48:17 nm-openvpn[5689]: Decoding PKCS12 failed. Probably wrong password or unsupported/legacy encryption

I didn't try running the config directly via OpenVPN yet. Will try that tomorrow morning.

What I tried in the meantime is a certificate without the password, which successfully creates a VPN connection.

Comment 4 Cajus Pollmeier 2022-05-19 06:19:04 UTC
Ok. Just found the time to dig a bit deeper. As I had a working "all in one without password test" config from the server guys, I tried to get the ordinary config working again:

The original certificates/keys/ca are rolled out as single p12 file by ejbca. As the test above uses inline PEM, I tried to convert it to single PEM snippets:

8<---
$ openssl pkcs12 -in cajus.p12 -out cajus.crt.pem -clcerts -nokeys
Enter Import Password:
Error outputting keys and certificates
40DCE05F6D7F0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:349:Global default library context, Algorithm (RC2-40-CBC : 0), Properties ()
8<---

So OpenSSL is picky there. Adding "-legacy" creates the desired certificate. I did the same for the remaining stuff (key/etc).

After feeding this in the ovpn config, it works. Propagating it back to the network manager applet works to.


So it's resolved for me. With my non in depth expertise on OpenSSL/OpenVPN, it feels like it doesn't want these "whatever legacy is" thingies in the p12 file.

Comment 5 Gert Doering 2022-05-19 07:11:49 UTC
Thanks for your tests.  These do indeed confirm that it's OpenSSL 3.0.x' handling of "legacy" algorithms.

The updated OpenVPN package (2.5.6+patches) David provides has two new switches to cope with this:

 --provider legacy default

(to load RC2 and other "legacy" algorithms)

 --tls-cert-profile insecure

(to tell OpenSSL that "yes, SHA1 certificates are ok, please let me connect!").

So, could you test the original .p12 with an openvpn invocation that includes "--tls-cert-profile insecure --provider legacy default"?  This *should* make it work fine - and this is something we need to spread the word about...

gert

Comment 6 Cajus Pollmeier 2022-05-19 07:54:21 UTC
Could you point me to the package to be used? Neither https://koji.fedoraproject.org/koji/taskinfo?taskID=87198883 from above, nor the latest and greatest openvpn-2.5.6-1.fc36.x86_64 seem to have these parameters.

Options error: Unrecognized option or missing or extra parameter(s) in [CMD-LINE]:1: provider (2.5.6)
Use --help for more information.

Comment 7 Gert Doering 2022-05-19 08:21:52 UTC
Apologies, typo on my side.  It needs to be --provider*s*.  Not enough coffee.

Copying from my 2.5 test case  '--tls-cert-profile insecure --providers legacy default'

It needs David's pre-release binary, which should identify itself with "--version" as git:release/2.5/1f54811e92c89fe0.

The current official FC36 binary (2.5.6-1) does not have these two options yet that are needed to make OpenSSL 3 do what we need.

Comment 8 Cajus Pollmeier 2022-05-19 08:33:00 UTC
Not enough coffee on my side, too. That's something that I could have found by myself.

The connection works with these flags and the old PKCS12 file. Thanks for your time!

Comment 9 Gert Doering 2022-05-19 09:06:01 UTC
Ah, perfect.  This means we can do a 2.5.7 release early next week, with FC36 updates, and get this class of problems fixed (or at least "bring the options to make it work").

Thanks for testing!

Comment 10 Christian Kujau 2022-05-28 00:05:45 UTC
Same here:

===============================================================
$ grep -A3 pkcs config.ovpn
pkcs12     cert.p12
ca         ca.crt
tls-auth   ta.key 1

$ openvpn --config config.ovpn
🔐 Enter Private Key Password: ************
Enter Auth Username: alice
🔐 Enter Auth Password: ******
[...]
2022-05-28 01:47:34 us=158354 OpenSSL: error:11800071:PKCS12 routines::mac verify failure
2022-05-28 01:47:34 us=158424 OpenSSL: error:0308010C:digital envelope routines::unsupported
2022-05-28 01:47:34 us=158449 Decoding PKCS12 failed. Probably wrong password or unsupported/legacy encryption
2022-05-28 01:47:34 us=158511 Error: private key password verification failed
2022-05-28 01:47:34 us=158544 Exiting due to fatal error
===============================================================

And indeed, the provided key appears to be too old to be decrypted properly:

===============================================================
$ openssl pkcs12 -in cert.p12 -info 
Enter Import Password:
MAC: sha1, Iteration 2048
MAC length: 20, salt length: 8
PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 2048
Error outputting keys and certificates
407C42ED8E7F0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:349:Global default library context, Algorithm (RC2-40-CBC : 0), Properties ()
===============================================================

When passing -legacy to the openssl command, the PKCS#12 information is shown.



And yes, with openvpn-2.5_git1f54811e92c-1.fc36.x86_64 and both --providers and --tls-cert-profile options set, the connection is working again. So, please do a proper release soon! :-)



Another workaround: decrypt the PKCS12 key into a password-less (!) key to prevent OpenVPN from decrypting the key at all, thus preventing the decoding error above:

===============================================================
$ openssl pkcs12 -in cert.p12 -nodes -out cert_plain.pem -legacy
Enter Import Password: ************

$ openssl pkcs12 -in cert_plain.pem -export -out cert_plain.p12
Enter Export Password:  <press Enter>
Verifying - Enter Export Password: <press Enter>
===============================================================

Use cert_plain.p12 in the OpenVPN config and just press "Enter" when OpenVPN prompts for "Enter Private Key Password".

Comment 11 David Sommerseth 2022-05-31 15:41:23 UTC
*** Bug 2068425 has been marked as a duplicate of this bug. ***

Comment 12 Fedora Update System 2022-05-31 15:42:32 UTC
FEDORA-EPEL-2022-7a1c159683 has been submitted as an update to Fedora EPEL 9. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-7a1c159683

Comment 13 Fedora Update System 2022-05-31 15:42:34 UTC
FEDORA-2022-31409e24bc has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-31409e24bc

Comment 14 Fedora Update System 2022-05-31 15:42:35 UTC
FEDORA-2022-c1123ef770 has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2022-c1123ef770

Comment 15 Fedora Update System 2022-06-01 02:29:42 UTC
FEDORA-2022-31409e24bc has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-31409e24bc`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-31409e24bc

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 16 Fedora Update System 2022-06-01 02:42:56 UTC
FEDORA-2022-7750ee6e19 has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-7750ee6e19`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-7750ee6e19

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 17 Fedora Update System 2022-06-01 02:58:46 UTC
FEDORA-2022-c1123ef770 has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-c1123ef770`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-c1123ef770

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 18 Fedora Update System 2022-06-01 03:10:32 UTC
FEDORA-EPEL-2022-7a1c159683 has been pushed to the Fedora EPEL 9 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-7a1c159683

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 19 Fedora Update System 2022-06-02 01:33:30 UTC
FEDORA-2022-31409e24bc has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 20 Fedora Update System 2022-06-09 00:41:28 UTC
FEDORA-EPEL-2022-7a1c159683 has been pushed to the Fedora EPEL 9 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 21 Fedora Update System 2022-06-09 12:24:28 UTC
FEDORA-2022-7750ee6e19 has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.