Bug 1157260 (MD5certificateverify)

Summary: openvpn/openssl certificate verify failed
Product: [Fedora] Fedora Reporter: Karsten Kretschmer <kkretschmer>
Component: opensslAssignee: Tomas Mraz <tmraz>
Status: CLOSED WONTFIX QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 21CC: bugzilla, davids, gwync, huzaifas, ikke, kkretschmer, steve, tmraz, zoltank
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-11-05 14:17:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Description Flags
CA certificate
server certificate
DO NOT USE - patch which re-enables md5 cert checking unconditionally none

Description Karsten Kretschmer 2014-10-26 18:56:19 UTC
Description of problem:

An openvpn connection using X.509 certificates for authentication no longer works after updating from Fedora 20 to Fedora 21. OpenSSL does not verify the certificate.

Version-Release number of selected component (if applicable):


How reproducible:

Fails consistently against my OpenVPN server (openvpn-2.3.4, OpenSSL 1.0.1h-freebsd 5 Jun 2014 on FreeBSD 10-STABLE i386)

Steps to Reproduce:
1. try to start an OpenVPN connection using X.509 certificates (in my case with NetworkManager, but the behaviour is the same when openvpn is started manually)
2. journalctl -t nm-openvpn

Actual results:

nm-openvpn[7015]: OpenVPN 2.3.4 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [MH] [IPv6] built on Aug 26 2014
nm-openvpn[7015]: library versions: OpenSSL 1.0.1j-fips 15 Oct 2014, LZO 2.08
nm-openvpn[7015]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
nm-openvpn[7015]: Control Channel Authentication: using '/etc/pki/tls/private/tls-auth.soekris' as a OpenVPN static key file
nm-openvpn[7015]: UDPv4 link local: [undef]
nm-openvpn[7015]: UDPv4 link remote: [AF_INET]
nm-openvpn[7015]: VERIFY ERROR: depth=0, error=certificate signature failure: /CN=soekris.my.net/C=DE/L=Hohenbrunn/ST=Bayern
nm-openvpn[7015]: TLS_ERROR: BIO read tls_read_plaintext error: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
nm-openvpn[7015]: TLS Error: TLS object -> incoming plaintext read error
nm-openvpn[7015]: TLS Error: TLS handshake failed
nm-openvpn[7015]: SIGUSR1[soft,tls-error] received, process restarting

Expected results:

nm-openvpn[4807]: OpenVPN 2.3.4 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [MH] [IPv6] built on Aug 26 2014
nm-openvpn[4807]: library versions: OpenSSL 1.0.1e-fips 11 Feb 2013, LZO 2.08
nm-openvpn[4807]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
nm-openvpn[4807]: Control Channel Authentication: using '/etc/pki/tls/private/tls-auth.soekris' as a OpenVPN static key file
nm-openvpn[4807]: UDPv4 link local: [undef]
nm-openvpn[4807]: UDPv4 link remote: [AF_INET]
nm-openvpn[4807]: [soekris.my.net] Peer Connection Initiated with [AF_INET]
nm-openvpn[4807]: TUN/TAP device tun0 opened
nm-openvpn-service-openvpn-helper tun0 1500 1558 init
nm-openvpn[4807]: Initialization Sequence Completed

Additional info:

If I use libcrypto.so from Fedora 20 instead (openssl-libs-1.0.1e-40.fc20.x86_64, via a wrapper script that sets LD_LIBRARY_PATH), the connection is established successfully. Vice versa, if I use Fedora 21's libcrypto.so on Fedora 20, I observe the same certificate verification failure.

Comment 1 David Sommerseth 2014-10-27 10:23:33 UTC
The reason for the failure is found in this log line:

VERIFY ERROR: depth=0, error=certificate signature failure: /CN=soekris.my.net/C=DE/L=Hohenbrunn/ST=Bayern

This error comes from the OpenSSL library and OpenVPN lets the SSL do all the certificate verification first.  OpenVPN can do additional verifications via the config file (--verify-x509-name, --verify-hash), via an additional script (--tls-verify) or a --plugin (OPENVPN_PLUGIN_TLS_VERIFY).  But neither of these additional checks in OpenVPN can return such an error message.

It would be helpful of you could generate a new set of failing test certificates (CA, server and client) which you could upload to this bz.

I'll move this bz over to the OpenSSL people, as they can hopefully help debugging this better.

Comment 2 Tomas Mraz 2014-10-29 09:48:55 UTC
Can you please attach the certificate (not the private key!) here or paste the output of 'openssl x509 -text -noout -in <the-certificate-file>' here?

Comment 3 Karsten Kretschmer 2014-10-29 10:27:35 UTC
Created attachment 951736 [details]
CA certificate

Comment 4 Karsten Kretschmer 2014-10-29 10:28:34 UTC
Created attachment 951737 [details]
server certificate

Comment 5 Tomas Mraz 2014-10-29 10:49:01 UTC
Both these certificates are signed with use of the MD5 hash algorithm which is insecure. It should be sufficient to issue new client certificate with use of SHA1 or better SHA256 hash for signing. The CA certificate signature is not verified in the process of authentication so it does not have to be replaced.

You can also set OPENSSL_ENABLE_MD5_VERIFY environment variable in the openvpn client and server and it should allow verification of certificates with MD5 hashes but I would not recommend it.

Comment 6 Karsten Kretschmer 2014-10-29 17:57:57 UTC
Thanks! Yes, the OPENSSL_ENABLE_MD5_VERIFY workaround works for me. I will update my certificates. Where could I have read about this change in OpenSSL behaviour? Will this be mentioned in the F21 release notes?

Comment 7 Tomas Mraz 2014-10-30 08:01:39 UTC
I created release note request bug 1158767

Comment 8 Zoltan Kota 2014-11-04 22:04:20 UTC
Oops, I'm in a similar situation. Until I get our sysadmins generate new keys I should use the workaround. On my pre-F21 (test)machine I use gnome with Networkmanager(-openvpn). How can I pass the above environment variable for Networkmanager?

Comment 9 Zoltan Kota 2014-11-05 14:14:01 UTC
Sorry for reopening the bug. I don't know if there is an issue on F21 or I misuse something with the "workaround" or other staff.

I've tried to apply the environment variable workaround for NetworkManager, but it does not work for me.

In a virtual machine with F20 I updated the F20 openssl-1.0.1e-40.fc20 to openssl-1.0.1e-40.fc21. After the update I get the error as expected. But after adding "Environment="OPENSSL_ENABLE_MD5_VERIFY=1" to /usr/lib/systemd/system/NetworkManager.service the openvpn connection started to work.

The same change under F21 does not help, I still get the error. What can I check/test?

Comment 10 Tomas Mraz 2014-11-05 14:17:19 UTC
The environment variable is respected in current openssl on F21 and Rawhide as well. There must be some difference in NetworkManager or systemd that makes the workaround not work for you.

Comment 11 David Mansfield 2014-12-14 20:54:47 UTC
Workaround doesn't work for me either for NetworkManager.  I have the same problem in Centos 7 and the workaround does work there.

In comment#9, the poster said it did work in F20 and not in F21.  

Thus it seems F21 is broken with respect to the documented workaround.  Any ideas how to track this down?

I have verified in /proc/<pid>/environ that the environment variable is being set properly in both the NetworkManager and child openvpn process.

Personally I think this warrants re-opening but I'll leave that to the discretion of the owners.

Comment 12 David Mansfield 2014-12-14 21:29:46 UTC
Created attachment 968680 [details]
DO NOT USE - patch which re-enables md5 cert checking unconditionally

Patch is NOT recommended for production.

I rebuilt openssl with the attached patch (which just forces the re-enabling of md5 cert checking) and now openvpn connects.  So it's the activation of the workaround which is broken.

Comment 13 Ilkka Tengvall 2014-12-16 21:27:23 UTC
I also verify this bug on F21. I have it happening on two of my machines. It's very unfortunate. If you are not in control of the server end, and still should do work over it, what should one do? Take a vacation? Change fedoras to something else? I don't understand why it's disabled without a warning. The workaround should still be available.

I verified that setting the comment 9 does not work with F21. I verified from /proc/<pid>/environ that both the network manager and openvpn process have the variable in their envs. It still won't work. On two different machines.

Luckily it works from command line, so at least people can get to do their work until server provider updates all the certs for all the clients and the server.

This bug should be re-opened, and fixed so that there is a workaround in settings.

Comment 14 Tomas Mraz 2014-12-17 08:49:04 UTC
If the workaround works from command-line, the culprit in breaking the workaround must lie in Network Manager. Please open a bug against it.

I certainly will not enable the MD5 verification by default in openssl again. MD5 signatures in certificates are insecure. You should notify your server operator to fix their insecure setup as well.

Comment 15 Ilkka Tengvall 2014-12-17 12:37:58 UTC
Ok, I agree it's on Network Manager. I'm not asking to enable it by default, but to have a workaround available. The place for workaround should be in Network Manager. I will create a ticket there.

Comment 16 Tomas Mraz 2014-12-17 12:41:35 UTC
Good, thanks - just for a note in bug 1174122 the workaround for a similar problem in wpa_supplicant works fine.

Comment 17 David Mansfield 2014-12-17 13:40:40 UTC
The code in openssl that implements the workaround uses "secure_getenv".  Since comment#13 and comment#11 both confirm the env variable is set, secure_getenv must be not returning anything.  It returns NULL when (from the manpage):

       *  the process's effective user ID did not match its real  user  ID  or
          the  process's  effective  group  ID did not match its real group ID
          (typically this is the result of executing  a  set-user-ID  or  set-
          group-ID program);

       *  the effective capability bit was set on the executable file; or

       *  the process has a nonempty permitted capability set.

So one of these must be happening when run under NetworkManager (and not from command line usage).

Since this issue is migrating to NetworkManager, does someone have the bug number for that?

Comment 18 Ilkka Tengvall 2014-12-18 13:44:45 UTC
I created a bug report yesterday to Network Manager, as discussed earlier.


Please join there.