Description of problem: An openvpn connection using X.509 certificates for authentication no longer works after updating from Fedora 20 to Fedora 21. OpenSSL does not verify the certificate. Version-Release number of selected component (if applicable): openssl-libs-1.0.1j-1.fc21.x86_64 openvpn-2.3.4-4.fc21.x86_64 How reproducible: Fails consistently against my OpenVPN server (openvpn-2.3.4, OpenSSL 1.0.1h-freebsd 5 Jun 2014 on FreeBSD 10-STABLE i386) Steps to Reproduce: 1. try to start an OpenVPN connection using X.509 certificates (in my case with NetworkManager, but the behaviour is the same when openvpn is started manually) 2. journalctl -t nm-openvpn Actual results: nm-openvpn[7015]: OpenVPN 2.3.4 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [MH] [IPv6] built on Aug 26 2014 nm-openvpn[7015]: library versions: OpenSSL 1.0.1j-fips 15 Oct 2014, LZO 2.08 nm-openvpn[7015]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts nm-openvpn[7015]: Control Channel Authentication: using '/etc/pki/tls/private/tls-auth.soekris' as a OpenVPN static key file nm-openvpn[7015]: UDPv4 link local: [undef] nm-openvpn[7015]: UDPv4 link remote: [AF_INET]188.174.125.137:33434 nm-openvpn[7015]: VERIFY ERROR: depth=0, error=certificate signature failure: /CN=soekris.my.net/C=DE/L=Hohenbrunn/ST=Bayern nm-openvpn[7015]: TLS_ERROR: BIO read tls_read_plaintext error: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed nm-openvpn[7015]: TLS Error: TLS object -> incoming plaintext read error nm-openvpn[7015]: TLS Error: TLS handshake failed nm-openvpn[7015]: SIGUSR1[soft,tls-error] received, process restarting Expected results: nm-openvpn[4807]: OpenVPN 2.3.4 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [MH] [IPv6] built on Aug 26 2014 nm-openvpn[4807]: library versions: OpenSSL 1.0.1e-fips 11 Feb 2013, LZO 2.08 nm-openvpn[4807]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts nm-openvpn[4807]: Control Channel Authentication: using '/etc/pki/tls/private/tls-auth.soekris' as a OpenVPN static key file nm-openvpn[4807]: UDPv4 link local: [undef] nm-openvpn[4807]: UDPv4 link remote: [AF_INET]188.174.57.133:33434 nm-openvpn[4807]: [soekris.my.net] Peer Connection Initiated with [AF_INET]188.174.57.133:33434 nm-openvpn[4807]: TUN/TAP device tun0 opened nm-openvpn-service-openvpn-helper tun0 1500 1558 10.190.102.30 10.190.102.29 init nm-openvpn[4807]: Initialization Sequence Completed Additional info: If I use libcrypto.so from Fedora 20 instead (openssl-libs-1.0.1e-40.fc20.x86_64, via a wrapper script that sets LD_LIBRARY_PATH), the connection is established successfully. Vice versa, if I use Fedora 21's libcrypto.so on Fedora 20, I observe the same certificate verification failure.
The reason for the failure is found in this log line: VERIFY ERROR: depth=0, error=certificate signature failure: /CN=soekris.my.net/C=DE/L=Hohenbrunn/ST=Bayern This error comes from the OpenSSL library and OpenVPN lets the SSL do all the certificate verification first. OpenVPN can do additional verifications via the config file (--verify-x509-name, --verify-hash), via an additional script (--tls-verify) or a --plugin (OPENVPN_PLUGIN_TLS_VERIFY). But neither of these additional checks in OpenVPN can return such an error message. It would be helpful of you could generate a new set of failing test certificates (CA, server and client) which you could upload to this bz. I'll move this bz over to the OpenSSL people, as they can hopefully help debugging this better.
Can you please attach the certificate (not the private key!) here or paste the output of 'openssl x509 -text -noout -in <the-certificate-file>' here?
Created attachment 951736 [details] CA certificate
Created attachment 951737 [details] server certificate
Both these certificates are signed with use of the MD5 hash algorithm which is insecure. It should be sufficient to issue new client certificate with use of SHA1 or better SHA256 hash for signing. The CA certificate signature is not verified in the process of authentication so it does not have to be replaced. You can also set OPENSSL_ENABLE_MD5_VERIFY environment variable in the openvpn client and server and it should allow verification of certificates with MD5 hashes but I would not recommend it.
Thanks! Yes, the OPENSSL_ENABLE_MD5_VERIFY workaround works for me. I will update my certificates. Where could I have read about this change in OpenSSL behaviour? Will this be mentioned in the F21 release notes?
I created release note request bug 1158767
Oops, I'm in a similar situation. Until I get our sysadmins generate new keys I should use the workaround. On my pre-F21 (test)machine I use gnome with Networkmanager(-openvpn). How can I pass the above environment variable for Networkmanager?
Sorry for reopening the bug. I don't know if there is an issue on F21 or I misuse something with the "workaround" or other staff. I've tried to apply the environment variable workaround for NetworkManager, but it does not work for me. In a virtual machine with F20 I updated the F20 openssl-1.0.1e-40.fc20 to openssl-1.0.1e-40.fc21. After the update I get the error as expected. But after adding "Environment="OPENSSL_ENABLE_MD5_VERIFY=1" to /usr/lib/systemd/system/NetworkManager.service the openvpn connection started to work. The same change under F21 does not help, I still get the error. What can I check/test?
The environment variable is respected in current openssl on F21 and Rawhide as well. There must be some difference in NetworkManager or systemd that makes the workaround not work for you.
Workaround doesn't work for me either for NetworkManager. I have the same problem in Centos 7 and the workaround does work there. In comment#9, the poster said it did work in F20 and not in F21. Thus it seems F21 is broken with respect to the documented workaround. Any ideas how to track this down? I have verified in /proc/<pid>/environ that the environment variable is being set properly in both the NetworkManager and child openvpn process. Personally I think this warrants re-opening but I'll leave that to the discretion of the owners.
Created attachment 968680 [details] DO NOT USE - patch which re-enables md5 cert checking unconditionally Patch is NOT recommended for production. I rebuilt openssl with the attached patch (which just forces the re-enabling of md5 cert checking) and now openvpn connects. So it's the activation of the workaround which is broken.
I also verify this bug on F21. I have it happening on two of my machines. It's very unfortunate. If you are not in control of the server end, and still should do work over it, what should one do? Take a vacation? Change fedoras to something else? I don't understand why it's disabled without a warning. The workaround should still be available. I verified that setting the comment 9 does not work with F21. I verified from /proc/<pid>/environ that both the network manager and openvpn process have the variable in their envs. It still won't work. On two different machines. Luckily it works from command line, so at least people can get to do their work until server provider updates all the certs for all the clients and the server. This bug should be re-opened, and fixed so that there is a workaround in settings.
If the workaround works from command-line, the culprit in breaking the workaround must lie in Network Manager. Please open a bug against it. I certainly will not enable the MD5 verification by default in openssl again. MD5 signatures in certificates are insecure. You should notify your server operator to fix their insecure setup as well.
Ok, I agree it's on Network Manager. I'm not asking to enable it by default, but to have a workaround available. The place for workaround should be in Network Manager. I will create a ticket there.
Good, thanks - just for a note in bug 1174122 the workaround for a similar problem in wpa_supplicant works fine.
The code in openssl that implements the workaround uses "secure_getenv". Since comment#13 and comment#11 both confirm the env variable is set, secure_getenv must be not returning anything. It returns NULL when (from the manpage): * the process's effective user ID did not match its real user ID or the process's effective group ID did not match its real group ID (typically this is the result of executing a set-user-ID or set- group-ID program); * the effective capability bit was set on the executable file; or * the process has a nonempty permitted capability set. So one of these must be happening when run under NetworkManager (and not from command line usage). Since this issue is migrating to NetworkManager, does someone have the bug number for that?
I created a bug report yesterday to Network Manager, as discussed earlier. https://bugzilla.redhat.com/show_bug.cgi?id=1175481 Please join there.