Bug 1473971 - Fedora 26: sendmail STARTTLS no longer supports EC certs using SECP curves
Fedora 26: sendmail STARTTLS no longer supports EC certs using SECP curves
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: sendmail (Show other bugs)
26
x86_64 Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Jaroslav Škarvada
Fedora Extras Quality Assurance
: Regression
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-07-22 17:09 EDT by Andrew
Modified: 2017-08-07 13:19 EDT (History)
5 users (show)

See Also:
Fixed In Version: sendmail-8.15.2-16.fc26
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-08-07 13:19:42 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Andrew 2017-07-22 17:09:02 EDT
Description of problem:

I use an EC cert (secp384r1) signed by my own local CA for httpd and sendmail.  After upgrading to Fedora 26 from Fedora 25 (which involves the upgrade to OpenSSL 1.1.0), the cert still works fine with httpd, as well as "openssl s_server" and "openssl s_client" together, but completely breaks sendmail STARTTLS.

I replaced with an RSA cert as a test and sendmail STARTTLS works like a charm. NIST-P384 (secp384r1) and NIST P-521 (secp521r1) certs break it.


Version-Release number of selected component (if applicable):

     sendmail-8.15.2-14.fc26.x86_64
     openssl-1.1.0f-7.fc26.x86_64


How reproducible:

    Every time.


Steps to Reproduce:

1.Configure sendmail to use a secp384r1 cert
2.STARTTLS fails

Added following macros in sendmail.mc:

define(`confCACERT_PATH', `/etc/pki/ca-trust/extracted/pem')dnl
define(`confCACERT', `/etc/mail/certs/cacert.pem')dnl
define(`confSERVER_CERT', `/etc/mail/certs/testcert.pem')dnl
define(`confSERVER_KEY', `/etc/mail/certs/testkey.pem')dnl


Actual results:

Log from sendmail:

Jul 22 13:58:09 localhost sendmail[7321]: STARTTLS=server, error: accept failed=-1, reason=no shared cipher, SSL_error=1, errno=0, retry=-1, relay=localhost [127.0.0.1]

Log from "openssl s_client -connect localhost:25 -state -starttls smtp"

CONNECTED(00000003)
SSL_connect:before SSL initialization
SSL_connect:SSLv3/TLS write client hello
SSL3 alert read:fatal:handshake failure
SSL_connect:error in SSLv3/TLS write client hello
140599398188800:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:ssl/record/rec_layer_s3.c:1399:SSL alert number 40
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 280 bytes and written 277 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1500757089
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---



Expected results:

At least get past client_hello.


Additional info:

Test results when replacing cert with a test RSA-8192 cert, signed by same CA with same signature algorithm (ecdsa-with-SHA512):

sendmail log:

Jul 22 14:02:26 localhost sendmail[7911]: STARTTLS=server, relay=localhost [127.0.0.1], version=TLSv1.2, verify=OK, cipher=ECDHE-RSA-CHACHA20-POLY1305, bits=256/256


output from "openssl s_client -connect localhost:25 -state -starttls smtp"



CONNECTED(00000003)
SSL_connect:before SSL initialization
SSL_connect:SSLv3/TLS write client hello
SSL_connect:SSLv3/TLS write client hello
SSL_connect:SSLv3/TLS read server hello
depth=1 [clip cert info]
verify return:1
depth=0 [clip cert info]
verify return:1
SSL_connect:SSLv3/TLS read server certificate
SSL_connect:SSLv3/TLS read server key exchange
SSL_connect:SSLv3/TLS read server certificate request
SSL_connect:SSLv3/TLS read server done
SSL_connect:SSLv3/TLS write client certificate
SSL_connect:SSLv3/TLS write client key exchange
SSL_connect:SSLv3/TLS write change cipher spec
SSL_connect:SSLv3/TLS write finished
SSL_connect:SSLv3/TLS write finished
SSL_connect:SSLv3/TLS read server session ticket
SSL_connect:SSLv3/TLS read change cipher spec
SSL_connect:SSLv3/TLS read finished
---
Certificate chain
 0 [clip cert info]
 1 [clip cert info]
---
Server certificate
-----BEGIN CERTIFICATE-----
[clip cert info]
-----END CERTIFICATE-----
subject=[clip subject info]
issuer=[clip issuer info]
---
Acceptable client certificate CA names
/C=[clip]
/emailAddress=[clip]
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: RSA+SHA512:DSA+SHA512:ECDSA+SHA512:RSA+SHA384:DSA+SHA384:ECDSA+SHA384:RSA+SHA256:DSA+SHA256:ECDSA+SHA256:RSA+SHA224:DSA+SHA224:ECDSA+SHA224:RSA+SHA1:DSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA+SHA512:DSA+SHA512:ECDSA+SHA512:RSA+SHA384:DSA+SHA384:ECDSA+SHA384:RSA+SHA256:DSA+SHA256:ECDSA+SHA256:RSA+SHA224:DSA+SHA224:ECDSA+SHA224:RSA+SHA1:DSA+SHA1:ECDSA+SHA1
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 4341 bytes and written 407 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-CHACHA20-POLY1305
Server public key is 8192 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-CHACHA20-POLY1305
    Session-ID: E8D7C1191EFB9FACAF42457C50A221AC224246A1607493EF232FE158E1A6E822
    Session-ID-ctx:
    Master-Key: 2DB8CD09244A9CB0BB2305E52C2B7F2BFFCB697C1F153CA28D3E44B51C20EBFB41E4325A5DA8B91E37AE658AE0A7AD41
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 1 (seconds)
    TLS session ticket:
    0000 - e1 d6 55 a3 96 b2 69 12-7e d4 33 95 cb 75 cd 2d   ..U...i.~.3..u.-
    0010 - 12 1f c2 c1 1b cf 7b d2-77 40 b5 f6 74 2a 4c 94   ......{.w@..t*L.
    0020 - 84 cd 3e e0 90 c5 5a 5c-23 90 84 a7 93 c2 8c 96   ..>...Z\#.......
    0030 - 38 75 01 54 fa 3c 3f 76-e0 33 44 e3 d0 d8 78 53   8u.T.<?v.3D...xS
    0040 - 9f f9 cb c7 2e 70 7d b7-8e 06 b8 16 be 3a 86 7d   .....p}......:.}
    0050 - 9a 06 0b 08 bc 46 b7 f5-94 ce 80 3c dc 7d de 04   .....F.....<.}..
    0060 - fb f8 bd 80 45 3f 62 aa-04 30 c4 fd 2d 31 34 0c   ....E?b..0..-14.
    0070 - e4 01 ac 6f ff fd 75 28-68 84 e7 fb 0c 0f 82 83   ...o..u(h.......
    0080 - f0 30 e2 4b aa 99 93 c4-b4 44 9e aa e3 ef 72 7e   .0.K.....D....r~
    0090 - dd 5d f3 66 73 26 b5 1e-f1 e2 4a fd ad bf 3a 6a   .].fs&....J...:j
    00a0 - 05 0b 70 35 ac 6c ca 6a-d2 ae eb c9 87 52 cb d6   ..p5.l.j.....R..

    Start Time: 1500757346
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
---
250 HELP
Comment 1 Roman Žilka 2017-07-24 10:58:31 EDT
FWIW, F26's sendmail works OK for me with a prime256v1-based cert (signature: ecdsa+SHA256).

Pkg versions:
openssl-1.1.0f-7.fc26.x86_64
sendmail-8.15.2-14.fc26.x86_64

In sendmail.mc:
define(`confCACERT_PATH', `/etc/pki/tls/certs')dnl
define(`confCACERT', `/etc/pki/tls/certs/ca-bundle.crt')dnl
define(`confSERVER_CERT', `/etc/pki/tls/certs/sendmail_cert.pem')dnl
define(`confSERVER_KEY', `/etc/pki/tls/certs/sendmail_eckey.pem')dnl

# openssl s_client -starttls smtp -connect localhost:25
   (...)
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: RSA+SHA512:DSA+SHA512:ECDSA+SHA512:RSA+SHA384:DSA+SHA384:ECDSA+SHA384:RSA+SHA256:DSA+SHA256:ECDSA+SHA256:RSA+SHA224:DSA+SHA224:ECDSA+SHA224:RSA+SHA1:DSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA+SHA512:DSA+SHA512:ECDSA+SHA512:RSA+SHA384:DSA+SHA384:ECDSA+SHA384:RSA+SHA256:DSA+SHA256:ECDSA+SHA256:RSA+SHA224:DSA+SHA224:ECDSA+SHA224:RSA+SHA1:DSA+SHA1:ECDSA+SHA1
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 18293 bytes and written 425 bytes
Verification error: self signed certificate
---
New, TLSv1.2, Cipher is ECDHE-ECDSA-AES256-GCM-SHA384
Server public key is 256 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-ECDSA-AES256-GCM-SHA384
   (...)
Comment 2 Roman Žilka 2017-07-24 11:29:13 EDT
Just tried secp384r1: getting a failure too.
Comment 4 Andrew 2017-07-24 13:10:50 EDT
Interesting data point; I hadn't tried all of the curves.  I took a few minutes to try certs based on all the EC curves enabled in Fedora 26:

secp224r1:   FAIL
secp256k1:   FAIL
secp384r1:   FAIL
secp521r1:   FAIL
prime256v1:  SUCCESS

BTW, I'm happy to provide any additional information and/or run troubleshooting steps - anything to help sort this out.
Comment 5 Jaroslav Škarvada 2017-07-25 08:28:46 EDT
This is not regression in sendmail, but maybe in openssl, i.e.:
OpenSSL_1_0_2k - works OK
OpenSSL_1_1_0f - doesn't work

Maybe it's related to https://github.com/openssl/openssl/issues/2033, I am going to investigate.
Comment 6 Andrew 2017-07-25 10:59:55 EDT
I saw that openssl issue and also wondered if it was related.  To clarify one data point from my original post, if I use "openssl s_server" with an secp384r1 cert and use "openssl s_client" to connect to it, it works fine...so openssl seems to be able to handle the cert.  But, sendmail + openssl breaks; so it may have something to do with the integration/interaction of the two.
Comment 7 Jaroslav Škarvada 2017-07-25 11:03:29 EDT
I think it's because sendmail uses prime256v1 for the tempkey always which probably confuses the openssl stack regarding the handshake of supported algorithms - just a preliminary theory :)
Comment 8 Andrew 2017-07-25 16:54:54 EDT
I believe you are right! I replaced the following line in tls.c, recompiled sendmail, and it works like a charm with my secp384r1 cert.  It negotiated an X25519 temp key ("Server Temp Key: X25519, 253 bits").

-  SSL_CTX_set_tmp_ecdh(*ctx, ecdh);
+  SSL_CTX_set_ecdh_auto(*ctx, 1);


Note that I'm not a coder and I am not certain this is a complete solution, but it does seem to validate your preliminary theory.
Comment 9 Jaroslav Škarvada 2017-07-26 04:54:21 EDT
(In reply to Andrew from comment #8)
> I believe you are right! I replaced the following line in tls.c, recompiled
> sendmail, and it works like a charm with my secp384r1 cert.  It negotiated
> an X25519 temp key ("Server Temp Key: X25519, 253 bits").
> 
> -  SSL_CTX_set_tmp_ecdh(*ctx, ecdh);
> +  SSL_CTX_set_ecdh_auto(*ctx, 1);
> 
> 
> Note that I'm not a coder and I am not certain this is a complete solution,
> but it does seem to validate your preliminary theory.

Thanks, this was on my todo list to check, now trying to bisect openssl, I wonder what they changed and why.
Comment 10 Roman Žilka 2017-07-26 10:38:16 EDT
(In reply to Andrew from comment #8)
> -  SSL_CTX_set_tmp_ecdh(*ctx, ecdh);
> +  SSL_CTX_set_ecdh_auto(*ctx, 1);

Randomly found this - FYI:
https://github.com/openssl/openssl/commit/2ecb9f2d18614fb7b7b42830a358b7163ed43221
Comment 11 Jaroslav Škarvada 2017-07-26 11:32:33 EDT
(In reply to Roman Žilka from comment #10)
> (In reply to Andrew from comment #8)
> > -  SSL_CTX_set_tmp_ecdh(*ctx, ecdh);
> > +  SSL_CTX_set_ecdh_auto(*ctx, 1);
> 
> Randomly found this - FYI:
> https://github.com/openssl/openssl/commit/
> 2ecb9f2d18614fb7b7b42830a358b7163ed43221

Thanks, it seems the SSL_CTX_set_ecdh_auto is useless.
Comment 12 Andrew 2017-07-26 12:11:10 EDT
Yup, I saw after I posted that comment that 'SSL_CTX_set_ecdh_auto' is a no-op compatibility macro.  Apparently, openssl 1.1.0 enables ECDH auto by default.  I commented out the 'SSL_CTX_set_ecdh_auto' line and that had the same effect - secp384r2 cert worked.

Sorry for the red herring, but I think it still validates your hypothesis.
Comment 13 Jaroslav Škarvada 2017-07-26 12:19:15 EDT
It seems whole openssl-1.1.0 branch exhibits this behavior.

The following sendmail patch make it also work:
--- a/sendmail/tls.c
+++ b/sendmail/tls.c
@@ -1257,13 +1257,17 @@ inittls(ctx, req, options, srv, certfile, keyfile, cacertpath, cacertfile, dhpar
 		}
 
 #if _FFR_TLS_EC
-		ecdh = EC_KEY_new_by_curve_name(NID_X9_62_prime256v1);
+		ecdh = EC_KEY_new_by_curve_name(NID_secp384r1);
 		if (ecdh != NULL)
 		{
 			SSL_CTX_set_options(*ctx, SSL_OP_SINGLE_ECDH_USE);
Comment 14 Andrew 2017-07-26 12:23:12 EDT
Makes sense that forcing it to the secp384r1 curve would work for a cert that uses that curve, though I imagine it would be better (more compatible) to allow openssl to negotiate which curve to use, no?

I actually think the whole '#if _FFR_TLS_EC' section could be commented out and it would work properly - at least with openssl 1.1.0, based on openssl 1.1.0 defaults.
Comment 15 Jaroslav Škarvada 2017-07-26 12:38:41 EDT
The comment 13 supports my initial theory. So I think it's openssl bug.

The same sendmail code worked with openssl-1.0.0, but it doesn't work in openssl-1.1.0. I wasn't able to find anything related, i.e. why the code shouldn't work, so reassigning to openssl to get at least information, what we are doing wrong in sendmail.
Comment 16 Tomas Mraz 2017-07-27 04:22:05 EDT
The OpenSSL bug mentioned in comment 5 is related but the real fix is to really replace the SSL_CTX_set_tmp_ecdh() with SSL_CTX_set_ecdh_auto(*ctx, 1). This can be done for both 1.1.0 and 1.0.2. Or if you want to reflect the fact that the SSL_CTX_set_ecdh_auto() is no-op on 1.1.0 you can just drop the SSL_CTX_set_tmp_ecdh() call on 1.1.0. (It should be still replaced with the _auto() call on 1.0.2 for better compatibility with other clients though.)

The current code artificially restraints the ECDH curve to NID_X9_62_prime256v1 which in combination with the OpenSSL 1.1.0 bug from comment 5 causes this failure with NID_secp384r1 ECDSA certificates.

Even if the bug in openssl is solved the change in sendmail is desirable.
Comment 17 Jaroslav Škarvada 2017-07-27 04:58:39 EDT
(In reply to Tomas Mraz from comment #16)
> The OpenSSL bug mentioned in comment 5 is related but the real fix is to
> really replace the SSL_CTX_set_tmp_ecdh() with SSL_CTX_set_ecdh_auto(*ctx,
> 1). This can be done for both 1.1.0 and 1.0.2. Or if you want to reflect the
> fact that the SSL_CTX_set_ecdh_auto() is no-op on 1.1.0 you can just drop
> the SSL_CTX_set_tmp_ecdh() call on 1.1.0. (It should be still replaced with
> the _auto() call on 1.0.2 for better compatibility with other clients
> though.)
> 
> The current code artificially restraints the ECDH curve to
> NID_X9_62_prime256v1 which in combination with the OpenSSL 1.1.0 bug from
> comment 5 causes this failure with NID_secp384r1 ECDSA certificates.
> 
> Even if the bug in openssl is solved the change in sendmail is desirable.

Thanks for info, I am going to fix it in Fedora and propose upstream. Regarding RHEL-7 it works correctly because there is openssl-1.0.2k, hopefully the rebase to 1.1.0 will not happen there :)
Comment 18 Tomas Mraz 2017-07-27 05:06:20 EDT
(In reply to Jaroslav Škarvada from comment #17)
> Thanks for info, I am going to fix it in Fedora and propose upstream.
> Regarding RHEL-7 it works correctly because there is openssl-1.0.2k,
> hopefully the rebase to 1.1.0 will not happen there :)

Nope, that would break laws of physics. :D
Comment 19 Roman Žilka 2017-07-27 06:43:50 EDT
By the way, while you're updating openssl usage, you may want to look at the SSLv23_server_method() and SSLv23_client_method() calls in the same sourcefile (sendmail/tls.c) - SSLv23_server_method(3) says they're deprecated in openssl-1.1.
Comment 20 Fedora Update System 2017-07-27 11:12:13 EDT
sendmail-8.15.2-15.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-f4431ab4a2
Comment 21 Jaroslav Škarvada 2017-07-27 11:38:44 EDT
Fixed in sendmail-8.15.2-16.fc26
Comment 22 Fedora Update System 2017-07-28 20:53:31 EDT
sendmail-8.15.2-16.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-f4431ab4a2
Comment 23 Fedora Update System 2017-08-07 13:19:42 EDT
sendmail-8.15.2-16.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.