Bug 1170339

Summary: use the default min/max TLS version provided by NSS [RHEL-7]
Product: Red Hat Enterprise Linux 7 Reporter: Eric Rich <erich>
Component: curlAssignee: Kamil Dudka <kdudka>
Status: CLOSED ERRATA QA Contact: Stefan Kremen <skremen>
Severity: urgent Docs Contact:
Priority: high    
Version: 7.0CC: andy.fletcher, bknudson, blumel, bugproxy, erich, fleite, fweimer, hannsj_uhl, janarula, jduncan, jjarvis, jkachuck, kdudka, kengert, ksrot, lmiksik, mmckinst, ovasik, pdwyer, sauchter, skremen, spider, srandhaw, thoger, ville, wharris, wtogami
Target Milestone: rcKeywords: FutureFeature, Reopened, Security
Target Release: 7.2   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: curl-7.29.0-24.el7 Doc Type: Bug Fix
Doc Text:
Previously, versions 1.1 and 1.2 of the TLS protocol were disabled by default in libcurl. Consequently, the users were required to explicitly enable these TLS versions in utilities based on libcurl in order to allow these utilities to securely communicate with servers that do not accept SSL 3.0 and TLS 1.0 connections. With this update, TLS 1.1 and TLS 1.2 are no longer disabled by default in libcurl. The user can, however, explicitly disable them through the libcurl API.
Story Points: ---
Clone Of:
: 1171318 (view as bug list) Environment:
Last Closed: 2015-11-19 07:07:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1171318    
Bug Blocks: 1105185, 1118366, 1133060, 1192015, 1252514, 1289205    

Description Eric Rich 2014-12-03 20:52:42 UTC
Description of problem:

Curl does not negotiate an SSL/TLS connections from strongest down to weakest as presented by the server. 

Version-Release number of selected component (if applicable): 
RHEL 7
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.15.4 zlib/1.2.7 libidn/1.28 libssh2/1.4.3

How reproducible: Very

Steps to Reproduce:
1. Install httpd
2. setup SSL and define the following:
   SSLProtocol TLSv1 TLSv1.1 TLSv1.2
   SSLCipherSuite HIGH
3. curl -IL https://$(hostname) -v

Actual results:

* About to connect() to $(hostname) port 443 (#0)
*   Trying XXX.XXX.XXX.XXX...
* Connected to $(hostname) (10.13.213.65) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/cacert.crt
  CApath: none
* NSS error -12190 (SSL_ERROR_PROTOCOL_VERSION_ALERT)
* Peer reports incompatible or unsupported protocol version.
* Error in TLS handshake, trying SSLv3...
> HEAD / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: $(hostname)
> Accept: */*
> 
* Connection died, retrying a fresh connect
* Closing connection 0
* Issue another request to this URL: 'https://$(hostname)'
* About to connect() to $(hostname) port 443 (#1)
*   Trying XXX.XXX.XXX.XXX...
* Connected to $(hostname) (XXX.XXX.XXX.XXX) port 443 (#1)
* TLS disabled due to previous handshake failure
*   CAfile: /etc/pki/tls/certs/cacert.crt
  CApath: none
* NSS error -12286 (SSL_ERROR_NO_CYPHER_OVERLAP)
* Cannot communicate securely with peer: no common encryption algorithm(s).
* Closing connection 1
curl: (35) Peer reports incompatible or unsupported protocol version.
Expected results:

A valid HTTP responce code (depending on content of httpd server). 

Additional info:

Comment 1 Eric Rich 2014-12-03 20:57:33 UTC
http://tools.ietf.org/html/rfc5246 indicates that clients should negotiate SSL/TLS protocols from strongest down to weakest.

Comment 3 Kamil Dudka 2014-12-04 07:53:26 UTC
RHEL-7 (lib)curl does not enable TLS > 1.0 by default.  Please use the --tlsv1 option of curl to negotiate the highest TLS version supported by client/server.

Comment 4 Eric Rich 2014-12-04 12:23:48 UTC
(In reply to Kamil Dudka from comment #3)
> RHEL-7 (lib)curl does not enable TLS > 1.0 by default.  Please use the
> --tlsv1 option of curl to negotiate the highest TLS version supported by
> client/server.

I can see curl does not enable TLS > 1.0 by default, however the issue (as denoted by this bug) is that this is against the RFC, for how a client should act.

What is the justification for curl doing this?

Comment 5 Kamil Dudka 2014-12-04 12:51:58 UTC
(In reply to Eric Rich from comment #4)
> What is the justification for curl doing this?

Compatibility.  We prefer not to change the default behavior in Enterprise Linux while we allow applications to use new features if explicitly requested. This is not specific to libcurl.  NSS (the library used by libcurl to implement TLS) itself does not negotiate TLS > 1.0 by default.  Please refer to the following bugs for details:

    bug #1012136 comment #23

    bug #994599 comment #7

Comment 6 Eric Rich 2014-12-04 13:10:35 UTC
(In reply to Kamil Dudka from comment #5)
> (In reply to Eric Rich from comment #4)
> > What is the justification for curl doing this?
> 
> Compatibility.  We prefer not to change the default behavior in Enterprise
> Linux while we allow applications to use new features if explicitly
> requested. This is not specific to libcurl.  NSS (the library used by
> libcurl to implement TLS) itself does not negotiate TLS > 1.0 by default. 
> Please refer to the following bugs for details:

I have to argue for the customer on this one. The Protocol (not the RFC) is designed so that a negotiation is done when determining the protocol. 

So server who only allow or implement older protocols (SSLv3), can work with newer clients (or clients who support newer protocols).

In short the argument that bug #1012136 comment #23, and bug #994599 comment #7 makes is that a server admin is allowing a protocol on there server that is not 100% implemented or not properly implemented, so we should have clients use lower less secure protocols (unless they explicitly state to use something stronger). 

With this said defaulting to SSLv3 makes curl vulnerable to https://access.redhat.com/articles/1232123 so I am marking this a security bug.

Comment 7 Kamil Dudka 2014-12-04 13:37:09 UTC
(In reply to Eric Rich from comment #6)
> In short the argument that bug #1012136 comment #23, and bug #994599 comment
> #7 makes is that a server admin is allowing a protocol on there server that
> is not 100% implemented or not properly implemented, so we should have
> clients use lower less secure protocols (unless they explicitly state to use
> something stronger).

Not sure I understand your comment.  There are servers out there that you can connect with curl using the default configuration, but you cannot connect them with the --tlsv1 option.  Yes, this is caused by bugs in server implementations, not in RFCs.  Still customers expect curl to connect those servers after a minor update of RHEL.

> With this said defaulting to SSLv3 makes curl vulnerable to
> https://access.redhat.com/articles/1232123 so I am marking this a security
> bug.

Nope, curl does not default to SSLv3.  It enables both TLS 1.0 and SSL 3.0 by default.  See also bug #1154060.

Comment 8 Eric Rich 2014-12-04 14:01:39 UTC
(In reply to Kamil Dudka from comment #7)
> (In reply to Eric Rich from comment #6)
> Nope, curl does not default to SSLv3.  It enables both TLS 1.0 and SSL 3.0
> by default.  See also bug #1154060.

Then why do I see a protocol error (on SSLv3) when I run the command: 

    curl -IL https://$(hostname) -v

If TLS 1.0 is enabled, curl should prefer it over SSLv3 as the httpd server configuration is presenting TLSv1.[0|1|2] protocols. 

This is the argument for a bug in curl. If curl enables both TLS 1.0 and SSL 3.0 this error would not occur, because the RFC (for TLS) would have been followed and curl would have negotiated a TLSv1 connection over the SSLv3 connection (that is disabled/not presented on the server). 

bug #1154060 can not be possible if bug #1012136 comment #23, and bug #994599 comment #7 comments stand as these bugs state the curl should only be defaulting to SSLv3. Simply having an option to force a higher protocol is ensuring the curl negotiates the strongest protocol.

Based on my review of these bugs it seems that the messaging and direction on what should be done to resolve this is not clear. The argument our customer(s) is/are making is that our client tools are not following the RFC - or the specification for how tools should operate, and that this should be addressed.

Comment 9 Kamil Dudka 2014-12-04 14:26:56 UTC
(In reply to Eric Rich from comment #8)
> If TLS 1.0 is enabled, curl should prefer it over SSLv3 as the httpd server
> configuration is presenting TLSv1.[0|1|2] protocols. 

RHEL-7 curl sends TLSv1 Client Hello by default.  You can check it by wireshark.

> This is the argument for a bug in curl. If curl enables both TLS 1.0 and SSL
> 3.0 this error would not occur, because the RFC (for TLS) would have been
> followed and curl would have negotiated a TLSv1 connection over the SSLv3
> connection (that is disabled/not presented on the server).

Could you please expose your testing server for me to repeat the bug?

> bug #1154060 can not be possible if bug #1012136 comment #23, and bug
> #994599 comment #7 comments stand as these bugs state the curl should only
> be defaulting to SSLv3. Simply having an option to force a higher protocol
> is ensuring the curl negotiates the strongest protocol.

There must be some misunderstanding.  None of the above bugs says anything about curl defaulting to SSLv3.  Bug #1154060 asks for disabling an out-of-protocol _fallback_ to SSLv3, which is exactly what happens in your case -- see the messages "Error in TLS handshake, trying SSLv3..." and "TLS disabled due to previous handshake failure" in the verbose output of curl (comment #0).

> Based on my review of these bugs it seems that the messaging and direction
> on what should be done to resolve this is not clear. The argument our
> customer(s) is/are making is that our client tools are not following the RFC
> - or the specification for how tools should operate, and that this should be
> addressed.

I believe that curl (without additional options) is compliant with the TLS 1.0 specification.  The implementation of TLS 1.2 came after RHEL-7 GA, thus needs to be explicitly enabled by the application.

Comment 13 Kamil Dudka 2014-12-04 16:41:09 UTC
Thank you for granting me the access to your testing server!  I believe it is just a mistake in the configuration of Apache what prevents curl from connecting without --tlsv1 (or --tlsv1.2).

You need to replace the line:

    SSLProtocol TLSv1 TLSv1.1 TLSv1.2

... by the line:

    SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2

... in your /etc/httpd/conf.d/ssl.conf and restart the httpd service.

Comment 14 Eric Rich 2014-12-04 17:44:39 UTC
(In reply to Kamil Dudka from comment #13)
> Thank you for granting me the access to your testing server!  I believe it
> is just a mistake in the configuration of Apache what prevents curl from
> connecting without --tlsv1 (or --tlsv1.2).
> 
> You need to replace the line:
> 
>     SSLProtocol TLSv1 TLSv1.1 TLSv1.2
> 
> ... by the line:
> 
>     SSLProtocol +TLSv1 +TLSv1.1 +TLSv1.2
> 
> ... in your /etc/httpd/conf.d/ssl.conf and restart the httpd service.

It seems you are correct!

Comment 15 Eric Rich 2014-12-04 20:44:20 UTC
re-opening this as switching the protocol's on the server size in this way helps fix the issue as httpd can now offer any of the TLS protocol version and curl can then (as a client) select TLSv1.0. 

However if I make the following change: 

    SSLProtocol +TLSv1.2

curl still give me the same error (as before). Again caused because the client is defaulting to only TLSv1 or SSLv3 as the strongest protocol it supports. 

The same configuration on RHEL 6 works. IE: curl can connect to a server only presenting at TLSv1.2 connection (with out the -1 flag).

Comment 16 Kamil Dudka 2014-12-04 22:34:32 UTC
(In reply to Eric Rich from comment #15)
> re-opening this as switching the protocol's on the server size in this way
> helps fix the issue as httpd can now offer any of the TLS protocol version
> and curl can then (as a client) select TLSv1.0. 
> 
> However if I make the following change: 
> 
>     SSLProtocol +TLSv1.2

Not a change really.  It is exactly what Apache read from your original configuration (stated in comment #0), isn't it?

> curl still give me the same error (as before). Again caused because the
> client is defaulting to only TLSv1 or SSLv3 as the strongest protocol it
> supports.

... which is the expected default behavior.  We are not going to change the default behavior in RHEL-7 (unless Security Response insists on the change).

> The same configuration on RHEL 6 works. IE: curl can connect to a server
> only presenting at TLSv1.2 connection (with out the -1 flag).

This is not supposed to work.  Please file a separate bug report against RHEL-6 with the precise NVR of libcurl and nss* packages, and steps to reproduce.

Comment 17 Eric Rich 2014-12-04 22:44:55 UTC
(In reply to Kamil Dudka from comment #16)
> Not a change really.  It is exactly what Apache read from your original
> configuration (stated in comment #0), isn't it?

Yes this is fundamentaly what I had in comment #0 because httpd does:

> In: httpd/modules/ssl/ssl_engine_config.c:

> 1338        if (action == '-') {
> 1339            *options &= ~thisopt;
> 1340        }
> 1341        else if (action == '+') {
> 1342            *options |= thisopt;
> 1343        }
> 1344        else {
> 1345            *options = thisopt;
> 1346        }

So it picks the last item in the list. 
 
> ... which is the expected default behavior.  We are not going to change the
> default behavior in RHEL-7 (unless Security Response insists on the change).

So it is expected that curl does not follow the RFC for negotiating the protocol from highest to lowest.
 
> This is not supposed to work.  Please file a separate bug report against
> RHEL-6 with the precise NVR of libcurl and nss* packages, and steps to
> reproduce.

Steps are the same as those listed here, however why would I file a bug for a RHEL product that is following the RFC as it should?

Comment 18 Kamil Dudka 2014-12-05 07:27:54 UTC
(In reply to Eric Rich from comment #17)
> So it is expected that curl does not follow the RFC for negotiating the
> protocol from highest to lowest.

RHEL-7 libcurl does follow RFC but not necessarily the latest one.

> Steps are the same as those listed here, however why would I file a bug for
> a RHEL product that is following the RFC as it should?

It would be an unintended change in behavior, which we normally avoid in RHEL.  Anyway, you have not shown us yet that your statement about RHEL-6 is correct.

Comment 19 Eric Rich 2014-12-05 13:38:15 UTC
(In reply to Kamil Dudka from comment #18)

> RHEL-7 libcurl does follow RFC but not necessarily the latest one.

So RHEL-7 libcurl follows the TLSv1 RFC but not the TLSv1.2 RFC, is this what you are saying? 

> It would be an unintended change in behavior, which we normally avoid in
> RHEL.  Anyway, you have not shown us yet that your statement about RHEL-6 is
> correct.

It seems my testing on RHEL6 was a bit flawed, as I tried on a fresh system today and curl on RHEL 6 only  negotiations on protocols from TLSv1 down to SSLv2 
   - Note: [which v2 and v3 are often disabled] leaving you with only TLSv1

The same behavior is what we are seeing on RHEL 7. However where curl on RHEL 7 differs its that it allows for TLSv1.1 and TLSv1.2 connections to be specified on the command line (RHEL 6 does not - from what I can tell). 

As this is the case I will move this to an RFE for RHEL 7 libcurl to follow the latest protocol versions RFC.

Comment 27 Kamil Dudka 2015-04-07 11:37:07 UTC
*** Bug 1183171 has been marked as a duplicate of this bug. ***

Comment 28 bknudson 2015-04-07 12:55:33 UTC
This bug makes it so that admins can't set up an https yum repo that disables SSL protocols worse than TLS 1.2, which might be required for security. This is because yum uses libcurl and there's no way to pass the --tlsv1 option through yum.

Comment 29 IBM Bug Proxy 2015-04-07 13:52:05 UTC
------- Comment From edmondsw.com 2015-04-07 13:49 EDT-------
bug # 1171318 doesn't appear to be public, but if that's for nss not using TLS 1.2 by default then it definitely needs to be fixed. TLS 1.0 should no longer be considered secure (e.g. BEAST if you use CBC ciphers, Bar Mitzvah if you use RC4) and will be disabled by more and more servers.

Comment 31 Hanns-Joachim Uhl 2015-04-10 10:05:23 UTC
Hello Red Hat / Kamil,
... can you please provide your thoughts about comment #28 and comment #29 
in this Red Hat bugzilla ..?
As outlined we think that the required usage of TLS 1.2 will become more and more mandatory (e.g. also for FIPS 140-2 compliance ...)
and therefore please provide your thoughts how this issue can be resolved
as soon as possible 
and whether you think that this issue could also be resolved with an RHN update
earlier than RHEL7.2 ..? 
Please advise as soon as possible ...
Thanks in advance for your support.

Comment 32 Kamil Dudka 2015-04-10 12:07:34 UTC
(In reply to Hanns-Joachim Uhl from comment #31)
> Hello Red Hat / Kamil,
> ... can you please provide your thoughts about comment #28

I believe that globally switching the default TLS version of curl/nss is not the only way to make yum speak TLS 1.2.  We can either introduce a yum option to enable TLS 1.2, or introduce another way to enable it explicitly.  The first option would probably imply update of the python-pycurl and python-urlgrabber packages.  The second option could be a documented environment variable or a configuration file somewhere on the file system.

Note that I am speaking purely about technical matters of the issue.  However, I am not the one to decide which approach will be taken in the end.

> and comment #29 in this Red Hat bugzilla ..?

See CVE-2011-3389 and CVE-2015-2808 for details.

> As outlined we think that the required usage of TLS 1.2 will become more and
> more mandatory (e.g. also for FIPS 140-2 compliance ...)
> and therefore please provide your thoughts how this issue can be resolved
> as soon as possible 
> and whether you think that this issue could also be resolved with an RHN
> update earlier than RHEL7.2 ..?

Switching the default in RHEL-7.2 is currently being evaluated by Product Management.  From engineering point of view, switching the default would be a tiny patch but there is a high risk of regression the patch could introduce.  In my personal opinion, switching the default TLS version before RHEL-7.2 is very unlikely to happen.

Comment 33 IBM Bug Proxy 2015-04-10 15:20:21 UTC
------- Comment From edmondsw.com 2015-04-10 15:12 EDT-------
Kamil, there isn't a single default, is there? I was expecting curl/nss would simply add TLS to a list of what is used by default (e.g. try TLS 1.2 first, then TLS 1.1...). I don't think we're suggesting you make TLS 1.2 the one and only default. If you just add it to the list, how would that involve a high risk of regression?

Comment 34 Kamil Dudka 2015-04-10 15:30:57 UTC
It has already been said privately, so reposting here for public...

The problem are so-called TLS intolerant servers.  If a client tells it supports TLS 1.2, the server simply closes the connection with no response about what TLS versions it supports.  You can read some details e.g. here:

http://blog.nexcess.net/2014/04/04/tls-1-2-and-openssl-1-0-1e-frequently-asked-questions/
http://blogs.msdn.com/b/ieinternals/archive/2011/03/25/misbehaving-https-servers-impair-tls-1.1-and-tls-1.2.aspx
http://www.ietf.org/mail-archive/web/tls/current/msg10657.html

There might be applications out there that use libcurl to securely communicate with such servers using TLS 1.0.  If we change the default to use TLS 1.2, those application will stop working after update of libcurl.

Comment 35 Mark McKinstry 2015-04-13 19:22:41 UTC
When I wrote http://blog.nexcess.net/2014/04/04/tls-1-2-and-openssl-1-0-1e-frequently-asked-questions/ , we had two clients who were having issues with two different API providers. A third client had problems, but it was with the same API provider as one of the first two clients. The blog posting was written as an explanation for any future problems we might see so we can refer clients or API providers to it for an explanation.

The two API providers that were having problems were both because the ClientHello was > 255 bytes due to openssl 1.0.1e adding a bunch more ciphers to its list. One API provider confirmed the problem was with their F5 BigIP device (https://www.imperialviolet.org/2013/10/07/f5update.html) and updated it. The other API provider confirmed the problem was on their end, but provided no technical details. The API provider that provided no technical details was also intolerant to TLSv1.2 which was incorrectly attributed as part of the problem, since we didn't realize libcurl was making TLSv1.0 connections at the time (we were debugging using 'openssl s_client' on the command line). Both API providers have resolved the issues.

The problem we're now seeing is the one Hanns-Joachim Uhl described in comment #31, some API providers are disabling TLSv1.0 and only supporting TLSv1.1 and TLSv1.2. Previously we could point out that the API providers weren't following the TLS RFC and they'd fix the problem on their end. Now the API providers point out to us that we're not following the TLS RFC and we have to fix it on our end.

In Ivan Ristic's 2013 message (https://www.ietf.org/mail-archive/web/tls/current/msg10657.html) 0.9% of servers he tested were intolerant to TLSv1.2. I'd guess that is even lower now given most people have had to update openssl and/or re-examine how they have TLS set up with heartbleed and the other recent CVEs. I think for the tiny minority of servers out there that are still TLS intolerant, the burden should be placed on them to follow the RFC rather than RHEL forcing use of an older version of TLS unless an application has been modified to request a newer version.

Comment 38 Warren Togami 2015-05-04 23:21:50 UTC
curl-7.29.0-19.el7.x86_64
curl-7.37.0-13.fc21.x86_64

This problem is becoming far more common, affecting all of the newer sites issued served by Cloudflare SSL.  Both RHEL7 and Fedora 21 curl are affected.

# curl -v https://sslspdy.com
* About to connect() to sslspdy.com port 443 (#0)
*   Trying 192.184.89.66...
* Connected to sslspdy.com (192.184.89.66) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* NSS error -12286 (SSL_ERROR_NO_CYPHER_OVERLAP)
* Cannot communicate securely with peer: no common encryption algorithm(s).
* Error in TLS handshake, trying SSLv3...
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: sslspdy.com
> Accept: */*
> 
* Connection died, retrying a fresh connect
* Closing connection 0
* Issue another request to this URL: 'https://sslspdy.com'
* About to connect() to sslspdy.com port 443 (#1)
*   Trying 192.184.89.66...
* Connected to sslspdy.com (192.184.89.66) port 443 (#1)
* TLS disabled due to previous handshake failure
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* NSS error -12286 (SSL_ERROR_NO_CYPHER_OVERLAP)
* Cannot communicate securely with peer: no common encryption algorithm(s).
* Closing connection 1
curl: (35) Cannot communicate securely with peer: no common encryption algorithm(s).

Comment 39 Warren Togami 2015-05-04 23:22:33 UTC
curl-7.29.0-19.el7.x86_64
curl-7.37.0-13.fc21.x86_64

Workaround:
# curl -v --ciphers ecdhe_ecdsa_aes_128_sha https://sslspdy.com

Comment 40 Mark McKinstry 2015-05-05 00:06:02 UTC
Warren,

The problem you're describing is a different problem caused when a site has an ECC SSL certificate, see bug #1185708 . The bug is for Fedora but RHEL has the same problem.

Comment 46 IBM Bug Proxy 2015-10-16 16:02:02 UTC
------- Comment From chavez.com 2015-10-16 16:01 EDT-------
I finally found someone with a RHEL 7.2 setup. It does look like this is resolved there. :)

Comment 48 errata-xmlrpc 2015-11-19 07:07:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-2159.html

Comment 53 Kamil Dudka 2016-03-08 09:36:55 UTC
*** Bug 1314162 has been marked as a duplicate of this bug. ***