Bug 1022017

This came about while investigating BZ 1010111

Assigning to Andrew to take a look.

Hi Jesus, can you try a workaround while we try to fix this issue?

You need to edit /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.45.x86_64/jre/lib/security/java.security

In there, change the priorities to make sun.security.provider.Sun #1 and so on i.e. change:

security.provider.1=sun.security.pkcs11.SunPKCS11 ${java.home}/lib/security/nss.cfg
security.provider.2=sun.security.provider.Sun
security.provider.3=sun.security.rsa.SunRsaSign
...
...
security.provider.10=sun.security.smartcardio.SunPCSC

to:

security.provider.1=sun.security.provider.Sun
security.provider.2=sun.security.rsa.SunRsaSign
...
...
security.provider.10=sun.security.pkcs11.SunPKCS11 ${java.home}/lib/security/nss.cfg

(In reply to Deepak Bhole from comment #3)
> Hi Jesus, can you try a workaround while we try to fix this issue?
> 
> You need to edit
> /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.45.x86_64/jre/lib/security/java.
> security
> 
> In there, change the priorities to make sun.security.provider.Sun #1 and so
> on i.e. change:
> 
> security.provider.1=sun.security.pkcs11.SunPKCS11
> ${java.home}/lib/security/nss.cfg
> security.provider.2=sun.security.provider.Sun
> security.provider.3=sun.security.rsa.SunRsaSign
> ...
> ...
> security.provider.10=sun.security.smartcardio.SunPCSC
> 
> to:
> 
> security.provider.1=sun.security.provider.Sun
> security.provider.2=sun.security.rsa.SunRsaSign
> ...
> ...
> security.provider.10=sun.security.pkcs11.SunPKCS11
> ${java.home}/lib/security/nss.cfg

my java.security now has the following:

security.provider.1=sun.security.provider.Sun
security.provider.2=sun.security.rsa.SunRsaSign
security.provider.3=sun.security.ec.SunEC
security.provider.4=com.sun.net.ssl.internal.ssl.Provider
security.provider.5=com.sun.crypto.provider.SunJCE
security.provider.6=sun.security.jgss.SunProvider
security.provider.7=com.sun.security.sasl.Provider
security.provider.8=org.jcp.xml.dsig.internal.dom.XMLDSigRI
security.provider.9=sun.security.smartcardio.SunPCSC
security.provider.10=sun.security.pkcs11.SunPKCS11 ${java.home}/lib/security/nss.cfg

Same error:

%% Initialized:  [Session-4, SSL_NULL_WITH_NULL_NULL]
http-8443-2, handling exception: java.security.ProviderException: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_DOMAIN_PARAMS_INVALID
%% Invalidated:  [Session-4, SSL_NULL_WITH_NULL_NULL]

Yes, CKR_DOMAIN_PARAMS_INVALID is an error coming from NSS, not OpenJDK.

Jesus, what happens if you remove the security.provider.10 line altogether?

Using the following providers in java.security:

security.provider.1=sun.security.provider.Sun
security.provider.2=sun.security.rsa.SunRsaSign
security.provider.3=sun.security.ec.SunEC
security.provider.4=com.sun.net.ssl.internal.ssl.Provider
security.provider.5=com.sun.crypto.provider.SunJCE
security.provider.6=sun.security.jgss.SunProvider
security.provider.7=com.sun.security.sasl.Provider
security.provider.8=org.jcp.xml.dsig.internal.dom.XMLDSigRI
security.provider.9=sun.security.smartcardio.SunPCSC
# the NSS security provider was not enabled for this build; it can be enabled
# if NSS (libnss3) is available on the machine. The nss.cfg file may need
# editing to reflect the location of the NSS installation.
#security.provider.1=sun.security.pkcs11.SunPKCS11 ${java.home}/lib/security/nss.cfg

I no longer get the CKR_DOMAIN_PARAMS_INVALID:

%% Initialized:  [Session-4, SSL_NULL_WITH_NULL_NULL]
%% Negotiating:  [Session-4, TLS_RSA_WITH_AES_256_CBC_SHA]
*** ServerHello, TLSv1.2

And when I connect with RestClient I get the actual JSON response I expected:

RestClient.get("https://localhost:8443/candlepin/status")
=> "{\"result\":true,\"version\":\"0.8.26\",\"rulesVersion\":\"4.2\",\"release\":\"1\",\"standalone\":true,\"timeUTC\":\"2013-10-25T18:17:16.391+0000\",\"managerCapabilities\":[\"cores\",\"ram\",\"instance_multiplier\",\"derived_product\",\"cert_v3\"],\"rulesSource\":\"DEFAULT\"}"

Interesting.  What is happening here is the following:

1.  The client tries to get a crypto provider that supports SSL_NULL_WITH_NULL_NULL.
2.  If the NSS provider is not present, no provider responds to this request and the client requests TLS_RSA_WITH_AES_256_CBC_SHA instead.
3.  If the NSS provider is present, instead of saying it doesn't support the algorithm it throws an exception which is carried up to the client.

So the NSS provider is giving the wrong response.

So is this a bug in NSS then, or do you mean NSS provider as in OpenJDK code that interfaces with NSS that is misinterpreting what low-level NSS libraries are saying?

Seems like the latter, but I won't know 100% until I have time to look into this fully.

I think https://bugzilla.redhat.com/show_bug.cgi?id=1022950 is related, if not the same issue.

This is the difference on Jesus' machine when the PKCS11 NSS provider is enabled and when it isn't:

+TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
+TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
 TLS_RSA_WITH_AES_256_CBC_SHA256
+TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384
+TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384
 TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
 TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
+TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
+TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
 TLS_RSA_WITH_AES_256_CBC_SHA
+TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA
+TLS_ECDH_RSA_WITH_AES_256_CBC_SHA
 TLS_DHE_RSA_WITH_AES_256_CBC_SHA
 TLS_DHE_DSS_WITH_AES_256_CBC_SHA
+TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
+TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
 TLS_RSA_WITH_AES_128_CBC_SHA256
+TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256
+TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256
 TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
 TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
+TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
+TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
 TLS_RSA_WITH_AES_128_CBC_SHA
+TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA
+TLS_ECDH_RSA_WITH_AES_128_CBC_SHA
 TLS_DHE_RSA_WITH_AES_128_CBC_SHA
 TLS_DHE_DSS_WITH_AES_128_CBC_SHA
+TLS_ECDHE_ECDSA_WITH_RC4_128_SHA
+TLS_ECDHE_RSA_WITH_RC4_128_SHA
 SSL_RSA_WITH_RC4_128_SHA
+TLS_ECDH_ECDSA_WITH_RC4_128_SHA
+TLS_ECDH_RSA_WITH_RC4_128_SHA
+TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA
+TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
 SSL_RSA_WITH_3DES_EDE_CBC_SHA
+TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA
+TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA
 SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA
 SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA
 SSL_RSA_WITH_RC4_128_MD5
 TLS_EMPTY_RENEGOTIATION_INFO_SCSV
 TLS_DH_anon_WITH_AES_256_CBC_SHA256
+TLS_ECDH_anon_WITH_AES_256_CBC_SHA
 TLS_DH_anon_WITH_AES_256_CBC_SHA
 TLS_DH_anon_WITH_AES_128_CBC_SHA256
+TLS_ECDH_anon_WITH_AES_128_CBC_SHA
 TLS_DH_anon_WITH_AES_128_CBC_SHA
+TLS_ECDH_anon_WITH_RC4_128_SHA
 SSL_DH_anon_WITH_RC4_128_MD5
+TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA
 SSL_DH_anon_WITH_3DES_EDE_CBC_SHA
 TLS_RSA_WITH_NULL_SHA256
+TLS_ECDHE_ECDSA_WITH_NULL_SHA
+TLS_ECDHE_RSA_WITH_NULL_SHA
 SSL_RSA_WITH_NULL_SHA
+TLS_ECDH_ECDSA_WITH_NULL_SHA
+TLS_ECDH_RSA_WITH_NULL_SHA
+TLS_ECDH_anon_WITH_NULL_SHA
 SSL_RSA_WITH_NULL_MD5
 SSL_RSA_WITH_DES_CBC_SHA
 SSL_DHE_RSA_WITH_DES_CBC_SHA

So, with it enabled, the SSL connection is trying to use TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 and failing because NSS doesn't actually support it.

I didn't get the ECC algorithms on my local RHEL machine (latest 6.4).  Has there been a change in NSS?

* Sun May 26 2013 Elio Maldonado <emaldona> - 3.14.3-5
- Syncup with uptream changes for aes gcm and ecc suiteb
- Enable ecc support for suite b
- Apply several upstream AES GCM fixes
- Use the pristine nss upstream sources with ecc included
- Export NSS_ENABLE_ECC=1 in both the build and the check sections
- Make failed requests for unsupoprted ssl pkcs 11 bypass non fatal
- Resolves: rhbz#882408 - NSS_NO_PKCS11_BYPASS must preserve ABI
- Related:  rhbz#918950  - rebase nss to 3.14.3

6.4 is on 3.14.3-4, the change before the above.
6.5 is on 3.15.1-15, so includes the above.

(In reply to Andrew John Hughes from comment #16)
> I think https://bugzilla.redhat.com/show_bug.cgi?id=1022950 is related, if
> not the same issue.
> 
> This is the difference on Jesus' machine when the PKCS11 NSS provider is
> enabled and when it isn't:
> 
>[snip]
> 
> So, with it enabled, the SSL connection is trying to use
> TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 and failing because NSS doesn't
> actually support it.
> 
> I didn't get the ECC algorithms on my local RHEL machine (latest 6.4).  Has
> there been a change in NSS?

yes, NSS in 6.5 introduced support for TLSv1.2 and ECC.
But the support is not complete.

In case of TLSv1.2 two features are not supported:
 * GCM
 * SHA384 as MAC
In case of ECC, only three curves are supported: nistp256, nistp384, nistp521.

so TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 won't work

Yeah, that's what I just worked out, tracing through the code :)

Can you tell us which of the additions in the list above should work, if any?  It looks like we're going to need to find where this list is created and patch out the ones our NSS doesn't support.

We came across an issue with the curves before.  Oracle's upstream test cases for this provider expect additional curves.  See:

http://icedtea.classpath.org/bugzilla/show_bug.cgi?id=469

There are basically only four suites that won't work:

> +TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
> +TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
> +TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384
> +TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384

rest of additions should work.

Created attachment 821065 [details]
Specify only the three supported curves

The attached patch should resolve the issue.

Looking at the code, I don't think NSS is being used for its TLS 1.2 implementation at all:

"* TLS 1.2 uses a different hash algorithm than 1.0/1.1 for the PRF calculations.  As of 2010, there is no PKCS11-level support for TLS 1.2 PRF calculations, and no known OS's have an internal variant we could use.  Therefore for TLS 1.2, we are updating JSSE to request different provider algorithms (e.g. "SunTls12Prf"), and currently only SunJCE has these TLS 1.2 algorithms. If we reused the names such as "SunTlsPrf", the PKCS11 providers would need be updated to fail correctly when presented with the wrong version number (via Provider.Service.supportsParameters()), and we would also need to add the appropriate supportsParamters() checks into KeyGenerators (not currently there).In the future, if PKCS11 support is added, we will restructure this."

                                                                           
which is where the mismatch occurs.  Instead of probing providers for which curves they support, the OpenJDK TLS code hardcodes a set list and uses that in its implementation of the Hello ECC curves extension.

The fix just cuts that list down to the three NSS curves.  I don't know why this wasn't designed so as to use an API to ask providers for the curves they support.  We obviously can't add it to 7 now.

Why did you change the assignment?

As Lukas mentioned above, using the java from comment #23, java-1.7.0-openjdk-1.7.0.45-2.4.3.4.el6.x86_64 I get:

candlepin      FAIL   SSL_connect returned=1 errno=0 state=SSLv3 read server key exchange B: EC lib 
candlepin_auth FAIL   SSL_connect returned=1 errno=0 state=SSLv3 read server key exchange B: EC lib 


This is slightly different than the *old* message which was a generic tlsv1 internal error:

candlepin      FAIL   SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: tlsv1 alert internal error 
candlepin_auth FAIL   SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: tlsv1 alert internal error

Using the RestClient method I get the following:

# scl enable ruby193 'irb'
irb> require 'rest-client'
irb> RestClient.get("https://localhost:8443/candlepin/status")
OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv3 read server key exchange B: EC lib

Created attachment 823039 [details]
SSL debug from tomcat6 for comment #26

The third attachment contains the SSL debug from tomcat for the comment run on comment #26.

I'm a little baffled that this:

Extension elliptic_curves, curve names: {sect571r1, sect571k1, secp521r1, sect409k1, sect409r1, secp384r1, sect283k1, sect283r1, secp256k1, secp256r1, sect239k1, sect233k1, sect233r1, secp224k1, secp224r1, sect193r1, sect193r2, secp192k1, secp192r1, sect163k1, sect163r1, sect163r2, secp160k1, secp160r1, secp160r2}

is still present in the output.  Is this definitely from the current run?  Have both the server and client been restarted?

(In reply to Andrew John Hughes from comment #28)
> I'm a little baffled that this:
> 
> Extension elliptic_curves, curve names: {sect571r1, sect571k1, secp521r1,
> sect409k1, sect409r1, secp384r1, sect283k1, sect283r1, secp256k1, secp256r1,
> sect239k1, sect233k1, sect233r1, secp224k1, secp224r1, sect193r1, sect193r2,
> secp192k1, secp192r1, sect163k1, sect163r1, sect163r2, secp160k1, secp160r1,
> secp160r2}
> 
> is still present in the output.  Is this definitely from the current run? 
> Have both the server and client been restarted?

Yeah, I rebooted the machine completely. Same result.

java -version
java version "1.7.0_45"
OpenJDK Runtime Environment (rhel-2.4.3.4.el6-x86_64 u45-b15)
OpenJDK 64-Bit Server VM (build 24.45-b08, mixed mode)

rpm -q nss
nss-3.15.1-15.el6.x86_64

Yeah, I don't think that's the cause, just wondering why it shows up.

Once I can upgrade to 6.5, I'll start testing/debugging this locally.

Ok, I have 6.5 now (after updating nearly broke my entire install...) but:

]# yum install katello-headpin-all
Loaded plugins: auto-update-debuginfo, presto, product-id, refresh-packagekit, security, subscription-manager
This system is receiving updates from Red Hat Subscription Management.
Setting up Install Process
No package katello-headpin-all available.

Is there an extra repo I need for this?

# /usr/bin/java -version
java version "1.7.0_45"
OpenJDK Runtime Environment (rhel-2.4.3.3.el6-x86_64 u45-b15)
OpenJDK 64-Bit Server VM (build 24.45-b08, mixed mode)

# scl enable ruby193 'irb'
irb(main):001:0> require 'rest-client'
=> true
irb(main):002:0> RestClient.get("https://localhost:8443/candlepin/status")
=> "{\"result\":true,\"version\":\"0.8.26\",\"rulesVersion\":\"4.2\",\"release\":\"1\",\"standalone\":true,\"timeUTC\":\"2013-11-28T00:00:16.894+0000\",\"managerCapabilities\":[\"cores\",\"ram\",\"instance_multiplier\",\"derived_product\",\"cert_v3\"],\"rulesSource\":\"DEFAULT\"}"

# rpm -q nss
nss-3.15.1-15.el6.x86_64

# rpm -q java-1.7.0-openjdk
java-1.7.0-openjdk-1.7.0.45-2.4.3.3.el6.x86_64

I'm not seeing the error.

I've managed to reproduce it on there too, but I can't see any obvious differences.  It's mystifying.

Are both ends of the transmission Java clients?  I did notice that lsof shows postgres, sendmail, cups and ssh to have references to a deleted version of NSS.

If anyone has a simpler reproducer, it would be very welcome :)

As a workaround, there is a property we could change the default of and thus turn off ECC.  Thoughts?

Ok, I'm pretty sure I've found the issue here and it's nothing to do with either Java or even NSS.

I can access the URL fine on  both machines with wget and curl.  The same issue I patched earlier has also been fixed in OpenSSL (used by your ruby client) recently:

* Thu Oct 31 2013 Tomáš Mráz <tmraz> 1.0.1e-16
- do not advertise ECC curves we do not support
- fix CPU identification on Cyrix CPUs

My box:

Name        : openssl                      Relocations: (not relocatable)
Version     : 1.0.1e                            Vendor: Red Hat, Inc.
Release     : 16.el6_5                      Build Date: Mon 04 Nov 2013 16:10:09 GMT

Box from Lukas:

Name        : openssl                      Relocations: (not relocatable)
Version     : 1.0.1e                            Vendor: Red Hat, Inc.
Release     : 15.el6                        Build Date: Fri 27 Sep 2013 10:13:23 EDT

After a yum upgrade of openssl, surprise, surprise...

irb(main):001:0> require 'rest-client'
=> true
irb(main):002:0> RestClient.get("https://localhost:8443/candlepin/status")
=> "{\"result\":true,\"version\":\"0.8.26\",\"rulesVersion\":\"4.2\",\"release\":\"1\",\"standalone\":true,\"timeUTC\":\"2013-12-04T14:06:23.328+0000\",\"managerCapabilities\":[\"cores\",\"ram\",\"instance_multiplier\",\"derived_product\",\"cert_v3\"],\"rulesSource\":\"DEFAULT\"}"

i.e. basically your OpenSSL implementation was saying it supports nist512 when it doesn't.  Thus, when the Java side sent it data encrypted with a nist512 curve, it failed.

Jesus, can you confirm that everything works now on your end with the new OpenSSL? If so, we will close this issue as NOTABUG.

Deepak, there was still a bug to begin with.  It just means our patch did fix it.

(In reply to Andrew John Hughes from comment #40)
> Deepak, there was still a bug to begin with.  It just means our patch did
> fix it.

Ah, right, sorry.

Jesus, if this works for you then, can you please close it as CURRENTRELEASE?

Yes that is correct.  The patch changes it to the shorter output to match NSS.

Could you guys please send the bugfix, or a link to this bugreport to the openssl guys? Your bugfix is quite worthless to the rest of the world if you don't report this bug upstream and keeping the fix to yourselves. I had to disable nss via. the java.security file in order to use mcabber with my jabber server, because I had this exact exception on a debian box with openfire running on top of it.

(In reply to rhbugzilla.5.nebuchadnezar from comment #46)
> Could you guys please send the bugfix, or a link to this bugreport to the
> openssl guys? Your bugfix is quite worthless to the rest of the world if you
> don't report this bug upstream and keeping the fix to yourselves. I had to
> disable nss via. the java.security file in order to use mcabber with my
> jabber server, because I had this exact exception on a debian box with
> openfire running on top of it.

The fix is not applicable to upstream. Upstream does does not remove support for certain Elliptic Curves. That is something specific to RHEL packages.

(In reply to rhbugzilla.5.nebuchadnezar from comment #46)
> Could you guys please send the bugfix, or a link to this bugreport to the
> openssl guys? Your bugfix is quite worthless to the rest of the world if you
> don't report this bug upstream and keeping the fix to yourselves. I had to
> disable nss via. the java.security file in order to use mcabber with my
> jabber server, because I had this exact exception on a debian box with
> openfire running on top of it.

This bug is not about OpenSSL.  You need to comment on their bug report.

It is clearly about openssl, or nss. At least the same reason why this bug report was opened. The workaround proposed here, worked for openfire too. It is something about the handling of tls 1.2. Openfire can't do anything about it, if the error is somewhere below the jvm, isn't it. Unfortunately I'm not well versed with nss/openssl, so I don't really know to whom I should report this, but I figured, due to the fact that the workaround provided here worked too, I can only conclude that the source of the problem is the same; or at least something very similar. Sorry, if I sounded angry before, but it looked to me like, .. nah don't bother with reporting this upstream, we just patch it ourselves, but whatever; I'm a bit helpless here to whom this should be reported. At least, the workaround helped.

(In reply to rhbugzilla.5.nebuchadnezar from comment #50)
> It is clearly about openssl, or nss. At least the same reason why this bug
> report was opened. The workaround proposed here, worked for openfire too. It
> is something about the handling of tls 1.2. Openfire can't do anything about
> it, if the error is somewhere below the jvm, isn't it. Unfortunately I'm not
> well versed with nss/openssl, so I don't really know to whom I should report
> this, but I figured, due to the fact that the workaround provided here
> worked too, I can only conclude that the source of the problem is the same;
> or at least something very similar. Sorry, if I sounded angry before, but it
> looked to me like, .. nah don't bother with reporting this upstream, we just
> patch it ourselves, but whatever; I'm a bit helpless here to whom this
> should be reported. At least, the workaround helped.

This is a bug related to java-1.7.0-openjdk, so no it's not "clearly about openssl or nss".

As far as RHEL is concerned, this issue is "resolved", in the sense that we have reverted the use of NSS as the default provider due to other issues with it. Marking it so.

This is also tracked on Launchpad:

https://bugs.launchpad.net/centos/+bug/1006776

Thanks for the additional information regarding that this is caused by a change in NSS (not Java™ or so) adding partial/incomplete ECC support.

Re-opening this as we appear to have regressed on this issue with the java-1.8.0-openjdk package. We now ship the SunEC provider which also uses NSS (see bug 1208307) and the same three curves, but the patch from comment #21 was never forward-ported to the OpenJDK 8 packages. It was already in the OpenJDK 7 packages when SunEC support was added, so it was missed as not being one of the changes we made then.

I saw upstream tracks this issue in https://bugs.openjdk.java.net/browse/JDK-8160342 and marka as fix version 9.1 [ 18400 ]

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0571.html