Note that this also happens on the client side as well:

 - TLS_DHE_RSA_WITH_AES_128_CBC_SHA :: Failed
Exception in thread "main" javax.net.ssl.SSLException: java.security.ProviderException: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_ATTRIBUTE_VALUE_INVALID
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
	at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
	at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1903)
	at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1886)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1402)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
	at Client.testHandshake(Client.java:377)
	at Client.main(Client.java:425)
Caused by: java.security.ProviderException: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_ATTRIBUTE_VALUE_INVALID
	at sun.security.pkcs11.P11KeyPairGenerator.generateKeyPair(P11KeyPairGenerator.java:424)
	at java.security.KeyPairGenerator$Delegate.generateKeyPair(KeyPairGenerator.java:697)
	at sun.security.ssl.ECDHCrypt.<init>(ECDHCrypt.java:78)
	at sun.security.ssl.ClientHandshaker.serverKeyExchange(ClientHandshaker.java:783)
	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:302)
	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
	at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
	... 15 more
Caused by: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_ATTRIBUTE_VALUE_INVALID
	at sun.security.pkcs11.wrapper.PKCS11.C_GenerateKeyPair(Native Method)
	at sun.security.pkcs11.P11KeyPairGenerator.generateKeyPair(P11KeyPairGenerator.java:416)
	... 24 more


I have no clue why though.

If you hack Crypto-Policies to allow non-PFS cipher suites (like TLS_RSA_WITH_AES_128_CBC_SHA -- which https://google.com and the JDK both support), you get a similar error, but this time when trying to derive the premaster secret:

*** ServerHelloDone
[read] MD5 and SHA1 hashes:  len = 4
0000: 0E 00 00 00                                        ....
Provider: KeyGenerator.SunTls12RsaPremasterSecret algorithm from: SunPKCS11-NSS-FIPS
main, handling exception: java.security.ProviderException: Could not generate premaster secret
%% Invalidated:  [Session-9, TLS_RSA_WITH_AES_128_CBC_SHA]
main, SEND TLSv1.2 ALERT:  fatal, description = internal_error
main, WRITE: TLSv1.2 Alert, length = 2
[Raw write]: length = 7
0000: 15 03 03 00 02 02 50                               ......P
main, called closeSocket()
 - TLS_RSA_WITH_AES_128_CBC_SHA :: Failed
java.security.ProviderException: Could not generate premaster secret
javax.net.ssl.SSLException: java.security.ProviderException: Could not generate premaster secret
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
    at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
    at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1903)
    at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1886)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1402)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396)
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355)
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373)
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
    at Client.testHandshake(Client.java:375)
    at Client.main(Client.java:424)
Caused by: java.security.ProviderException: Could not generate premaster secret
    at sun.security.pkcs11.P11TlsRsaPremasterSecretGenerator.engineGenerateKey(P11TlsRsaPremasterSecretGenerator.java:114)
    at javax.crypto.KeyGenerator.generateKey(KeyGenerator.java:540)
    at sun.security.ssl.RSAClientKeyExchange.<init>(RSAClientKeyExchange.java:81)
    at sun.security.ssl.ClientHandshaker.serverHelloDone(ClientHandshaker.java:972)
    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:369)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
    at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
    ... 15 more
Caused by: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_ATTRIBUTE_VALUE_INVALID
    at sun.security.pkcs11.wrapper.PKCS11.C_GenerateKey(Native Method)
    at sun.security.pkcs11.P11TlsRsaPremasterSecretGenerator.engineGenerateKey(P11TlsRsaPremasterSecretGenerator.java:108)
    ... 24 more


So I think this isn't specific to DHE / ECDHE but instead to C_GenerateKey.

Created attachment 1643656 [details]
Reproducer - TLS Client side

Check README.txt in the archive.

@Alex: thanks for providing the steps to reproduce.

I could not reproduce in my local development environment with the legacy DB. Will test on a clean RHEL 8.2 with the new DB now.

Note: from the list of ciphersuites on the test, I removed those containing GCM mode which is currently unimplemented in java-1.8.0-openjdk's SunPKCS11. 4 ciphersuites were successful. 8 ciphersuites were rejected by the server.

A couple of questions meanwhile:

 1) Does it reproduce on java-11-openjdk?

 2) Have you made any change to nss.fips.cfg file in addition to the NSS DB path? In particular, I had the hypothesis of NSS returning CKR_ATTRIBUTE_VALUE_INVALID on C_GenerateKey and C_GenerateKeyPair calls because of the public/private key template containing CKA_SENSITIVE to "false" (which can be caused by using "attributes = compatibility" in the NSS cfg file).

Thanks,
Martin.-

Ah sorry, thought I added that to this bug.

My nss.fips.cfg was:

> name = NSS-FIPS
> nssLibraryDirectory = /usr/lib64
> nssSecmodDirectory = /nssdb
> nssDbMode = readWrite
> nssModule = fips
> attributes = compatibility


Removing

> attributes = compatibility

does indeed fix this. My understanding was that this was equivalent to NSS's NSS_INIT_COOPERATE flag, but I guess that isn't correct.


Is there any way to set NSS_INIT_COOPERATE flag? I want to open two separate NSS DBs (effecitvely initializing NSS twice -- once for the SunPKCS11 provider and once for JSS). I can access the JSS DB if I initialize it first, but then the SunPKCS11 provider fails with:

> Caused by: sun.security.validator.ValidatorException: No trusted certificate found

Because it doesn't open the NSS DB in nss.fips.cfg.

If I initialize SunPKCS11 provider first, then it can only see certificates in the first NSS DB. 



I guess it'd be sufficient to set nssSecmodDirectory to the same value as what we're using for JSS, but that'd require bz#1780335 being solved to not break HSM and ACME installs.

- Alex

attributes = compatibility adds a few attributes to the template for generated keys. Among these attributes there is CKA_SENSITIVE set to false, which is convenient to extract key values and use them on other security providers but is not allowed when the NSS software token is configured in FIPS mode.

In regards to NSS_INIT_COOPERATE, looks to me that OpenJDK does not pass this flag when initializing NSS (only NSS_INIT_OPTIMIZESPACE, NSS_INIT_READONLY, NSS_INIT_NOCERTDB, NSS_INIT_NOMODDB, NSS_INIT_FORCEOPEN, NSS_INIT_NOROOTINIT may be passed depending on the configuration set).

I'm not sure of understanding your use-case correctly. Do you want SunPKCS11 to be able to access both DBs?

Did SunPKCS11 have access to JSS's DB even when its nssSecmodDirectory configuration was pointing to its own DB? This looks unexpected to me. I'd be grateful if you can elaborate more on that.

> Do you want SunPKCS11 to be able to access both DBs?

No; I want NSS SunPKCS11 to access a separate NSSDB from JSS.

In particular, because of bz#1780335, I'd like the following NSS DB setup (pardon the notation):

 /var/lib/pki/pki-tomcat/alias  --- Accessed from JSS only
   - CA certificate
   - CA key
   - (other certificates and keys)
   - <possibly HSM module>
   - <possibly p11-kit-trust module>

 /var/lib/pki/pki-tomcat/sunpkcs11   --- Accessed from SunPKCS11 only
   - CA certificate
   - (Intermediate CA certificate)
   - SSL Server certificate
   - SSL Server key
   - <no PKCS#11 modules besides softoken>

Our use case for SunPKCS11 is only for handling Tomcat's inbound (server-side) SSL connections.

 - bz#1780335 means that our usual NSS DB doesn't work (because it potentially has other modules in it)
 - bz#1759335 means that we can't easily control selection of certificates; since our NSS DB can potentially have lots of certificates for lots of services, selecting the certificate by alias is important.

> Did SunPKCS11 have access to JSS's DB even when its nssSecmodDirectory configuration was pointing to its own DB?

No. But both had access to a common DB when nssSecmodDirectory == JSS's NSS DB. That's--I believe--one of the things NSS_INIT_COOPERATE is supposed to help with ensuring no conflicts happen.

But when that common DB is a production NSS DB and not a sample one created for testing... We run into the above two bugs.


--- 

And even if we go the route of not using SunJSSE + SunPKCS11-NSS-FIPS for the SSLContext/SSLEngine and build it in JSS... it still seems very... fragile and brittle that if someone beats us and accidentally initializes NSS before us... we could get loaded with /etc/pki/nssdb (the default and almost always empty NSS DB) instead of the NSS DB we actually want. The only way around that would be to disable FIPS mode for the JDK, always (in case y'all re-enable it by default and quit making it conditional on -Dcom.redhat.fips=true being passed). Which'd raise an eyebrow or two and would mean we'd have to carefully police dependencies to ensure they use our (Mozilla-JSS) provider instead of the JDK's providers. Or we'd have to give SunPKCS11-NSS-FIPS access to our NSS DB (which then we hit bz#1780335 again).

---

Either way, seems best to figure out how to get these two providers to play nicely together. Maybe we need an RFE against NSS, idk. I'll have to ask Bob and play with it some more. 


- A

Yes, I see the problem you are facing. OpenJDK and NSS in FIPS mode are quite limited in regards to interacting with other security providers.

In summary:

 1) SunPKCS11-NSS-FIPS is not currently able to use non-PKCS11 keystores (bz#1759335)
 2) SunPKCS11-NSS-FIPS is not currently able to use NSSDBs when external modules are installed (bz#1780335)
 3) SunPKCS11-NSS-FIPS does not support "attributes = compatibility" (which is CKA_SENSIVE = false) to extract and move keys easily (bz#1780339)

Most of these limitations -if not all of them- come from the NSS side. We can try to workaround them from OpenJDK (as proposed for bz#1759335) or use a different way of providing FIPS compliance in OpenJDK in the future.

I understand that moving certificates/keys to SunPKCS11-NSS-FIPS's NSSDB has downsides and is inconvenient to you.

I will see how can we work on #1 or #2. That said, #1 won't be ready for RHEL 8.2 and #2 is still uncertain to me (will try to investigate before the end of this week).

Does JSS support NSS in FIPS mode? One option would be that you don't use SunPKCS11-NSS-FIPS at all (-Dcom.redhat.fips=false) and you go your own way. At the end of the day, you would be using NSS (which is a FIPS compliant library in RHEL). If you get crypto-team approval, that should be fine.

Thanks for your time and investigations!


> Does JSS support NSS in FIPS mode?

Yes and it has for a long while. It's supported HSMs as well. I thought one of the benefits of using PKCS#11 bindings was that it made using external modules easier, not harder... :-)

One of the key technical differences between JSS and the JDK is that JSS isn't shy of using all NSS features. We embrace NSS as a dependency and make full use of it. For instance, JSS's org.mozilla.jss.ssl.SSLSocket (which predates the creation of the JDK's own javax.net.ssl.SSLSocket) uses NSS's high level SSL_* functions, rather than doing TLS with primitives from PKCS#11 like javax.net.ssl.SSLEngine does. We also generally prefer NSS's high-level PK11_* functions over directly calling into softoken (or maintaining our own PKCS#11 layer). 

This has admittedly lead to some issues while creating an SSLEngine of our own... But it also means we don't care as much when e.g., a new TLS release drops or recommended cipher suites change. It also makes writing the high-level certificate management and other functionality (CMAC, KBKDF integration) much easier.

For context, this whole conversation sprung out of a Tomcat decision: Tomcat <= 8.0 supports the older BIO adapters, but Tomcat 8.5+ (shipping Tomcat 9.0) only supports NIO -- concretely, that means we need an SSLEngine now. JSS hasn't had one, though one is in-progress. So while our product does a lot of other crypto, we'd have liked to insert SunPKCS11-NSS-FIPS _only_ in the SSLEngine slot for Tomcat. 

So perhaps we are best off writing and maintaining our own....

I'll close this ticket as the reported behavior is the one expected when using the configuration "attributes = compatibility", as previously discussed.