Bug 1964109

TLS_ECDHE* is broken when java-latest-openjdk is configured to use SunPKCS11-NSS-FIPS provider (in configuration similar to one used for JDK8 and JDK11 rhel-8 packages in fips mode) on fips enabled RHEL-8.

Steps to reproduce:
Can be reproduced by ssl-tests by running:

export JAVA_HOME=/usr/lib/jvm/java-16-openjdk
make clean && make TEST_PKCS11_FIPS=1 SSLTESTS_CUSTOM_JAVA_PARAMS="-Djdk.tls.ephemeralDHKeySize=2048" SSLTESTS_IGNORE_PROTOCOLS=TLSv1.3 SSLTESTS_SSL_CONFIG_FILTER=SunJSSE,Default,TLSv1.2,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
...
javax.net.ssl.SSLException: java.security.InvalidAlgorithmParameterException: init() failed
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:369)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:312)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:307)
	at java.base/sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1690)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:454)
	at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:904)
	at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:995)
	at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:955)
	at SSLSocketServer.serverLoop(SSLSocketServer.java:133)
	at SSLSocketServer$1.run(SSLSocketServer.java:75)
	at java.base/java.lang.Thread.run(Thread.java:831)
Caused by: java.security.ProviderException: java.security.InvalidAlgorithmParameterException: init() failed
	at java.base/sun.security.ssl.SSLMasterKeyDerivation$LegacyMasterKeyDerivation.deriveKey(SSLMasterKeyDerivation.java:160)
	at java.base/sun.security.ssl.KAKeyDerivation.t12DeriveKey(KAKeyDerivation.java:89)
	at java.base/sun.security.ssl.KAKeyDerivation.deriveKey(KAKeyDerivation.java:61)
	at java.base/sun.security.ssl.ECDHClientKeyExchange$ECDHEClientKeyExchangeConsumer.consume(ECDHClientKeyExchange.java:518)
	at java.base/sun.security.ssl.ClientKeyExchange$ClientKeyExchangeConsumer.consume(ClientKeyExchange.java:110)
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:458)
	at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:199)
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:171)
	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1498)
	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1404)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:441)
	... 6 more
Caused by: java.security.InvalidAlgorithmParameterException: init() failed
	at jdk.crypto.cryptoki/sun.security.pkcs11.P11TlsMasterSecretGenerator.engineInit(P11TlsMasterSecretGenerator.java:110)
	at java.base/javax.crypto.KeyGenerator.init(KeyGenerator.java:476)
	at java.base/javax.crypto.KeyGenerator.init(KeyGenerator.java:452)
	at java.base/sun.security.ssl.SSLMasterKeyDerivation$LegacyMasterKeyDerivation.deriveKey(SSLMasterKeyDerivation.java:149)
	... 18 more
Caused by: java.security.InvalidKeyException: Could not create key
	at jdk.crypto.cryptoki/sun.security.pkcs11.P11SecretKeyFactory.createKey(P11SecretKeyFactory.java:285)
	at jdk.crypto.cryptoki/sun.security.pkcs11.P11SecretKeyFactory.convertKey(P11SecretKeyFactory.java:190)
	at jdk.crypto.cryptoki/sun.security.pkcs11.P11SecretKeyFactory.convertKey(P11SecretKeyFactory.java:122)
	at jdk.crypto.cryptoki/sun.security.pkcs11.P11TlsMasterSecretGenerator.engineInit(P11TlsMasterSecretGenerator.java:108)
	... 21 more
Caused by: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_ATTRIBUTE_VALUE_INVALID
	at jdk.crypto.cryptoki/sun.security.pkcs11.wrapper.PKCS11.C_CreateObject(Native Method)
	at jdk.crypto.cryptoki/sun.security.pkcs11.P11SecretKeyFactory.createKey(P11SecretKeyFactory.java:280)
	... 24 more
...
FAILED: SunJSSE/Default: TLSv1.2 + TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384

(alternatively SSLTESTS_SSL_CONFIG_FILTER can be removed to test all cipher suites)

Package tested was java-latest-openjdk-16.0.1.0.9-1.rolling.el8.x86_64, but
same can be reproduced with local build of latest openjdk.
( Btw. using ssl-test, this can also be reproduced on fedora, with same command. )


More details:
After some debugging problem seems to be, that "key agreement" used does not come from pkcs11 provider, which later causes problems due to use of "raw" secrets.

Key agreement which is created in KAKeyDerivation class [2] actually uses sun.security.ec.XDHKeyAgreement class (latest jdk), see:

Breakpoint hit: "thread=main", sun.security.ssl.KAKeyDerivation.t12DeriveKey(), line=89 bci=100
...
main[1] dump ka
 ka = {
    debug: null
    pdebug: null
    skipDebug: false
    provider: instance of sun.security.ec.SunEC(id=2144)
    spi: instance of sun.security.ec.XDHKeyAgreement(id=2145)
    algorithm: "XDH"
    firstService: null
    serviceIterator: null
    lock: instance of java.lang.Object(id=2146)
    warnCount: 10
    I_NO_PARAMS: 1
    I_PARAMS: 2
}


Current java-11-openjdk package uses sun.security.pkcs11.P11ECDHKeyAgreement class here instead:
Breakpoint hit: "thread=main", sun.security.ssl.KAKeyDerivation.t12DeriveKey(), line=89 bci=100

main[1] dump ka
 ka = {
    debug: null
    pdebug: null
    skipDebug: false
    provider: instance of sun.security.pkcs11.SunPKCS11(id=2017)
    spi: instance of sun.security.pkcs11.P11ECDHKeyAgreement(id=2018)
    algorithm: "ECDH"
    firstService: null
    serviceIterator: null
    lock: null
    warnCount: 10
    I_NO_PARAMS: 1
    I_PARAMS: 2
}

Problem was probably introduced by JDK-8171279 [3], which added "XDH" algorithm and XDH* classes, while this does not seem to be supported by PKCS11 provider. This was also backported to JDK11, so issue will probably also appear in JDK11 rpms in the future.

[1] https://github.com/zzambers/ssl-tests
[2] https://github.com/openjdk/jdk/blob/739769c8fc4b496f08a92225a12d07414537b6c0/src/java.base/share/classes/sun/security/ssl/KAKeyDerivation.java#L73
[3] https://bugs.openjdk.java.net/browse/JDK-8171279