Bug 1882185

+++ This bug was initially created as a clone of Bug #1882168 +++

Problem:
System crypto policy fails to increase DH key size [1], based on crypto policy requirements [2].

Seems like crypto-policies are actually trying to increase DH key size using jdk.tls.ephemeralDHKeySize property:
cat /etc/crypto-policies/back-ends/java.config | grep DHKeySize
jdk.tls.ephemeralDHKeySize=2048

However this property is not applied since this property ends up in Security properties [3], but it needs to be in System properties [4] to be applied.

Issue was discovered wile running ssl-tests [5] in FIPS mode, where crypto policy fails to set bigger key size, but constrains on key size as enforced by nss (used by pkcs11 provider) are causing this exception:

Caused by: java.security.ProviderException: sun.security.pkcs11.wrapper.PKCS11Exception: CKR_ARGUMENTS_BAD
    at sun.security.pkcs11.P11KeyPairGenerator.generateKeyPair(P11KeyPairGenerator.java:424)
    at java.security.KeyPairGenerator$Delegate.generateKeyPair(KeyPairGenerator.java:697)
    at sun.security.ssl.DHKeyExchange$DHEPossession.generateDHKeyPair(DHKeyExchange.java:181)
    at sun.security.ssl.DHKeyExchange$DHEPossession.<init>(DHKeyExchange.java:139)
    at sun.security.ssl.DHKeyExchange$DHEPossessionGenerator.createPossession(DHKeyExchange.java:389
...
FAILED: SunJSSE/Default: TLSv1.2 + TLS_DHE_RSA_WITH_AES_256_CBC_SHA256

(This one was thrown on servers side, I am not sure if exception can also be thrown on client side.)


Workaround:
Set this property manually using command line argument, eg.:
java -Djdk.tls.ephemeralDHKeySize=2048 ...


[1] https://hg.openjdk.java.net/jdk-updates/jdk11u/file/e16ef8c081c6/src/java.base/share/classes/sun/security/ssl/DHKeyExchange.java#l318
[2] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/using-the-system-wide-cryptographic-policies_security-hardening
[3] https://docs.oracle.com/javase/8/docs/api/java/security/Security.html
[4] https://docs.oracle.com/javase/8/docs/api/java/lang/System.html
[5] https://github.com/zzambers/ssl-tests

--- Additional comment from  on 2020-09-24 03:01:15 CEST ---

But it seems there is also bug in LEGACY crypto policy (invalid key size of 1023):

cat /usr/share/crypto-policies/LEGACY/java.txt | grep DHKeySize
jdk.tls.ephemeralDHKeySize=1023

which would cause this exception, if it was actually used:

Exception in thread "Thread-0" java.lang.ExceptionInInitializerError
	at sun.security.ssl.DHKeyExchange.<clinit>(DHKeyExchange.java:57)
	at sun.security.ssl.SSLKeyExchange$T12KeyAgreement.<clinit>(SSLKeyExchange.java:348)
	at sun.security.ssl.SSLKeyExchange$1.<clinit>(SSLKeyExchange.java:411)
	at sun.security.ssl.SSLKeyExchange.valueOf(SSLKeyExchange.java:203)
	at sun.security.ssl.ServerHello$T12ServerHelloProducer.chooseCipherSuite(ServerHello.java:421)
	at sun.security.ssl.ServerHello$T12ServerHelloProducer.produce(ServerHello.java:295)
	at sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:421)
	at sun.security.ssl.ClientHello$T12ClientHelloConsumer.consume(ClientHello.java:1020)
	at sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:727)
	at sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:693)
	at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377)
	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422)
	at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182)
	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149)
	at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143)
	at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394)
	at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708)
	at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72)
	at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791)
	at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:750)
	at SSLSocketServer.serverLoop(SSLSocketServer.java:132)
	at SSLSocketServer$1.run(SSLSocketServer.java:74)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Unsupported customized DH key size: 1023. The key size must be multiple of 64, and range from 1024 to 8192 (inclusive)
	at sun.security.ssl.DHKeyExchange$DHEPossessionGenerator.<clinit>(DHKeyExchange.java:280)
	... 25 more

[1] https://hg.openjdk.java.net/jdk-updates/jdk11u/file/e16ef8c081c6/src/java.base/share/classes/sun/security/ssl/DHKeyExchange.java#l280