Description of problem:

As part of the FIPS auto-configuration feature in RHEL's OpenJDK package, two non-FIPS compliant security providers are enabled: 1) SUN, and 2) SunEC. The former is needed to handle PKCS#12 trust-stores (where trusted certificate anchors are located) and X.509 certificates (including certificate path validation). The latter is needed as a SunPKCS11 helper to handle public EC key values and EC algorithm parameters.

The problem with enabling the aforementioned security providers is that they bring with them several non-FIPS-compliant crypto primitives. In example, SUN offers Signature services with algorithms such as SHA<X> with DSA. To mitigate this issue, non-FIPS compliant providers are lower in the priority list. A user requiring an algorithm which is implemented in SunPKCS11 will get a FIPS-compliant service by default. At the same time, users are advised not to get crypto services from non-FIPS compliant security providers by explicitly specifying their name in the ::getInstance services APIs.

The first thing to notice here is that if an algorithm is not implemented in SunPKCS11 but it is in SUN or SunEC, a non-FIPS compliant implementation may be obtained. The most likely development here is that the private key (which is 'sensitive' in PKCS#11 terms) cannot be used there because it cannot be extracted out of the NSS Software Token in plain (for a 'translation'), and an error will be thrown. While this can be immediately noticed, it can turn into a problem for the TLS engine as I will describe below.

After JDK-13, the TLS engine cannot be setup to force its crypto to be obtained from a single security provider (i.e. SunPKCS11). This is because the 'Experimental FIPS mode' was removed in the context of JDK-8217835 [1]. Thus, a TLS client may decide that a ciphersuite is available because there is an implementation for the algorithm provided by SUN or SunEC, and offer it to the server. The server may choose it and the Handshake later fails on the client side, while trying to use a PKCS#11 key on the non-FIPS compliant algorithm implementation. There are no re-tries at this point and the connection won't be established.

Proposed solution:

RHEL's OpenJDK has already the capability of detecting if the system is configured in FIPS mode or not -and, implicitly, if system FIPS alignment was not disabled by the user-. That can be achieved by means of calling 'SharedSecrets.getJavaSecuritySystemConfiguratorAccess().isSystemFipsEnabled()'. The proposed solution is to modify the services registration in SUN and SunEC providers to enable only the services required when in FIPS mode. In example, under FIPS mode the SUN provider won't be registering any Signature service (which includes SHA-<X> with DSA among other algorithms); but it will register certificate-related services.

--
[1] - https://bugs.openjdk.java.net/browse/JDK-8217835