Description of problem:
FIPS 186-4, section 5 "The RSA Digital Signature Algorithm", subsection 5.5 "PKCS #1" says:

For RSASSA-PSS […] the length (in bytes) of the salt (sLen) shall satisfy 0 ≤ sLen ≤ hLen, where hLen is the length of the hash function output block (in bytes).

Our current copy of OpenSSL does not enforce this.

Version-Release number of selected component (if applicable):
3.0.1-43.el9_0

How reproducible:


Steps to Reproduce:
1. openssl genpkey -algorithm rsa -pkeyopt rsa_keygen_bits:2048 -out rsa2048.pem
2. should work: openssl dgst -sha256 -sign rsa2048.pem -sigopt rsa_padding_mode:pss -sigopt rsa_pss_saltlen:32 -out rsa2048.pem.sha256sig rsa2048.pem
3. should not work: openssl dgst -sha256 -sign rsa2048.pem -sigopt rsa_padding_mode:pss -sigopt rsa_pss_saltlen:33 -out rsa2048.pem.sha256sig rsa2048.pem, because the output length of SHA256 is 32 bytes.

Actual results:
Both commands succeed

Expected results:
Signature parameter error "rsa_pss_saltlen:33"
80BB4A493E7F0000:error:1C800070:Provider routines:rsa_set_ctx_params:invalid salt length:providers/implementations/signature/rsa_sig.c:1302:
or an indication that this is not a FIPS-approved operation

Additional info:
Older versions of OpenSSL default to using the maximum salt length supported by the RSA key length, so implementing an indicator may be preferable here.