Created attachment 1926129 [details]
Reproducer program that uses EVP_PKEY_encapsulate() and EVP_PKEY_decapsulate()

Description of problem:
NIST SP 800-131Ar2 table 5 "Approval Status for the RSA-based Key Agreement and Key Transport Schemes" lists a minimum RSA key length of 2048 bits for SP 800-56B Key Agreement and Key Transport schemes, which is what OpenSSL implements in EVP_KEM-RSA (https://www.openssl.org/docs/man3.0/man7/EVP_KEM-RSA.html) to be used with EVP_PKEY_encapsulate() and EVP_PKEY_decapsulate().

While encapsulation correctly fails when invoked with short key lengths, decapsulation still accepts keys >= 1024 bits.

Version-Release number of selected component (if applicable):
3.0.1-44.el9 (!)

How reproducible:
Run the attached reproducer

Steps to Reproduce:
1. cc -std=c99 -Wall -Werror -pedantic -D_XOPEN_SOURCE=600 -o rsakem rsakem.c -lcrypto
2. ./rsakem 1536  # fails as expected
3. ./rsakem 1536 '?fips!=yes' ''  # succeeds, but should not when the ?fips!=yes property query only applies to encapsulation

Actual results:
$> $(head -1 rsakem.c | sed -E 's#^// ##g') && ./rsakem 1536
ERR EVP_PKEY_encapsulate_init(ctx, {OSSL_KEM_PARAM_OPERATION: RSASVE}): error:1C800069:Provider routines::invalid key length (in function ossl_rsa_check_key in providers/common/securitycheck.c:67): operation: 4096
$> $(head -1 rsakem.c | sed -E 's#^// ##g') && ./rsakem 1536 '?fips!=yes' ''
OK     recovered secret: 0x755bcab13bcf07294658ba7877e42385313fddcedce87f72f2af5fdc19f8503aa88ec61b3140eccb8a5726454f4f079aadc6f84598fc4b04d24151ca3fe63d6428c0b86c066e879d5a4a7c54f713451d9b2d6c97d6061bc5b37bb3cd670934b08e90d5a1283ffa312cdc9b1f2717535004e112a7ea770dd7012546e2c7dd80ce2311b908dc738c7e5ab31c7586b6c59bdfadc3926d130b7a0e1d2ac61453b73b6278b160ba186db6bbf8a2c387103ed6af43156defa4715ade9f4e050bb08cae
OK      original secret: 0x755bcab13bcf07294658ba7877e42385313fddcedce87f72f2af5fdc19f8503aa88ec61b3140eccb8a5726454f4f079aadc6f84598fc4b04d24151ca3fe63d6428c0b86c066e879d5a4a7c54f713451d9b2d6c97d6061bc5b37bb3cd670934b08e90d5a1283ffa312cdc9b1f2717535004e112a7ea770dd7012546e2c7dd80ce2311b908dc738c7e5ab31c7586b6c59bdfadc3926d130b7a0e1d2ac61453b73b6278b160ba186db6bbf8a2c387103ed6af43156defa4715ade9f4e050bb08cae

Expected results:
$> $(head -1 rsakem.c | sed -E 's#^// ##g') && ./rsakem 1536
ERR EVP_PKEY_encapsulate_init(ctx, {OSSL_KEM_PARAM_OPERATION: RSASVE}): error:1C800069:Provider routines::invalid key length (in function ossl_rsa_check_key in providers/common/securitycheck.c:67): operation: 4096
$> $(head -1 rsakem.c | sed -E 's#^// ##g') && ./rsakem 1536 '?fips!=yes' ''
ERR EVP_PKEY_decapsulate_init(ctx, {OSSL_KEM_PARAM_OPERATION: RSASVE}): error:1C800069:Provider routines::invalid key length (in function ossl_rsa_check_key in providers/common/securitycheck.c:67): operation: 4096


Additional info:
https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-131Ar2.pdf for the limitation
https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-56Br2.pdf section 7.2.1 for RSASVE