Description of problem: When the openssl smime command is used to encrypt email in FIPS mode, it still uses 3DES. Since that algorithm is not allowed in FIPS mode, the operation fails. Version-Release number of selected component (if applicable): openssl-3.0.7-2.el9.x86_64 How reproducible: always Steps to Reproduce: Steps to Reproduce: 1. wget https://raw.githubusercontent.com/redhat-qe-security/certgen/master/certgen/lib.sh 2. source lib.sh 3. x509KeyGen ca 4. x509KeyGen client 5. x509SelfSign ca 6. x509CertSign --CA ca -t webclient client 7. echo 'Hello world' > message.txt 8. openssl smime -encrypt -in message.txt -out message.enc client/cert.pem Actual results: Error creating PKCS#7 structure 408C0F8F017F0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:373:Global default library context, Algorithm (DES-EDE3-CBC : 27), Properties () 408C0F8F017F0000:error:108C0100:PKCS7 routines:PKCS7_final:malloc failure:crypto/pkcs7/pk7_smime.c:80: Expected results: Email encrypted with an AES cipher and RSA-OAEP Additional info: setting -aes128 explicitly doesn't help as (I'm assuming) it's trying to use PKCS#1 v1.5 encryption with the provided RSA key, instead of RSA-OEAP: Error creating PKCS#7 structure 40ACB4D62A7F0000:error:1C8000A5:Provider routines:rsa_encrypt:illegal or unsupported padding mode:providers/implementations/asymciphers/rsa_enc.c:163: 40ACB4D62A7F0000:error:108C0100:PKCS7 routines:PKCS7_final:malloc failure:crypto/pkcs7/pk7_smime.c:80:
Same issue for cms command: # openssl cms -encrypt -in message.txt -out message.enc client/cert.pem 403C7406297F0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:373:Global default library context, Algorithm (DES-EDE3-CBC : 27), Properties () 403C7406297F0000:error:17000065:CMS routines:ossl_cms_EncryptedContent_init_bio:cipher initialisation error:crypto/cms/cms_enc.c:78: 403C7406297F0000:error:17000068:CMS routines:CMS_final:cms lib:crypto/cms/cms_smime.c:882: # openssl cms -encrypt -in message.txt -out message.enc -aes128 client/cert.pem 404C8805B77F0000:error:1C8000A5:Provider routines:rsa_encrypt:illegal or unsupported padding mode:providers/implementations/asymciphers/rsa_enc.c:163: 404C8805B77F0000:error:17000074:CMS routines:cms_EnvelopedData_Encryption_init_bio:error setting recipientinfo:crypto/cms/cms_env.c:1142: 404C8805B77F0000:error:17000068:CMS routines:CMS_final:cms lib:crypto/cms/cms_smime.c:882: