Bug 2160797

Summary:	openssl smime and cms commands default to 3DES and PKCS#1v1.5 encryption in FIPS mode
Product:	Red Hat Enterprise Linux 9	Reporter:	Hubert Kario <hkario>
Component:	openssl	Assignee:	Dmitry Belyavskiy <dbelyavs>
Status:	VERIFIED ---	QA Contact:	Hubert Kario <hkario>
Severity:	low	Docs Contact:	Mirek Jahoda <mjahoda>
Priority:	low
Version:	9.0	CC:	cllang
Target Milestone:	rc	Keywords:	Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openssl-3.0.7-18.el9	Doc Type:	Known Issue
Doc Text:	Cause: OpenSSL cms and smime commands default to legacy algorithms, like 3DES for symmetric encryption and PKCS#1 v1.5 for asymmetric encryption. Consequence: Those algorithms are forbidden in FIPS mode, thus the operations fail. Workaround (if any): There is no workaround for use of PKCS#1 v1.5 encryption in smime command. For cms command, it's possible to specify the recipient using the -recip switch and then set the padding used using -keyopt rsa_padding_mode:oaep to use OAEP. For use of 3DES, both commands accept accept -aes128 and -aes256 switch to change the symmetric algorithm. Result: Encrypting files using cms and smime commands in FIPS mode doesn't work with default settings.	Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Hubert Kario 2023-01-13 18:14:09 UTC

Description of problem:
When the openssl smime command is used to encrypt email in FIPS mode, it still uses 3DES. Since that algorithm is not allowed in FIPS mode, the operation fails.

Version-Release number of selected component (if applicable):
openssl-3.0.7-2.el9.x86_64

How reproducible:
always

Steps to Reproduce:
Steps to Reproduce:
1. wget https://raw.githubusercontent.com/redhat-qe-security/certgen/master/certgen/lib.sh
2. source lib.sh
3. x509KeyGen ca
4. x509KeyGen client
5. x509SelfSign ca
6. x509CertSign --CA ca -t webclient client
7. echo 'Hello world' > message.txt
8. openssl smime -encrypt -in message.txt -out message.enc client/cert.pem

Actual results:
Error creating PKCS#7 structure
408C0F8F017F0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:373:Global default library context, Algorithm (DES-EDE3-CBC : 27), Properties ()
408C0F8F017F0000:error:108C0100:PKCS7 routines:PKCS7_final:malloc failure:crypto/pkcs7/pk7_smime.c:80:

Expected results:
Email encrypted with an AES cipher and RSA-OAEP

Additional info:

setting -aes128 explicitly doesn't help as (I'm assuming) it's trying to use PKCS#1 v1.5 encryption with the provided RSA key, instead of RSA-OEAP:

Error creating PKCS#7 structure
40ACB4D62A7F0000:error:1C8000A5:Provider routines:rsa_encrypt:illegal or unsupported padding mode:providers/implementations/asymciphers/rsa_enc.c:163:
40ACB4D62A7F0000:error:108C0100:PKCS7 routines:PKCS7_final:malloc failure:crypto/pkcs7/pk7_smime.c:80:

Comment 1 Hubert Kario 2023-01-13 19:52:07 UTC

Same issue for cms command:

# openssl cms -encrypt -in message.txt -out message.enc client/cert.pem

403C7406297F0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:373:Global default library context, Algorithm (DES-EDE3-CBC : 27), Properties ()
403C7406297F0000:error:17000065:CMS routines:ossl_cms_EncryptedContent_init_bio:cipher initialisation error:crypto/cms/cms_enc.c:78:
403C7406297F0000:error:17000068:CMS routines:CMS_final:cms lib:crypto/cms/cms_smime.c:882:


# openssl cms -encrypt -in message.txt -out message.enc -aes128 client/cert.pem

404C8805B77F0000:error:1C8000A5:Provider routines:rsa_encrypt:illegal or unsupported padding mode:providers/implementations/asymciphers/rsa_enc.c:163:
404C8805B77F0000:error:17000074:CMS routines:cms_EnvelopedData_Encryption_init_bio:error setting recipientinfo:crypto/cms/cms_env.c:1142:
404C8805B77F0000:error:17000068:CMS routines:CMS_final:cms lib:crypto/cms/cms_smime.c:882: