Bug 2215366

Summary: OpenSSL CMS encryption with ECDSA keys doesn't work
Product: Red Hat Enterprise Linux 9 Reporter: Hubert Kario <hkario>
Component: opensslAssignee: Dmitry Belyavskiy <dbelyavs>
Status: NEW --- QA Contact: BaseOS QE Security Team <qe-baseos-security>
Severity: low Docs Contact:
Priority: low    
Version: 9.2CC: cllang
Target Milestone: rcKeywords: Regression, Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Hubert Kario 2023-06-15 17:26:18 UTC 
Description of problem:
When the `openssl cms` command is used to encrypt data, the command aborts when the recipient key is ECDSA

Version-Release number of selected component (if applicable):
openssl-3.0.1-20.el9_0.x86_64
openssl-3.0.7-20.el9.x86_64

How reproducible:
always

Steps to Reproduce:
openssl ecparam -out p256.pem -name P-256
openssl req -x509 -newkey ec:p256.pem -keyout root.key -out root.crt -subj /CN=localhost -batch -nodes -days 36500 -sha256
echo something to encrypt > message.txt
openssl cms -encrypt -in message.txt -out message.enc root.crt

Actual results:
(for 3.0.7-20.el9)
409C3F54CC7F0000:error:17000074:CMS routines:cms_EnvelopedData_Encryption_init_bio:error setting recipientinfo:crypto/cms/cms_env.c:1159:
409C3F54CC7F0000:error:17000068:CMS routines:CMS_final:cms lib:crypto/cms/cms_smime.c:882:

Expected results:
CMS file encrypted with ECDSA/ECDH

Additional info:
The command does succeed in FIPS mode (and produces a file encrypted with `:dhSinglePass-stdDH-sha1kdf-scheme`) and works if the KDF is explicitly set to SHA-256:
openssl cms -encrypt -in message.txt -out message.enc -recip root.crt -keyopt ecdh_kdf_md:sha256
The operation also works in RHEL-8