Bug 1962355

Summary: OpenSSL 3.x does not work if context is on stack
Product: Red Hat Enterprise Linux 9 Reporter: Honza Horak <hhorak>
Component: opensslAssignee: Sahana Prasad <sahana>
Status: CLOSED NOTABUG QA Contact: BaseOS QE Security Team <qe-baseos-security>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: CentOS StreamCC: bstinson, dbelyavs, fweimer, jwboyer
Target Milestone: beta   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-21 12:11:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1952951, 1962047    
Attachments:
Description Flags
Reproducer extracted from MariaDB 10.5.9 none

Description Honza Horak 2021-05-19 19:53:20 UTC 
Description of problem:
One of the reasons why MariaDB and MySQL components fail to build with OpenSSL 3.x is a change in how context of digest functions can be used.

I understand that using those functions is deprecated, but it doesn't mean it should result in segmentation fault.

MySQL uses for example something like this:

static void my_md5_hash(unsigned char *digest, unsigned const char *buf,
                        int len) {
  MD5_CTX ctx;
  MD5_Init(&ctx);
  MD5_Update(&ctx, buf, len);
  MD5_Final(digest, &ctx);
}

It worked with OpenSSL 1.x but not with 3.x. 

Version-Release number of selected component (if applicable):
openssl-3.0.0-0.alpha16.1.ssl3.x86_64
Comment 1 Honza Horak 2021-05-19 20:29:02 UTC 
Created attachment 1784946 [details]
Reproducer extracted from MariaDB 10.5.9

This is another reproducer, this time what MariaDB does, extracted to a standalone file.

Usage:
$ gcc -DHAVE_OPENSSL -lssl -lcrypto md5-reproducer.c && ./a.out 
Segmentation fault (core dumped)

It looks like again, char ctx_buf[EVP_MD_CTX_SIZE]; on stack is a problem with new OpenSSL 3.x.