RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2039993 - httpd fails to start with double free after updating to openssl-1.0.2k-23.el7_9
Summary: httpd fails to start with double free after updating to openssl-1.0.2k-23.el7_9
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: openssl
Version: 7.9
Hardware: x86_64
OS: Linux
urgent
urgent
Target Milestone: rc
: ---
Assignee: Sahana Prasad
QA Contact: Stanislav Zidek
URL:
Whiteboard:
: 2040476 2040853 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-01-12 21:42 UTC by Aaron Ogburn
Modified: 2022-09-19 04:07 UTC (History)
39 users (show)

Fixed In Version: openssl-1.0.2k-24.el7_9
Doc Type: Bug Fix
Doc Text:
Cause: A bug in the function append_ia5() causes a double free of the emtmp parameter. Consequence: When OCSP Stapling is enabled, and OpenSSL version is updated to openssl-1.0.2k-23.el7_9, httpd and nginx webservers crash, and the service cannot be reloaded/restarted. Fix: The function append_ia5() is fixed to free emtmp on push failure only. Result: No crashes are seen when httpd or nginx services are restarted/reloaded when OCSP stapling is enabled.
Clone Of:
Environment:
Last Closed: 2022-01-18 09:12:16 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker CRYPTO-6037 0 None None None 2022-01-13 15:17:31 UTC
Red Hat Issue Tracker RHELPLAN-107632 0 None None None 2022-01-12 21:46:52 UTC
Red Hat Knowledge Base (Solution) 6637201 0 None None None 2022-01-13 12:35:25 UTC
Red Hat Product Errata RHBA-2022:0156 0 None None None 2022-01-18 09:12:19 UTC

Internal Links: 2040853

Description Aaron Ogburn 2022-01-12 21:42:05 UTC 
Description of problem:

It seems like https://access.redhat.com/errata/RHSA-2022:0064 is faulty, because after updating to openssl-1.0.2k-23.el7_9.x86_64, the Apache webserver only segfaults. Downgrading to openssl-1.0.2k-22.el7_9.x86_64 leads to successful httpd startups again.

httpd error_logs show the following double free backtrace:

*** Error in `/usr/sbin/httpd': double free or corruption (fasttop): 0x00005637ad2df900 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81329)[0x7fae1fa2d329]
/lib64/libcrypto.so.10(CRYPTO_free+0x1d)[0x7fae149db96d]
/lib64/libcrypto.so.10(sk_pop_free+0x30)[0x7fae14a93360]
/lib64/libcrypto.so.10(+0x170ed5)[0x7fae14added5]
/lib64/libcrypto.so.10(X509_get1_ocsp+0x80)[0x7fae14ade120]
/etc/httpd/modules/mod_ssl.so(+0x22afb)[0x7fae15064afb]
/etc/httpd/modules/mod_ssl.so(+0xfb0e)[0x7fae15051b0e]
/etc/httpd/modules/mod_ssl.so(+0x116ee)[0x7fae150536ee]
/etc/httpd/modules/mod_ssl.so(+0x128ec)[0x7fae150548ec]
/usr/sbin/httpd(ap_run_post_config+0x59)[0x5637ab0fab69]
/usr/sbin/httpd(main+0x8b8)[0x5637ab0d7ec8]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7fae1f9ce555]
/usr/sbin/httpd(+0x1f1df)[0x5637ab0d81df]

Version-Release number of selected component (if applicable):

openssl-1.0.2k-23.el7_9.x86_64
httpd-2.4.6-97.el7_9.2.x86_64
mod_ssl-2.4.6-97.el7_9.2.x86_64
Comment 5 RaidM 2022-01-13 07:29:58 UTC 
This also affected Nginx webservers too. All previous build  or a custom source build of nginx also fail with this bug.

# /usr/local/nginx/sbin/nginx -t
*** Error in `/usr/local/nginx/sbin/nginx': double free or corruption (fasttop): 0x0000000002b8aa60 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81329)[0x7fca27588329]
/lib64/libcrypto.so.10(CRYPTO_free+0x1d)[0x7fca27b5996d]
/lib64/libcrypto.so.10(sk_pop_free+0x30)[0x7fca27c11360]
/lib64/libcrypto.so.10(+0x170ed5)[0x7fca27c5bed5]
/lib64/libcrypto.so.10(X509_get1_ocsp+0x80)[0x7fca27c5c120]
/usr/local/nginx/sbin/nginx(ngx_ssl_stapling+0x4a9)[0x44377d]
/usr/local/nginx/sbin/nginx[0x48ce52]
/usr/local/nginx/sbin/nginx[0x4453b6]
/usr/local/nginx/sbin/nginx(ngx_conf_parse+0xc73)[0x426581]
/usr/local/nginx/sbin/nginx(ngx_init_cycle+0x71a)[0x423d55]
/usr/local/nginx/sbin/nginx(main+0x9b6)[0x41295c]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7fca27529555]
/usr/local/nginx/sbin/nginx[0x411279]
======= Memory map: ========
00400000-004d5000 r-xp 00000000 08:03 6030667                            /usr/local/nginx/sbin/nginx
006d5000-006d6000 r--p 000d5000 08:03 6030667                            /usr/local/nginx/sbin/nginx
006d6000-006f1000 rw-p 000d6000 08:03 6030667                            /usr/local/nginx/sbin/nginx
006f1000-00710000 rw-p 00000000 00:00 0 
016a7000-02b9b000 rw-p 00000000 00:00 0                                  [heap]
7fca20000000-7fca20021000 rw-p 00000000 00:00 0 


To solve temporary simply downgrade the openssl

# yum downgrade openssl openssl-libs openssl-devel
Comment 8 Robert Scheck 2022-01-13 10:17:04 UTC 
We are experiencing the same issue and filed yesterday the cases #03123297, #03123183 and #03123276 (different Red Hat customers) in the Red Hat Customer Portal for it.
Comment 18 troy.engel 2022-01-13 14:13:29 UTC 
Our initial investigations yesterday seem to point to the OCSP Stapling being broken; the bug can be recreated like so:

## Obtain the SSL certificate from www.redhat.com

  openssl s_client -connect www.redhat.com:443 < /dev/null 2>&1 | sed -n '/-----BEGIN/,/-----END/p' > redhat.pem

## Ask openssl client for the OCSP Stapling URL from redhat.pem

  openssl x509 -noout -ocsp_uri -in redhat.pem 

  *** Error in `openssl': double free or corruption (fasttop): 0x0000000001052d50 ***
  ======= Backtrace: =========
  (... same backtrace as everyone else)

You may be able to work around this issue by disabling the Apache/Nginx configuration to disable OCSP Stapling instead of downgrading the openssl packages (both solutions seem to work, but only minimal testing done).
Comment 20 RaidM 2022-01-13 15:00:59 UTC 
Disabling OCSP sapling is not an option . The easy solution is to downgrade openssl
Comment 21 Robert Scheck 2022-01-13 15:09:17 UTC 
I am in doubt that disabling OCSP stapling is a good idea, in some cases it's IMHO even not possible at all, due to OCSP must-staple set by the certificate authority.
Comment 23 Orion Poplawski 2022-01-13 16:08:01 UTC 
We are seeing crashes in alpine - apparently related to s/mime certs.
Comment 24 Sahana Prasad 2022-01-13 16:10:08 UTC 
@orion Hi Orion, do you see the same backtrace as mentioned in the comments earlier?
Comment 28 Sahana Prasad 2022-01-14 09:03:54 UTC 
*** Bug 2040476 has been marked as a duplicate of this bug. ***
Comment 30 Michael Buchfink 2022-01-14 14:27:08 UTC 
This is also affected (Duplicate Bug 2040476)

openssl x509 -noout -email -in cert_with_email.pem

*** Error in `openssl': double free or corruption (fasttop): 0x0000000001f101e0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81329)[0x7f508880c329]
...
Comment 32 Sara Golemon 2022-01-14 20:16:13 UTC 
Evidently I just filed a duplicate of this bug, though mine has the identified cause and how to fix it: https://bugzilla.redhat.com/show_bug.cgi?id=2040853
Comment 33 Sahana Prasad 2022-01-17 03:37:09 UTC 
*** Bug 2040853 has been marked as a duplicate of this bug. ***
Comment 41 errata-xmlrpc 2022-01-18 09:12:16 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (openssl bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:0156
Note You need to log in before you can comment on or make changes to this bug.