RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1976080 - mod_ssl: Unable to load encrypted keys (AH02565: Certificate and private key do not match)
Summary: mod_ssl: Unable to load encrypted keys (AH02565: Certificate and private key ...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: httpd
Version: 9.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: beta
: ---
Assignee: Luboš Uhliarik
QA Contact: Branislav Náter
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-06-25 07:04 UTC by Christian Heimes
Modified: 2021-12-07 22:00 UTC (History)
7 users (show)

Fixed In Version: httpd-2.4.48-5.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-12-07 21:57:54 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Christian Heimes 2021-06-25 07:04:31 UTC 
Description of problem:
IPA server fails to install on RHEL 9 beta with OpenSSL 3.0.0 because mod_ssl is unable to load the encrypted private RSA key of the server certificate.

Version-Release number of selected component (if applicable):
mod_ssl-2.4.48-3.el9.x86_64
openssl-3.0.0-0.alpha16.4.el9.x86_64

How reproducible:
always

Steps to Reproduce:
1. dnf install ipa-server
2. ipa-server-install --domain=ipa.test --realm=IPA.TEST --ds-password=password --admin-password=password --hostname=$(hostname -f) -U

Actual results:
- installation fails
- /var/log/httpd/error_log contains error message
[ssl:emerg] [pid 35708:tid 35708] AH02565: Certificate and private key server.example.com:443:0 from /var/lib/ipa/certs/httpd.crt and /var/lib/ipa/private/httpd.key do not match

Expected results:
installation is successful

Additional info:

OpenSSL's s_server loads encrypted key and cert just fine:

    openssl s_server -pass file:/var/lib/ipa/passwds/server.example.com-443-RSA -cert /var/lib/ipa/certs/httpd.crt -key /var/lib/ipa/private/httpd.key -port 443 -debug

The private key is encrypted with modern, supported algorithms (PBES2 with PBKDF2, SHA-256 and AES-128-CBC).

Modulus of private key and cert's public key match.

I'm convinced that code https://github.com/apache/httpd/blob/2.4.48/modules/ssl/ssl_engine_init.c#L1345-L1348 does not work correctly on OpenSSL 3.0.0.

        else if ((SSL_CTX_use_PrivateKey_file(mctx->ssl_ctx, keyfile,
                                              SSL_FILETYPE_PEM) < 1)
                 && (ERR_GET_FUNC(ERR_peek_last_error())
                     != X509_F_X509_CHECK_PRIVATE_KEY)) {

SSL_CTX_use_PrivateKey_file() returns 0 and pushes 0xa080009 on the error stack. ERR_GET_FUNC(0xa080009) == 128, X509_F_X509_CHECK_PRIVATE_KEY is 128, so the function never tries to load encrypted private keys with ssl_load_encrypted_pkey().
Comment 2 Joe Orton 2021-06-25 11:48:25 UTC 
How does IPA configure pass phrase entry?
Comment 3 Joe Orton 2021-06-25 11:50:40 UTC 
Looks like the CHECK_PRIVATE_KEY error constant is in the deprecated list so at least that needs updating, not sure it should matter to the test failure.

./include/openssl/cryptoerr_legacy.h:#  define X509_F_X509_CHECK_PRIVATE_KEY                    0
./include/openssl/cryptoerr_legacy.h:#  define X509_F_X509_REQ_CHECK_PRIVATE_KEY                0
Comment 4 Christian Heimes 2021-06-25 11:52:58 UTC 
IPA sets a global password reader script with

    SSLPassPhraseDialog exec:/usr/libexec/ipa/ipa-httpd-pwdreader

It's a simple Python script that turns host:port and keytype arguments into a file path, then reads the file content from /var/lib/ipa/passwds/server.example.com-443-RSA and prints the content on stdout.
Comment 5 Joe Orton 2021-06-25 13:32:07 UTC 
(In reply to Joe Orton from comment #3)
> Looks like the CHECK_PRIVATE_KEY error constant is in the deprecated list so
> at least that needs updating, not sure it should matter to the test failure.
> 
> ./include/openssl/cryptoerr_legacy.h:#  define X509_F_X509_CHECK_PRIVATE_KEY
> 0
> ./include/openssl/cryptoerr_legacy.h:#  define
> X509_F_X509_REQ_CHECK_PRIVATE_KEY                0

https://www.openssl.org/docs/manmaster/man3/ERR_GET_FUNC.html

"Starting with OpenSSL 3.0.0, the function code is always set to zero"

... well who could predict that would have surprising effects???  Seriously...
Comment 9 Joe Orton 2021-06-25 15:32:50 UTC 
Good to hear.  Great job for correctly identifying the problematic lines of code, thanks :)
Note You need to log in before you can comment on or make changes to this bug.