Bug 1564537

Summary: httpd-2.4.33-2 won't start with SSL certificate error
Product: [Fedora] Fedora Reporter: Sjoerd Mullender <sjoerd>
Component: httpdAssignee: Luboš Uhliarik <luhliari>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 27CC: alex.ploumistos, debaditya, franzzemen, irlapati, jkaluza, jorton, j, luhliari, pahan, redhat-bugzilla, sjoerd
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: httpd-2.4.33-4.fc26 httpd-2.4.33-5.fc27 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-05-05 22:27:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sjoerd Mullender 2018-04-06 14:56:27 UTC
Description of problem:
After upgrade to 2.4.33-2, httpd won't start.

I had a working httpd with version 2.4.29-1, using certificates from letsencrypt (maintained by certbot).  After the upgrade, the only actual errors I see are in /var/log/httpd/error_log (journalctl -xe and systemctl status httpd.service basically give no information).

The log complains about there not being a certificate for one of the subdomains.  The previous version had no problems with the certificate.  My certificate is for one name (example.com) with a few alternate names (mail.example.com, www.example.com).  (Domain name changed to protect the innocent.)

Using strace I can see that httpd can open the certificate files, so there is no selinux issue here.

Version-Release number of selected component (if applicable):
httpd.x86_64                            2.4.33-2.fc27                    updates
httpd-filesystem.noarch                 2.4.33-2.fc27                    updates
httpd-manual.noarch                     2.4.33-2.fc27                    updates
httpd-tools.x86_64                      2.4.33-2.fc27                    updates
mod_ssl.x86_64                          1:2.4.33-2.fc27                  updates


How reproducible:
100%

Steps to Reproduce:
1.upgrade
2.
3.

Actual results:
httpd doesn't start

Expected results:
httpd does start

Additional info:

Log message in /var/log/httpd/error_log:

[Fri Apr 06 14:52:52.434244 2018] [core:notice] [pid 12253:tid 140296309571584] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Fri Apr 06 14:52:52.435864 2018] [suexec:notice] [pid 12253:tid 140296309571584] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Fri Apr 06 14:52:52.436152 2018] [ssl:emerg] [pid 12253:tid 140296309571584] AH02572: Failed to configure at least one certificate and key for mail.example.com:80
[Fri Apr 06 14:52:52.436172 2018] [ssl:emerg] [pid 12253:tid 140296309571584] SSL Library Error: error:140A80B1:SSL routines:SSL_CTX_check_private_key:no certificate assigned
[Fri Apr 06 14:52:52.436176 2018] [ssl:emerg] [pid 12253:tid 140296309571584] AH02312: Fatal error initialising mod_ssl, exiting.
AH00016: Configuration Failed

Comment 1 Jason Tibbitts 2018-04-09 20:10:53 UTC
I am seeing the same thing; this broke a few servers here.  The error messages I receive are identical except for the domain name.

However, one important difference with my situation is that I'm using a wildcard certificate issued by an SSL vendor, not a LetsEncrypt cert.  The certificate is valid and has always worked just fine across different operating systems and pieces of software.  I'm at a loss to understand what this new version of httpd doesn't like about it.

Simply downgrading httpd to the previous version (which will also downgrade various httpd-* packages and mod_ssl) is sufficient to get things working again.

Comment 2 Joe Orton 2018-04-10 07:14:17 UTC
I assume neither of you installed mod_md as well? (Just trying to eliminate root causes)

Can you give me some config details and /var/log/httpd/ssl_error_log output?  If you can share with me privately (e-mail is fine) but not in bugzilla that's fine.

Comment 3 Joe Orton 2018-04-10 07:16:25 UTC
If you can remove "LogLevel warn" from /etc/httpd/conf.d/ssl.conf and set "LogLevel ssl_module:trace7" globally then capture /var/log/httpd/ssl_error_log that'd be even better - again e-mail or private attachments here is fine.

Comment 4 Joe Orton 2018-04-10 07:24:39 UTC
Specifically details of the vhost config might be important. Ideally "httpd -t -DDUMP_CONFIG" output.

Comment 5 Sjoerd Mullender 2018-04-10 08:18:06 UTC
Having or not having mod_md installed doesn't make a difference.

After changing the LogLevel as you requested and attempting to start httpd, the ssl_error_log file wasn't modified: no messages.  There is no point sending an empty file, so I won't.  The error_log file only received the messages that I quoted already.

I'll email the DUMP_CONFIG output separately (I don't see a way of marking an attachment as private).  This is on a system without vhosts.

Comment 7 Joe Orton 2018-04-12 07:18:56 UTC
Package: httpd-2.4.33-3.fc27
Build: https://koji.fedoraproject.org/koji/buildinfo?buildID=1069164

Comment 8 Joe Orton 2018-04-12 07:39:54 UTC
There is a major behavioural change in 2.4.33 which affects how mod_ssl configuration is merged to vhosts. Upstream discussion is here:

https://marc.info/?t=152337738200004&r=1&w=2

The issue is where you have a multiple VirtualHosts, some without an Explicit "SSLEngine on" or any other SSL configuration, and expect all to inherit the SSL configuration from a default *:443 vhost.

I have a partial workaround, which will allow the server to start, but depending on the configuration it may behave differently at runtime, since mod_ssl will now treat the secondary vhosts as not having SSL enabled at config level.

https://koji.fedoraproject.org/koji/buildinfo?buildID=1069164

I'm not sure I can fix this 2.4.33 regression without breaking other behaviour to be honest; you may need to adjust your configurations, copying the SSLCertificateFile (etc) from the default *:443 vhost into the other :443 vhosts.

Comment 9 Sjoerd Mullender 2018-04-12 12:33:19 UTC
(In reply to Joe Orton from comment #8)
> The issue is where you have a multiple VirtualHosts, some without an
> Explicit "SSLEngine on" or any other SSL configuration, and expect all to
> inherit the SSL configuration from a default *:443 vhost.

I can easily add "SSLEngine on" to all VirtualHost sections for 443 ports, but the question then is: what other configuration variables should I copy from the _default_ entry.  Is there any documentation about this anywhere?

Comment 10 Sjoerd Mullender 2018-04-12 13:39:54 UTC
I am now running httpd-2.4.33-2.fc27.x86_64 on my servers after having added all configuration variables from the _default_ section to the other vhost sections.

Comment 11 Jason Tibbitts 2018-04-13 00:02:29 UTC
Thank you for your efforts here.

Interestingly enough, I have never put any SSL directives into vhost definitions.  Since I use wildcard certs and rarely have more than one vhost defined anyway (since most things are in VMs) I just edited ssl.conf to reference the certificate directly.  I make no claims that's the right way to do it, only that I basically messed with it until it worked.  And it did work, until this update.

On another machine, I guess I thought that was unpleasant so I commented the SSLCertificateFile directives from ssl.conf and stuck the new values in a separate file in /etc/httpd/conf.d.  And for some reason the machine where that was done doesn't have this issue. The only other difference I can see in how the two servers were configured is that the order of the two virtualhost directives I use (one on port 80 redirecting to https, and the other on port 443) are in a different order, and one host has a Protocols line with "h2 h2c http/1.1" because I was playing with that.  None of the virtualhost definitions mentions anything SSL-related at all.

So I'm guessing is that the first defined vhost is special and inherits stuff, but none of the other vhosts do.  So I guess the end result is that if you only have one port 443 vhost and it's the first one then you're good but only by luck.  If you have anything that goes before it then that configuration is broken (but worked fine and without even warnings until 2.4.33).

Comment 12 Joe Orton 2018-04-13 11:27:36 UTC
Yes, the first defined vhost is special for SSL for a given port is special in that it will be the one which catches TCP connections to that port, and the name-based matching only happens later.

With some further analysis I think the workaround I put in this build:

https://koji.fedoraproject.org/koji/buildinfo?buildID=1069164

...should be safe, should fix the regression, and shouldn't cause any other issues.  So I'd really appreciate some testing on that build against unchanged configs which worked with 2.4.29.

Comment 13 Sjoerd Mullender 2018-04-13 17:19:29 UTC
I tried it on one of the systems where I had problems using the original configuration.  It works without a hitch.

Comment 14 Jason Tibbitts 2018-04-14 01:00:13 UTC
I concur; I installed the packages from that build on three servers which had the problem and all are working fine now.

Comment 15 Deb Mukherjee 2018-04-22 00:28:48 UTC
I am suffering from the 2.4.33 regression. Will try the workaround here, specifically Jason Tibbitts, if that does not work will try  Joe Orton's "alpha" situation in this post 

https://marc.info/?l=apache-httpd-dev&m=152361791112315&w=2

mostly, logging this to be notified about the progress.

Comment 16 Deb Mukherjee 2018-04-23 03:48:50 UTC
Tested the workaround in comment 12 by Joe Orton, works with the unchanged configs from 2.4.29 and httpd fires up without any error messages. Thanks.

Comment 17 Alexander Ploumistos 2018-04-30 05:46:32 UTC
Would it be possible to get an updated build for f28 as well?

Comment 18 Joe Orton 2018-05-01 09:04:17 UTC
Package: httpd-2.4.33-5.fc28
Build: https://koji.fedoraproject.org/koji/buildinfo?buildID=1077998

Comment 19 Joe Orton 2018-05-01 20:47:29 UTC
Package: httpd-2.4.33-5.fc27
Build: https://koji.fedoraproject.org/koji/buildinfo?buildID=1078253

Comment 20 Joe Orton 2018-05-01 20:55:05 UTC
Package: httpd-2.4.33-4.fc26
Build: https://koji.fedoraproject.org/koji/buildinfo?buildID=1078254

Comment 21 Fedora Update System 2018-05-01 21:32:31 UTC
httpd-2.4.33-5.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-5e105755d6

Comment 22 Fedora Update System 2018-05-01 21:32:42 UTC
httpd-2.4.33-5.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-5e105755d6

Comment 23 Fedora Update System 2018-05-01 21:32:55 UTC
httpd-2.4.33-5.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-7155c69687

Comment 24 Joe Orton 2018-05-01 21:41:51 UTC
Thanks a lot to everyone who provided feedback on the builds, I really appreciate it.  Anybody hanging back because of my earlier hesitance in comments above, please don't, I'm reasonably confident there are no regressions in the patch we're using here (which should also be in upstream 2.4.34).

Don't hesitate to let us know if you see further issues, but updates for f26..28 now filed with this patch.

Comment 25 Fedora Update System 2018-05-02 10:12:17 UTC
httpd-2.4.33-5.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-5e105755d6

Comment 26 Fedora Update System 2018-05-02 10:53:09 UTC
httpd-2.4.33-5.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.

Comment 27 Fedora Update System 2018-05-05 22:27:48 UTC
httpd-2.4.33-5.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

Comment 28 Franz Zemen 2018-05-29 02:38:43 UTC
Adding comment for anyone searching for AWS Linux Apache 2.4.33 upgrade SSL breaking change.

This thread was a god-send.  Had the same issue with 2.4.33 (after a yum update) on AWS Linux.  I copied everything relevant fro conf.d/ssl.conf into conf/httpd.conf for every *:443 virtual host and it was resolved.  

Feels like a major step backwards but I'd rather be up and running than scratching my head.