Description of problem:
After upgrade to 2.4.33-2, httpd won't start.
I had a working httpd with version 2.4.29-1, using certificates from letsencrypt (maintained by certbot). After the upgrade, the only actual errors I see are in /var/log/httpd/error_log (journalctl -xe and systemctl status httpd.service basically give no information).
The log complains about there not being a certificate for one of the subdomains. The previous version had no problems with the certificate. My certificate is for one name (example.com) with a few alternate names (mail.example.com, www.example.com). (Domain name changed to protect the innocent.)
Using strace I can see that httpd can open the certificate files, so there is no selinux issue here.
Version-Release number of selected component (if applicable):
httpd.x86_64 2.4.33-2.fc27 updates
httpd-filesystem.noarch 2.4.33-2.fc27 updates
httpd-manual.noarch 2.4.33-2.fc27 updates
httpd-tools.x86_64 2.4.33-2.fc27 updates
mod_ssl.x86_64 1:2.4.33-2.fc27 updates
Steps to Reproduce:
httpd doesn't start
httpd does start
Log message in /var/log/httpd/error_log:
[Fri Apr 06 14:52:52.434244 2018] [core:notice] [pid 12253:tid 140296309571584] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Fri Apr 06 14:52:52.435864 2018] [suexec:notice] [pid 12253:tid 140296309571584] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Fri Apr 06 14:52:52.436152 2018] [ssl:emerg] [pid 12253:tid 140296309571584] AH02572: Failed to configure at least one certificate and key for mail.example.com:80
[Fri Apr 06 14:52:52.436172 2018] [ssl:emerg] [pid 12253:tid 140296309571584] SSL Library Error: error:140A80B1:SSL routines:SSL_CTX_check_private_key:no certificate assigned
[Fri Apr 06 14:52:52.436176 2018] [ssl:emerg] [pid 12253:tid 140296309571584] AH02312: Fatal error initialising mod_ssl, exiting.
AH00016: Configuration Failed
I am seeing the same thing; this broke a few servers here. The error messages I receive are identical except for the domain name.
However, one important difference with my situation is that I'm using a wildcard certificate issued by an SSL vendor, not a LetsEncrypt cert. The certificate is valid and has always worked just fine across different operating systems and pieces of software. I'm at a loss to understand what this new version of httpd doesn't like about it.
Simply downgrading httpd to the previous version (which will also downgrade various httpd-* packages and mod_ssl) is sufficient to get things working again.
I assume neither of you installed mod_md as well? (Just trying to eliminate root causes)
Can you give me some config details and /var/log/httpd/ssl_error_log output? If you can share with me privately (e-mail is fine) but not in bugzilla that's fine.
If you can remove "LogLevel warn" from /etc/httpd/conf.d/ssl.conf and set "LogLevel ssl_module:trace7" globally then capture /var/log/httpd/ssl_error_log that'd be even better - again e-mail or private attachments here is fine.
Specifically details of the vhost config might be important. Ideally "httpd -t -DDUMP_CONFIG" output.
Having or not having mod_md installed doesn't make a difference.
After changing the LogLevel as you requested and attempting to start httpd, the ssl_error_log file wasn't modified: no messages. There is no point sending an empty file, so I won't. The error_log file only received the messages that I quoted already.
I'll email the DUMP_CONFIG output separately (I don't see a way of marking an attachment as private). This is on a system without vhosts.
There is a major behavioural change in 2.4.33 which affects how mod_ssl configuration is merged to vhosts. Upstream discussion is here:
The issue is where you have a multiple VirtualHosts, some without an Explicit "SSLEngine on" or any other SSL configuration, and expect all to inherit the SSL configuration from a default *:443 vhost.
I have a partial workaround, which will allow the server to start, but depending on the configuration it may behave differently at runtime, since mod_ssl will now treat the secondary vhosts as not having SSL enabled at config level.
I'm not sure I can fix this 2.4.33 regression without breaking other behaviour to be honest; you may need to adjust your configurations, copying the SSLCertificateFile (etc) from the default *:443 vhost into the other :443 vhosts.
(In reply to Joe Orton from comment #8)
> The issue is where you have a multiple VirtualHosts, some without an
> Explicit "SSLEngine on" or any other SSL configuration, and expect all to
> inherit the SSL configuration from a default *:443 vhost.
I can easily add "SSLEngine on" to all VirtualHost sections for 443 ports, but the question then is: what other configuration variables should I copy from the _default_ entry. Is there any documentation about this anywhere?
I am now running httpd-2.4.33-2.fc27.x86_64 on my servers after having added all configuration variables from the _default_ section to the other vhost sections.
Thank you for your efforts here.
Interestingly enough, I have never put any SSL directives into vhost definitions. Since I use wildcard certs and rarely have more than one vhost defined anyway (since most things are in VMs) I just edited ssl.conf to reference the certificate directly. I make no claims that's the right way to do it, only that I basically messed with it until it worked. And it did work, until this update.
On another machine, I guess I thought that was unpleasant so I commented the SSLCertificateFile directives from ssl.conf and stuck the new values in a separate file in /etc/httpd/conf.d. And for some reason the machine where that was done doesn't have this issue. The only other difference I can see in how the two servers were configured is that the order of the two virtualhost directives I use (one on port 80 redirecting to https, and the other on port 443) are in a different order, and one host has a Protocols line with "h2 h2c http/1.1" because I was playing with that. None of the virtualhost definitions mentions anything SSL-related at all.
So I'm guessing is that the first defined vhost is special and inherits stuff, but none of the other vhosts do. So I guess the end result is that if you only have one port 443 vhost and it's the first one then you're good but only by luck. If you have anything that goes before it then that configuration is broken (but worked fine and without even warnings until 2.4.33).
Yes, the first defined vhost is special for SSL for a given port is special in that it will be the one which catches TCP connections to that port, and the name-based matching only happens later.
With some further analysis I think the workaround I put in this build:
...should be safe, should fix the regression, and shouldn't cause any other issues. So I'd really appreciate some testing on that build against unchanged configs which worked with 2.4.29.
I tried it on one of the systems where I had problems using the original configuration. It works without a hitch.
I concur; I installed the packages from that build on three servers which had the problem and all are working fine now.
I am suffering from the 2.4.33 regression. Will try the workaround here, specifically Jason Tibbitts, if that does not work will try Joe Orton's "alpha" situation in this post
mostly, logging this to be notified about the progress.
Tested the workaround in comment 12 by Joe Orton, works with the unchanged configs from 2.4.29 and httpd fires up without any error messages. Thanks.
Would it be possible to get an updated build for f28 as well?
httpd-2.4.33-5.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-5e105755d6
httpd-2.4.33-5.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-7155c69687
Thanks a lot to everyone who provided feedback on the builds, I really appreciate it. Anybody hanging back because of my earlier hesitance in comments above, please don't, I'm reasonably confident there are no regressions in the patch we're using here (which should also be in upstream 2.4.34).
Don't hesitate to let us know if you see further issues, but updates for f26..28 now filed with this patch.
httpd-2.4.33-5.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-5e105755d6
httpd-2.4.33-5.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.
httpd-2.4.33-5.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.
Adding comment for anyone searching for AWS Linux Apache 2.4.33 upgrade SSL breaking change.
This thread was a god-send. Had the same issue with 2.4.33 (after a yum update) on AWS Linux. I copied everything relevant fro conf.d/ssl.conf into conf/httpd.conf for every *:443 virtual host and it was resolved.
Feels like a major step backwards but I'd rather be up and running than scratching my head.