Bug 1564537
Summary: | httpd-2.4.33-2 won't start with SSL certificate error | ||
---|---|---|---|
Product: | [Fedora] Fedora | Reporter: | Sjoerd Mullender <sjoerd> |
Component: | httpd | Assignee: | Luboš Uhliarik <luhliari> |
Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
Severity: | unspecified | Docs Contact: | |
Priority: | unspecified | ||
Version: | 27 | CC: | alex.ploumistos, debaditya, franzzemen, irlapati, jkaluza, jorton, j, luhliari, pahan, redhat-bugzilla, sjoerd |
Target Milestone: | --- | ||
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | httpd-2.4.33-4.fc26 httpd-2.4.33-5.fc27 | Doc Type: | If docs needed, set a value |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2018-05-05 22:27:48 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Sjoerd Mullender
2018-04-06 14:56:27 UTC
I am seeing the same thing; this broke a few servers here. The error messages I receive are identical except for the domain name. However, one important difference with my situation is that I'm using a wildcard certificate issued by an SSL vendor, not a LetsEncrypt cert. The certificate is valid and has always worked just fine across different operating systems and pieces of software. I'm at a loss to understand what this new version of httpd doesn't like about it. Simply downgrading httpd to the previous version (which will also downgrade various httpd-* packages and mod_ssl) is sufficient to get things working again. I assume neither of you installed mod_md as well? (Just trying to eliminate root causes) Can you give me some config details and /var/log/httpd/ssl_error_log output? If you can share with me privately (e-mail is fine) but not in bugzilla that's fine. If you can remove "LogLevel warn" from /etc/httpd/conf.d/ssl.conf and set "LogLevel ssl_module:trace7" globally then capture /var/log/httpd/ssl_error_log that'd be even better - again e-mail or private attachments here is fine. Specifically details of the vhost config might be important. Ideally "httpd -t -DDUMP_CONFIG" output. Having or not having mod_md installed doesn't make a difference. After changing the LogLevel as you requested and attempting to start httpd, the ssl_error_log file wasn't modified: no messages. There is no point sending an empty file, so I won't. The error_log file only received the messages that I quoted already. I'll email the DUMP_CONFIG output separately (I don't see a way of marking an attachment as private). This is on a system without vhosts. Package: httpd-2.4.33-3.fc27 Build: https://koji.fedoraproject.org/koji/buildinfo?buildID=1069164 There is a major behavioural change in 2.4.33 which affects how mod_ssl configuration is merged to vhosts. Upstream discussion is here: https://marc.info/?t=152337738200004&r=1&w=2 The issue is where you have a multiple VirtualHosts, some without an Explicit "SSLEngine on" or any other SSL configuration, and expect all to inherit the SSL configuration from a default *:443 vhost. I have a partial workaround, which will allow the server to start, but depending on the configuration it may behave differently at runtime, since mod_ssl will now treat the secondary vhosts as not having SSL enabled at config level. https://koji.fedoraproject.org/koji/buildinfo?buildID=1069164 I'm not sure I can fix this 2.4.33 regression without breaking other behaviour to be honest; you may need to adjust your configurations, copying the SSLCertificateFile (etc) from the default *:443 vhost into the other :443 vhosts. (In reply to Joe Orton from comment #8) > The issue is where you have a multiple VirtualHosts, some without an > Explicit "SSLEngine on" or any other SSL configuration, and expect all to > inherit the SSL configuration from a default *:443 vhost. I can easily add "SSLEngine on" to all VirtualHost sections for 443 ports, but the question then is: what other configuration variables should I copy from the _default_ entry. Is there any documentation about this anywhere? I am now running httpd-2.4.33-2.fc27.x86_64 on my servers after having added all configuration variables from the _default_ section to the other vhost sections. Thank you for your efforts here. Interestingly enough, I have never put any SSL directives into vhost definitions. Since I use wildcard certs and rarely have more than one vhost defined anyway (since most things are in VMs) I just edited ssl.conf to reference the certificate directly. I make no claims that's the right way to do it, only that I basically messed with it until it worked. And it did work, until this update. On another machine, I guess I thought that was unpleasant so I commented the SSLCertificateFile directives from ssl.conf and stuck the new values in a separate file in /etc/httpd/conf.d. And for some reason the machine where that was done doesn't have this issue. The only other difference I can see in how the two servers were configured is that the order of the two virtualhost directives I use (one on port 80 redirecting to https, and the other on port 443) are in a different order, and one host has a Protocols line with "h2 h2c http/1.1" because I was playing with that. None of the virtualhost definitions mentions anything SSL-related at all. So I'm guessing is that the first defined vhost is special and inherits stuff, but none of the other vhosts do. So I guess the end result is that if you only have one port 443 vhost and it's the first one then you're good but only by luck. If you have anything that goes before it then that configuration is broken (but worked fine and without even warnings until 2.4.33). Yes, the first defined vhost is special for SSL for a given port is special in that it will be the one which catches TCP connections to that port, and the name-based matching only happens later. With some further analysis I think the workaround I put in this build: https://koji.fedoraproject.org/koji/buildinfo?buildID=1069164 ...should be safe, should fix the regression, and shouldn't cause any other issues. So I'd really appreciate some testing on that build against unchanged configs which worked with 2.4.29. I tried it on one of the systems where I had problems using the original configuration. It works without a hitch. I concur; I installed the packages from that build on three servers which had the problem and all are working fine now. I am suffering from the 2.4.33 regression. Will try the workaround here, specifically Jason Tibbitts, if that does not work will try Joe Orton's "alpha" situation in this post https://marc.info/?l=apache-httpd-dev&m=152361791112315&w=2 mostly, logging this to be notified about the progress. Tested the workaround in comment 12 by Joe Orton, works with the unchanged configs from 2.4.29 and httpd fires up without any error messages. Thanks. Would it be possible to get an updated build for f28 as well? Package: httpd-2.4.33-5.fc28 Build: https://koji.fedoraproject.org/koji/buildinfo?buildID=1077998 Package: httpd-2.4.33-5.fc27 Build: https://koji.fedoraproject.org/koji/buildinfo?buildID=1078253 Package: httpd-2.4.33-4.fc26 Build: https://koji.fedoraproject.org/koji/buildinfo?buildID=1078254 httpd-2.4.33-5.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-5e105755d6 httpd-2.4.33-5.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-5e105755d6 httpd-2.4.33-5.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-7155c69687 Thanks a lot to everyone who provided feedback on the builds, I really appreciate it. Anybody hanging back because of my earlier hesitance in comments above, please don't, I'm reasonably confident there are no regressions in the patch we're using here (which should also be in upstream 2.4.34). Don't hesitate to let us know if you see further issues, but updates for f26..28 now filed with this patch. httpd-2.4.33-5.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-5e105755d6 httpd-2.4.33-5.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report. httpd-2.4.33-5.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report. Adding comment for anyone searching for AWS Linux Apache 2.4.33 upgrade SSL breaking change. This thread was a god-send. Had the same issue with 2.4.33 (after a yum update) on AWS Linux. I copied everything relevant fro conf.d/ssl.conf into conf/httpd.conf for every *:443 virtual host and it was resolved. Feels like a major step backwards but I'd rather be up and running than scratching my head. |