Red Hat Bugzilla – Bug 1442477
latest httpd update 2.4.6-45.el7_3.4 is breaking Spacewalk
Last modified: 2017-05-09 11:00:09 EDT
Description of problem: I'm running two instances of Spacewalk - 2.4 and 2.6. On both installations after the httpd update clients stopped to receive yum metadata and rpm packages with error 400. Rolling back this update fixed this problem immediately. Version-Release number of selected component (if applicable): 2.4.6-45.el7_3.4 How reproducible: always Steps to Reproduce: 1. update httpd on Spacewalk server 2. run yum update on spacewalk client system Actual results: On client you will have: failed to retrieve repodata/repomd.xml from epel-x86_64-7 error was [Errno 14] HTTPS Error 400 - Bad Request In Spacewalk server logs you will have: 192.168.1.78 - - [14/Apr/2017:17:53:11 +0000] "GET /XMLRPC//GET-REQ/centos-x86_64-7/repodata/repomd.xml HTTP/1.1" 400 226 Expected results: spacewalk client should be able to get yum repodata and updates Additional info: Turnend on debug, I found following in httpd logs: [Fri Apr 14 18:11:12.193177 2017] [core:debug] [pid 6486] protocol.c(956): (22)Invalid argument: [client 192.168.1.78:42834] Failed to read request header line X-RHN-Auth-User-Id: [Fri Apr 14 18:11:12.193220 2017] [core:debug] [pid 6486] protocol.c(1309): [client 192.168.1.78:42834] AH00567: request failed: error reading the headers Since the policy of header processing was changed in patch httpd-2.4.6-CVE-2016-8743.patch, I tried to recompile httpd with this patch commented out, and it fixed the problem.
Have you also updated yum-rhn-plugin ?
Leonid, I just spoke to SW developers, and they told me, fix for this issue will be included in SW 2.7. So you have 2 options: a) roll back httpd on your SW instance b) apply the changes from fdf2063e14fc2539f09efcb7e72dd85060ab9ae5 yourself
Hello Luboš, Yes, I verified that yum-rhn-plugin-2.0.1-6.1.el7_3.noarch works with new httpd, but it's unavailable for el6 and most of my clients are using yum-rhn-plugin-2.6.3-1. I found other solution: instead of rolling back httpd I sat an option HttpProtocolOptions Unsafe in /etc/httpd/conf.d/zz-spacewalk-server-wsgi.conf It restored behavior of the header parser.
So what's happening here is the result of an update to httpd, that enforces strict whitespace rules, that uncovered a bug in yum-rhn-plugin that was introduced in 2010. (The spacewalk commit that started us down this path, for anyone who is interested, is 213b97e38dd75007281b37e3180dc68b736f2fac ) yum-rhn-plugin has been fixed, but a client without the fix yet won't be able to call home to *get* the fix, if the server has the new httpd. There are two workarounds available: One is to downgrade httpd The other is to add an httpd config option that *turns off" strict-whitespace-enforcing until all your clients have a corrected yum-rhn-plugin. You can do that with the following: # echo “HTTPProtocolOptions unsafe” >> /etc/httpd/conf.d/123unsafe.conf; systemctl restart httpd That will put httpd back to its pre-2.4.25 behavior. spacewalk-nightly deploys the config-option reliably for Fedora and RHEL6. We're still working on finding a way to deploy that config-change reliably on RHEL7. The issue is that RHEL7's httpd change is in the release, not the version, so <IfVersion> can't be used to 'protect' the change - and trying to use HttpProtocolOptions on an httpd that doesn't support it, causes httpd to fail and exit. Which is rude.
Setting “HTTPProtocolOptions unsafe” in /etc/httpd/conf.d/zz-spacewalk-server-wsgi.conf also works :)
We'll be able to improve the workaround for future Spacewalk + RHEL7 httpd, but I'm closing this out since it's not a bug, Grant's commentary in comment 5 covers the options/workarounds.