Description of problem:
I'm running two instances of Spacewalk - 2.4 and 2.6. On both installations after the httpd update clients stopped to receive yum metadata and rpm packages with error 400. Rolling back this update fixed this problem immediately.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. update httpd on Spacewalk server
2. run yum update on spacewalk client system
On client you will have:
failed to retrieve repodata/repomd.xml from epel-x86_64-7
error was [Errno 14] HTTPS Error 400 - Bad Request
In Spacewalk server logs you will have:
192.168.1.78 - - [14/Apr/2017:17:53:11 +0000] "GET /XMLRPC//GET-REQ/centos-x86_64-7/repodata/repomd.xml HTTP/1.1" 400 226
spacewalk client should be able to get yum repodata and updates
Turnend on debug, I found following in httpd logs:
[Fri Apr 14 18:11:12.193177 2017] [core:debug] [pid 6486] protocol.c(956): (22)Invalid argument: [client 192.168.1.78:42834] Failed to read request header line X-RHN-Auth-User-Id:
[Fri Apr 14 18:11:12.193220 2017] [core:debug] [pid 6486] protocol.c(1309): [client 192.168.1.78:42834] AH00567: request failed: error reading the headers
Since the policy of header processing was changed in patch httpd-2.4.6-CVE-2016-8743.patch, I tried to recompile httpd with this patch commented out, and it fixed the problem.
Have you also updated yum-rhn-plugin ?
I just spoke to SW developers, and they told me, fix for this issue will be included in SW 2.7.
So you have 2 options:
a) roll back httpd on your SW instance
b) apply the changes from fdf2063e14fc2539f09efcb7e72dd85060ab9ae5 yourself
Yes, I verified that yum-rhn-plugin-2.0.1-6.1.el7_3.noarch works with new httpd, but it's unavailable for el6 and most of my clients are using yum-rhn-plugin-2.6.3-1. I found other solution: instead of rolling back httpd I sat an option
It restored behavior of the header parser.
So what's happening here is the result of an update to httpd, that enforces strict whitespace rules, that uncovered a bug in yum-rhn-plugin that was introduced in 2010.
(The spacewalk commit that started us down this path, for anyone who is interested, is 213b97e38dd75007281b37e3180dc68b736f2fac )
yum-rhn-plugin has been fixed, but a client without the fix yet won't be able to call home to *get* the fix, if the server has the new httpd.
There are two workarounds available:
One is to downgrade httpd
The other is to add an httpd config option that *turns off" strict-whitespace-enforcing until all your clients have a corrected yum-rhn-plugin. You can do that with the following:
# echo “HTTPProtocolOptions unsafe” >> /etc/httpd/conf.d/123unsafe.conf; systemctl restart httpd
That will put httpd back to its pre-2.4.25 behavior.
spacewalk-nightly deploys the config-option reliably for Fedora and RHEL6. We're still working on finding a way to deploy that config-change reliably on RHEL7. The issue is that RHEL7's httpd change is in the release, not the version, so <IfVersion> can't be used to 'protect' the change - and trying to use HttpProtocolOptions on an httpd that doesn't support it, causes httpd to fail and exit. Which is rude.
Setting “HTTPProtocolOptions unsafe” in /etc/httpd/conf.d/zz-spacewalk-server-wsgi.conf also works :)
We'll be able to improve the workaround for future Spacewalk + RHEL7 httpd, but I'm closing this out since it's not a bug, Grant's commentary in comment 5 covers the options/workarounds.