Bug 1442477

Summary: latest httpd update 2.4.6-45.el7_3.4 is breaking Spacewalk
Product: Red Hat Enterprise Linux 7 Reporter: Leonid Kanter <leon>
Component: httpdAssignee: Luboš Uhliarik <luhliari>
Status: CLOSED NOTABUG QA Contact: BaseOS QE - Apps <qe-baseos-apps>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.3CC: ggainey, jorton, leon, luhliari, nuno
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-05-09 15:00:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Leonid Kanter 2017-04-14 22:19:20 UTC
Description of problem:

I'm running two instances of Spacewalk - 2.4 and 2.6. On both installations after the httpd update clients stopped to receive yum metadata and rpm packages with error 400. Rolling back this update fixed this problem immediately.

Version-Release number of selected component (if applicable):

2.4.6-45.el7_3.4

How reproducible: 

always

Steps to Reproduce:
1. update httpd on Spacewalk server
2. run yum update on spacewalk client system

Actual results:

On client you will have:

failed to retrieve repodata/repomd.xml from epel-x86_64-7
error was [Errno 14] HTTPS Error 400 - Bad Request

In Spacewalk server logs you will have:

192.168.1.78 - - [14/Apr/2017:17:53:11 +0000] "GET /XMLRPC//GET-REQ/centos-x86_64-7/repodata/repomd.xml HTTP/1.1" 400 226

Expected results:

spacewalk client should be able to get yum repodata and updates

Additional info:

Turnend on debug, I found following in httpd logs:

[Fri Apr 14 18:11:12.193177 2017] [core:debug] [pid 6486] protocol.c(956): (22)Invalid argument: [client 192.168.1.78:42834] Failed to read request header line X-RHN-Auth-User-Id:
[Fri Apr 14 18:11:12.193220 2017] [core:debug] [pid 6486] protocol.c(1309): [client 192.168.1.78:42834] AH00567: request failed: error reading the headers

Since the policy of header processing was changed in patch httpd-2.4.6-CVE-2016-8743.patch, I tried to recompile httpd with this patch commented out, and it fixed the problem.

Comment 2 Luboš Uhliarik 2017-04-18 15:47:51 UTC
Have you also updated yum-rhn-plugin ?

Comment 3 Luboš Uhliarik 2017-04-18 16:04:07 UTC
Leonid,

I just spoke to SW developers, and they told me, fix for this issue will be included in SW 2.7. 

So you have 2 options:

a) roll back httpd on your SW instance

b) apply the changes from fdf2063e14fc2539f09efcb7e72dd85060ab9ae5 yourself

Comment 4 Leonid Kanter 2017-04-18 18:40:08 UTC
Hello Luboš,

Yes, I verified that yum-rhn-plugin-2.0.1-6.1.el7_3.noarch works with new httpd, but it's unavailable for el6 and most of my clients are using yum-rhn-plugin-2.6.3-1. I found other solution: instead of rolling back httpd I sat an option


HttpProtocolOptions Unsafe

in /etc/httpd/conf.d/zz-spacewalk-server-wsgi.conf

It restored behavior of the header parser.

Comment 5 Grant Gainey 2017-04-26 13:35:44 UTC
So what's happening here is the result of an update to httpd, that enforces strict whitespace rules, that uncovered a bug in yum-rhn-plugin that was introduced in 2010.

(The spacewalk commit that started us down this path, for anyone who is interested, is 213b97e38dd75007281b37e3180dc68b736f2fac )

yum-rhn-plugin has been fixed, but a client without the fix yet won't be able to call home to *get* the fix, if the server has the new httpd.

There are two workarounds available:

One is to downgrade httpd

The other is to add an httpd config option that *turns off" strict-whitespace-enforcing until all your clients have a corrected yum-rhn-plugin. You can do that with the following:

# echo “HTTPProtocolOptions unsafe” >> /etc/httpd/conf.d/123unsafe.conf; systemctl restart httpd

That will put httpd back to its pre-2.4.25 behavior.

spacewalk-nightly deploys the config-option reliably for Fedora and RHEL6. We're still working on finding a way to deploy that config-change reliably on RHEL7. The issue is that RHEL7's httpd change is in the release, not the version, so <IfVersion> can't be used to 'protect' the change - and trying to use HttpProtocolOptions on an httpd that doesn't support it, causes httpd to fail and exit. Which is rude.

Comment 6 Grant Gainey 2017-04-26 13:36:34 UTC
Setting “HTTPProtocolOptions unsafe” in /etc/httpd/conf.d/zz-spacewalk-server-wsgi.conf also works :)

Comment 12 Joe Orton 2017-05-09 15:00:09 UTC
We'll be able to improve the workaround for future Spacewalk + RHEL7 httpd, but I'm closing this out since it's not a bug, Grant's commentary in comment 5 covers the options/workarounds.