Bug 1442477 - latest httpd update 2.4.6-45.el7_3.4 is breaking Spacewalk
Summary: latest httpd update 2.4.6-45.el7_3.4 is breaking Spacewalk
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: httpd
Version: 7.3
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Luboš Uhliarik
QA Contact: BaseOS QE - Apps
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-04-14 22:19 UTC by Leonid Kanter
Modified: 2017-05-09 15:00 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-05-09 15:00:09 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1445428 0 unspecified CLOSED httpd: HTTP Error 400 - Bad Request 2021-02-22 00:41:40 UTC

Internal Links: 1445428

Description Leonid Kanter 2017-04-14 22:19:20 UTC
Description of problem:

I'm running two instances of Spacewalk - 2.4 and 2.6. On both installations after the httpd update clients stopped to receive yum metadata and rpm packages with error 400. Rolling back this update fixed this problem immediately.

Version-Release number of selected component (if applicable):

2.4.6-45.el7_3.4

How reproducible: 

always

Steps to Reproduce:
1. update httpd on Spacewalk server
2. run yum update on spacewalk client system

Actual results:

On client you will have:

failed to retrieve repodata/repomd.xml from epel-x86_64-7
error was [Errno 14] HTTPS Error 400 - Bad Request

In Spacewalk server logs you will have:

192.168.1.78 - - [14/Apr/2017:17:53:11 +0000] "GET /XMLRPC//GET-REQ/centos-x86_64-7/repodata/repomd.xml HTTP/1.1" 400 226

Expected results:

spacewalk client should be able to get yum repodata and updates

Additional info:

Turnend on debug, I found following in httpd logs:

[Fri Apr 14 18:11:12.193177 2017] [core:debug] [pid 6486] protocol.c(956): (22)Invalid argument: [client 192.168.1.78:42834] Failed to read request header line X-RHN-Auth-User-Id:
[Fri Apr 14 18:11:12.193220 2017] [core:debug] [pid 6486] protocol.c(1309): [client 192.168.1.78:42834] AH00567: request failed: error reading the headers

Since the policy of header processing was changed in patch httpd-2.4.6-CVE-2016-8743.patch, I tried to recompile httpd with this patch commented out, and it fixed the problem.

Comment 2 Luboš Uhliarik 2017-04-18 15:47:51 UTC
Have you also updated yum-rhn-plugin ?

Comment 3 Luboš Uhliarik 2017-04-18 16:04:07 UTC
Leonid,

I just spoke to SW developers, and they told me, fix for this issue will be included in SW 2.7. 

So you have 2 options:

a) roll back httpd on your SW instance

b) apply the changes from fdf2063e14fc2539f09efcb7e72dd85060ab9ae5 yourself

Comment 4 Leonid Kanter 2017-04-18 18:40:08 UTC
Hello Luboš,

Yes, I verified that yum-rhn-plugin-2.0.1-6.1.el7_3.noarch works with new httpd, but it's unavailable for el6 and most of my clients are using yum-rhn-plugin-2.6.3-1. I found other solution: instead of rolling back httpd I sat an option


HttpProtocolOptions Unsafe

in /etc/httpd/conf.d/zz-spacewalk-server-wsgi.conf

It restored behavior of the header parser.

Comment 5 Grant Gainey 2017-04-26 13:35:44 UTC
So what's happening here is the result of an update to httpd, that enforces strict whitespace rules, that uncovered a bug in yum-rhn-plugin that was introduced in 2010.

(The spacewalk commit that started us down this path, for anyone who is interested, is 213b97e38dd75007281b37e3180dc68b736f2fac )

yum-rhn-plugin has been fixed, but a client without the fix yet won't be able to call home to *get* the fix, if the server has the new httpd.

There are two workarounds available:

One is to downgrade httpd

The other is to add an httpd config option that *turns off" strict-whitespace-enforcing until all your clients have a corrected yum-rhn-plugin. You can do that with the following:

# echo “HTTPProtocolOptions unsafe” >> /etc/httpd/conf.d/123unsafe.conf; systemctl restart httpd

That will put httpd back to its pre-2.4.25 behavior.

spacewalk-nightly deploys the config-option reliably for Fedora and RHEL6. We're still working on finding a way to deploy that config-change reliably on RHEL7. The issue is that RHEL7's httpd change is in the release, not the version, so <IfVersion> can't be used to 'protect' the change - and trying to use HttpProtocolOptions on an httpd that doesn't support it, causes httpd to fail and exit. Which is rude.

Comment 6 Grant Gainey 2017-04-26 13:36:34 UTC
Setting “HTTPProtocolOptions unsafe” in /etc/httpd/conf.d/zz-spacewalk-server-wsgi.conf also works :)

Comment 12 Joe Orton 2017-05-09 15:00:09 UTC
We'll be able to improve the workaround for future Spacewalk + RHEL7 httpd, but I'm closing this out since it's not a bug, Grant's commentary in comment 5 covers the options/workarounds.


Note You need to log in before you can comment on or make changes to this bug.