Bug 1445428 - httpd: HTTP Error 400 - Bad Request
Summary: httpd: HTTP Error 400 - Bad Request
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Spacewalk
Classification: Community
Component: Server
Version: 2.6
Hardware: Unspecified
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Tomáš Kašpárek
QA Contact: Red Hat Satellite QA List
URL:
Whiteboard:
Depends On:
Blocks: space27
TreeView+ depends on / blocked
 
Reported: 2017-04-25 16:09 UTC by vinzenz.meier
Modified: 2017-09-28 18:07 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-06-19 08:17:36 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1442477 0 unspecified CLOSED latest httpd update 2.4.6-45.el7_3.4 is breaking Spacewalk 2021-02-22 00:41:40 UTC

Internal Links: 1442477

Description vinzenz.meier 2017-04-25 16:09:56 UTC
Description of problem:
after updating httpd, XMLRPC calls return '400 Bad request' 

Version-Release number of selected component (if applicable):
  Updating   : httpd-tools-2.4.6-45.el7_3.4.x86_64                                                                                                 1/6 
  Updating   : httpd-2.4.6-45.el7_3.4.x86_64                                                                                                       2/6 
  Updating   : 1:mod_ssl-2.4.6-45.el7_3.4.x86_64                                                                                                   3/6 
  Cleanup    : 1:mod_ssl-2.4.6-45.el7.x86_64                                                                                                       4/6 
  Cleanup    : httpd-2.4.6-45.el7.x86_64                                                                                                           5/6 
  Cleanup    : httpd-tools-2.4.6-45.el7.x86_64                                                                                                     6/6


How reproducible:


Steps to Reproduce:
1.[server] yum update httpd
2.[server] service httpd restart
3.[client] yumdownloader kernel

Actual results:
client:
# rm -f kernel-3.10.0-514.16.1.el7.x86_64.rpm ; yumdownloader kernel
Loaded plugins: product-id, rhnplugin
This system is receiving updates from RHN Classic or Red Hat Satellite.
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
kernel-3.10.0-514.16.1.el7.x86_64: failed to retrieve getPackage/kernel-3.10.0-514.16.1.el7.x86_64.rpm from rhel-x86_64-server-7
error was [Errno 14] HTTP Error 400 - Bad Request


Expected results:
# rm -f kernel-3.10.0-514.16.1.el7.x86_64.rpm ; yumdownloader kernel
Loaded plugins: product-id, rhnplugin
This system is receiving updates from RHN Classic or Red Hat Satellite.
kernel-3.10.0-514.16.1.el7.x86_64.rpm                                                                                                                                                                                                         |  37 MB  00:00:00     

Additional info:
client:
# rpm -qa|egrep 'rhn|spacewalk'
yum-rhn-plugin-2.6.3-1.el7.noarch
rhn-client-tools-2.6.8-1.el7.noarch
rhn-check-2.6.8-1.el7.noarch
rhnsd-5.0.25-1.el7.x86_64
rhn-setup-2.6.8-1.el7.noarch
rhnlib-2.6.3-1.el7.noarch

server:
# rpm -qa|egrep 'httpd|spacewalk|rhn'|sort
httpd-2.4.6-45.el7_3.4.x86_64
httpd-tools-2.4.6-45.el7_3.4.x86_64
rhn-check-2.6.8-1.el7.noarch
rhn-client-tools-2.6.8-1.el7.noarch
rhnlib-2.6.3-1.el7.noarch
rhnpush-5.5.101-1.el7.noarch
rhnsd-5.0.25-1.el7.x86_64
rhn-setup-2.6.8-1.el7.noarch
spacewalk-admin-2.6.1-1.el7.noarch
spacewalk-backend-2.6.77-1.el7.noarch
spacewalk-backend-app-2.6.77-1.el7.noarch
spacewalk-backend-applet-2.6.77-1.el7.noarch
spacewalk-backend-config-files-2.6.77-1.el7.noarch
spacewalk-backend-config-files-common-2.6.77-1.el7.noarch
spacewalk-backend-config-files-tool-2.6.77-1.el7.noarch
spacewalk-backend-iss-2.6.77-1.el7.noarch
spacewalk-backend-iss-export-2.6.77-1.el7.noarch
spacewalk-backend-libs-2.6.77-1.el7.noarch
spacewalk-backend-package-push-server-2.6.77-1.el7.noarch
spacewalk-backend-server-2.6.77-1.el7.noarch
spacewalk-backend-sql-2.6.77-1.el7.noarch
spacewalk-backend-sql-postgresql-2.6.77-1.el7.noarch
spacewalk-backend-tools-2.6.77-1.el7.noarch
spacewalk-backend-usix-2.6.77-1.el7.noarch
spacewalk-backend-xml-export-libs-2.6.77-1.el7.noarch
spacewalk-backend-xmlrpc-2.6.77-1.el7.noarch
spacewalk-base-2.6.6-1.el7.noarch
spacewalk-base-minimal-2.6.6-1.el7.noarch
spacewalk-base-minimal-config-2.6.6-1.el7.noarch
spacewalk-branding-2.5.3-1.el7.noarch
spacewalk-certs-tools-2.5.3-1.el7.noarch
spacewalk-common-2.6.1-1.el7.noarch
spacewalk-config-2.6.5-1.el7.noarch
spacewalk-dobby-2.6.6-1.el7.noarch
spacewalk-doc-indexes-2.6.2-1.el7.noarch
spacewalk-html-2.6.6-1.el7.noarch
spacewalk-java-2.6.49-1.el7.noarch
spacewalk-java-config-2.6.49-1.el7.noarch
spacewalk-java-lib-2.6.49-1.el7.noarch
spacewalk-java-postgresql-2.6.49-1.el7.noarch
spacewalk-jpp-workaround-2.3.5-1.el7.noarch
spacewalk-postgresql-2.6.1-1.el7.noarch
spacewalk-reports-2.6.3-1.el7.noarch
spacewalk-schema-2.6.17-1.el7.noarch
spacewalk-search-2.6.1-1.el7.noarch
spacewalk-selinux-2.3.2-1.el7.noarch
spacewalk-setup-2.6.2-1.el7.noarch
spacewalk-setup-jabberd-2.3.2-1.el7.noarch
spacewalk-setup-postgresql-2.6.2-1.el7.noarch
spacewalk-taskomatic-2.6.49-1.el7.noarch
spacewalk-utils-2.6.17-1.el7.noarch
yum-rhn-plugin-2.6.3-1.el7.noarch


xx - - [25/Apr/2017:17:53:58 +0200] "GET /XMLRPC/GET-REQ/rhel-x86_64-server-7/getPackage/kernel-3.10.0-514.16.1.el7.x86_64.rpm HTTP/1.1" 200 39093376 "-" "rhn.rpclib.py/2.6.3-1.el7"
xx - - [25/Apr/2017:17:54:41 +0200] "GET /XMLRPC/GET-REQ/rhel-x86_64-server-7/getPackage/kernel-3.10.0-514.16.1.el7.x86_64.rpm HTTP/1.1" 400 226 "-" "rhn.rpclib.py/2.6.3-1.el7"

Comment 1 c.lilwah 2017-04-26 03:50:51 UTC
I too am having this issue.
I did a fresh install of spacewalk yesterday

apachectl -v
Server version: Apache/2.4.6 (CentOS)
Server built:   Apr 12 2017 21:03:28


On the client, if I edit the serverURL in /etc/sysconfig/rhn/up2date to either http or https it gives me the same error on yum check-update

http:
failed to retrieve repodata/repomd.xml from ****
error was [Errno 14] HTTP Error 400 - Bad Request

https
failed to retrieve repodata/repomd.xml from ****
error was [Errno 14] HTTPS Error 400 - Bad Request

additionally, I get D: local action status: ((6,), 'Fatal error in Python code occurred', {}) from rhn_check -vv

rhn_check -vv

D: do_call packages.checkNeedUpdate('rhnsd=1',){}
Loaded plugins: fastestmirror, rhnplugin
D: login(forceUpdate=False) invoked
D: readCachedLogin invoked
D: Checking pickled loginInfo, currentTime=1493177702.55, createTime=1493177520.76, expire-offset=3600.0
D: readCachedLogin(): using pickled loginInfo set to expire at 1493181120.76
D: rpcServer: Calling XMLRPC up2date.listChannels
This system is receiving updates from RHN Classic or Red Hat Satellite.
Loading mirror speeds from cached hostfile
 * base: mirror.netflash.net
 * epel: mirror.steadfast.net
 * extras: centos.mirror.rafal.ca
 * updates: mirror.csclub.uwaterloo.ca
updateLoginInfo() login info
D: login(forceUpdate=True) invoked
logging into up2date server
D: rpcServer: Calling XMLRPC up2date.login
D: writeCachedLogin() invoked
D: Wrote pickled loginInfo at 1493177705.25 with expiration of 1493181305.25 seconds.
successfully retrieved authentication token from up2date server
D: logininfo:{'X-RHN-Server-Id': 1000010002, 'X-RHN-Auth-Server-Time': '1493177706.24', 'X-RHN-Auth-Channels': [['centos7_parent', '20170425161252', '1', '1'], ['centos7', '20170425161948', '0', '1']], 'X-RHN-Auth': 'CXy/TcybsxQtQG1DkALcWy2GDW/ceJz9flgU4lXOsRs=', 'X-RHN-Auth-User-Id': '', 'X-RHN-Auth-Expire-Offset': '3600.0'}
updateLoginInfo() login info
D: login(forceUpdate=True) invoked
logging into up2date server
D: rpcServer: Calling XMLRPC up2date.login
D: writeCachedLogin() invoked
D: Wrote pickled loginInfo at 1493177705.29 with expiration of 1493181305.29 seconds.
successfully retrieved authentication token from up2date server
D: logininfo:{'X-RHN-Server-Id': 1000010002, 'X-RHN-Auth-Server-Time': '1493177706.27', 'X-RHN-Auth-Channels': [['centos7_parent', '20170425161252', '1', '1'], ['centos7', '20170425161948', '0', '1']], 'X-RHN-Auth': 'w9GnU+NtfqOO5uc7e6VQGJCd664a4kzfNJ/pBgO4ya8=', 'X-RHN-Auth-User-Id': '', 'X-RHN-Auth-Expire-Offset': '3600.0'}
updateLoginInfo() login info
D: login(forceUpdate=True) invoked
logging into up2date server
D: rpcServer: Calling XMLRPC up2date.login
D: writeCachedLogin() invoked
D: Wrote pickled loginInfo at 1493177705.34 with expiration of 1493181305.34 seconds.
successfully retrieved authentication token from up2date server
D: logininfo:{'X-RHN-Server-Id': 1000010002, 'X-RHN-Auth-Server-Time': '1493177706.33', 'X-RHN-Auth-Channels': [['centos7_parent', '20170425161252', '1', '1'], ['centos7', '20170425161948', '0', '1']], 'X-RHN-Auth': 'J0Lvv2Vwu3SNSMw4y5ws+k7tMyLcWwzZLPorNEPJYU8=', 'X-RHN-Auth-User-Id': '', 'X-RHN-Auth-Expire-Offset': '3600.0'}
D: local action status: ((6,), 'Fatal error in Python code occurred', {})
D: rpcServer: Calling XMLRPC registration.welcome_message



also tried visiting the url in web browser. output is below.

Request URL:http://[my spacewalk fqdn]/XMLRPC/GET-REQ/centos7/repodata/repomd.xml
Request Method:GET
Status Code:405 Method Not Allowed
Remote Address:[spacewalk server ip address]:80
Referrer Policy:no-referrer-when-downgrade

same response with wget.


for comparison to the above packages, here is my package list as well

client:
rpm -qa|egrep 'rhn|spacewalk'
rhn-client-tools-2.6.8-1.el7.noarch
rhn-setup-2.6.8-1.el7.noarch
rhn-check-2.6.8-1.el7.noarch
rhn-org-trusted-ssl-cert-1.0-1.noarch
spacewalk-backend-usix-2.6.77-1.el7.noarch
rhncfg-client-5.10.99-1.el7.noarch
rhncfg-management-5.10.99-1.el7.noarch
spacewalk-client-repo-2.6-0.el7.noarch
rhnlib-2.6.3-1.el7.noarch
yum-rhn-plugin-2.6.3-1.el7.noarch
rhnsd-5.0.25-1.el7.x86_64
rhncfg-5.10.99-1.el7.noarch
rhncfg-actions-5.10.99-1.el7.noarch


server:
rpm -qa|egrep 'httpd|spacewalk|rhn'|sort
httpd-2.4.6-45.el7.centos.4.x86_64
httpd-tools-2.4.6-45.el7.centos.4.x86_64
rhn-client-tools-2.6.8-1.el7.noarch
rhnlib-2.6.3-1.el7.noarch
rhn-org-httpd-ssl-key-pair-spacewalk.lab-1.0-1.noarch
rhnpush-5.5.101-1.el7.noarch
spacewalk-admin-2.6.1-1.el7.noarch
spacewalk-backend-2.6.77-1.el7.noarch
spacewalk-backend-app-2.6.77-1.el7.noarch
spacewalk-backend-applet-2.6.77-1.el7.noarch
spacewalk-backend-config-files-2.6.77-1.el7.noarch
spacewalk-backend-config-files-common-2.6.77-1.el7.noarch
spacewalk-backend-config-files-tool-2.6.77-1.el7.noarch
spacewalk-backend-iss-2.6.77-1.el7.noarch
spacewalk-backend-iss-export-2.6.77-1.el7.noarch
spacewalk-backend-libs-2.6.77-1.el7.noarch
spacewalk-backend-package-push-server-2.6.77-1.el7.noarch
spacewalk-backend-server-2.6.77-1.el7.noarch
spacewalk-backend-sql-2.6.77-1.el7.noarch
spacewalk-backend-sql-postgresql-2.6.77-1.el7.noarch
spacewalk-backend-tools-2.6.77-1.el7.noarch
spacewalk-backend-usix-2.6.77-1.el7.noarch
spacewalk-backend-xml-export-libs-2.6.77-1.el7.noarch
spacewalk-backend-xmlrpc-2.6.77-1.el7.noarch
spacewalk-base-2.6.6-1.el7.noarch
spacewalk-base-minimal-2.6.6-1.el7.noarch
spacewalk-base-minimal-config-2.6.6-1.el7.noarch
spacewalk-branding-2.5.3-1.el7.noarch
spacewalk-certs-tools-2.5.3-1.el7.noarch
spacewalk-common-2.6.1-1.el7.noarch
spacewalk-config-2.6.5-1.el7.noarch
spacewalk-doc-indexes-2.6.2-1.el7.noarch
spacewalk-html-2.6.6-1.el7.noarch
spacewalk-java-2.6.49-1.el7.noarch
spacewalk-java-config-2.6.49-1.el7.noarch
spacewalk-java-lib-2.6.49-1.el7.noarch
spacewalk-java-postgresql-2.6.49-1.el7.noarch
spacewalk-jpp-workaround-2.3.5-1.el7.noarch
spacewalk-postgresql-2.6.1-1.el7.noarch
spacewalk-repo-2.6-0.el7.noarch
spacewalk-schema-2.6.17-1.el7.noarch
spacewalk-search-2.6.1-1.el7.noarch
spacewalk-selinux-2.3.2-1.el7.noarch
spacewalk-setup-2.6.2-1.el7.noarch
spacewalk-setup-jabberd-2.3.2-1.el7.noarch
spacewalk-setup-postgresql-2.6.2-1.el7.noarch
spacewalk-taskomatic-2.6.49-1.el7.noarch


i hope this additional information helps

Comment 2 Grant Gainey 2017-04-26 13:31:29 UTC
So what's happening here is the result of an update to httpd, that enforces strict whitespace rules, that uncovered a bug in yum-rhn-plugin that was introduced in 2010.

(THhe spacewalk commit that started us down this path, for anyone who is interested, is 213b97e38dd75007281b37e3180dc68b736f2fac )

yum-rhn-plugin has been fixed, but a client without the fix yet won't be able to call home to *get* the fix, if the server has the new httpd.

There are two workarounds available:

One is to downgrade httpd to something *before* 2.4.25.

The other is to add an httpd config option that *turns off" strict-whitespace-enforcing until all your clients have a corrected yum-rhn-plugin. You can do that with the following:

# echo “HTTPProtocolOptions unsafe” >> /etc/httpd/conf.d/123unsafe.conf; systemctl restart httpd

That will put httpd back to its pre-2.4.25 behavior.

spacewalk-nightly has both fixes, so they will be standard when 2.7 is released

Comment 3 c.lilwah 2017-04-26 14:04:41 UTC
Hi Grant

I have tested and confirmed it works. I did believe that it was related to the strict whitespace rules but then I saw in 

/etc/httpd/conf.d/aa-spacewalk-server.conf

<IfVersion >= 2.4.25>
  HTTPProtocolOptions unsafe
</IfVersion>

<IfDefine _RH_HAS_HTTPPROTOCOLOPTIONS>
  HTTPProtocolOptions unsafe
</IfDefine>

so I thought it was already covered by those.

the above fix seems more global.

thanks for your help

Comment 4 vinzenz.meier 2017-04-26 14:11:10 UTC
The latest httpd on RHEL7 is 
Available Packages
Name        : httpd
Arch        : x86_64
Version     : 2.4.6
Release     : 45.el7_3.4
Size        : 1.2 M
Repo        : rhel-x86_64-server-7
Summary     : Apache HTTP Server
License     : ASL 2.0
Description : The Apache HTTP Server is a powerful, efficient, and extensible
            : web server.


So the strict whitespaceing must have been backported in this latest release - and the version macro cannot catch this

Comment 5 Grant Gainey 2017-04-26 14:38:12 UTC
(In reply to vinzenz.meier from comment #4)
> The latest httpd on RHEL7 is 
> Available Packages
> Name        : httpd
> Arch        : x86_64
> Version     : 2.4.6
> Release     : 45.el7_3.4
> Size        : 1.2 M
> Repo        : rhel-x86_64-server-7
> Summary     : Apache HTTP Server
> License     : ASL 2.0
> Description : The Apache HTTP Server is a powerful, efficient, and extensible
>             : web server.
> 
> 
> So the strict whitespaceing must have been backported in this latest release
> - and the version macro cannot catch this

Hey Vinzenz - yes indeed. See the last paragraph in https://bugzilla.redhat.com/show_bug.cgi?id=1442477#c5 , we're investigating the 'right way' to make this work for RHEL7.

Comment 6 Grant Gainey 2017-04-27 13:30:49 UTC
Just published a kbase on the details of what's going on here - see https://access.redhat.com/articles/3013361 for all the fun.

Comment 7 Matt Wilkinson 2017-05-25 17:58:07 UTC
Grant - is there any way for the current yum-rhn-plugin to get updated for Spacewalk Client 2.6 repo? We'd like to fix our clients now and have the httpd updated on the server in a secure fashion. 

If not, I assume we have to wait for SW 2.7?

Comment 8 Matt Wilkinson 2017-06-01 13:58:59 UTC
Looks like it was indeed updated. Thanks!

Comment 9 Grant Gainey 2017-06-01 14:04:58 UTC
(In reply to Matt Wilkinson from comment #8)
> Looks like it was indeed updated. Thanks!

Ah yes - sorry, forgot to update here when we mashed out the client update :)

Comment 10 Tomáš Kašpárek 2017-06-19 08:17:36 UTC
As the update is already present in Spacewalk 2.6 client. I am closing this BZ.

Comment 11 Eric Herget 2017-09-28 18:07:27 UTC
This BZ closed some time during 2.5, 2.6 or 2.7.  Adding to 2.7 tracking bug.


Note You need to log in before you can comment on or make changes to this bug.