Bug 1002007 - proxy "Exception type exceptions.KeyError" and "XMLRPC ProtocolError"
proxy "Exception type exceptions.KeyError" and "XMLRPC ProtocolError"
Status: CLOSED CURRENTRELEASE
Product: Spacewalk
Classification: Community
Component: Proxy Server (Show other bugs)
2.0
x86_64 Linux
unspecified Severity urgent
: ---
: ---
Assigned To: Michael Mráka
Red Hat Satellite QA List
:
Depends On:
Blocks: space20
  Show dependency treegraph
 
Reported: 2013-08-28 06:13 EDT by Sebastien Foucou
Modified: 2013-12-19 07:00 EST (History)
4 users (show)

See Also:
Fixed In Version: spacewalk-proxy-2.0.2-1
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1002441 (view as bug list)
Environment:
Last Closed: 2013-12-19 06:58:43 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Sebastien Foucou 2013-08-28 06:13:06 EDT
Description of problem:

We upgraded Spacewalk to version 2.0. the master instance looks fine and everything looks fine.

But we have problems after upgrade of the proxy instance in our datacenter.

Now all servers behind the proxy are not able to query the master instance with error: “XMLRPC ProtocolError: <ProtocolError for ***proxy*** /XMLRPC: 500 Internal Server Error>”

Version-Release number of selected component (if applicable):

CentOS release 5.9 (Final)
x86_64

spacewalk-ssl-cert-check-2.3-1.el5
spacewalk-base-minimal-2.0.3-1.el5
spacewalk-proxy-installer-2.0.1-1.el5
spacewalk-proxy-package-manager-2.0.1-1.el5
spacewalk-proxy-redirect-2.0.1-1.el5
spacewalk-backend-libs-2.0.3-1.el5
spacewalk-proxy-docs-2.0.1-1.el5
spacewalk-backend-2.0.3-1.el5
spacewalk-setup-jabberd-2.0.1-1.el5
spacewalk-monitoring-selinux-2.0.1-1.el5
spacewalk-proxy-html-2.0.1-1.el5
spacewalk-proxy-common-2.0.1-1.el5
spacewalk-proxy-selinux-2.0.1-1.el5
spacewalk-certs-tools-2.0.1-1.el5
spacewalk-proxy-monitoring-2.0.1-1.el5
spacewalk-proxy-broker-2.0.1-1.el5
spacewalk-proxy-management-2.0.1-1.el5
rhncfg-5.10.55-1.el5
rhncfg-management-5.10.55-1.el5
yum-rhn-plugin-2.0.1-1.el5
rhncfg-actions-5.10.55-1.el5
rhn-check-2.0.1-1.el5
rhn-client-tools-2.0.1-1.el5
rhncfg-client-5.10.55-1.el5
rhnsd-5.0.13-1.el5
rhnlib-2.5.65-1.el5
rhnpush-5.5.65-1.el5
rhn-setup-2.0.1-1.el5
rhn-org-httpd-ssl-key-pair-spwkf1p1-1.0-11

How reproducible:

doing "yum repolist" on client system

Actual results:

Mail report from proxy server : 

Exception reported from spwkf1p1
Time: Tue Aug 27 16:06:45 2013
Exception type exceptions.KeyError
Exception while handling function handler Request object information:
URI: /XMLRPC
Remote Host:  xxx.xxx.xxx.xxx
Server Name: spwkf1p1.secret.local:443
Headers passed in:
	Accept-Encoding: identity
	Content-Length: 2709
	Host: spwkf1p1.secret.local
	content-type: text/xml
	user-agent: rhn.rpclib.py/$Revision$
	x-client-version: 1
	x-info: RPC Processor (C) Red Hat, Inc (version $Revision$)
	x-rhn-client-capability: packages.verify(1)=1,packages.extended_profile(2)=1,configfiles.base64_enc(1)=1,reboot.reboot(1)=1,configfiles.deploy(1)=1,caneatCheese(1)=1,packages.rollBack(1)=1,configfiles.mtime_upload(1)=1,configfiles.diff(1)=1,packages.update(2)=2,script.run(1)=1,configfiles.upload(1)=1,packages.runTransaction(1)=1,packages.verify(1)=1,packages.extended_profile(2)=1,configfiles.base64_enc(1)=1,reboot.reboot(1)=1,configfiles.deploy(1)=1,caneatCheese(1)=1,packages.rollBack(1)=1,configfiles.mtime_upload(1)=1,configfiles.diff(1)=1,packages.update(2)=2,script.run(1)=1,configfiles.upload(1)=1,packages.runTransaction(1)=1
	x-rhn-transport-capability: follow-redirects=3
	x-transport-info: Extended Capabilities Transport (C) Red Hat, Inc (version $Revision$)
	x-up2date-version: 1.7.14-1.el6
Extra information about this error:
Unhandled exception type

Exception Handler Information
Traceback (most recent call last):
  File "/usr/share/rhn/proxy/apacheServer.py", line 62, in __call__
    ret = f(req)
  File "/usr/share/rhn/proxy/apacheHandler.py", line 355, in handler
    ret = handlerObj.handler()
  File "/usr/share/rhn/proxy/broker/rhnBroker.py", line 215, in handler
    status = self._serverCommo()       # part 2
  File "/usr/share/rhn/proxy/rhnShared.py", line 201, in _serverCommo
    status, headers, bodyFd = self._proxy2server()
  File "/usr/share/rhn/proxy/rhnShared.py", line 366, in _proxy2server
    http_connection.send(self.req.headers_in['wsgi.input'])
KeyError: 'wsgi.input'

Local variables by frame
Frame _proxy2server in /usr/share/rhn/proxy/rhnShared.py at line 366
	     http_connection = <type 'instance'> <rhn.connections.HTTPSConnection instance at 0x2aefa4f55200>
	                hdrs = <type 'instance'> {'content-length': '2709', 'x-client-version': '1', 'accept-encoding': 'identity', 'x-rhn-proxy-version': '5.5.0', 'x-transport-info': 'Extended Capabilities Transport (C) Red Hat, Inc (version $Revision$)', 'x-up2date-version': '1.7.14-1.el6', 'user-agent': 'rhn.rpclib.py/$Revision$', 'x-rhn-ip-path': ' xxx.xxx.xxx.xxx', 'host': 'spwkf1p1.secret.local', 'x-rhn-transport-capability': 'follow-redirects=3', 'x-rhn-proxy-auth': '1000010002::1377594702.78:21600.0:1ti06mIfYFzXS4uChdC2Mg==:spwkf1p1.secret.local', 'x-info': 'RPC Processor (C) Red Hat, Inc (version $Revision$)', 'content-type': 'text/xml', 'x-rhn-client-capability': 'packages.verify(1)=1,packages.extended_profile(2)=1,configfiles.base64_enc(1)=1,reboot.reboot(1)=1,configfiles.deploy(1)=1,caneatCheese(1)=1,packages.rollBack(1)=1,configfiles.mtime_upload(1)=1,configfiles.diff(1)=1,packages.update(2)=2,script.run(1)=1,configfiles.upload(1)=1,packages.runTransaction(1)=1,packages.ver
 ify(1)=1,packages.extended_profile(2)=1,configfiles.base64_enc(1)=1,reboot.reboot(1)=1,configfiles.deploy(1)=1,caneatCheese(1)=1,packages.rollBack(1)=1,configfiles.mtime_upload(1)=1,configfiles.diff(1)=1,packages.update(2)=2,script.run(1)=1,configfiles.upload(1)=1,packages.runTransaction(1)=1'}
	                   k = <type 'str'> X-RHN-Proxy-Auth
	                   v = <type 'str'> 1000010002::1377594702.78:21600.0:1ti06mIfYFzXS4uChdC2Mg==:spwkf1p1.secret.local
	                vals = <type 'list'> ['1000010002::1377594702.78:21600.0:1ti06mIfYFzXS4uChdC2Mg==:spwkf1p1.secret.local']
	                self = <type 'instance'> <proxy.broker.rhnBroker.BrokerHandler instance at 0x2aefa4e4ef38>
	                size = <type 'int'> 2709

Frame _serverCommo in /usr/share/rhn/proxy/rhnShared.py at line 214
	                self = <type 'instance'> <proxy.broker.rhnBroker.BrokerHandler instance at 0x2aefa4e4ef38>

Frame handler in /usr/share/rhn/proxy/broker/rhnBroker.py at line 215
	           authToken = <type 'str'> 1000010002::1377594702.78:21600.0:1ti06mIfYFzXS4uChdC2Mg==:spwkf1p1.secret.local
	           client_ip = <type 'str'>  xxx.xxx.xxx.xxx
	                self = <type 'instance'> <proxy.broker.rhnBroker.BrokerHandler instance at 0x2aefa4e4ef38>
	                _oto = <type 'instance'> {'content-length': '2709', 'x-client-version': '1', 'accept-encoding': 'identity', 'x-rhn-proxy-version': '5.5.0', 'x-transport-info': 'Extended Capabilities Transport (C) Red Hat, Inc (version $Revision$)', 'x-up2date-version': '1.7.14-1.el6', 'user-agent': 'rhn.rpclib.py/$Revision$', 'x-rhn-ip-path': ' xxx.xxx.xxx.xxx', 'host': 'spwkf1p1.secret.local', 'x-rhn-transport-capability': 'follow-redirects=3', 'x-rhn-proxy-auth': '1000010002::1377594702.78:21600.0:1ti06mIfYFzXS4uChdC2Mg==:spwkf1p1.secret.local', 'x-info': 'RPC Processor (C) Red Hat, Inc (version $Revision$)', 'content-type': 'text/xml', 'x-rhn-client-capability': 'packages.verify(1)=1,packages.extended_profile(2)=1,configfiles.base64_enc(1)=1,reboot.reboot(1)=1,configfiles.deploy(1)=1,caneatCheese(1)=1,packages.rollBack(1)=1,configfiles.mtime_upload(1)=1,configfiles.diff(1)=1,packages.update(2)=2,script.run(1)=1,configfiles.upload(1)=1,packages.runTransaction(1)=1,packages.ver
 ify(1)=1,packages.extended_profile(2)=1,configfiles.base64_enc(1)=1,reboot.reboot(1)=1,configfiles.deploy(1)=1,caneatCheese(1)=1,packages.rollBack(1)=1,configfiles.mtime_upload(1)=1,configfiles.diff(1)=1,packages.update(2)=2,script.run(1)=1,configfiles.upload(1)=1,packages.runTransaction(1)=1'}
	           getResult = <type 'NoneType'> None
	              tokens = <type 'list'> ['1000010002::1377594702.78:21600.0:1ti06mIfYFzXS4uChdC2Mg==:spwkf1p1.secret.local']
	             ip_path = <type 'str'>  xxx.xxx.xxx.xxx
	                  _i = <type 'int'> 0

Frame handler in /usr/share/rhn/proxy/apacheHandler.py at line 357
	          handlerObj = <type 'instance'> <proxy.broker.rhnBroker.BrokerHandler instance at 0x2aefa4e4ef38>
	                self = <type 'instance'> <proxy.apacheHandler.apacheHandler instance at 0x2aefa4d9bb48>
	                 req = <type 'mp_request'> <mp_request object at 0x2aefa4ade050>
	                 ret = <type 'int'> 0
	           rhnBroker = <type 'module'> <module 'proxy.broker.rhnBroker' from '/usr/share/rhn/proxy/broker/rhnBroker.pyc'>

Frame __call__ in /usr/share/rhn/proxy/apacheServer.py at line 68
	                self = <type 'instance'> <proxy.apacheServer.HandlerWrap instance at 0x2aefa4d0d908>
	                 req = <type 'mp_request'> <mp_request object at 0x2aefa4ade050>
	                   f = <type 'instancemethod'> <bound method apacheHandler.handler of <proxy.apacheHandler.apacheHandler instance at 0x2aefa4d9bb48>>

Frame HandlerDispatch in /usr/lib64/python2.4/site-packages/mod_python/apache.py at line 299
	                 req = <type 'mp_request'> <mp_request object at 0x2aefa4ade050>
	              config = <type 'mp_table'> {'PythonPath': 'sys.path+['/usr/share/rhn']'}
	                self = <type 'instance'> <mod_python.apache.CallBack instance at 0x2aefa4ad6bd8>
	              object = <type 'instance'> <proxy.apacheServer.HandlerWrap instance at 0x2aefa4d0d908>
	                   l = <type 'list'> ['proxy.apacheServer', 'Handler']
	              module = <type 'module'> <module 'proxy.apacheServer' from '/usr/share/rhn/proxy/apacheServer.pyc'>
	               hlist = <type 'mp_hlist'> {'handler:'proxy.apacheServer::Handler','directory':'/XMLRPC/','silent':0}
	          object_str = <type 'str'> Handler
	               debug = <type 'int'> 0
	         module_name = <type 'str'> proxy.apacheServer
	          pathstring = <type 'str'> sys.path+['/usr/share/rhn']
	              result = <type 'int'> 500


Environment for PID=16456 on exception:
LANG = C
PATH = /sbin:/usr/sbin:/bin:/usr/bin
PWD = /
SHLVL = 2
TERM = xterm
_ = /usr/sbin/httpd



Expected results:


Additional info:

Database :
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
Comment 1 Lukas Pramuk 2013-08-29 12:49:02 EDT
The broker is not broking, 

/etc/httpd/error_log:

Exception Handler Information
Traceback (most recent call last):
  File "/usr/share/rhn/proxy/apacheServer.py", line 62, in __call__
    ret = f(req)
  File "/usr/share/rhn/proxy/apacheHandler.py", line 348, in handler
    handlerObj = rhnBroker.BrokerHandler(req)
  File "/usr/share/rhn/proxy/broker/rhnBroker.py", line 70, in __init__
    my_ip_addr = req.headers_in['SERVER_ADDR']
KeyError: 'SERVER_ADDR'
Comment 2 Lukas Pramuk 2013-08-29 13:47:34 EDT
Oops. I meant /var/log/httpd/error_log:
Comment 3 Stephen Herr 2013-08-29 16:00:06 EDT
Lukas, can you tell me specifically what you did to get that error?
Comment 4 Lukas Pramuk 2013-08-30 03:57:23 EDT
client -> proxy (el5) -> satellite

[@client]# rhnreg_ks --username=$USER --password=$PASS --serverUrl=https://$PROXY/XMLRPC --sslCACert=/usr/share/rhn/${PROXY}-TRUSTED-SSL-CERT --force
An error has occurred:
Error communicating with server. The message was:
Internal Server Error
See /var/log/up2date for more information
Comment 5 Lukas Pramuk 2013-08-30 04:02:01 EDT
/var/log/up2date:
...
  File "/usr/share/rhn/up2date_client/rhnserver.py", line 160, in __get_capabilities
    self.registration.welcome_message()
  File "/usr/share/rhn/up2date_client/rhnserver.py", line 51, in __call__
    return rpcServer.doCall(method, *args, **kwargs)
 File "/usr/share/rhn/up2date_client/rpcServer.py", line 278, in doCall
    raise up2dateErrors.CommunicationError(e.errmsg)
<class 'up2date_client.up2dateErrors.CommunicationError'>: Error communicating with server. The message was:
Internal Server Error
Comment 6 Michael Mráka 2013-08-30 07:41:27 EDT
I changed proxy spec to use mod_wsgi on RHEL5 but there are still some issues; investigating difference between RHEL6 and RHEL5...

commit 34c3e9c2c0177345a8f0c0ce2a2a9757e136ec67
    1002007 - use mod_wsgi even on RHEL5
Comment 7 Michael Mráka 2013-08-30 08:35:58 EDT
Fixed python 2.4 httplib issues in Spacewalk master:

commit dafdc3d2245f8f8ec02ddbe92be316a3ab2505a8
    Automatic commit of package [spacewalk-proxy] release [2.1.9-1].
commit cd5f85734304214dc40470b5e4ef6b96c89ffc47
    1002007 - don't send empty data
    addressing
    Traceback (most recent call last):
      File "/usr/share/rhn/proxy/apacheServer.py", line 62, in __call__
        ret = f(req)
      File "/usr/share/rhn/proxy/apacheHandler.py", line 355, in handler
        ret = handlerObj.handler()
      File "/usr/share/rhn/proxy/redirect/rhnRedirect.py", line 92, in handler
        status = self._serverCommo()       # part 2
      File "/usr/share/rhn/proxy/rhnShared.py", line 201, in _serverCommo
        status, headers, bodyFd = self._proxy2server()
      File "/usr/share/rhn/proxy/rhnShared.py", line 370, in _proxy2server
        http_connection.send(data)
      File "/usr/lib64/python2.4/httplib.py", line 664, in send
        self.sock.sendall(str)
      File "/usr/lib/python2.4/site-packages/rhn/SSL.py", line 217, in write
        sent = self._connection.send(data)
    SysCallError: (-1, 'Unexpected EOF')
Comment 8 Michael Mráka 2013-08-30 09:49:59 EDT
Oops, pylint fix

commit eaaa9d48395d4b1b706ee0b48d366f66e9926fda
    fixed pylint error
Comment 9 Sebastien Foucou 2013-09-02 05:02:53 EDT
Hi,

I started this morning an update proxy

First, i installed and activated the mod_wsgi module for apache

Second, i installed the proxy update

The installed version is repository "nightly":
spacewalk-ssl-cert-check-2.3-1.el5
spacewalk-proxy-broker-2.1.10-1.el5
spacewalk-proxy-docs-2.0.1-1.el5
spacewalk-setup-jabberd-2.0.1-1.el5
spacewalk-base-minimal-config-2.1.16-1.el5
spacewalk-proxy-common-2.1.10-1.el5
spacewalk-proxy-management-2.1.10-1.el5
spacewalk-monitoring-selinux-2.0.1-1.el5
spacewalk-proxy-html-2.0.1-1.el5
spacewalk-proxy-selinux-2.0.1-1.el5
spacewalk-certs-tools-2.1.1-1.el5
spacewalk-base-minimal-2.1.16-1.el5
spacewalk-backend-2.1.14-1.el5
spacewalk-proxy-redirect-2.1.10-1.el5
spacewalk-proxy-monitoring-2.0.1-1.el5
spacewalk-backend-libs-2.1.14-1.el5
spacewalk-proxy-installer-2.1.1-1.el5
spacewalk-proxy-package-manager-2.1.10-1.el5

 
I unsubscribed the proxy server on spacewalk : # rhn-proxy-activate --deactivate
I started the proxy configuration: # /usr/sbin/configure-proxy.sh

I did a test from a client and everything works perfectly now.

Thank you very much for your help!

NB: the spacewalk server remained in version 2.0, do you recommend an update to 2.1?
Comment 10 Michael Mráka 2013-09-02 05:30:32 EDT
(In reply to Sebastien Foucou from comment #9)
> Hi,
> 
> I started this morning an update proxy
...
> I did a test from a client and everything works perfectly now.
> 
> Thank you very much for your help!
> 
> NB: the spacewalk server remained in version 2.0, do you recommend an update
> to 2.1?

Hello Sebastien,

Thank you for reporting back.

Please do not upgrade your server, current nightly version is not supposed to be stable and there might not be an easy way to upgrade to final 2.1 once it will be released.

Conversely we feel this is a serious bug and we work on backporting it to current 2.0 proxy version.

Regards,
Michael
Comment 11 Michael Mráka 2013-12-19 06:58:43 EST
The fixed version spacewalk-proxy-2.0.2-1 has been released as a part of Spacewalk 2.0 release.

If the solution does not work for you, please open a new bug report.

Note You need to log in before you can comment on or make changes to this bug.