Bug 1002441

+++ This bug was initially created as a clone of Bug #1002007 +++

Description of problem:

We upgraded Spacewalk to version 2.0. the master instance looks fine and everything looks fine.

But we have problems after upgrade of the proxy instance in our datacenter.

Now all servers behind the proxy are not able to query the master instance with error: “XMLRPC ProtocolError: <ProtocolError for ***proxy*** /XMLRPC: 500 Internal Server Error>”

Version-Release number of selected component (if applicable):

CentOS release 5.9 (Final)
x86_64

spacewalk-ssl-cert-check-2.3-1.el5
spacewalk-base-minimal-2.0.3-1.el5
spacewalk-proxy-installer-2.0.1-1.el5
spacewalk-proxy-package-manager-2.0.1-1.el5
spacewalk-proxy-redirect-2.0.1-1.el5
spacewalk-backend-libs-2.0.3-1.el5
spacewalk-proxy-docs-2.0.1-1.el5
spacewalk-backend-2.0.3-1.el5
spacewalk-setup-jabberd-2.0.1-1.el5
spacewalk-monitoring-selinux-2.0.1-1.el5
spacewalk-proxy-html-2.0.1-1.el5
spacewalk-proxy-common-2.0.1-1.el5
spacewalk-proxy-selinux-2.0.1-1.el5
spacewalk-certs-tools-2.0.1-1.el5
spacewalk-proxy-monitoring-2.0.1-1.el5
spacewalk-proxy-broker-2.0.1-1.el5
spacewalk-proxy-management-2.0.1-1.el5
rhncfg-5.10.55-1.el5
rhncfg-management-5.10.55-1.el5
yum-rhn-plugin-2.0.1-1.el5
rhncfg-actions-5.10.55-1.el5
rhn-check-2.0.1-1.el5
rhn-client-tools-2.0.1-1.el5
rhncfg-client-5.10.55-1.el5
rhnsd-5.0.13-1.el5
rhnlib-2.5.65-1.el5
rhnpush-5.5.65-1.el5
rhn-setup-2.0.1-1.el5
rhn-org-httpd-ssl-key-pair-spwkf1p1-1.0-11

How reproducible:

doing "yum repolist" on client system

Actual results:

Mail report from proxy server : 

Exception reported from spwkf1p1
Time: Tue Aug 27 16:06:45 2013
Exception type exceptions.KeyError
Exception while handling function handler Request object information:
URI: /XMLRPC
Remote Host:  xxx.xxx.xxx.xxx
Server Name: spwkf1p1.secret.local:443
Headers passed in:
	Accept-Encoding: identity
	Content-Length: 2709
	Host: spwkf1p1.secret.local
	content-type: text/xml
	user-agent: rhn.rpclib.py/$Revision$
	x-client-version: 1
	x-info: RPC Processor (C) Red Hat, Inc (version $Revision$)
	x-rhn-client-capability: packages.verify(1)=1,packages.extended_profile(2)=1,configfiles.base64_enc(1)=1,reboot.reboot(1)=1,configfiles.deploy(1)=1,caneatCheese(1)=1,packages.rollBack(1)=1,configfiles.mtime_upload(1)=1,configfiles.diff(1)=1,packages.update(2)=2,script.run(1)=1,configfiles.upload(1)=1,packages.runTransaction(1)=1,packages.verify(1)=1,packages.extended_profile(2)=1,configfiles.base64_enc(1)=1,reboot.reboot(1)=1,configfiles.deploy(1)=1,caneatCheese(1)=1,packages.rollBack(1)=1,configfiles.mtime_upload(1)=1,configfiles.diff(1)=1,packages.update(2)=2,script.run(1)=1,configfiles.upload(1)=1,packages.runTransaction(1)=1
	x-rhn-transport-capability: follow-redirects=3
	x-transport-info: Extended Capabilities Transport (C) Red Hat, Inc (version $Revision$)
	x-up2date-version: 1.7.14-1.el6
Extra information about this error:
Unhandled exception type

Exception Handler Information
Traceback (most recent call last):
  File "/usr/share/rhn/proxy/apacheServer.py", line 62, in __call__
    ret = f(req)
  File "/usr/share/rhn/proxy/apacheHandler.py", line 355, in handler
    ret = handlerObj.handler()
  File "/usr/share/rhn/proxy/broker/rhnBroker.py", line 215, in handler
    status = self._serverCommo()       # part 2
  File "/usr/share/rhn/proxy/rhnShared.py", line 201, in _serverCommo
    status, headers, bodyFd = self._proxy2server()
  File "/usr/share/rhn/proxy/rhnShared.py", line 366, in _proxy2server
    http_connection.send(self.req.headers_in['wsgi.input'])
KeyError: 'wsgi.input'

Local variables by frame
Frame _proxy2server in /usr/share/rhn/proxy/rhnShared.py at line 366
	     http_connection = <type 'instance'> <rhn.connections.HTTPSConnection instance at 0x2aefa4f55200>
	                hdrs = <type 'instance'> {'content-length': '2709', 'x-client-version': '1', 'accept-encoding': 'identity', 'x-rhn-proxy-version': '5.5.0', 'x-transport-info': 'Extended Capabilities Transport (C) Red Hat, Inc (version $Revision$)', 'x-up2date-version': '1.7.14-1.el6', 'user-agent': 'rhn.rpclib.py/$Revision$', 'x-rhn-ip-path': ' xxx.xxx.xxx.xxx', 'host': 'spwkf1p1.secret.local', 'x-rhn-transport-capability': 'follow-redirects=3', 'x-rhn-proxy-auth': '1000010002::1377594702.78:21600.0:1ti06mIfYFzXS4uChdC2Mg==:spwkf1p1.secret.local', 'x-info': 'RPC Processor (C) Red Hat, Inc (version $Revision$)', 'content-type': 'text/xml', 'x-rhn-client-capability': 'packages.verify(1)=1,packages.extended_profile(2)=1,configfiles.base64_enc(1)=1,reboot.reboot(1)=1,configfiles.deploy(1)=1,caneatCheese(1)=1,packages.rollBack(1)=1,configfiles.mtime_upload(1)=1,configfiles.diff(1)=1,packages.update(2)=2,script.run(1)=1,configfiles.upload(1)=1,packages.runTransaction(1)=1,packages.ver
 ify(1)=1,packages.extended_profile(2)=1,configfiles.base64_enc(1)=1,reboot.reboot(1)=1,configfiles.deploy(1)=1,caneatCheese(1)=1,packages.rollBack(1)=1,configfiles.mtime_upload(1)=1,configfiles.diff(1)=1,packages.update(2)=2,script.run(1)=1,configfiles.upload(1)=1,packages.runTransaction(1)=1'}
	                   k = <type 'str'> X-RHN-Proxy-Auth
	                   v = <type 'str'> 1000010002::1377594702.78:21600.0:1ti06mIfYFzXS4uChdC2Mg==:spwkf1p1.secret.local
	                vals = <type 'list'> ['1000010002::1377594702.78:21600.0:1ti06mIfYFzXS4uChdC2Mg==:spwkf1p1.secret.local']
	                self = <type 'instance'> <proxy.broker.rhnBroker.BrokerHandler instance at 0x2aefa4e4ef38>
	                size = <type 'int'> 2709

Frame _serverCommo in /usr/share/rhn/proxy/rhnShared.py at line 214
	                self = <type 'instance'> <proxy.broker.rhnBroker.BrokerHandler instance at 0x2aefa4e4ef38>

Frame handler in /usr/share/rhn/proxy/broker/rhnBroker.py at line 215
	           authToken = <type 'str'> 1000010002::1377594702.78:21600.0:1ti06mIfYFzXS4uChdC2Mg==:spwkf1p1.secret.local
	           client_ip = <type 'str'>  xxx.xxx.xxx.xxx
	                self = <type 'instance'> <proxy.broker.rhnBroker.BrokerHandler instance at 0x2aefa4e4ef38>
	                _oto = <type 'instance'> {'content-length': '2709', 'x-client-version': '1', 'accept-encoding': 'identity', 'x-rhn-proxy-version': '5.5.0', 'x-transport-info': 'Extended Capabilities Transport (C) Red Hat, Inc (version $Revision$)', 'x-up2date-version': '1.7.14-1.el6', 'user-agent': 'rhn.rpclib.py/$Revision$', 'x-rhn-ip-path': ' xxx.xxx.xxx.xxx', 'host': 'spwkf1p1.secret.local', 'x-rhn-transport-capability': 'follow-redirects=3', 'x-rhn-proxy-auth': '1000010002::1377594702.78:21600.0:1ti06mIfYFzXS4uChdC2Mg==:spwkf1p1.secret.local', 'x-info': 'RPC Processor (C) Red Hat, Inc (version $Revision$)', 'content-type': 'text/xml', 'x-rhn-client-capability': 'packages.verify(1)=1,packages.extended_profile(2)=1,configfiles.base64_enc(1)=1,reboot.reboot(1)=1,configfiles.deploy(1)=1,caneatCheese(1)=1,packages.rollBack(1)=1,configfiles.mtime_upload(1)=1,configfiles.diff(1)=1,packages.update(2)=2,script.run(1)=1,configfiles.upload(1)=1,packages.runTransaction(1)=1,packages.ver
 ify(1)=1,packages.extended_profile(2)=1,configfiles.base64_enc(1)=1,reboot.reboot(1)=1,configfiles.deploy(1)=1,caneatCheese(1)=1,packages.rollBack(1)=1,configfiles.mtime_upload(1)=1,configfiles.diff(1)=1,packages.update(2)=2,script.run(1)=1,configfiles.upload(1)=1,packages.runTransaction(1)=1'}
	           getResult = <type 'NoneType'> None
	              tokens = <type 'list'> ['1000010002::1377594702.78:21600.0:1ti06mIfYFzXS4uChdC2Mg==:spwkf1p1.secret.local']
	             ip_path = <type 'str'>  xxx.xxx.xxx.xxx
	                  _i = <type 'int'> 0

Frame handler in /usr/share/rhn/proxy/apacheHandler.py at line 357
	          handlerObj = <type 'instance'> <proxy.broker.rhnBroker.BrokerHandler instance at 0x2aefa4e4ef38>
	                self = <type 'instance'> <proxy.apacheHandler.apacheHandler instance at 0x2aefa4d9bb48>
	                 req = <type 'mp_request'> <mp_request object at 0x2aefa4ade050>
	                 ret = <type 'int'> 0
	           rhnBroker = <type 'module'> <module 'proxy.broker.rhnBroker' from '/usr/share/rhn/proxy/broker/rhnBroker.pyc'>

Frame __call__ in /usr/share/rhn/proxy/apacheServer.py at line 68
	                self = <type 'instance'> <proxy.apacheServer.HandlerWrap instance at 0x2aefa4d0d908>
	                 req = <type 'mp_request'> <mp_request object at 0x2aefa4ade050>
	                   f = <type 'instancemethod'> <bound method apacheHandler.handler of <proxy.apacheHandler.apacheHandler instance at 0x2aefa4d9bb48>>

Frame HandlerDispatch in /usr/lib64/python2.4/site-packages/mod_python/apache.py at line 299
	                 req = <type 'mp_request'> <mp_request object at 0x2aefa4ade050>
	              config = <type 'mp_table'> {'PythonPath': 'sys.path+['/usr/share/rhn']'}
	                self = <type 'instance'> <mod_python.apache.CallBack instance at 0x2aefa4ad6bd8>
	              object = <type 'instance'> <proxy.apacheServer.HandlerWrap instance at 0x2aefa4d0d908>
	                   l = <type 'list'> ['proxy.apacheServer', 'Handler']
	              module = <type 'module'> <module 'proxy.apacheServer' from '/usr/share/rhn/proxy/apacheServer.pyc'>
	               hlist = <type 'mp_hlist'> {'handler:'proxy.apacheServer::Handler','directory':'/XMLRPC/','silent':0}
	          object_str = <type 'str'> Handler
	               debug = <type 'int'> 0
	         module_name = <type 'str'> proxy.apacheServer
	          pathstring = <type 'str'> sys.path+['/usr/share/rhn']
	              result = <type 'int'> 500


Environment for PID=16456 on exception:
LANG = C
PATH = /sbin:/usr/sbin:/bin:/usr/bin
PWD = /
SHLVL = 2
TERM = xterm
_ = /usr/sbin/httpd



Expected results:


Additional info:

Database :
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production