Bug 976816
| Summary: | bootstrap.sh rhnreg_ks rhn_register all failed client registration only partial registration.. | ||
|---|---|---|---|
| Product: | [Community] Spacewalk | Reporter: | James Edwards <1homeforjames> |
| Component: | Clients | Assignee: | Michael Mráka <mmraka> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Red Hat Satellite QA List <satqe-list> |
| Severity: | urgent | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 1.9 | CC: | 1homeforjames, matt, psherren |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-07-18 09:10:40 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Attachments: | |||
I will need to see the server side logs from this error, in particular /var/log/rhn/rhn_server_xmlrpc.log when registering the problematic machine. Thank you. Created attachment 764567 [details]
spacewalk1 errors
i'm sending multiple spacewalk1 error file but only 1 from spacewalk2
I will send in zipp format then again 1 file at a time
Created attachment 764568 [details]
spacewalk1 and spacewalk2's error file
Created attachment 764569 [details]
space1 file1
Created attachment 764570 [details]
space1 file2
Created attachment 764571 [details]
space1 file3
Created attachment 764572 [details]
space1 file4
Created attachment 764574 [details]
spacewalk2 file1
Spacewalk2 will allow me to register a file an will set up channel correctly and captures all relevant System Info.. but fails to talk to the systems when I attempt a yum update
2013/06/19 11:57:19 -04:00 7688 192.65.40.112: xmlrpc/registration.welcome_message('lang: None',)
2013/06/19 11:57:20 -04:00 7109 192.65.40.112: xmlrpc/registration.create_system("token = '1-centos6'", '6', 'x86_64')
2013/06/19 11:57:20 -04:00 7109 192.65.40.112: rhnServer/server_token.process_token(1000010259, 'enterprise_entitled,monitoring_entitled,provisioning_entitled')
2013/06/19 11:57:21 -04:00 7682 192.65.40.112: xmlrpc/registration.welcome_message('lang: None',)
This is the IP of the system I've been using to test troubouleshoot this issue on both spacewalk serves
After doing a complete rebuild of the CentOS 6.4 channel on spacewalk2 the communication issues still persist.
I've also tried to update the spacewalk server and client packages on spacewalk2 only and the client "192.65.40.112" this broke spacewalk even further.
Comment on attachment 764571 [details]
space1 file3
IP 192.65.40.112 is where the issue is occuring..
again the systems doesn't complety capture the needed information to fully register to spacewalk
Comment on attachment 764571 [details]
space1 file3
IP 192.65.40.112 is where the issue is occuring..
again the systems doesn't complety capture the needed information to fully register to spacewalk
Comment on attachment 764571 [details]
space1 file3
IP 192.65.40.112 is where the issue is occuring..
again the systems doesn't complety capture the needed information to fully register to spacewalk
Created attachment 765166 [details]
more troubleshooting errors from spacewalk1.. looking like CERT in no longer recognized
more troubleshooting errors from spacewalk1.. looking like CERT in no longer recognized
We're having the same issue, but only -after- switching httpd from prefork to worker MPM... we're running into capacity issues managing 7,000+ systems and I'm trying whatever I can to get it tuned as well as possible. Worker seems to be a much better deal, but it seems that there is something, not sure what, that causes rhnreg_ks to fail when httpd is in worker mode. Switching back to prefork fixed the issue immediately on one system (a RHEL6.6 box). I forgot to mention we're on Spacewalk 2.3 release. This was the only other mention I could find so far that matches my own issue as closely.
[me@work ~]$ sudo /usr/sbin/rhnreg_ks --activationkey "1-rhel6-prod"
An error has occurred:
rhn-plugin: Error communicating with server. The message was:
Internal Server Error
See /var/log/up2date for more information
and /var/log/up2date has:
[Thu Aug 27 22:35:59 2015] up2date A protocol error occurred: Internal Server Error , attempt #1,
[Thu Aug 27 22:35:59 2015] up2date
Traceback (most recent call last):
File "/usr/sbin/rhnreg_ks", line 218, in <module>
cli.run()
File "/usr/share/rhn/up2date_client/rhncli.py", line 96, in run
sys.exit(self.main() or 0)
File "/usr/sbin/rhnreg_ks", line 151, in main
rhnreg.sendPackages(systemId, packageList)
File "/usr/share/rhn/up2date_client/rhnreg.py", line 482, in sendPackages
s.registration.add_packages(systemId, packageList)
File "/usr/share/rhn/up2date_client/rhnserver.py", line 63, in __call__
return rpcServer.doCall(method, *args, **kwargs)
File "/usr/share/rhn/up2date_client/rpcServer.py", line 203, in doCall
ret = method(*args, **kwargs)
File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
return self.__send(self.__name, args)
File "/usr/share/rhn/up2date_client/rpcServer.py", line 37, in _request1
ret = self._request(methodname, params)
File "/usr/lib/python2.6/site-packages/rhn/rpclib.py", line 384, in _request
self._handler, request, verbose=self._verbose)
File "/usr/lib/python2.6/site-packages/rhn/transports.py", line 170, in request
headers, fd = req.send_http(host, handler)
File "/usr/lib/python2.6/site-packages/rhn/transports.py", line 729, in send_http
response.status, response.reason, response.msg)
<class 'up2date_client.up2dateErrors.CommunicationError'>: rhn-plugin: Error communicating with server. The message was:
Internal Server Error
This issue can't be reproduced in current Spacewalk 2.9 version. If you are still able to see it on your setup you are encouraged to update the reproducer and reopen it. |
Created attachment 763869 [details] shows error for spacewalk1 and success for spacewalk2 plus what I think is going on Description of problem: Upon registration with either bootstrap.sh or rhnreg_ks or rhn_register.. the client will fail to fully register to the spacewalk server. The registration process does produce a failed message however. Still I see the client in on the spacewalk server but it's not complete.. It's missing the kernel, IP, and Hostname. This issue never corrects. Version-Release number of selected component (if applicable): Spacewalk 1.9 running on CentOS 6.4.. issue started after recent update of OS packages. How reproducible: Steps to Reproduce: 1. cp bootstrap.sh to /tmp and execute 2. rhnreg_ks 3. rhn_register Actual results: [Wed Jun 19 11:57:21 2013] up2date A protocol error occurred: Internal Server Error , attempt #1, [Wed Jun 19 11:57:21 2013] up2date Traceback (most recent call last): File "/usr/sbin/rhnreg_ks", line 218, in <module> cli.run() File "/usr/share/rhn/up2date_client/rhncli.py", line 96, in run sys.exit(self.main() or 0) File "/usr/sbin/rhnreg_ks", line 148, in main rhnreg.sendHardware(systemId, hardwareList) File "/usr/share/rhn/up2date_client/rhnreg.py", line 475, in sendHardware s.registration.add_hw_profile(systemId, _encode_characters(hardwareList)) File "/usr/share/rhn/up2date_client/rhnserver.py", line 63, in __call__ return rpcServer.doCall(method, *args, **kwargs) File "/usr/share/rhn/up2date_client/rpcServer.py", line 203, in doCall ret = method(*args, **kwargs) File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__ return self.__send(self.__name, args) File "/usr/share/rhn/up2date_client/rpcServer.py", line 38, in _request1 ret = self._request(methodname, params) File "/usr/lib/python2.6/site-packages/rhn/rpclib.py", line 381, in _request self._handler, request, verbose=self._verbose) File "/usr/lib/python2.6/site-packages/rhn/transports.py", line 167, in request headers, fd = req.send_http(host, handler) File "/usr/lib/python2.6/site-packages/rhn/transports.py", line 707, in send_http response.status, response.reason, response.msg) <class 'up2date_client.up2dateErrors.CommunicationError'>: Error communicating with server. The message was: Internal Server Error [Wed Jun 19 11:59:01 2013] up2date A protocol error occurred: Internal Server Error , attempt #1, [Wed Jun 19 11:59:01 2013] up2date Traceback (most recent call last): File "/usr/sbin/rhnreg_ks", line 218, in <module> cli.run() File "/usr/share/rhn/up2date_client/rhncli.py", line 96, in run sys.exit(self.main() or 0) File "/usr/sbin/rhnreg_ks", line 148, in main rhnreg.sendHardware(systemId, hardwareList) File "/usr/share/rhn/up2date_client/rhnreg.py", line 475, in sendHardware s.registration.add_hw_profile(systemId, _encode_characters(hardwareList)) File "/usr/share/rhn/up2date_client/rhnserver.py", line 63, in __call__ return rpcServer.doCall(method, *args, **kwargs) File "/usr/share/rhn/up2date_client/rpcServer.py", line 203, in doCall ret = method(*args, **kwargs) File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__ return self.__send(self.__name, args) File "/usr/share/rhn/up2date_client/rpcServer.py", line 38, in _request1 ret = self._request(methodname, params) File "/usr/lib/python2.6/site-packages/rhn/rpclib.py", line 381, in _request self._handler, request, verbose=self._verbose) File "/usr/lib/python2.6/site-packages/rhn/transports.py", line 167, in request headers, fd = req.send_http(host, handler) File "/usr/lib/python2.6/site-packages/rhn/transports.py", line 707, in send_http response.status, response.reason, response.msg) <class 'up2date_client.up2dateErrors.CommunicationError'>: Error communicating with server. The message was: Internal Server Error [Wed Jun 19 12:05:59 2013] up2date logging into up2date server [Wed Jun 19 12:06:08 2013] up2date logging into up2date server [Wed Jun 19 12:06:09 2013] up2date logging into up2date server [Wed Jun 19 12:06:10 2013] up2date logging into up2date server [Wed Jun 19 12:06:13 2013] up2date logging into up2date server [Wed Jun 19 12:06:23 2013] up2date logging into up2date server [Wed Jun 19 12:06:23 2013] up2date successfully retrieved authentication token from up2date server [Wed Jun 19 12:06:24 2013] up2date Updating package profile [Wed Jun 19 12:08:34 2013] up2date A protocol error occurred: Internal Server Error , attempt #1, [Wed Jun 19 12:08:34 2013] up2date Traceback (most recent call last): File "/usr/sbin/rhnreg_ks", line 218, in <module> cli.run() File "/usr/share/rhn/up2date_client/rhncli.py", line 96, in run sys.exit(self.main() or 0) File "/usr/sbin/rhnreg_ks", line 148, in main rhnreg.sendHardware(systemId, hardwareList) File "/usr/share/rhn/up2date_client/rhnreg.py", line 475, in sendHardware s.registration.add_hw_profile(systemId, _encode_characters(hardwareList)) File "/usr/share/rhn/up2date_client/rhnserver.py", line 63, in __call__ return rpcServer.doCall(method, *args, **kwargs) File "/usr/share/rhn/up2date_client/rpcServer.py", line 203, in doCall ret = method(*args, **kwargs) File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__ return self.__send(self.__name, args) File "/usr/share/rhn/up2date_client/rpcServer.py", line 38, in _request1 ret = self._request(methodname, params) File "/usr/lib/python2.6/site-packages/rhn/rpclib.py", line 381, in _request self._handler, request, verbose=self._verbose) File "/usr/lib/python2.6/site-packages/rhn/transports.py", line 167, in request headers, fd = req.send_http(host, handler) File "/usr/lib/python2.6/site-packages/rhn/transports.py", line 707, in send_http response.status, response.reason, response.msg) <class 'up2date_client.up2dateErrors.CommunicationError'>: Error communicating with server. The message was: Internal Server Error [Wed Jun 19 12:11:40 2013] up2date A protocol error occurred: Internal Server Error , attempt #1, [Wed Jun 19 12:11:40 2013] up2date Traceback (most recent call last): File "/usr/sbin/rhnreg_ks", line 218, in <module> cli.run() File "/usr/share/rhn/up2date_client/rhncli.py", line 96, in run sys.exit(self.main() or 0) File "/usr/sbin/rhnreg_ks", line 148, in main rhnreg.sendHardware(systemId, hardwareList) File "/usr/share/rhn/up2date_client/rhnreg.py", line 475, in sendHardware s.registration.add_hw_profile(systemId, _encode_characters(hardwareList)) File "/usr/share/rhn/up2date_client/rhnserver.py", line 63, in __call__ return rpcServer.doCall(method, *args, **kwargs) File "/usr/share/rhn/up2date_client/rpcServer.py", line 203, in doCall ret = method(*args, **kwargs) File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__ return self.__send(self.__name, args) File "/usr/share/rhn/up2date_client/rpcServer.py", line 38, in _request1 ret = self._request(methodname, params) File "/usr/lib/python2.6/site-packages/rhn/rpclib.py", line 381, in _request self._handler, request, verbose=self._verbose) File "/usr/lib/python2.6/site-packages/rhn/transports.py", line 167, in request headers, fd = req.send_http(host, handler) File "/usr/lib/python2.6/site-packages/rhn/transports.py", line 707, in send_http response.status, response.reason, response.msg) <class 'up2date_client.up2dateErrors.CommunicationError'>: Error communicating with server. The message was: Internal Server Error [Wed Jun 19 12:31:14 2013] up2date A protocol error occurred: Internal Server Error , attempt #1, [Wed Jun 19 12:31:14 2013] up2date Traceback (most recent call last): File "/usr/sbin/rhnreg_ks", line 218, in <module> cli.run() File "/usr/share/rhn/up2date_client/rhncli.py", line 96, in run sys.exit(self.main() or 0) File "/usr/sbin/rhnreg_ks", line 148, in main rhnreg.sendHardware(systemId, hardwareList) File "/usr/share/rhn/up2date_client/rhnreg.py", line 475, in sendHardware s.registration.add_hw_profile(systemId, _encode_characters(hardwareList)) File "/usr/share/rhn/up2date_client/rhnserver.py", line 63, in __call__ return rpcServer.doCall(method, *args, **kwargs) File "/usr/share/rhn/up2date_client/rpcServer.py", line 203, in doCall ret = method(*args, **kwargs) File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__ return self.__send(self.__name, args) File "/usr/share/rhn/up2date_client/rpcServer.py", line 38, in _request1 ret = self._request(methodname, params) File "/usr/lib/python2.6/site-packages/rhn/rpclib.py", line 381, in _request self._handler, request, verbose=self._verbose) File "/usr/lib/python2.6/site-packages/rhn/transports.py", line 167, in request headers, fd = req.send_http(host, handler) File "/usr/lib/python2.6/site-packages/rhn/transports.py", line 707, in send_http response.status, response.reason, response.msg) <class 'up2date_client.up2dateErrors.CommunicationError'>: Error communicating with server. The message was: Internal Server Error [Thu Jun 20 07:59:21 2013] up2date A protocol error occurred: Internal Server Error , attempt #1, [Thu Jun 20 07:59:21 2013] up2date Traceback (most recent call last): File "/usr/sbin/rhnreg_ks", line 218, in <module> cli.run() File "/usr/share/rhn/up2date_client/rhncli.py", line 96, in run sys.exit(self.main() or 0) File "/usr/sbin/rhnreg_ks", line 148, in main rhnreg.sendHardware(systemId, hardwareList) File "/usr/share/rhn/up2date_client/rhnreg.py", line 475, in sendHardware s.registration.add_hw_profile(systemId, _encode_characters(hardwareList)) File "/usr/share/rhn/up2date_client/rhnserver.py", line 63, in __call__ return rpcServer.doCall(method, *args, **kwargs) File "/usr/share/rhn/up2date_client/rpcServer.py", line 203, in doCall ret = method(*args, **kwargs) File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__ return self.__send(self.__name, args) File "/usr/share/rhn/up2date_client/rpcServer.py", line 38, in _request1 ret = self._request(methodname, params) File "/usr/lib/python2.6/site-packages/rhn/rpclib.py", line 381, in _request self._handler, request, verbose=self._verbose) File "/usr/lib/python2.6/site-packages/rhn/transports.py", line 167, in request headers, fd = req.send_http(host, handler) File "/usr/lib/python2.6/site-packages/rhn/transports.py", line 707, in send_http response.status, response.reason, response.msg) <class 'up2date_client.up2dateErrors.CommunicationError'>: Error communicating with server. The message was: Internal Server Error [Thu Jun 20 08:07:16 2013] up2date logging into up2date server [Thu Jun 20 08:07:18 2013] up2date logging into up2date server [Thu Jun 20 08:09:26 2013] up2date logging into up2date server [Thu Jun 20 08:10:44 2013] up2date logging into up2date server [Thu Jun 20 08:12:22 2013] up2date logging into up2date server [Thu Jun 20 08:12:35 2013] up2date logging into up2date server [Thu Jun 20 08:16:37 2013] up2date logging into up2date server [Thu Jun 20 08:16:51 2013] up2date logging into up2date server [Thu Jun 20 08:17:20 2013] up2date A protocol error occurred: Internal Server Error , attempt #1, [Thu Jun 20 08:17:20 2013] up2date Traceback (most recent call last): File "/usr/sbin/rhnreg_ks", line 218, in <module> cli.run() File "/usr/share/rhn/up2date_client/rhncli.py", line 96, in run sys.exit(self.main() or 0) File "/usr/sbin/rhnreg_ks", line 148, in main rhnreg.sendHardware(systemId, hardwareList) File "/usr/share/rhn/up2date_client/rhnreg.py", line 475, in sendHardware s.registration.add_hw_profile(systemId, _encode_characters(hardwareList)) File "/usr/share/rhn/up2date_client/rhnserver.py", line 63, in __call__ return rpcServer.doCall(method, *args, **kwargs) File "/usr/share/rhn/up2date_client/rpcServer.py", line 203, in doCall ret = method(*args, **kwargs) File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__ return self.__send(self.__name, args) File "/usr/share/rhn/up2date_client/rpcServer.py", line 38, in _request1 ret = self._request(methodname, params) File "/usr/lib/python2.6/site-packages/rhn/rpclib.py", line 381, in _request self._handler, request, verbose=self._verbose) File "/usr/lib/python2.6/site-packages/rhn/transports.py", line 167, in request headers, fd = req.send_http(host, handler) File "/usr/lib/python2.6/site-packages/rhn/transports.py", line 707, in send_http response.status, response.reason, response.msg) <class 'up2date_client.up2dateErrors.CommunicationError'>: Error communicating with server. The message was: Internal Server Error [Thu Jun 20 09:21:55 2013] up2date A protocol error occurred: Internal Server Error , attempt #1, [Thu Jun 20 09:21:55 2013] up2date Traceback (most recent call last): File "/usr/sbin/rhnreg_ks", line 218, in <module> cli.run() File "/usr/share/rhn/up2date_client/rhncli.py", line 96, in run sys.exit(self.main() or 0) File "/usr/sbin/rhnreg_ks", line 148, in main rhnreg.sendHardware(systemId, hardwareList) File "/usr/share/rhn/up2date_client/rhnreg.py", line 475, in sendHardware s.registration.add_hw_profile(systemId, _encode_characters(hardwareList)) File "/usr/share/rhn/up2date_client/rhnserver.py", line 63, in __call__ return rpcServer.doCall(method, *args, **kwargs) File "/usr/share/rhn/up2date_client/rpcServer.py", line 203, in doCall ret = method(*args, **kwargs) File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__ return self.__send(self.__name, args) File "/usr/share/rhn/up2date_client/rpcServer.py", line 38, in _request1 ret = self._request(methodname, params) File "/usr/lib/python2.6/site-packages/rhn/rpclib.py", line 381, in _request self._handler, request, verbose=self._verbose) File "/usr/lib/python2.6/site-packages/rhn/transports.py", line 167, in request headers, fd = req.send_http(host, handler) File "/usr/lib/python2.6/site-packages/rhn/transports.py", line 707, in send_http response.status, response.reason, response.msg) <class 'up2date_client.up2dateErrors.CommunicationError'>: Error communicating with server. The message was: Internal Server Error [Thu Jun 20 10:21:16 2013] up2date A protocol error occurred: Internal Server Error , attempt #1, [Thu Jun 20 10:21:16 2013] up2date Traceback (most recent call last): File "/usr/sbin/rhnreg_ks", line 218, in <module> cli.run() File "/usr/share/rhn/up2date_client/rhncli.py", line 96, in run sys.exit(self.main() or 0) File "/usr/sbin/rhnreg_ks", line 148, in main rhnreg.sendHardware(systemId, hardwareList) File "/usr/share/rhn/up2date_client/rhnreg.py", line 475, in sendHardware s.registration.add_hw_profile(systemId, _encode_characters(hardwareList)) File "/usr/share/rhn/up2date_client/rhnserver.py", line 63, in __call__ return rpcServer.doCall(method, *args, **kwargs) File "/usr/share/rhn/up2date_client/rpcServer.py", line 203, in doCall ret = method(*args, **kwargs) File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__ return self.__send(self.__name, args) File "/usr/share/rhn/up2date_client/rpcServer.py", line 38, in _request1 ret = self._request(methodname, params) File "/usr/lib/python2.6/site-packages/rhn/rpclib.py", line 381, in _request self._handler, request, verbose=self._verbose) File "/usr/lib/python2.6/site-packages/rhn/transports.py", line 167, in request headers, fd = req.send_http(host, handler) File "/usr/lib/python2.6/site-packages/rhn/transports.py", line 707, in send_http response.status, response.reason, response.msg) <class 'up2date_client.up2dateErrors.CommunicationError'>: Error communicating with server. The message was: Internal Server Error [Thu Jun 20 10:42:24 2013] up2date logging into up2date server [Thu Jun 20 10:42:24 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 10:42:26 2013] up2date Updating package profile [Thu Jun 20 10:42:29 2013] up2date updateLoginInfo() login info [Thu Jun 20 10:42:29 2013] up2date logging into up2date server [Thu Jun 20 10:42:29 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 10:42:29 2013] up2date Updating package profile [Thu Jun 20 11:19:16 2013] up2date Updating package profile [Thu Jun 20 11:23:13 2013] up2date Updating package profile [Thu Jun 20 11:27:56 2013] up2date A protocol error occurred: Internal Server Error , attempt #1, [Thu Jun 20 11:27:56 2013] up2date Traceback (most recent call last): File "/usr/sbin/rhnreg_ks", line 218, in <module> cli.run() File "/usr/share/rhn/up2date_client/rhncli.py", line 96, in run sys.exit(self.main() or 0) File "/usr/sbin/rhnreg_ks", line 148, in main rhnreg.sendHardware(systemId, hardwareList) File "/usr/share/rhn/up2date_client/rhnreg.py", line 475, in sendHardware s.registration.add_hw_profile(systemId, _encode_characters(hardwareList)) File "/usr/share/rhn/up2date_client/rhnserver.py", line 63, in __call__ return rpcServer.doCall(method, *args, **kwargs) File "/usr/share/rhn/up2date_client/rpcServer.py", line 203, in doCall ret = method(*args, **kwargs) File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__ return self.__send(self.__name, args) File "/usr/share/rhn/up2date_client/rpcServer.py", line 38, in _request1 ret = self._request(methodname, params) File "/usr/lib/python2.6/site-packages/rhn/rpclib.py", line 381, in _request self._handler, request, verbose=self._verbose) File "/usr/lib/python2.6/site-packages/rhn/transports.py", line 167, in request headers, fd = req.send_http(host, handler) File "/usr/lib/python2.6/site-packages/rhn/transports.py", line 707, in send_http response.status, response.reason, response.msg) <class 'up2date_client.up2dateErrors.CommunicationError'>: Error communicating with server. The message was: Internal Server Error [Thu Jun 20 11:47:13 2013] up2date A protocol error occurred: Internal Server Error , attempt #1, [Thu Jun 20 11:47:13 2013] up2date Traceback (most recent call last): File "/usr/sbin/rhnreg_ks", line 218, in <module> cli.run() File "/usr/share/rhn/up2date_client/rhncli.py", line 96, in run sys.exit(self.main() or 0) File "/usr/sbin/rhnreg_ks", line 148, in main rhnreg.sendHardware(systemId, hardwareList) File "/usr/share/rhn/up2date_client/rhnreg.py", line 475, in sendHardware s.registration.add_hw_profile(systemId, _encode_characters(hardwareList)) File "/usr/share/rhn/up2date_client/rhnserver.py", line 63, in __call__ return rpcServer.doCall(method, *args, **kwargs) File "/usr/share/rhn/up2date_client/rpcServer.py", line 203, in doCall ret = method(*args, **kwargs) File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__ return self.__send(self.__name, args) File "/usr/share/rhn/up2date_client/rpcServer.py", line 38, in _request1 ret = self._request(methodname, params) File "/usr/lib/python2.6/site-packages/rhn/rpclib.py", line 381, in _request self._handler, request, verbose=self._verbose) File "/usr/lib/python2.6/site-packages/rhn/transports.py", line 167, in request headers, fd = req.send_http(host, handler) File "/usr/lib/python2.6/site-packages/rhn/transports.py", line 707, in send_http response.status, response.reason, response.msg) <class 'up2date_client.up2dateErrors.CommunicationError'>: Error communicating with server. The message was: Internal Server Error [Thu Jun 20 11:52:47 2013] up2date logging into up2date server [Thu Jun 20 11:55:00 2013] up2date logging into up2date server [Thu Jun 20 11:55:39 2013] up2date logging into up2date server [Thu Jun 20 11:56:40 2013] up2date logging into up2date server [Thu Jun 20 11:57:12 2013] up2date logging into up2date server [Thu Jun 20 12:08:18 2013] up2date logging into up2date server [Thu Jun 20 12:09:46 2013] up2date logging into up2date server [Thu Jun 20 14:02:30 2013] up2date A protocol error occurred: Internal Server Error , attempt #1, [Thu Jun 20 14:02:30 2013] up2date Traceback (most recent call last): File "/usr/sbin/rhnreg_ks", line 218, in <module> cli.run() File "/usr/share/rhn/up2date_client/rhncli.py", line 96, in run sys.exit(self.main() or 0) File "/usr/sbin/rhnreg_ks", line 148, in main rhnreg.sendHardware(systemId, hardwareList) File "/usr/share/rhn/up2date_client/rhnreg.py", line 475, in sendHardware s.registration.add_hw_profile(systemId, _encode_characters(hardwareList)) File "/usr/share/rhn/up2date_client/rhnserver.py", line 63, in __call__ return rpcServer.doCall(method, *args, **kwargs) File "/usr/share/rhn/up2date_client/rpcServer.py", line 203, in doCall ret = method(*args, **kwargs) File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__ return self.__send(self.__name, args) File "/usr/share/rhn/up2date_client/rpcServer.py", line 38, in _request1 ret = self._request(methodname, params) File "/usr/lib/python2.6/site-packages/rhn/rpclib.py", line 381, in _request self._handler, request, verbose=self._verbose) File "/usr/lib/python2.6/site-packages/rhn/transports.py", line 167, in request headers, fd = req.send_http(host, handler) File "/usr/lib/python2.6/site-packages/rhn/transports.py", line 707, in send_http response.status, response.reason, response.msg) <class 'up2date_client.up2dateErrors.CommunicationError'>: Error communicating with server. The message was: Internal Server Error Expected results: [Thu Jun 20 14:05:30 2013] up2date logging into up2date server [Thu Jun 20 14:10:18 2013] up2date logging into up2date server [Thu Jun 20 14:10:19 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:10:19 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:10:19 2013] up2date logging into up2date server [Thu Jun 20 14:10:19 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:10:19 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:10:19 2013] up2date logging into up2date server [Thu Jun 20 14:10:20 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:10:20 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:10:20 2013] up2date logging into up2date server [Thu Jun 20 14:10:20 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:11:54 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:11:54 2013] up2date logging into up2date server [Thu Jun 20 14:11:54 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:11:54 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:11:54 2013] up2date logging into up2date server [Thu Jun 20 14:11:54 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:11:54 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:11:54 2013] up2date logging into up2date server [Thu Jun 20 14:11:54 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:13:00 2013] up2date logging into up2date server [Thu Jun 20 14:13:00 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:13:00 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:13:00 2013] up2date logging into up2date server [Thu Jun 20 14:13:00 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:13:00 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:13:00 2013] up2date logging into up2date server [Thu Jun 20 14:13:01 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:13:01 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:13:01 2013] up2date logging into up2date server [Thu Jun 20 14:13:01 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:13:07 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:13:07 2013] up2date logging into up2date server [Thu Jun 20 14:13:07 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:13:07 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:13:07 2013] up2date logging into up2date server [Thu Jun 20 14:13:07 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:13:07 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:13:07 2013] up2date logging into up2date server [Thu Jun 20 14:13:07 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:13:13 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:13:13 2013] up2date logging into up2date server [Thu Jun 20 14:13:13 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:13:13 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:13:13 2013] up2date logging into up2date server [Thu Jun 20 14:13:13 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:13:13 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:13:13 2013] up2date logging into up2date server [Thu Jun 20 14:13:13 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:16:27 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:16:27 2013] up2date logging into up2date server [Thu Jun 20 14:16:27 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:16:27 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:16:27 2013] up2date logging into up2date server [Thu Jun 20 14:16:27 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:16:27 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:16:27 2013] up2date logging into up2date server [Thu Jun 20 14:16:27 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:16:32 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:16:32 2013] up2date logging into up2date server [Thu Jun 20 14:16:32 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:16:32 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:16:32 2013] up2date logging into up2date server [Thu Jun 20 14:16:32 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:16:32 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:16:32 2013] up2date logging into up2date server [Thu Jun 20 14:16:32 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:18:54 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:18:54 2013] up2date logging into up2date server [Thu Jun 20 14:18:54 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:18:54 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:18:54 2013] up2date logging into up2date server [Thu Jun 20 14:18:54 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:18:54 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:18:54 2013] up2date logging into up2date server [Thu Jun 20 14:18:54 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:19:04 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:19:04 2013] up2date logging into up2date server [Thu Jun 20 14:19:04 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:19:04 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:19:04 2013] up2date logging into up2date server [Thu Jun 20 14:19:04 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:19:04 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:19:04 2013] up2date logging into up2date server [Thu Jun 20 14:19:04 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:19:08 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:19:08 2013] up2date logging into up2date server [Thu Jun 20 14:19:08 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:19:08 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:19:08 2013] up2date logging into up2date server [Thu Jun 20 14:19:08 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:19:08 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:19:08 2013] up2date logging into up2date server [Thu Jun 20 14:19:08 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:19:25 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:19:25 2013] up2date logging into up2date server [Thu Jun 20 14:19:25 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:19:25 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:19:25 2013] up2date logging into up2date server [Thu Jun 20 14:19:25 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:19:25 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:19:25 2013] up2date logging into up2date server [Thu Jun 20 14:19:25 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:20:06 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:20:06 2013] up2date logging into up2date server [Thu Jun 20 14:20:06 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:20:06 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:20:06 2013] up2date logging into up2date server [Thu Jun 20 14:20:06 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:20:06 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:20:06 2013] up2date logging into up2date server [Thu Jun 20 14:20:06 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:20:15 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:20:15 2013] up2date logging into up2date server [Thu Jun 20 14:20:15 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:20:15 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:20:15 2013] up2date logging into up2date server [Thu Jun 20 14:20:15 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:20:15 2013] up2date updateLoginInfo() login info [Thu Jun 20 14:20:15 2013] up2date logging into up2date server [Thu Jun 20 14:20:15 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:22:23 2013] up2date logging into up2date server [Thu Jun 20 14:22:23 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 14:23:16 2013] up2date logging into up2date server [Thu Jun 20 14:23:16 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 16:50:12 2013] up2date logging into up2date server [Thu Jun 20 16:50:12 2013] up2date successfully retrieved authentication token from up2date server [Thu Jun 20 16:50:14 2013] up2date Updating package profile [Thu Jun 20 20:50:12 2013] up2date logging into up2date server [Thu Jun 20 20:50:12 2013] up2date successfully retrieved authentication token from up2date server [Fri Jun 21 00:50:12 2013] up2date logging into up2date server [Fri Jun 21 00:50:13 2013] up2date successfully retrieved authentication token from up2date server [Fri Jun 21 04:50:13 2013] up2date logging into up2date server [Fri Jun 21 04:50:13 2013] up2date successfully retrieved authentication token from up2date server [Fri Jun 21 07:39:46 2013] up2date logging into up2date server [Fri Jun 21 07:39:46 2013] up2date successfully retrieved authentication token from up2date server Additional info: I've attempted to register the same server on two different spacewalk servers spacewalk1 fails ... spacewalk 2 works still neither server will are able to manage clients. spacewalk2 the registration works but when I do a yum repolist after disabling the local repostories I get the following errors: ============================================================================ I will delete the channel contents entirely and rebuild it... this issue on spacealk2 may not be related to my problem on spacewalk1 but I wanted to mention it becuase it was also patched.. And though this system was patched it would only perform the update if I ignored all tomcat6 packages. The system where Registration is totally failing "spacewalk1" did accept the tomcat6 package updates. I've already tried backing out the tomcat 6 pacakges.. And removing them completely and applying a previous version of the tomcat6 packages. From there I changed the necessary tomcat6 configuration files on spacewalk1; making them match spacewalk2.. Still registration fails..( or is incomplete) ==================================================================== When I register a systems on spacealk2 this is what is happening: =================================================================== Subscribed Channels (Alter Channel Subscriptions) Kickstart CentOS 6.4 - Base - x86_64 Kickstart CentOS 6 - EPEL - x86_64 Kickstart CentOS 6.4 - Spacewalk-1.9-Client RHEL6 - x86_64 Kickstart CentOS 6.4-Updates-x86_64 yum repolist Loaded plugins: fastestmirror, refresh-packagekit, rhnplugin This system is receiving updates from RHN Classic or RHN Satellite. Determining fastest mirrors repo id repo name status kickstart-6.4-updates-x86_64 Kickstart CentOS 6.4-Updates-x86_64 0 kickstart-centos-6.4-base-x86-64bit Kickstart CentOS 6.4 - Base - x86_64 0 kickstart-centos-6.4-spacewalk-1.9-client-rhel6-x86_64 Kickstart CentOS 6.4 - Spacewalk-1.9-Client RHEL6 - x86_64 0 kickstart-epel6-x86_64 Kickstart CentOS 6 - EPEL - x86_64 0 repolist: 0 yum repolist -v Loading "rhnplugin" plugin Loading "fastestmirror" plugin Loading "refresh-packagekit" plugin Config time: 0.044 This system is receiving updates from RHN Classic or RHN Satellite. Looking for repo options for [main] Looking for repo options for [kickstart-centos-6.4-base-x86-64bit] Repo 'kickstart-centos-6.4-base-x86-64bit' setting option 'enabled' = '1' Repo 'kickstart-centos-6.4-base-x86-64bit' setting option 'gpgcheck' = '1' Looking for repo options for [kickstart-6.4-updates-x86_64] Repo 'kickstart-6.4-updates-x86_64' setting option 'enabled' = '1' Repo 'kickstart-6.4-updates-x86_64' setting option 'gpgcheck' = '1' Looking for repo options for [kickstart-centos-6.4-spacewalk-1.9-client-rhel6-x86_64] Repo 'kickstart-centos-6.4-spacewalk-1.9-client-rhel6-x86_64' setting option 'enabled' = '1' Repo 'kickstart-centos-6.4-spacewalk-1.9-client-rhel6-x86_64' setting option 'gpgcheck' = '1' Looking for repo options for [kickstart-epel6-x86_64] Repo 'kickstart-epel6-x86_64' setting option 'enabled' = '1' Repo 'kickstart-epel6-x86_64' setting option 'gpgcheck' = '1' Yum Version: 3.2.29 Loading mirror speeds from cached hostfile Error: Cannot retrieve repository metadata (repomd.xml) for repository: kickstart-6.4-updates-x86_64. Please verify its path and try again I'm remove the Updates repo from the client and the issue moves to the base... Cleaning up Everything Cleaning up list of fastest mirrors yum repolist Loaded plugins: fastestmirror, refresh-packagekit, rhnplugin This system is receiving updates from RHN Classic or RHN Satellite. Determining fastest mirrors repo id repo name status kickstart-centos-6.4-base-x86-64bit Kickstart CentOS 6.4 - Base - x86_64 0 kickstart-centos-6.4-spacewalk-1.9-client-rhel6-x86_64 Kickstart CentOS 6.4 - Spacewalk-1.9-Client RHEL6 - x86_64 0 kickstart-epel6-x86_64 Kickstart CentOS 6 - EPEL - x86_64 0 repolist: 0 yum.repos.d]# yum repolist -v Loading "rhnplugin" plugin Loading "fastestmirror" plugin Loading "refresh-packagekit" plugin Config time: 0.042 This system is receiving updates from RHN Classic or RHN Satellite. Looking for repo options for [main] Looking for repo options for [kickstart-centos-6.4-base-x86-64bit] Repo 'kickstart-centos-6.4-base-x86-64bit' setting option 'enabled' = '1' Repo 'kickstart-centos-6.4-base-x86-64bit' setting option 'gpgcheck' = '1' Looking for repo options for [kickstart-centos-6.4-spacewalk-1.9-client-rhel6-x86_64] Repo 'kickstart-centos-6.4-spacewalk-1.9-client-rhel6-x86_64' setting option 'enabled' = '1' Repo 'kickstart-centos-6.4-spacewalk-1.9-client-rhel6-x86_64' setting option 'gpgcheck' = '1' Looking for repo options for [kickstart-epel6-x86_64] Repo 'kickstart-epel6-x86_64' setting option 'enabled' = '1' Repo 'kickstart-epel6-x86_64' setting option 'gpgcheck' = '1' Yum Version: 3.2.29 Loading mirror speeds from cached hostfile Error: Cannot retrieve repository metadata (repomd.xml) for repository: kickstart-centos-6.4-base-x86-64bit. Please verify its path and try again