Bug 1401819

Summary:	rhnreg_ks fails with ISE: ERROR: current transaction is aborted
Product:	[Community] Spacewalk	Reporter:	Ales Dujicek <adujicek>
Component:	Server	Assignee:	Tomáš Kašpárek <tkasparek>
Status:	CLOSED EOL	QA Contact:	Red Hat Satellite QA List <satqe-list>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	2.5
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-03-06 14:27:49 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Comment 1 Ales Dujicek 2016-12-06 08:11:05 UTC

Time to time we see in our automation failed client registration:

# rhnreg_ks --username=admin --password=password --serverUrl=https://spacewalk/XMLRPC --sslCACert=/usr/share/rhn/cert --force
An error has occurred:
rhn-plugin: Error communicating with server. The message was:
Internal Server Error
See /var/log/up2date for more information


/var/log/up2date :

[Mon Dec  5 17:06:41 2016] up2date A protocol error occurred: Internal Server Error , attempt #1,
[Mon Dec  5 17:06:41 2016] up2date 
Traceback (most recent call last):
  File "/usr/sbin/rhnreg_ks", line 217, in <module>
    cli.run()
  File "/usr/share/rhn/up2date_client/rhncli.py", line 96, in run
    sys.exit(self.main() or 0)
  File "/usr/sbin/rhnreg_ks", line 152, in main
    rhnreg.sendPackages(systemId, packageList)
  File "/usr/share/rhn/up2date_client/rhnreg.py", line 488, in sendPackages
    s.registration.add_packages(systemId, packageList)
  File "/usr/share/rhn/up2date_client/rhnserver.py", line 67, in __call__
    return rpcServer.doCall(method, *args, **kwargs)
  File "/usr/share/rhn/up2date_client/rpcServer.py", line 290, in doCall
    raise_with_tb(up2dateErrors.CommunicationError(e.errmsg))
  File "/usr/share/rhn/up2date_client/rpcServer.py", line 214, in doCall
    ret = method(*args, **kwargs)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__
    return self.__send(self.__name, args)
  File "/usr/share/rhn/up2date_client/rpcServer.py", line 48, in _request1
    ret = self._request(methodname, params)
  File "/usr/lib/python2.7/site-packages/rhn/rpclib.py", line 394, in _request
    self._handler, request, verbose=self._verbose)
  File "/usr/lib/python2.7/site-packages/rhn/transports.py", line 177, in request
    headers, fd = req.send_http(host, handler)
  File "/usr/lib/python2.7/site-packages/rhn/transports.py", line 740, in send_http
    response.status, response.reason, response.msg)
<class 'up2date_client.up2dateErrors.CommunicationError'>: rhn-plugin: Error communicating with server. The message was:
Internal Server Error


on server side:

/var/log/httpd/error_log

[Mon Dec 05 17:06:40 2016] [error] Spacewalk 22874 2016/12/05 17:06:40 -04:00: ('rhnSQL.SQLSchemaError caught', SQLSchemaError(99999, 'ERROR:  current transaction is aborted, commands ignored until end of transaction block', '', InternalError('current transaction is aborted, commands ignored until end of transaction block\\n',)))
[Mon Dec 05 17:06:40 2016] [error] Exception reported from spacewalk.server
[Mon Dec 05 17:06:40 2016] [error] Time: Mon Dec  5 17:06:40 2016
[Mon Dec 05 17:06:40 2016] [error] Exception type <class 'spacewalk.server.rhnSQL.sql_base.SQLSchemaError'>
[Mon Dec 05 17:06:40 2016] [error] Exception while handling function registration.add_packages
[Mon Dec 05 17:06:40 2016] [error] Request object information:
[Mon Dec 05 17:06:40 2016] [error] URI: /XMLRPC
[Mon Dec 05 17:06:40 2016] [error] Remote Host: spacewalk.client
[Mon Dec 05 17:06:40 2016] [error] Server Name: spacewalk.server:443
[Mon Dec 05 17:06:40 2016] [error] Headers passed in:
[Mon Dec 05 17:06:40 2016] [error] \tAccept-Encoding: identity
[Mon Dec 05 17:06:40 2016] [error] \tCONTENT_LENGTH: 324414
[Mon Dec 05 17:06:40 2016] [error] \tCONTENT_TYPE: text/xml
[Mon Dec 05 17:06:40 2016] [error] \tDOCUMENT_ROOT: /var/www/html
[Mon Dec 05 17:06:40 2016] [error] \tGATEWAY_INTERFACE: CGI/1.1
[Mon Dec 05 17:06:40 2016] [error] \tHTTPS: 1
[Mon Dec 05 17:06:40 2016] [error] \tHTTP_ACCEPT_ENCODING: identity
[Mon Dec 05 17:06:40 2016] [error] \tHTTP_HOST: spacewalk.server
[Mon Dec 05 17:06:40 2016] [error] \tHTTP_USER_AGENT: rhn.rpclib.py/2.7.1-1.el7
[Mon Dec 05 17:06:40 2016] [error] \tHTTP_X_CLIENT_VERSION: 1
[Mon Dec 05 17:06:40 2016] [error] \tHTTP_X_INFO: RPC Processor (C) Red Hat, Inc (version 2.7.1-1.el7)
[Mon Dec 05 17:06:40 2016] [error] \tHTTP_X_RHN_CLIENT_CAPABILITY: kickstart_guest.initiate(1)=1,kickstart.initiate(1)=1,osad.ping(1)=1,scap.xccdf_eval(1)=1,packages.verify(1)=1,packages.extended_profile(2)=1,osad.rhn_check(1)=1,configfiles.base64_enc(1)=1,reboot.reboot(1)=1,configfiles.deploy(1)=1,caneatCheese(1)=1,packages.rollBack(1)=1,configfiles.mtime_upload(1)=1,configfiles.diff(1)=1,packages.update(2)=2,script.run(1)=1,abrt.check(1)=1,configfiles.upload(1)=1,packages.runTransaction(1)=1
[Mon Dec 05 17:06:40 2016] [error] \tHTTP_X_RHN_TRANSPORT_CAPABILITY: follow-redirects=3
[Mon Dec 05 17:06:40 2016] [error] \tHTTP_X_TRANSPORT_INFO: Extended Capabilities Transport (C) Red Hat, Inc (version 2.7.1-1.el7)
[Mon Dec 05 17:06:40 2016] [error] \tHTTP_X_UP2DATE_VERSION: 2.7.1-1.el7
[Mon Dec 05 17:06:40 2016] [error] \tHost: spacewalk.server
[Mon Dec 05 17:06:40 2016] [error] \tPATH_INFO: 
[Mon Dec 05 17:06:40 2016] [error] \tQUERY_STRING: 
[Mon Dec 05 17:06:40 2016] [error] \tREMOTE_ADDR: 10.16.69.150
[Mon Dec 05 17:06:40 2016] [error] \tREMOTE_PORT: 39568
[Mon Dec 05 17:06:40 2016] [error] \tREQUEST_METHOD: POST
[Mon Dec 05 17:06:40 2016] [error] \tREQUEST_URI: /XMLRPC
[Mon Dec 05 17:06:40 2016] [error] \tSCRIPT_FILENAME: /usr/share/rhn/wsgi/xmlrpc.py
[Mon Dec 05 17:06:40 2016] [error] \tSCRIPT_NAME: /XMLRPC
[Mon Dec 05 17:06:40 2016] [error] \tSCRIPT_URI: https://spacewalk.server/XMLRPC
[Mon Dec 05 17:06:40 2016] [error] \tSCRIPT_URL: /XMLRPC
[Mon Dec 05 17:06:40 2016] [error] \tSERVER_ADDR: 10.19.41.54
[Mon Dec 05 17:06:40 2016] [error] \tSERVER_ADMIN: root@localhost
[Mon Dec 05 17:06:40 2016] [error] \tSERVER_NAME: spacewalk.server
[Mon Dec 05 17:06:40 2016] [error] \tSERVER_PORT: 443
[Mon Dec 05 17:06:40 2016] [error] \tSERVER_PROTOCOL: HTTP/1.1
[Mon Dec 05 17:06:40 2016] [error] \tSERVER_SIGNATURE: <address>Apache Server at spacewalk.server Port 443</address>
[Mon Dec 05 17:06:40 2016] [error] 
[Mon Dec 05 17:06:40 2016] [error] \tSERVER_SOFTWARE: Apache
[Mon Dec 05 17:06:40 2016] [error] \tUser-Agent: rhn.rpclib.py/2.7.1-1.el7
[Mon Dec 05 17:06:40 2016] [error] \tX-Client-Version: 1
[Mon Dec 05 17:06:40 2016] [error] \tX-Info: RPC Processor (C) Red Hat, Inc (version 2.7.1-1.el7)
[Mon Dec 05 17:06:40 2016] [error] \tX-RHN-Client-Capability: kickstart_guest.initiate(1)=1,kickstart.initiate(1)=1,osad.ping(1)=1,scap.xccdf_eval(1)=1,packages.verify(1)=1,packages.extended_profile(2)=1,osad.rhn_check(1)=1,configfiles.base64_enc(1)=1,reboot.reboot(1)=1,configfiles.deploy(1)=1,caneatCheese(1)=1,packages.rollBack(1)=1,configfiles.mtime_upload(1)=1,configfiles.diff(1)=1,packages.update(2)=2,script.run(1)=1,abrt.check(1)=1,configfiles.upload(1)=1,packages.runTransaction(1)=1
[Mon Dec 05 17:06:40 2016] [error] \tX-RHN-Transport-Capability: follow-redirects=3
[Mon Dec 05 17:06:40 2016] [error] \tX-Transport-Info: Extended Capabilities Transport (C) Red Hat, Inc (version 2.7.1-1.el7)
[Mon Dec 05 17:06:40 2016] [error] \tX-Up2Date-Version: 2.7.1-1.el7
[Mon Dec 05 17:06:40 2016] [error] \tmod_ssl.is_https: <built-in method ssl_is_https of mod_wsgi.Adapter object at 0x7fa2e95eaaf8>
[Mon Dec 05 17:06:40 2016] [error] \tmod_ssl.var_lookup: <built-in method ssl_var_lookup of mod_wsgi.Adapter object at 0x7fa2e95eaaf8>
[Mon Dec 05 17:06:40 2016] [error] \tmod_wsgi.application_group: spacewalk.server|/xmlrpc
[Mon Dec 05 17:06:40 2016] [error] \tmod_wsgi.callable_object: application
[Mon Dec 05 17:06:40 2016] [error] \tmod_wsgi.handler_script: 
[Mon Dec 05 17:06:40 2016] [error] \tmod_wsgi.input_chunked: 0
[Mon Dec 05 17:06:40 2016] [error] \tmod_wsgi.listener_host: 
[Mon Dec 05 17:06:40 2016] [error] \tmod_wsgi.listener_port: 443
[Mon Dec 05 17:06:40 2016] [error] \tmod_wsgi.process_group: 
[Mon Dec 05 17:06:40 2016] [error] \tmod_wsgi.request_handler: wsgi-script
[Mon Dec 05 17:06:40 2016] [error] \tmod_wsgi.script_reloading: 1
[Mon Dec 05 17:06:40 2016] [error] \tmod_wsgi.version: (3, 2)
[Mon Dec 05 17:06:40 2016] [error] \twsgi.errors: <mod_wsgi.Log object at 0x7fa2e9688230>
[Mon Dec 05 17:06:40 2016] [error] \twsgi.file_wrapper: <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7fa2e95eaaf8>
[Mon Dec 05 17:06:40 2016] [error] \twsgi.input: <mod_wsgi.Input object at 0x7fa2e96881f0>
[Mon Dec 05 17:06:40 2016] [error] \twsgi.multiprocess: True
[Mon Dec 05 17:06:40 2016] [error] \twsgi.multithread: False
[Mon Dec 05 17:06:40 2016] [error] \twsgi.run_once: False
[Mon Dec 05 17:06:40 2016] [error] \twsgi.url_scheme: https
[Mon Dec 05 17:06:40 2016] [error] \twsgi.version: (1, 1)
[Mon Dec 05 17:06:40 2016] [error] Extra information about this error:
[Mon Dec 05 17:06:40 2016] [error] SQL Error generated: (99999, 'ERROR:  current transaction is aborted, commands ignored until end of transaction block', '', InternalError('current transaction is aborted, commands ignored until end of transaction block\\n',))
[Mon Dec 05 17:06:40 2016] [error] 
[Mon Dec 05 17:06:40 2016] [error] Exception Handler Information
[Mon Dec 05 17:06:40 2016] [error] Traceback (most recent call last):
[Mon Dec 05 17:06:40 2016] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/server/apacheRequest.py", line 135, in call_function
[Mon Dec 05 17:06:40 2016] [error]     response = func(*params)
[Mon Dec 05 17:06:40 2016] [error]   File "/usr/share/rhn/server/handlers/xmlrpc/registration.py", line 815, in add_packages
[Mon Dec 05 17:06:40 2016] [error]     server.save_packages()
[Mon Dec 05 17:06:40 2016] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnServer/server_wrapper.py", line 72, in save_packages
[Mon Dec 05 17:06:40 2016] [error]     ret = self.save_packages_byid(self.server["id"], schedule=schedule)
[Mon Dec 05 17:06:40 2016] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnServer/server_packages.py", line 252, in save_packages_byid
[Mon Dec 05 17:06:40 2016] [error]     update_errata_cache(sysid)
[Mon Dec 05 17:06:40 2016] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnServer/server_packages.py", line 335, in update_errata_cache
[Mon Dec 05 17:06:40 2016] [error]     update_needed_cache(server_id, 0)
[Mon Dec 05 17:06:40 2016] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnSQL/driver_postgresql.py", line 116, in __call__
[Mon Dec 05 17:06:40 2016] [error]     result = Function.__call__(self, *args)
[Mon Dec 05 17:06:40 2016] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnSQL/driver_postgresql.py", line 92, in __call__
[Mon Dec 05 17:06:40 2016] [error]     raise sql_base.SQLSchemaError(error_code, e.pgerror, e)
[Mon Dec 05 17:06:40 2016] [error] SQLSchemaError: (99999, 'ERROR:  current transaction is aborted, commands ignored until end of transaction block', '', InternalError('current transaction is aborted, commands ignored until end of transaction block\\n',))
[Mon Dec 05 17:06:40 2016] [error] 



Version-Release number of selected component (if applicable):
client:
rhn-setup.noarch 0:2.7.1-1.el7
server:
spacewalk-java-2.7.7-1.el6.noarch
spacewalk-schema-2.7.2-1.el6.noarch
spacewalk-backend-2.7.4-1.el6.noarch

How reproducible:
time to time

Steps to Reproduce:
1. rhnreg_ks --username=admin --password=password --serverUrl=https://spacewalk/XMLRPC --sslCACert=/usr/share/rhn/cert --force

Comment 2 Ales Dujicek 2017-02-27 07:49:10 UTC

maybe this could help too
it happens before tracebacks in comments#0-1

/var/log/httpd/error_log
[Sun Feb 26 16:47:14 2017] [error] SQL Error generated: (99999, 'ERROR:  query returned no rows', 'CONTEXT:  PL/pgSQL function "lookup_client_capability" line 17 at SQL statement\\n', InternalError('query returned no rows\\nCONTEXT:  PL/pgSQL function "lookup_client_capability" line 17 at SQL statement\\n',))
[Sun Feb 26 16:47:14 2017] [error] 
[Sun Feb 26 16:47:14 2017] [error] Exception Handler Information
[Sun Feb 26 16:47:14 2017] [error] Traceback (most recent call last):
[Sun Feb 26 16:47:14 2017] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/server/apacheRequest.py", line 135, in call_function
[Sun Feb 26 16:47:14 2017] [error]     response = func(*params)
[Sun Feb 26 16:47:14 2017] [error]   File "/usr/share/rhn/server/handlers/xmlrpc/registration.py", line 808, in add_packages
[Sun Feb 26 16:47:14 2017] [error]     packages = self._normalize_packages(system_id, packages)
[Sun Feb 26 16:47:14 2017] [error]   File "/usr/share/rhn/server/handlers/xmlrpc/registration.py", line 912, in _normalize_packages
[Sun Feb 26 16:47:14 2017] [error]     rhnCapability.update_client_capabilities(self.server_id)
[Sun Feb 26 16:47:14 2017] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnCapability.py", line 140, in update_client_capabilities
[Sun Feb 26 16:47:14 2017] [error]     h.executemany(**inserts)
[Sun Feb 26 16:47:14 2017] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnSQL/sql_base.py", line 160, in executemany
[Sun Feb 26 16:47:14 2017] [error]     return self._execute_wrapper(self._executemany, *p, **kw)
[Sun Feb 26 16:47:14 2017] [error]   File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnSQL/driver_postgresql.py", line 303, in _execute_wrapper
[Sun Feb 26 16:47:14 2017] [error]     raise sql_base.SQLSchemaError(error_code, e.pgerror, e)
[Sun Feb 26 16:47:14 2017] [error] SQLSchemaError: (99999, 'ERROR:  query returned no rows', 'CONTEXT:  PL/pgSQL function "lookup_client_capability" line 17 at SQL statement\\n', InternalError('query returned no rows\\nCONTEXT:  PL/pgSQL function "lookup_client_capability" line 17 at SQL statement\\n',))
[Sun Feb 26 16:47:14 2017] [error]

Comment 3 Michael Mráka 2020-03-06 14:27:49 UTC

Spacewalk 2.8 (and older) has already reached it's End Of Life.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before end of life. If you would still like
to see this bug fixed and are able to reproduce it against current version
of Spacewalk 2.9, you are encouraged change the 'version' and re-open it.