Bug 994531

Summary: rhn_check exception: OverflowError, "int exceeds XML-RPC limits"
Product: Red Hat Enterprise Linux 6 Reporter: Ales Zelinka <azelinka>
Component: rhn-client-toolsAssignee: Milan Zázrivec <mzazrivec>
Status: CLOSED ERRATA QA Contact: Lukas Pramuk <lpramuk>
Severity: high Docs Contact:
Priority: high    
Version: 6.5CC: jhutar, lpramuk, tlavigne
Target Milestone: rc   
Target Release: ---   
Hardware: ppc64   
OS: Unspecified   
Whiteboard:
Fixed In Version: rhn-client-tools-1.0.0.1-14.el6 Doc Type: Bug Fix
Doc Text:
Cause: Machine with many CPUs would potentially report large value for idle time for all its processors. Consequence: The idle value would not fit into XML-RPC's integer limits, running rhn_check on the problematic machine would result into a traceback. Fix: Simple enough. Result: rhn_check in the problematic scenarios works correctly now.
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-11-21 23:15:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 798710    
Attachments:
Description Flags
up2date log none

Description Ales Zelinka 2013-08-07 13:00:06 UTC
Created attachment 783895 [details]
up2date log

rhn_check tracebacks on rhel6.4/ppc64 (rhn-check-1.0.0.1-4.el6.noarch):

# rhn_check -vvv
D: opening  db environment /var/lib/rpm cdb:mpool:joinenv
D: opening  db index       /var/lib/rpm/Packages rdonly mode=0x0
D: locked   db index       /var/lib/rpm/Packages
D: loading keyring from pubkeys in /var/lib/rpm/pubkeys/*.key
D: couldn't find any keys in /var/lib/rpm/pubkeys/*.key
D: loading keyring from rpmdb
D: opening  db index       /var/lib/rpm/Name rdonly mode=0x0
D: added key gpg-pubkey-fd431d51-4ae0493b to keyring
D: added key gpg-pubkey-2fa658e0-45700c69 to keyring
D: Using legacy gpg-pubkey(s) from rpmdb
D: opening  db index       /var/lib/rpm/Providename rdonly mode=0x0
An error has occurred:
<type 'exceptions.OverflowError'>
See /var/log/up2date for more information
D: closed   db index       /var/lib/rpm/Providename
D: closed   db index       /var/lib/rpm/Name
D: closed   db index       /var/lib/rpm/Packages
D: closed   db environment /var/lib/rpm

from /var/log/up2date (full log attched):
[Wed Aug  7 08:07:12 2013] up2date 
Traceback (most recent call last):
  File "/usr/sbin/rhn_check", line 378, in <module>
    cli.run()
  File "/usr/share/rhn/up2date_client/rhncli.py", line 84, in run
    sys.exit(self.main() or 0)
  File "/usr/sbin/rhn_check", line 75, in main
    self.__run_remote_actions()
  File "/usr/sbin/rhn_check", line 169, in __run_remote_actions
    action = self.__get_action(status_report)
  File "/usr/sbin/rhn_check", line 87, in __get_action
    ACTION_VERSION, status_report)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/share/rhn/up2date_client/rpcServer.py", line 39, in _request1
    ret = self._request(methodname, params)
  File "/usr/lib/python2.6/site-packages/rhn/rpclib.py", line 379, in _request
    request = self._req_body(self._strip_characters(params), methodname)
  File "/usr/lib/python2.6/site-packages/rhn/rpclib.py", line 234, in _req_body
    return xmlrpclib.dumps(params, methodname, encoding=self._encoding)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1132, in dumps
    data = m.dumps(params)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 677, in dumps
    dump(v, write)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 699, in __dump
    f(self, value, write)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 780, in dump_struct
    dump(v, write)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 699, in __dump
    f(self, value, write)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 759, in dump_array
    dump(v, write)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 699, in __dump
    f(self, value, write)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 710, in dump_int
    raise OverflowError, "int exceeds XML-RPC limits"
<type 'exceptions.OverflowError'>: int exceeds XML-RPC limits

Let me know if you need additional info to be able to debug this.

Comment 2 Milan Zázrivec 2013-09-24 12:25:01 UTC
What happens here is:
1. machine with many cpus
2. the cpus spent a lot of time idle (sum of idle times)
3. rhn_check tries to send content of /proc/uptime to the parent Satellite
4. the idle value from /proc/uptime won't fit into the limits of XML-RPC
   integer (32 bit signed int)

This needs to be fixed, as there's no immediate resolution nor workaround
to this problem, other than rebooting the affected machine.

Comment 3 Milan Zázrivec 2013-09-25 15:52:43 UTC
spacewalk.git master: b258e06ea3329b6782fdd21db77939380b3a85ca

Comment 6 Lukas Pramuk 2013-10-31 16:43:28 UTC
VERIFIED. rhn-check-1.0.0.1-16.el6.noarch

For the reproducer you need a machine where idle value (2nd number from /proc/uptime) is greater than max signed int (2^31-1)
# IDLE=$(cat /proc/uptime | sed "s/^.* \(.\+\)\..\+$/\1/")
# echo $(($IDLE > 2**31-1))
1

I've found 3 machines having idle values high enough to reproduce the issue (ppc machines seem to have high idle values):
ibm-p750e-01-lp1.rhts.eng.bos.redhat.com
ibm-p750e-02-lp1.rhts.eng.bos.redhat.com
ibm-p730-01-lp4.lab.eng.brq.redhat.com (used this)


REPRO:

# rhn_check -vv
An error has occurred:
<type 'exceptions.OverflowError'>
See /var/log/up2date for more information

[Thu Oct 31 09:53:12 2013] up2date 
Traceback (most recent call last):
  File "/usr/sbin/rhn_check", line 378, in <module>
    cli.run()
  File "/usr/share/rhn/up2date_client/rhncli.py", line 84, in run
    sys.exit(self.main() or 0)
  File "/usr/sbin/rhn_check", line 75, in main
    self.__run_remote_actions()
  File "/usr/sbin/rhn_check", line 169, in __run_remote_actions
    action = self.__get_action(status_report)
  File "/usr/sbin/rhn_check", line 87, in __get_action
    ACTION_VERSION, status_report)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/share/rhn/up2date_client/rpcServer.py", line 39, in _request1
    ret = self._request(methodname, params)
  File "/usr/lib/python2.6/site-packages/rhn/rpclib.py", line 379, in _request
    request = self._req_body(self._strip_characters(params), methodname)
  File "/usr/lib/python2.6/site-packages/rhn/rpclib.py", line 234, in _req_body
    return xmlrpclib.dumps(params, methodname, encoding=self._encoding)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1132, in dumps
    data = m.dumps(params)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 677, in dumps
    dump(v, write)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 699, in __dump
    f(self, value, write)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 780, in dump_struct
    dump(v, write)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 699, in __dump
    f(self, value, write)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 759, in dump_array
    dump(v, write)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 699, in __dump
    f(self, value, write)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 710, in dump_int
    raise OverflowError, "int exceeds XML-RPC limits"
<type 'exceptions.OverflowError'>: int exceeds XML-RPC limits


vs.


FIX:

# rhn_check -vv
D: do_call packages.checkNeedUpdate('rhnsd=1',){}
Loaded plugins: product-id, rhnplugin
D: login(forceUpdate=False) invoked
D: readCachedLogin invoked
D: Checking pickled loginInfo, currentTime=1383229117.12, createTime=1383228425.17, expire-offset=3600.0
D: readCachedLogin(): using pickled loginInfo set to expire at 1383232025.17
D: rpcServer: Calling XMLRPC up2date.listChannels
This system is receiving updates from RHN Classic or RHN Satellite.
D: local action status: (0, 'rpm database not modified since last update (or package list recently updated)', {})
D: rpcServer: Calling XMLRPC registration.welcome_message

Comment 7 errata-xmlrpc 2013-11-21 23:15:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1702.html