Bug 434168 - exceptions.OverflowError when up2date'ing rpm bigger than 2GB
exceptions.OverflowError when up2date'ing rpm bigger than 2GB
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: up2date (Show other bugs)
4.6
All Linux
low Severity low
: rc
: ---
Assigned To: Sayli Karmarkar
desktop-bugs@redhat.com
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-02-22 10:39 EST by Jan Pazdziora
Modified: 2015-03-22 21:09 EDT (History)
1 user (show)

See Also:
Fixed In Version: RHBA-2008-0771
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2008-07-24 16:05:58 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Jan Pazdziora 2008-02-22 10:39:56 EST
Description of problem:

When I run up2date -i huge-rpm, up2date fails with

An error has occurred:
exceptions.OverflowError

Version-Release number of selected component (if applicable):

# rpm -qf /usr/lib/python2.3/site-packages/rhn/transports.py
rhnlib-2.1.1-9.el4_6
# rpm -qf /usr/sbin/up2date
up2date-4.6.2-7.el4_6.1

How reproducible:

Deterministic.

Steps to Reproduce:
1. Create rpm bigger than 2GB.
2. Push it to Satellite.
3. Register RHEL 4 system to that Satellite (and to the channel to which you
pushed that huge rpm).
4. Run up2date -i huge-rpm.
  
Actual results:

Testing package set / solving RPM inter-dependencies...
########################################
huge-rpm-1.0-20080221.14221         
############################################                     11187 k/sec,
00:02:57
rem##########################################################################################
           11259 k/sec, 00:01:37
rem###################################################################################################
  11278 k/sec, 00:00:19
rem####################################################### Done.                   
An error has occurred:
exceptions.OverflowError
See /var/log/up2date for more information

Expected results:

No error, rpm installed.

Additional info:

# tail -25 /var/log/up2date
[Fri Feb 22 16:27:15 2008] up2date successfully retrieved authentication token
from up2date server
[Fri Feb 22 16:27:15 2008] up2date availablePackageList from network
[Fri Feb 22 16:31:15 2008] up2date   File "/usr/sbin/up2date", line 1273, in ?
    sys.exit(main() or 0)
   File "/usr/sbin/up2date", line 791, in main
    fullUpdate, dryRun=options.dry_run))
   File "/usr/sbin/up2date", line 1127, in batchRun
    batch.run()
   File "/usr/share/rhn/up2date_client/up2dateBatch.py", line 92, in run
    self.__getPackages()
   File "/usr/share/rhn/up2date_client/up2dateBatch.py", line 223, in __getPackages
    up2date.getPackage(pkg, wrapperUtils.printPkg, wrapperUtils.printRetrieveHash)
   File "/usr/share/rhn/up2date_client/up2date.py", line 180, in getPackage
    buf = rpcServer.doCall(repos.getPackage, pkg, msgCallback, progressCallback)
   File "/usr/share/rhn/up2date_client/rpcServer.py", line 238, in doCall
    ret = apply(method, args, kwargs)
   File "/usr/share/rhn/up2date_client/repoDirector.py", line 37, in getPackage
    return self.handlers[channel['type']].getPackage(pkg, msgCallback,
progressCallback)
   File "/usr/share/rhn/up2date_client/rpmSource.py", line 186, in getPackage
    package = source.getPackage(pkg, MsgCallback, progressCallback)
   File "/usr/share/rhn/up2date_client/repoBackends/up2dateRepo.py", line 214,
in getPackage
    f.write(fd.read())
   File "/usr/lib/python2.3/site-packages/rhn/transports.py", line 793, in read
    return fd.read()

Strace shows:

3703  read(7, "n\0a\357\36\255\223)x\235!r\363\256\226\345s\233MfU\362"...,
15904) = 15904
3703  write(9, "\34\243)\240+\357h \203U\333\356i\324\274\264\200\336\337"...,
4096) = 4096
3703  write(9, "\223\265_0\245\340\32\267v\246\324w\t\\!\322d.\266r H\36"...,
4096) = 4096
3703  gettimeofday({1203694275, 326623}, NULL) = 0
3703  write(9, "\353\2232\331R1\'\306c\326\336\352\233\31=\320:\220\224"...,
4096) = 4096
3703  write(9, "\247\231\275\33\276\243\253\340\177\367d\321\351\1\226"...,
4096) = 4096
3703  write(1, "#", 1)                  = 1
3703  write(1, "#", 1)                  = 1
3703  write(1, "#", 1)                  = 1
3703  write(1, "#", 1)                  = 1
3703  write(1, "#", 1)                  = 1
3703  write(1, "#", 1)                  = 1
3703  write(1, "#", 1)                  = 1
3703  write(1, "#", 1)                  = 1
3703  write(1, "#", 1)                  = 1
3703  write(1, "#", 1)                  = 1
3703  write(1, "#", 1)                  = 1
3703  write(1, "#", 1)                  = 1
3703  write(1, "#", 1)                  = 1
3703  write(1, "#", 1)                  = 1
3703  write(1, "#", 1)                  = 1
3703  write(1, "#", 1)                  = 1
3703  write(1, "#", 1)                  = 1
3703  write(1, "#", 1)                  = 1
3703  write(1, "#", 1)                  = 1
3703  write(1, "#", 1)                  = 1
3703  write(1, "#", 1)                  = 1
3703  write(1, "#", 1)                  = 1
3703  write(1, "#", 1)                  = 1
3703  write(1, "#", 1)                  = 1
3703  write(1, "#", 1)                  = 1
3703  write(1, "#", 1)                  = 1
3703  write(1, " Done.                   ", 25) = 25
3703  write(1, "\n", 1)                 = 1
3703  write(9, "7\5\204g\320l\332\374\330\3248}\t2\264H\205g\362\322N\v"...,
3584) = 3584
3703  _llseek(9, 0, [0], SEEK_SET)      = 0
3703  _llseek(9, 0, [0], SEEK_SET)      = 0
3703  _llseek(9, 0, [0], SEEK_SET)      = 0
3703  fstat64(9, {st_mode=S_IFREG|0755, st_size=2622242304, ...}) = 0
3703  _llseek(9, 0, [0], SEEK_CUR)      = 0
3703  write(1, "An error has occurred:", 22) = 22
3703  write(1, "\n", 1)                 = 1
3703  write(1, "exceptions.OverflowError", 24) = 24
3703  write(1, "\n", 1)                 = 1
3703  gettimeofday({1203694275, 336605}, NULL) = 0

A lsof run while the up2date was still running shows that the file was actually
correctly downloaded from the Satellite:

up2date 3703 root    0u   CHR  136,0                 2 /dev/pts/0
up2date 3703 root    1u   CHR  136,0                 2 /dev/pts/0
up2date 3703 root    2u   CHR  136,0                 2 /dev/pts/0
up2date 3703 root    3r   REG  253,0    8581120 340040 /var/lib/rpm/Packages
up2date 3703 root    4r   REG  253,0      12288 340046 /var/lib/rpm/Name
up2date 3703 root    5r   REG  253,0      94208 340042 /var/lib/rpm/Providename
up2date 3703 root    6r   REG  253,0          0 340250
/var/spool/up2date/huge-rpm-1.0-20080221.142211.noarch.rpm
up2date 3703 root    7u  IPv4   8482               TCP
vmware166.englab.brq.redhat.com:32798->xen34.englab.brq.redhat.com:https
(ESTABLISHED)
up2date 3703 root    8u   REG  253,0          0 340250
/var/spool/up2date/huge-rpm-1.0-20080221.142211.noarch.rpm
up2date 3703 root    9u   REG  253,0 2589831168 275287 /tmp/_rhn_transports-3703
(deleted)

^^ note the size of the /tmp/_rhn_transports-3703.
Comment 1 Jan Pazdziora 2008-02-25 07:22:01 EST
If I wrap that fd.read() with try except and print the exception, I get:

requested number of bytes is more than a Python string can hold
An error has occurred:
exceptions.OverflowError

So ... we use that SmartIO thingy to download the file to a temporary file in
tmp, and then try to read the whole content of that tmp file into memory. Is
that correct?
Comment 2 Jan Pazdziora 2008-02-25 08:28:07 EST
Hmmm. Plus, that

f.write(fd.read())

in /usr/share/rhn/up2date_client/repoBackends/up2dateRepo.py:214 does not look
like a good approach either. It slurps the whole file into memory and then
writes it to the filePath.

Why can't the self.s.getPackage get another argument, the directory to which to
store that temporary file, and then just do a rename in that directory?
Comment 3 RHEL Product and Program Management 2008-02-25 11:28:10 EST
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.
Comment 4 Jan Pazdziora 2008-03-27 16:51:20 EDT
If you need to generate some 2+GB rpm to test, look into bug 433795 for a spec file.
Comment 9 errata-xmlrpc 2008-07-24 16:05:58 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2008-0771.html

Note You need to log in before you can comment on or make changes to this bug.