Bug 612409

That's what the .xml says:

  <name>openoffice.org-debuginfo</name>
[...]
<size package="452133452" installed="-2031517036" archive="-2025743476"/>
<location href="openoffice.org-debuginfo-3.2.0-12.25.fc13.x86_64.rpm"/>

...which almost certainly makes this an rpm-python bug (using the wrong type).

Ok ... rpm doesn't appear to have UINT32 types, lib/gentagtbl.sh just does INT32 and there are lookup functions based on that ... *sigh*.

We can fix it up with something like:

def uint(x): return x & 0xFFFFFFFF

Ok ... I've just checked this on F-13 and rpm appears to dtrt (not sure how though :). Normal createrepo run looks like:

  <size package="452133452" installed="2263450260" archive="2269223820"/>

...which is "negative" as int32:

% nums 2263450260
 Input: 2263450260
    %#'x = 0x86E9,8294
    %#'u =   2,263,450,260
    %#'o = 0o20,672,301,224
    %#'b = 0b1000,0110,1110,1001,1000,0010,1001,0100

...not sure when this changed, but my guess is that the repodata was created on RHEL-5.

Panu can you confirm that hdr['size'] not being negative is fixed in recent rpm versions, and what version it was fixed in ... and I guess how hard it would be to backport that fix to RHEL-5 rpm?

Yup. Rpm >= 4.6.0 considers all integers from headers to be unsigned, but python bindings got signed numbers in some cases prior to 4.8.0. So RHEL 6 itself is not affected with mis-signedness etc.

The problem with backporting this to RHEL 5 is that rpm 4.4.x internally considers all integers from headers to be signed, limiting sizes to the ~2GB. The patch to change python to get unsigned values instead is not big or hard as such, but it is somewhat scary: it involves returning longs instead of ints in cases like this, and I've no idea whether RHN can deal with that. IIRC yum too needed some patching for rpm returning python longs.

Well, just scrap it, then. I thought it was going to be easy fix, like bad conversion somewhere, but if it requires changing half of our infrastructure... I can live with the negative number quite well :)

*** Bug 619206 has been marked as a duplicate of this bug. ***

*** Bug 625759 has been marked as a duplicate of this bug. ***

I'm going to close this as NaB, as any work around we do would be a major hack ... customers will just have to use RHEL-6 for createrepo, if they have _large_ rpms.

I can reproduce the same issue on RHEL6 running createrepo.

https://bugzilla.redhat.com/show_bug.cgi?id=625759

So EL6 will not solve the issue.

Grant, you are sure that createrepo was run on RHEL-6?

Can you provide the size data from the XML?

<package type="rpm">
  <name>ibm-winxp-kvm-3.01</name>
  <arch>noarch</arch>
  <version epoch="0" ver="1" rel="1"/>
  <checksum type="sha256" pkgid="YES">afd31d7d6b9aa80d104196685f92364a1818e508463b0cef037f689edae89ff4</checksum>
  <summary>IBM Client Windows XP KVM Image</summary>
  <description>Open Client for Linux, Microsoft Windows XP KVM Image.</description>
  <packager></packager>
  <url></url>
  <time file="1282279787" build="1282279335"/>
  <size package="3008586525" installed="3044372868" archive="3044374108"/>
<location href="ibm-winxp-kvm-3.01-1-1.noarch.rpm"/>
  <format>
    <rpm:license>IBM Internal Use Only</rpm:license>
    <rpm:vendor/>
    <rpm:group>Applications/Emulators</rpm:group>
    <rpm:buildhost>superrh</rpm:buildhost>
    <rpm:sourcerpm>ibm-winxp-kvm-3.01-1-1.nosrc.rpm</rpm:sourcerpm>
    <rpm:header-range start="280" end="3107"/>
    <rpm:provides>
      <rpm:entry name="ibm-winxp-kvm-3.01" flags="EQ" epoch="0" ver="1" rel="1"/>
    </rpm:provides>
    <rpm:requires>
      <rpm:entry name="sed"/>
      <rpm:entry name="coreutils"/>
      <rpm:entry name="/bin/sh" pre="1"/>
      <rpm:entry name="util-linux-ng"/>
    </rpm:requires>    <file>/etc/libvirt/qemu/ibmwinxp.xml</file>

  </format>
</package>

Ok ... I think Grant's problem is that when we do the XML => .sqlite conversion, we do:

p->size_package = strtol(value, NULL, 10);

...which is fine, but then:

sqlite3_bind_int  (handle, 20, p->size_package);

...which converts the number to signed 32bit.

James, is there something I can patch to test? I am willing to rebuild any package.

Sure, try this against yum-metadata-parser:

http://yum.baseurl.org/gitweb?p=yum-metadata-parser.git;a=commitdiff;h=2d8499cf272bf9027d015fae0d344998debfae69

...it should fix the above bug, but it's possible there are other ones lurking about (given it can't have ever worked), so any testing you can do would be helpful :).

Initial test on 6.0 works.
Any idea if this can still make the 6.0 GA Deadline or will this need to wait
till 6.1?

At this point IBM would need to push really hard to get this fixed for 6.0. I've already marked it to be considered for 6.1.

Uh, just realized this was against yum and not yum-metadata-parser.

This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated
in the current release, Red Hat is unfortunately unable to
address this request at this time. Red Hat invites you to
ask your support representative to propose this request, if
appropriate and relevant, in the next release of Red Hat
Enterprise Linux. If you would like it considered as an
exception in the current release, please ask your support
representative.

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Due to an error in the conversion of SQLite data for large packages, an attempt to install a package larger than 2GB caused Yum to incorrectly represent the package size as a negative number. This update ensures that yum-metadata-parser converts all data to SQLite correctly, and package sizes are now always represented as expected.

James, 
what is an expected behavior on 32-bit (i386) platform? I tried to install 3GB package 
which is recognized by yum as 2GB and the installation results to traceback.
64-bit platforms are working fine.

Installing:
 largepkg           noarch           1.0-1             bz612409           2.0 G

Transaction Summary
================================================================================
Install       1 Package(s)
Upgrade       0 Package(s)

Total download size: 2.0 G
Installed size: 2.0 G
Downloading Packages:
Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in <module>
    yummain.user_main(sys.argv[1:], exit_code=True)
  File "/usr/share/yum-cli/yummain.py", line 254, in user_main
    errcode = main(args)
  File "/usr/share/yum-cli/yummain.py", line 192, in main
    return_code = base.doTransaction()
  File "/usr/share/yum-cli/cli.py", line 409, in doTransaction
    problems = self.downloadPkgs(downloadpkgs, callback_total=self.download_callback_total_cb) 
  File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 1619, in downloadPkgs
    cache=po.repo.http_caching != 'none',
  File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 838, in getPackage
    size=package.size,
  File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 810, in _getFile
    size=size
  File "/usr/lib/python2.6/site-packages/urlgrabber/mirror.py", line 408, in urlgrab
    return self._mirror_try(func, url, kw)
  File "/usr/lib/python2.6/site-packages/urlgrabber/mirror.py", line 394, in _mirror_try
    return func_ref( *(fullurl,), **kwargs )
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 967, in urlgrab
    apply(cb_func, (obj, )+cb_args, cb_kwargs)        
  File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 1466, in verifyPkg
    if not po.verifyLocalPkg():
  File "/usr/lib/python2.6/site-packages/yum/packages.py", line 805, in verifyLocalPkg
    datasize=self.packagesize)
  File "/usr/lib/python2.6/site-packages/yum/misc.py", line 318, in checksum
    if datasize is not None and len(data) > datasize:
TypeError: __len__() should return an int
package largepkg is not installed

Uh, that's an annoying bug in python.

I can this to the checksum function:

  if datasize is not None and datasize >= (2 * 1024 * 1024 * 1024):
      datasize = None

...but I'm not sure if any other code will be doing something similar (can you try that?)

I was able to install, verify-rpm and remove the 3GB package on i386 using slightly modified condition:

    if datasize is not None and datasize == (2 * 1024 * 1024 * 1024 - 1):
        datasize = None

Of course >= would work but I am not sure whether to resign on the check on 64-bit platforms. On 32-bit there should be exactly 2*1024*1024*1024-1 I think.

Yeh, the check was just a hack to make sure nothing else would blow up when I fixed that problem.

The upstream fix for the problem, was:

commit aa94b544dfe5a9926ff980ccc5f972abc59e0970
Author: James Antill <james>
Date:   Fri Feb 18 12:56:13 2011 -0500

[...]
+    # Note that len(x) is assert limited to INT_MAX, which is 2GB on i686.
+    length = property(fget=lambda self: self._len)
[...]
-            if datasize is not None and len(data) > datasize:
+            if datasize is not None and data.length > datasize:

...which is nice in that there are no magic numbers, and size checking will still happen on i386 :).

The yum side is fixed in: yum-3.2.29-6.el6 ... I assume I shouldn't put a comma list in the "fixed in version" field?

James, 
I think there should be a new bug for the yum side.

To summarize it:
yum-metadata-parser update fixes the bug on 64-bit platforms. yum update is necessary to fix the bug on i686, this update will be available in RHEL6.1.

I have created a new bug
Bug 679760 - Cannot install >2 GB package on i686 platform.
to track the changes on yum side.

Found another bug, using strtol() instead of strtoll() for the XML parsing.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0781.html