Bug 612409 - yum shows negative installed size for big packages (y-m-p uses 32bit numbers for SQLite).
yum shows negative installed size for big packages (y-m-p uses 32bit numbers ...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: yum-metadata-parser (Show other bugs)
6.0
All Linux
medium Severity medium
: rc
: ---
Assigned To: James Antill
BaseOS QE Security Team
: Reopened
: 619206 625759 (view as bug list)
Depends On: 679760 704600
Blocks: 661279
  Show dependency treegraph
 
Reported: 2010-07-08 03:01 EDT by David Tardon
Modified: 2014-01-21 01:18 EST (History)
7 users (show)

See Also:
Fixed In Version: yum-metadata-parser-1.1.2-16.el6
Doc Type: Bug Fix
Doc Text:
Due to an error in the conversion of SQLite data for large packages, an attempt to install a package larger than 2GB caused Yum to incorrectly represent the package size as a negative number. This update ensures that yum-metadata-parser converts all data to SQLite correctly, and package sizes are now always represented as expected.
Story Points: ---
Clone Of:
: 661279 (view as bug list)
Environment:
Last Closed: 2011-05-19 10:31:15 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
yum output (950 bytes, application/octet-stream)
2010-07-08 03:01 EDT, David Tardon
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2011:0781 normal SHIPPED_LIVE yum-metadata-parser bug fix update 2011-05-18 14:08:22 EDT

  None (edit)
Description David Tardon 2010-07-08 03:01:53 EDT
Created attachment 430250 [details]
yum output

Description of problem:
When you try to install a big package, like openoffice.org-debuginfo, yum reports Installed size as a negative number.

Version-Release number of selected component (if applicable):
yum-3.2.27-12.el6.noarch

How reproducible:
always

Steps to Reproduce:
1. yum --enablerepo=*-debuginfo install openoffice.org-debuginfo
2. see Installed size: line in the output

Additional info:
rpm -q openoffice.org-debuginfo --queryformat='%{size}' shows (correctly) 2266262890 .
Comment 2 James Antill 2010-07-08 10:11:02 EDT
That's what the .xml says:

  <name>openoffice.org-debuginfo</name>
[...]
<size package="452133452" installed="-2031517036" archive="-2025743476"/>
<location href="openoffice.org-debuginfo-3.2.0-12.25.fc13.x86_64.rpm"/>

...which almost certainly makes this an rpm-python bug (using the wrong type).
Comment 3 James Antill 2010-07-08 10:36:40 EDT
Ok ... rpm doesn't appear to have UINT32 types, lib/gentagtbl.sh just does INT32 and there are lookup functions based on that ... *sigh*.

We can fix it up with something like:

def uint(x): return x & 0xFFFFFFFF
Comment 4 James Antill 2010-07-08 12:07:47 EDT
Ok ... I've just checked this on F-13 and rpm appears to dtrt (not sure how though :). Normal createrepo run looks like:

  <size package="452133452" installed="2263450260" archive="2269223820"/>

...which is "negative" as int32:

% nums 2263450260
 Input: 2263450260
    %#'x = 0x86E9,8294
    %#'u =   2,263,450,260
    %#'o = 0o20,672,301,224
    %#'b = 0b1000,0110,1110,1001,1000,0010,1001,0100

...not sure when this changed, but my guess is that the repodata was created on RHEL-5.
Comment 5 James Antill 2010-07-08 12:09:18 EDT
Panu can you confirm that hdr['size'] not being negative is fixed in recent rpm versions, and what version it was fixed in ... and I guess how hard it would be to backport that fix to RHEL-5 rpm?
Comment 6 Panu Matilainen 2010-07-09 02:24:12 EDT
Yup. Rpm >= 4.6.0 considers all integers from headers to be unsigned, but python bindings got signed numbers in some cases prior to 4.8.0. So RHEL 6 itself is not affected with mis-signedness etc.

The problem with backporting this to RHEL 5 is that rpm 4.4.x internally considers all integers from headers to be signed, limiting sizes to the ~2GB. The patch to change python to get unsigned values instead is not big or hard as such, but it is somewhat scary: it involves returning longs instead of ints in cases like this, and I've no idea whether RHN can deal with that. IIRC yum too needed some patching for rpm returning python longs.
Comment 7 David Tardon 2010-07-09 03:20:18 EDT
Well, just scrap it, then. I thought it was going to be easy fix, like bad conversion somewhere, but if it requires changing half of our infrastructure... I can live with the negative number quite well :)
Comment 10 James Antill 2010-07-28 17:31:30 EDT
*** Bug 619206 has been marked as a duplicate of this bug. ***
Comment 11 James Antill 2010-08-20 09:23:06 EDT
*** Bug 625759 has been marked as a duplicate of this bug. ***
Comment 12 James Antill 2010-08-20 09:24:35 EDT
I'm going to close this as NaB, as any work around we do would be a major hack ... customers will just have to use RHEL-6 for createrepo, if they have _large_ rpms.
Comment 13 Grant Williamson 2010-08-20 09:46:35 EDT
I can reproduce the same issue on RHEL6 running createrepo.

https://bugzilla.redhat.com/show_bug.cgi?id=625759

So EL6 will not solve the issue.
Comment 14 James Antill 2010-08-20 11:45:57 EDT
Grant, you are sure that createrepo was run on RHEL-6?

Can you provide the size data from the XML?
Comment 15 Grant Williamson 2010-08-20 13:06:30 EDT
<package type="rpm">
  <name>ibm-winxp-kvm-3.01</name>
  <arch>noarch</arch>
  <version epoch="0" ver="1" rel="1"/>
  <checksum type="sha256" pkgid="YES">afd31d7d6b9aa80d104196685f92364a1818e508463b0cef037f689edae89ff4</checksum>
  <summary>IBM Client Windows XP KVM Image</summary>
  <description>Open Client for Linux, Microsoft Windows XP KVM Image.</description>
  <packager></packager>
  <url></url>
  <time file="1282279787" build="1282279335"/>
  <size package="3008586525" installed="3044372868" archive="3044374108"/>
<location href="ibm-winxp-kvm-3.01-1-1.noarch.rpm"/>
  <format>
    <rpm:license>IBM Internal Use Only</rpm:license>
    <rpm:vendor/>
    <rpm:group>Applications/Emulators</rpm:group>
    <rpm:buildhost>superrh</rpm:buildhost>
    <rpm:sourcerpm>ibm-winxp-kvm-3.01-1-1.nosrc.rpm</rpm:sourcerpm>
    <rpm:header-range start="280" end="3107"/>
    <rpm:provides>
      <rpm:entry name="ibm-winxp-kvm-3.01" flags="EQ" epoch="0" ver="1" rel="1"/>
    </rpm:provides>
    <rpm:requires>
      <rpm:entry name="sed"/>
      <rpm:entry name="coreutils"/>
      <rpm:entry name="/bin/sh" pre="1"/>
      <rpm:entry name="util-linux-ng"/>
    </rpm:requires>    <file>/etc/libvirt/qemu/ibmwinxp.xml</file>

  </format>
</package>
Comment 16 James Antill 2010-08-20 14:26:34 EDT
Ok ... I think Grant's problem is that when we do the XML => .sqlite conversion, we do:

p->size_package = strtol(value, NULL, 10);

...which is fine, but then:

sqlite3_bind_int  (handle, 20, p->size_package);

...which converts the number to signed 32bit.
Comment 17 Grant Williamson 2010-08-21 08:53:48 EDT
James, is there something I can patch to test? I am willing to rebuild any package.
Comment 18 James Antill 2010-08-21 18:28:34 EDT
Sure, try this against yum-metadata-parser:

http://yum.baseurl.org/gitweb?p=yum-metadata-parser.git;a=commitdiff;h=2d8499cf272bf9027d015fae0d344998debfae69

...it should fix the above bug, but it's possible there are other ones lurking about (given it can't have ever worked), so any testing you can do would be helpful :).
Comment 19 Grant Williamson 2010-08-23 00:59:37 EDT
Initial test on 6.0 works.
Any idea if this can still make the 6.0 GA Deadline or will this need to wait
till 6.1?
Comment 20 James Antill 2010-08-23 09:20:05 EDT
At this point IBM would need to push really hard to get this fixed for 6.0. I've already marked it to be considered for 6.1.
Comment 21 James Antill 2010-09-16 13:52:30 EDT
Uh, just realized this was against yum and not yum-metadata-parser.
Comment 24 RHEL Product and Program Management 2011-01-07 10:35:59 EST
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated
in the current release, Red Hat is unfortunately unable to
address this request at this time. Red Hat invites you to
ask your support representative to propose this request, if
appropriate and relevant, in the next release of Red Hat
Enterprise Linux. If you would like it considered as an
exception in the current release, please ask your support
representative.
Comment 29 Jaromir Hradilek 2011-02-15 12:24:10 EST
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Due to an error in the conversion of SQLite data for large packages, an attempt to install a package larger than 2GB caused Yum to incorrectly represent the package size as a negative number. This update ensures that yum-metadata-parser converts all data to SQLite correctly, and package sizes are now always represented as expected.
Comment 30 Karel Srot 2011-02-17 08:03:59 EST
James, 
what is an expected behavior on 32-bit (i386) platform? I tried to install 3GB package 
which is recognized by yum as 2GB and the installation results to traceback.
64-bit platforms are working fine.

Installing:
 largepkg           noarch           1.0-1             bz612409           2.0 G

Transaction Summary
================================================================================
Install       1 Package(s)
Upgrade       0 Package(s)

Total download size: 2.0 G
Installed size: 2.0 G
Downloading Packages:
Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in <module>
    yummain.user_main(sys.argv[1:], exit_code=True)
  File "/usr/share/yum-cli/yummain.py", line 254, in user_main
    errcode = main(args)
  File "/usr/share/yum-cli/yummain.py", line 192, in main
    return_code = base.doTransaction()
  File "/usr/share/yum-cli/cli.py", line 409, in doTransaction
    problems = self.downloadPkgs(downloadpkgs, callback_total=self.download_callback_total_cb) 
  File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 1619, in downloadPkgs
    cache=po.repo.http_caching != 'none',
  File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 838, in getPackage
    size=package.size,
  File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 810, in _getFile
    size=size
  File "/usr/lib/python2.6/site-packages/urlgrabber/mirror.py", line 408, in urlgrab
    return self._mirror_try(func, url, kw)
  File "/usr/lib/python2.6/site-packages/urlgrabber/mirror.py", line 394, in _mirror_try
    return func_ref( *(fullurl,), **kwargs )
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 967, in urlgrab
    apply(cb_func, (obj, )+cb_args, cb_kwargs)        
  File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 1466, in verifyPkg
    if not po.verifyLocalPkg():
  File "/usr/lib/python2.6/site-packages/yum/packages.py", line 805, in verifyLocalPkg
    datasize=self.packagesize)
  File "/usr/lib/python2.6/site-packages/yum/misc.py", line 318, in checksum
    if datasize is not None and len(data) > datasize:
TypeError: __len__() should return an int
package largepkg is not installed
Comment 31 James Antill 2011-02-18 12:41:46 EST
Uh, that's an annoying bug in python.

I can this to the checksum function:

  if datasize is not None and datasize >= (2 * 1024 * 1024 * 1024):
      datasize = None

...but I'm not sure if any other code will be doing something similar (can you try that?)
Comment 32 Karel Srot 2011-02-21 07:20:56 EST
I was able to install, verify-rpm and remove the 3GB package on i386 using slightly modified condition:

    if datasize is not None and datasize == (2 * 1024 * 1024 * 1024 - 1):
        datasize = None

Of course >= would work but I am not sure whether to resign on the check on 64-bit platforms. On 32-bit there should be exactly 2*1024*1024*1024-1 I think.
Comment 33 James Antill 2011-02-21 14:34:20 EST
Yeh, the check was just a hack to make sure nothing else would blow up when I fixed that problem.

The upstream fix for the problem, was:

commit aa94b544dfe5a9926ff980ccc5f972abc59e0970
Author: James Antill <james@and.org>
Date:   Fri Feb 18 12:56:13 2011 -0500

[...]
+    # Note that len(x) is assert limited to INT_MAX, which is 2GB on i686.
+    length = property(fget=lambda self: self._len)
[...]
-            if datasize is not None and len(data) > datasize:
+            if datasize is not None and data.length > datasize:

...which is nice in that there are no magic numbers, and size checking will still happen on i386 :).
Comment 34 James Antill 2011-02-21 15:24:57 EST
The yum side is fixed in: yum-3.2.29-6.el6 ... I assume I shouldn't put a comma list in the "fixed in version" field?
Comment 35 Karel Srot 2011-02-22 05:23:21 EST
James, 
I think there should be a new bug for the yum side.

To summarize it:
yum-metadata-parser update fixes the bug on 64-bit platforms. yum update is necessary to fix the bug on i686, this update will be available in RHEL6.1.
Comment 36 Karel Srot 2011-02-23 08:01:45 EST
I have created a new bug
Bug 679760 - Cannot install >2 GB package on i686 platform.
to track the changes on yum side.
Comment 37 James Antill 2011-02-24 11:22:19 EST
Found another bug, using strtol() instead of strtoll() for the XML parsing.
Comment 40 errata-xmlrpc 2011-05-19 10:31:15 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0781.html

Note You need to log in before you can comment on or make changes to this bug.