Bug 152105

Summary: yum checksum error results in double-sized rpm
Product: [Fedora] Fedora Reporter: Fred New <fred.new2911>
Component: yumAssignee: Jeremy Katz <katzj>
Status: CLOSED NOTABUG QA Contact:
Severity: medium Docs Contact:
Priority: medium    
Version: 4CC: katzj
Target Milestone: ---   
Target Release: ---   
Hardware: i386   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2005-03-24 21:29:12 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Fred New 2005-03-24 21:17:53 UTC
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.6) Gecko/20050322 Firefox/1.0.1 Fedora/1.0.1-6

Description of problem:
Yum switched to a second mirror when downloading an update to openoffice.org-core.  When it finished downloading all 75 MB the second time, the resulting RPM was twice as large as expected:

[root@darth packages]# yum update openoffice.org\*
Setting up Update Process
...
Total download size: 91 M
Is this ok [y/N]: y
Downloading Packages:
(1/9): openoffice.org-lan 100% |=========================| 3.1 MB    00:15
(2/9): openoffice.org-cor 100% |=========================|  75 MB    06:33
http://sunsite.mff.cuni.cz/pub/fedora/development/i386/Fedora/RPMS/openoffice.org-core-1.9.87-2.i386.rpm: [Errno -1] Package does not match checksum
Trying other mirror.
(2/9): openoffice.org-cor 100% |=========================|  75 MB    12:05
http://ftp.dulug.duke.edu/pub/fedora/linux/core/development/i386/Fedora/RPMS/openoffice.org-core-1.9.87-2.i386.rpm: [Errno -1] Package does not match checksum
Trying other mirror.
http://distro.ibiblio.org/pub/linux/distributions/fedora/linux/core/development/i386/Fedora/RPMS/openoffice.org-core-1.9.87-2.i386.rpm: [Errno 4] IOError: HTTP Error 404: Date: Thu, 24 Mar 2005 20:53:45 GMT
Server: Apache/1.3.29 (Unix)
Transfer-Encoding: chunked
Content-Type: text/html; charset=iso-8859-1
Trying other mirror.
(3/9): openoffice.org-gra 100% |=========================| 140 kB    00:00
...
Error Downloading Packages:
  openoffice.org-core - 1:1.9.87-2.i386: failure: Fedora/RPMS/openoffice.org-core-1.9.87-2.i386.rpm from development: [Errno 256] No more mirrors to try.
[root@darth packages]# l openoffice.org-core-1.9.8*
-rw-r--r--  1 root  77664786 Mar 19 17:32 openoffice.org-core-1.9.85-1.i386.rpm
-rw-r--r--  1 root  78161753 Mar 22 01:31 openoffice.org-core-1.9.87-1.i386.rpm
-rw-r--r--  1 root 157426750 Mar 24 06:56 openoffice.org-core-1.9.87-2.i386.rpm
[root@darth packages]# pwd
/var/cache/yum/development/packages


Version-Release number of selected component (if applicable):
yum-2.3.1-3

How reproducible:
Didn't try

Steps to Reproduce:
Sorry, I don't know if you can reproduce this one.  The essential steps to reproduce it are listed above.  Please just check the logic to make sure that yum is deleting the downloaded file before trying to download it the second time.  It looks like it just appended to the bad file to create 150 MB of garbage.

Actual Results:  The downloaded RPM was 150 MB.

Expected Results:  The messages indicated that I should get a 75 MB RPM.

Additional info:

Comment 1 Seth Vidal 2005-03-24 21:29:12 UTC
This is the reget functionality. You had already downloaded half so it only had
to download another half.

Reget isn't supposed to and cannot check the content.

Comment 2 Fred New 2005-03-24 22:30:56 UTC
I disagree.  Yum should not be using reget in this case.  If this were rsync,
it could use the old file and figure out where the problem is; I would have
preferred this.  But instead, yum ignored the fact that the original RPM was
bad and appended a perfectly valid RPM to the end of it.

After filing this bug, I used dd to split the 150 MB in two, creating a valid
RPM from the second half.  As indicated by the above error, the first half
was bad:

[fred@darth updates]$ l openoffice.org-core-1.9.87-2.i386.rpm.orig
-rw-r--r--  1 fred 157426750 Mar 24 23:46 openoffice.org-core-1.9.87-2.i386.rpm.orig
[fred@darth updates]$ expr 157426750 / 2 / 125
629707
[fred@darth updates]$ dd if=openoffice.org-core-1.9.87-2.i386.rpm.orig
of=openoffice.org-core-1.9.87-2.i386.rpm.part1 bs=125 count=629707
629707+0 records in
629707+0 records out
[fred@darth updates]$ dd if=openoffice.org-core-1.9.87-2.i386.rpm.orig
of=openoffice.org-core-1.9.87-2.i386.rpm.part2 bs=125 skip=629707
629707+0 records in
629707+0 records out
[fred@darth updates]$ rpm -K openoffice.org-core-1.9.87-2.i386.rpm.part[12]
openoffice.org-core-1.9.87-2.i386.rpm.part1: sha1 MD5 NOT OK
openoffice.org-core-1.9.87-2.i386.rpm.part2: sha1 md5 OK
[fred@darth updates]$

Comment 3 Seth Vidal 2005-03-25 00:42:09 UTC
How is yum supposed to know that the previous parts are bad?

Seriously, either you err on the side of caution and always a delete it, and you
find out later if the file is good or not.

Or you assume the part you have is good, just incomplete and you test it once it
is done for validity. 

How do you think it should happen and still keep reget functionality?

Comment 4 Fred New 2005-03-26 13:12:56 UTC
If the RPM gets downloaded to the point where yum decides that it is time to run
sha1sum and the sha1sum fails, then it should delete the file and start again.

Comment 5 Fred New 2005-03-26 13:45:06 UTC
If you are expecting 78713375 bytes of data and you have downloaded 78713375
bytes of data and the checksum doesn't work, then there is nothing you can add
to the file that will correct the checksum and give you the correct number of bytes.

Comment 6 Seth Vidal 2005-03-26 18:38:00 UTC
to comment #4: what are you talking about?

Remember, this is a reget. If you're expecting 150M and you have 75M then the
bandwidth-sensitive assumption you make is that the first 75M are correct and
you should only download the next 75M.

I'm really confused as to what you think should be the correct behavior and when
you think yum has all the information necessary to checksum the package and to
KNOW it's corrupt and not just incomplete.

Please describe, in detail, what you think correct behavior is.


Comment 7 Fred New 2005-03-26 19:45:51 UTC
But I wasn't expecting 150 MB, I was expecting 75 MB.  I agree completely with
you if I was expecting 150 MB, but I wasn't.

Comment 8 Fred New 2005-03-27 07:54:22 UTC
What I don't understand is in __init__.py, why doesn't cursize == totsize?
Does os.stat(local)[6] return the number of bytes minus 1, while int(po.size())
returns the actual number of bytes?  Or doesn't os.unlink delete the file?

Comment 9 Fred New 2005-03-27 15:04:22 UTC
Please look at openoffice.org-core-1.9.87-2.i386.rpm on any mirror; you will see
that it is 75 MB.  Why is it 150 MB in my /var/yum/cache/development/packages
directory?

Comment 10 Fred New 2005-03-28 06:36:28 UTC
Changing summary to improve searchability.