Bug 135652 - DVD image download corruption; arithmetic overflow errors in ftp/lftp/http/wget/mirror sites/etc.?!
DVD image download corruption; arithmetic overflow errors in ftp/lftp/http/wg...
Status: CLOSED DUPLICATE of bug 134508
Product: Fedora
Classification: Fedora
Component: fedora-release (Show other bugs)
3
All Linux
medium Severity high
: ---
: ---
Assigned To: Ed Bailey
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2004-10-13 21:47 EDT by C.H.
Modified: 2007-11-30 17:10 EST (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2006-02-21 14:06:18 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description C.H. 2004-10-13 21:47:26 EDT
Description of problem: 
 
Please review the following and help document and resolve these 
problems. 
 
Someone on a fat pipe (T3+) can validate all this a lot better than I 
can 
and determine exactly what is broken where, and see to the 
appropriate bug 
reports / patches / advisories. 
 
There are four possibilities: 
(a) Specific client tool binaries on Fedora are broken 
(b) Specific server http/ftp binaries on the distribution sites are 
broken 
(c) Specific protocol errors exist in the specifications of FTP/HTTP? 
(d) Specific distribution / mirror site filesystems / operating 
systems are causing errors. 
 
The essence of the problem is that for the past several Fedora core 
and core test releases, the DVD images have been ~4.2 Gigabytes in  
size, and evidentally this is somehow too large for several of the 
HTTP / FTP servers on the distribution / mirror sites and the client 
tools included in Fedora to deal with without arithmetic overflow 
errors in determining the file size. 
 
The result is usually a truncated download of ~ 2.x billion bytes or 
so, and a completion status indicating a successful 100% complete 
download whenever one tries to download the i386 or x86_64 DVD images 
from the mirror sites or main distribution site. 
 
In the past I have been able to use 'ftp' which I assume 
was /usr/bin/ftp to pull the files successfully when running 
Fedora on i386.   
 
Presently I have tried (Running the stock i386 FC3T2 versions): 
/usr/bin/ftp 
/usr/kerberos/bin/ftp 
/usr/bin/wget 
/usr/bin/curl 
/usr/bin/lftp 
and I believe they all seemed that they probably fail to indicate the 
download size  
properly or to in fact execute the download to completion when they 
were let run either on an 
initial download or a 'restart / continue' download. 
 
I have tried to 'restart bytecount' in interactive ftp where 
bytecount is 
somewhat less than the actual disk size of a previously 'finished' 
improperly truncated download to try to retrieve the rest of the file 
and that has yielded incorrect and inconsistent results. 
 
Now I also have an Athlon 64 system running x86_64 Fedora Core 2 
Test 2, and I thought that since that was a 64 bit system and 
'sizeof ( int )' would be 64 bits, that the stock ftp / wget / lftp 
tools on THAT system would be able to handle the situation properly, 
and, to my dismay and surprise I find that those tools seem not to 
work properly either!  Maybe this is a bad interaction with the 
server-side 
server daemons?! 
 
I've downloaded the same 2 gigabyte 'initial chunks' of 
FC3-test3-i386-DVD.iso and FC3-test3-x86_64-DVD.iso three or 
so times now and have yet to find a way to get FC3T2's i386 or 
x86_64 tools to work.  I have had similar problems downloading 
FC2, FC3T1, FC3T2 in the past. 
 
Someone on a fat pipe PLEASE spend a few minutes testing this with 
wget, curl, lftp, ftp against 
 
ftp://download.fedora.redhat.com//pub/fedora/linux/core/test/2.92/i386/iso/ 
-rw-r--r--    1 ftp      ftp      2427422720 Oct 06 01:54 
FC3-test3-i386-DVD.iso 
ftp://download.fedora.redhat.com//pub/fedora/linux/core/test/2.92/i386/iso/FC3-test3-i386-DVD.iso 
 
ftp://download.fedora.redhat.com//pub/fedora/linux/core/test/2.92/x86_64/iso/ 
-rw-r--r--    1 ftp      ftp      2630025216 Oct 06 02:22 
FC3-test3-x86_64-DVD.iso 
ftp://download.fedora.redhat.com//pub/fedora/linux/core/test/2.92/x86_64/iso/FC3-test3-x86_64-DVD.iso 
 
ftp://limestone.uoregon.edu/fedora/test/2.92/x86_64/iso/ 
File: FC3-test3-x86_64-DVD.iso  2568384 KB 10/06/2004   02:22:00 AM 
ftp://limestone.uoregon.edu/fedora/test/2.92/x86_64/iso/FC3-test3-x86_64-DVD.iso 
 
ftp://limestone.uoregon.edu/fedora/test/2.92/i386/iso/ 
FC3-test3-i386-DVD.iso   2370530 KB   10/06/2004   
01:54:00 AM 
ftp://limestone.uoregon.edu/fedora/test/2.92/i386/iso/FC3-test3-i386-DVD.iso 
 
ftp://fedora.cat.pdx.edu/linux/core/test/2.92/i386/iso/ 
FC3-test3-i386-DVD.iso   2370530 KB   10/06/2004   
01:54:00 AM 
ftp://fedora.cat.pdx.edu/linux/core/test/2.92/i386/iso/FC3-test3-i386-DVD.iso 
 
ftp://fedora.cat.pdx.edu/linux/core/test/2.92/x86_64/iso/ 
FC3-test3-x86_64-DVD.iso   2568384 KB   10/06/2004   
02:22:00 AM 
ftp://fedora.cat.pdx.edu/linux/core/test/2.92/x86_64/iso/FC3-test3-x86_64-DVD.iso 
 
i386 Logs: uname -a Linux localhost.com 2.6.8-1.541 #1 Wed Sep 1 
18:01:20 EDT 2004 i686 i686 i386 GNU/Linux 
 
WGET BROKENNESS -- 
wget --mirror 
'ftp://download.fedora.redhat.com//pub/fedora/linux/core/test/2.92/x86_64/iso/FC3-test3-x86_64-DVD.iso' 
RETR FC3-test3-x86_64-DVD.iso ... done. 
Length: -1,664,942,080 
 
CURL BROKENNESS -- 
curl -v -O 
ftp://download.fedora.redhat.com//pub/fedora/linux/core/test/2.92/i386/iso/FC3-test3-i386-DVD.iso 
< 227 Entering Passive Mode (66,187,224,20,56,68) 
* About to connect() to download.fedora.redhat.com port 14404 
* Connecting to download.fedora.redhat.com (66.187.224.20) port 14404 
* Connected the data stream with PASV! 
> TYPE I 
< 200 Switching to Binary mode. 
> SIZE FC3-test3-i386-DVD.iso 
< 213 2427422720 
> RETR FC3-test3-i386-DVD.iso 
< 150 Opening BINARY mode data connection for FC3-test3-i386-DVD.iso 
(2427422720 bytes). 
* Getting file with size: 2427422720 
 
 
/USR/BIN/FTP BROKENNESS -- 
/usr/bin/ftp download.fedora.redhat.com 
ftp> bin 
ftp> cd /pub/fedora/linux/core/test/2.92/i386/iso/ 
ftp> debug 
Debugging on (debug=1). 
ftp> get FC3-test3-i386-DVD.iso 
local: FC3-test3-i386-DVD.iso remote: FC3-test3-i386-DVD.iso 
ftp: setsockopt (ignored): Permission denied 
---> PASV 
227 Entering Passive Mode (66,187,224,20,39,159) 
---> RETR FC3-test3-i386-DVD.iso 
150 Opening BINARY mode data connection for FC3-test3-i386-DVD.iso 
(2427422720 bytes). 
 
 
/USR/KERBEROS/BIN/FTP BROKENNESS -- 
/usr/kerberos/bin/ftp download.fedora.redhat.com 
Using binary mode to transfer files. 
ftp> debug 
ftp> cd /pub/fedora/linux/core/test/2.92/i386/iso/ 
ftp> get FC3-test3-i386-DVD.iso 
150 Opening BINARY mode data connection for FC3-test3-i386-DVD.iso 
(2427422720 bytes). 
 
 
This is a persistent (effecting every single DVD Fedora image that 
I've ever seen on any mirror on 
any version of Fedora) HORRIBLE HORRIBLE problem that should not 
exist in this day and age! 
Aren't intger overflows a leading cause of security flaws in some 
software?  How can it be 
permitted that the several most common client / server file transfer 
tools / protocols / filesystems be broken 
so that they can't even deal with a DVD image file without arithmetic 
overflow corruption problems?! 
 
Please make sure the i386, x86_64 HTTP / FTP client tools (and WWW 
browsers) are patched as needed 
to correctly determine, display, and handle downloads of this now 
very common size range! 
 
Please also ensure the officially listed mirror sites are correctly 
storing the DVD images 
(they are downloadable with non-broken client tools, and the expected 
MD5 / size matches), 
and that they display the correct file size in their directory 
listings and the sizes reported 
by their FTP / HTTP servers! 
 
If for some reason it's not possible to straighten out this horrible 
mess, please switch 
to providing distribution DVD file images split-up so that the 
downloadable pieces are all 2,000,000,000 bytes or less 
e.g. 
/usr/bin/split --bytes=2000000000 FC3-test3-i386-DVD.iso 
/usr/bin/split --bytes=2000000000 FC3-test3-x86_64-DVD.iso 
 
Please check the exact reported and actually downloaded byte counts, 
and the file MD5SUMs 
against the officially listed proper MD5sums for the DVD images and 
determine if there are 
presentation (debug output, directory listings, transfer progress et. 
al. errors), and 
transfer errors (the downloaded file size ends up wrong), and errors 
with the restart / continue 
operation attempting to resume a partially downloaded file past the 3 
GBy mark. 
 
Thanks! 
 
Version-Release number of selected component (if applicable): 
 
 
How reproducible: 
Every time 
 
Steps to Reproduce: 
1. see description; pick a server, pick a client, try a download, 
spin the wheel, take your chances. 
2. 
3. 
   
Actual results: 
truncated files and garbage user interface size listings / status 
reports! 
 
 
 
Expected results: 
Correct size listings and correctly downloaded files. 
 
 
Additional info:
Comment 1 C.H. 2004-10-14 06:58:35 EDT
From the original poster now using FC3T2 x86_64 stock distribution:
uname -a
Linux 2.6.8-1.541 #1 Wed Sep 1 18:01:46 EDT 2004 x86_64 x86_64 x86_64
GNU/Linux

WGET --
wget --mirror
'ftp://download.fedora.redhat.com//pub/fedora/linux/core/test/2.92/x86_64/iso/FC3-test3-x86_64-DVD.iso'
--19:34:37-- 
ftp://download.fedora.redhat.com//pub/fedora/linux/core/test/2.92/x86_64/iso/FC3-test3-x86_64-DVD.iso
           =>
`download.fedora.redhat.com/pub/fedora/linux/core/test/2.92/x86_64/iso/FC3-test3-x86_64-DVD.iso'
==> CWD not required.
==> PASV ... done.    ==> RETR FC3-test3-x86_64-DVD.iso ... done.
Length: 2,630,025,216
*** N.B. the different size printed than the i386 FC3T2 wget!

/USR/BIN/FTP --
/usr/bin/ftp download.fedora.redhat.com
ftp> cd /pub/fedora/linux/core/test/2.92/i386/iso/
ftp> get FC3-test3-i386-DVD.iso
150 Opening BINARY mode data connection for FC3-test3-i386-DVD.iso
(2427422720 bytes).

/USR/BIN/FTP --
ftp> cd /pub/fedora/linux/core/test/2.91/x86_64/iso
ftp> dir
227 Entering Passive Mode (66,187,224,20,43,127)
150 Here comes the directory listing.
-rw-r--r--    1 ftp      ftp      4889448448 Sep 16 05:04
FC3-test2-x86_64-DVD.iso
ftp> get FC3-test2-x86_64-DVD.iso
local: FC3-test2-x86_64-DVD.iso remote: FC3-test2-x86_64-DVD.iso
227 Entering Passive Mode (66,187,224,20,54,76)
150 Opening BINARY mode data connection for FC3-test2-x86_64-DVD.iso
(4889448448 bytes).

WGET AGAIN --
wget --mirror
'ftp://download.fedora.redhat.com/pub/fedora/linux/core/test/2.91/x86_64/iso/FC3-test2-x86_64-DVD.iso'
--19:41:41--  
==> PASV ... done.    ==> RETR FC3-test2-x86_64-DVD.iso ... done.
Length: 4,889,448,448

CURL --
 curl -v -O
'ftp://download.fedora.redhat.com/pub/fedora/linux/core/test/2.91/x86_64/iso/FC3-test2-x86_64-DVD.iso'
< 200 Switching to Binary mode.
> SIZE FC3-test2-x86_64-DVD.iso
< 213 4889448448
> RETR FC3-test2-x86_64-DVD.iso
< 150 Opening BINARY mode data connection for FC3-test2-x86_64-DVD.iso
(4889448448 bytes).
* Getting file with size: 4889448448

/USR/KERBEROS/BIN/FTP --
/usr/kerberos/bin/ftp download.fedora.redhat.com
ftp> cd /pub/fedora/linux/core/test/2.91/x86_64/iso
ftp> get FC3-test2-x86_64-DVD.iso
local: FC3-test2-x86_64-DVD.iso remote: FC3-test2-x86_64-DVD.iso
227 Entering Passive Mode (209,132,176,220,46,218)
150 Opening BINARY mode data connection for FC3-test2-x86_64-DVD.iso
(4889448448 bytes).

Ok from this it looks like maybe:
(a) the sizes of the FC3T3 DVD's may *REALLY* be 2.x GBy now, unlike
all past Fedora Core / Core Test releases.  Perhaps I was
extrapolating from past misbehaviour (errors in server directory
listings and errors in i386 client tools with truncations and bad size
reports) and assuming that the sizes of the FC3T3 DVD's were ALSO
4.x GBy... a reasonable but seemingly wrong presumption.  All this
underscores the needs to publish a text file listing the SIZES of
the files along with the ISO images and their MD5s so the user can
more easily tell if they downloaded properly.

(b) the X86_64 64 bit FC3T2 client tools in general might be working
(lftp, and web browsers weren't checked) after all.  It's reassuring
that they seem to list the true 4.x GBY size of the FC3T2 X86_64 DVD.

(c) I believe there still IS a genuine problem with some i386
Fedora distributed (as of FC3T2) WWW browsers, and WGET and possibly
other tools, because I and others (on fedora-test-list) did have 
the above sorts of truncation and size report errors downloading the
4.X GBy FC3T2 DVD images with those tools.   As you see above, the
i386 FC3T2 wget *IS* broken listing a NEGATIVE download size for even
the smaller FC3T3 DVD image, unlike the x86_64 FC3T2 wget.  
So although the 'test cases' and extrapolations I originally listed
may be a bit incorrect, I think there is an issue here still deserving
test / repair.

Here's something clearly broken that I noticed previously on FC3T2
x86_64 and now have a log of:
$ ls -al
-r--r--r--  1 2588073784 Oct 12 22:42 FC3-test3-x86_64-DVD.iso
-r--r--r--  1         59 Oct 13 20:02 FC3-test3-x86_64-DVD.iso.md5
(the ISO file was from a previous transfer attempt whuich
'finished' [on the same X86_64 box] and was kept to try a restart
subsequently)
$ cat FC3-test3-x86_64-DVD.iso.md5
6c717d6b03685151742ab77cbc3076b4 *FC3-test3-x86_64-DVD.iso
/usr/kerberos/bin/ftp limestone.uoregon.edu
 dir
227 Entering Passive Mode (128,223,162,20,218,15).
150 Opening ASCII mode data connection for file list
drwxr-xr-x   2 263      263           496 Oct  6 17:08 .
drwxr-xr-x   5 263      263           152 Oct  6 16:58 ..
-rw-r--r--   1 263      263      2630025216 Oct  6 02:22
FC3-test3-x86_64-DVD.iso
-rw-r--r--   1 263      263      586057728 Oct  6 02:13
FC3-test3-x86_64-SRPMS-disc1.iso
-rw-r--r--   1 263      263      586061824 Oct  6 02:15
FC3-test3-x86_64-SRPMS-disc2.iso
-rw-r--r--   1 263      263      586049536 Oct  6 02:16
FC3-test3-x86_64-SRPMS-disc3.iso
-rw-r--r--   1 263      263      586055680 Oct  6 02:17
FC3-test3-x86_64-SRPMS-disc4.iso
-rw-r--r--   1 263      263      662339584 Oct  6 02:08
FC3-test3-x86_64-disc1.iso
-rw-r--r--   1 263      263      653533184 Oct  6 02:10
FC3-test3-x86_64-disc2.iso
-rw-r--r--   1 263      263      669048832 Oct  6 02:11
FC3-test3-x86_64-disc3.iso
-rw-r--r--   1 263      263      646377472 Oct  6 02:12
FC3-test3-x86_64-disc4.iso
-rw-r--r--   1 263      263           807 Oct  6 17:08 MD5SUM
226 Transfer complete.
ftp> bin
200 Type set to I
ftp> pas
Passive mode off.
ftp> pas
Passive mode on.
ftp> restart 2500000000
restarting at 2500000000. execute get, put or append to initiate transfer
ftp> get FC3-test3-x86_64-DVD.iso
local: FC3-test3-x86_64-DVD.iso remote: FC3-test3-x86_64-DVD.iso
227 Entering Passive Mode (128,223,162,20,218,21).
350 Restarting at 2147483647. Send STORE or RETRIEVE to initiate transfer
150 Opening BINARY mode data connection for FC3-test3-x86_64-DVD.iso
(482541569 bytes)

So N.B. 
(a) that in the 'ftp dir' the file size is listed as
2630025216 Oct  6 02:22 FC3-test3-x86_64-DVD.iso

(b) restart 2500000000
restarting at 2500000000. execute get, put or append to initiate transfer
seems to accept the restart request, and specifies the given
offset to resume.  Note from above log that the DISK SIZE of the
file being restarted before FTP was run is: 
2588073784 Oct 12 22:42 FC3-test3-x86_64-DVD.iso

(d) Notice the 4.X GBY size listed in the 'get' command's logging
(in restart mode), which is VERY different than the 'ftp dir' command
output!  What
does this mean??

(e) Notice that the GET command says it is restarting at some totally
different location than was requested / indicated above when the
restart command was given and its response received.

ftp> get FC3-test3-x86_64-DVD.iso
local: FC3-test3-x86_64-DVD.iso remote: FC3-test3-x86_64-DVD.iso
227 Entering Passive Mode (128,223,162,20,218,21).
350 Restarting at 2147483647. Send STORE or RETRIEVE to initiate transfer
150 Opening BINARY mode data connection for FC3-test3-x86_64-DVD.iso
(482541569 bytes)
226 Transfer complete.
482541569 bytes received in 3.3e+03 seconds (1.4e+02 Kbytes/s)
...After this transfer finished --
$ ls -l FC3-test3-x86_64-DVD.iso
-rw-------  1 2982541569 Oct 13 21:35 FC3-test3-x86_64-DVD.iso

(a) The '(482541569 bytes)' and '482541569 bytes received' outputs
from the ftp program are just totally wrong and seem to have no
bearing on any reality.  It's both the historical truth of the
DVD images really being 4.X GB as well as this 4.X GBy size reported
at 'ftp restart' that had me convinced the FC3T3 DVDs were 4.X GBy
and that ftp/wget was breaking, and as it turns out only the latter
two seems to be the case. :)

(b) So the ftp restart caused the disk file to grow from
2588073784 bytes to 2982541569 bytes, an addition of 
394,467,785 bytes.  This behaviour seems inexplicable and quite
random.

Anyway it'd be useful to go back and look at the i386 FC3T2 utilities'
behaviours with the FC3T2 X86_64 DVD image which is truly 4.X GBy
and I suspect that beyond the demonstrated WGET bug there will be
other problems with other transfer utilities.

Comment 2 Ed Bailey 2004-10-15 16:54:00 EDT

*** This bug has been marked as a duplicate of 134508 ***
Comment 3 Red Hat Bugzilla 2006-02-21 14:06:18 EST
Changed to 'CLOSED' state since 'RESOLVED' has been deprecated.

Note You need to log in before you can comment on or make changes to this bug.