|Summary:||DVD image download corruption; arithmetic overflow errors in ftp/lftp/http/wget/mirror sites/etc.?!|
|Product:||[Fedora] Fedora||Reporter:||C.H. <rhbzfedora000>|
|Component:||fedora-release||Assignee:||Ed Bailey <ed>|
|Status:||CLOSED DUPLICATE||QA Contact:|
|Fixed In Version:||Doc Type:||Bug Fix|
|Doc Text:||Story Points:||---|
|Last Closed:||2006-02-21 19:06:18 UTC||Type:||---|
|oVirt Team:||---||RHEL 7.3 requirements from Atomic Host:|
Description C.H. 2004-10-14 01:47:26 UTC
Description of problem: Please review the following and help document and resolve these problems. Someone on a fat pipe (T3+) can validate all this a lot better than I can and determine exactly what is broken where, and see to the appropriate bug reports / patches / advisories. There are four possibilities: (a) Specific client tool binaries on Fedora are broken (b) Specific server http/ftp binaries on the distribution sites are broken (c) Specific protocol errors exist in the specifications of FTP/HTTP? (d) Specific distribution / mirror site filesystems / operating systems are causing errors. The essence of the problem is that for the past several Fedora core and core test releases, the DVD images have been ~4.2 Gigabytes in size, and evidentally this is somehow too large for several of the HTTP / FTP servers on the distribution / mirror sites and the client tools included in Fedora to deal with without arithmetic overflow errors in determining the file size. The result is usually a truncated download of ~ 2.x billion bytes or so, and a completion status indicating a successful 100% complete download whenever one tries to download the i386 or x86_64 DVD images from the mirror sites or main distribution site. In the past I have been able to use 'ftp' which I assume was /usr/bin/ftp to pull the files successfully when running Fedora on i386. Presently I have tried (Running the stock i386 FC3T2 versions): /usr/bin/ftp /usr/kerberos/bin/ftp /usr/bin/wget /usr/bin/curl /usr/bin/lftp and I believe they all seemed that they probably fail to indicate the download size properly or to in fact execute the download to completion when they were let run either on an initial download or a 'restart / continue' download. I have tried to 'restart bytecount' in interactive ftp where bytecount is somewhat less than the actual disk size of a previously 'finished' improperly truncated download to try to retrieve the rest of the file and that has yielded incorrect and inconsistent results. Now I also have an Athlon 64 system running x86_64 Fedora Core 2 Test 2, and I thought that since that was a 64 bit system and 'sizeof ( int )' would be 64 bits, that the stock ftp / wget / lftp tools on THAT system would be able to handle the situation properly, and, to my dismay and surprise I find that those tools seem not to work properly either! Maybe this is a bad interaction with the server-side server daemons?! I've downloaded the same 2 gigabyte 'initial chunks' of FC3-test3-i386-DVD.iso and FC3-test3-x86_64-DVD.iso three or so times now and have yet to find a way to get FC3T2's i386 or x86_64 tools to work. I have had similar problems downloading FC2, FC3T1, FC3T2 in the past. Someone on a fat pipe PLEASE spend a few minutes testing this with wget, curl, lftp, ftp against ftp://download.fedora.redhat.com//pub/fedora/linux/core/test/2.92/i386/iso/ -rw-r--r-- 1 ftp ftp 2427422720 Oct 06 01:54 FC3-test3-i386-DVD.iso ftp://download.fedora.redhat.com//pub/fedora/linux/core/test/2.92/i386/iso/FC3-test3-i386-DVD.iso ftp://download.fedora.redhat.com//pub/fedora/linux/core/test/2.92/x86_64/iso/ -rw-r--r-- 1 ftp ftp 2630025216 Oct 06 02:22 FC3-test3-x86_64-DVD.iso ftp://download.fedora.redhat.com//pub/fedora/linux/core/test/2.92/x86_64/iso/FC3-test3-x86_64-DVD.iso ftp://limestone.uoregon.edu/fedora/test/2.92/x86_64/iso/ File: FC3-test3-x86_64-DVD.iso 2568384 KB 10/06/2004 02:22:00 AM ftp://limestone.uoregon.edu/fedora/test/2.92/x86_64/iso/FC3-test3-x86_64-DVD.iso ftp://limestone.uoregon.edu/fedora/test/2.92/i386/iso/ FC3-test3-i386-DVD.iso 2370530 KB 10/06/2004 01:54:00 AM ftp://limestone.uoregon.edu/fedora/test/2.92/i386/iso/FC3-test3-i386-DVD.iso ftp://fedora.cat.pdx.edu/linux/core/test/2.92/i386/iso/ FC3-test3-i386-DVD.iso 2370530 KB 10/06/2004 01:54:00 AM ftp://fedora.cat.pdx.edu/linux/core/test/2.92/i386/iso/FC3-test3-i386-DVD.iso ftp://fedora.cat.pdx.edu/linux/core/test/2.92/x86_64/iso/ FC3-test3-x86_64-DVD.iso 2568384 KB 10/06/2004 02:22:00 AM ftp://fedora.cat.pdx.edu/linux/core/test/2.92/x86_64/iso/FC3-test3-x86_64-DVD.iso i386 Logs: uname -a Linux localhost.com 2.6.8-1.541 #1 Wed Sep 1 18:01:20 EDT 2004 i686 i686 i386 GNU/Linux WGET BROKENNESS -- wget --mirror 'ftp://download.fedora.redhat.com//pub/fedora/linux/core/test/2.92/x86_64/iso/FC3-test3-x86_64-DVD.iso' RETR FC3-test3-x86_64-DVD.iso ... done. Length: -1,664,942,080 CURL BROKENNESS -- curl -v -O ftp://download.fedora.redhat.com//pub/fedora/linux/core/test/2.92/i386/iso/FC3-test3-i386-DVD.iso < 227 Entering Passive Mode (66,187,224,20,56,68) * About to connect() to download.fedora.redhat.com port 14404 * Connecting to download.fedora.redhat.com (18.104.22.168) port 14404 * Connected the data stream with PASV! > TYPE I < 200 Switching to Binary mode. > SIZE FC3-test3-i386-DVD.iso < 213 2427422720 > RETR FC3-test3-i386-DVD.iso < 150 Opening BINARY mode data connection for FC3-test3-i386-DVD.iso (2427422720 bytes). * Getting file with size: 2427422720 /USR/BIN/FTP BROKENNESS -- /usr/bin/ftp download.fedora.redhat.com ftp> bin ftp> cd /pub/fedora/linux/core/test/2.92/i386/iso/ ftp> debug Debugging on (debug=1). ftp> get FC3-test3-i386-DVD.iso local: FC3-test3-i386-DVD.iso remote: FC3-test3-i386-DVD.iso ftp: setsockopt (ignored): Permission denied ---> PASV 227 Entering Passive Mode (66,187,224,20,39,159) ---> RETR FC3-test3-i386-DVD.iso 150 Opening BINARY mode data connection for FC3-test3-i386-DVD.iso (2427422720 bytes). /USR/KERBEROS/BIN/FTP BROKENNESS -- /usr/kerberos/bin/ftp download.fedora.redhat.com Using binary mode to transfer files. ftp> debug ftp> cd /pub/fedora/linux/core/test/2.92/i386/iso/ ftp> get FC3-test3-i386-DVD.iso 150 Opening BINARY mode data connection for FC3-test3-i386-DVD.iso (2427422720 bytes). This is a persistent (effecting every single DVD Fedora image that I've ever seen on any mirror on any version of Fedora) HORRIBLE HORRIBLE problem that should not exist in this day and age! Aren't intger overflows a leading cause of security flaws in some software? How can it be permitted that the several most common client / server file transfer tools / protocols / filesystems be broken so that they can't even deal with a DVD image file without arithmetic overflow corruption problems?! Please make sure the i386, x86_64 HTTP / FTP client tools (and WWW browsers) are patched as needed to correctly determine, display, and handle downloads of this now very common size range! Please also ensure the officially listed mirror sites are correctly storing the DVD images (they are downloadable with non-broken client tools, and the expected MD5 / size matches), and that they display the correct file size in their directory listings and the sizes reported by their FTP / HTTP servers! If for some reason it's not possible to straighten out this horrible mess, please switch to providing distribution DVD file images split-up so that the downloadable pieces are all 2,000,000,000 bytes or less e.g. /usr/bin/split --bytes=2000000000 FC3-test3-i386-DVD.iso /usr/bin/split --bytes=2000000000 FC3-test3-x86_64-DVD.iso Please check the exact reported and actually downloaded byte counts, and the file MD5SUMs against the officially listed proper MD5sums for the DVD images and determine if there are presentation (debug output, directory listings, transfer progress et. al. errors), and transfer errors (the downloaded file size ends up wrong), and errors with the restart / continue operation attempting to resume a partially downloaded file past the 3 GBy mark. Thanks! Version-Release number of selected component (if applicable): How reproducible: Every time Steps to Reproduce: 1. see description; pick a server, pick a client, try a download, spin the wheel, take your chances. 2. 3. Actual results: truncated files and garbage user interface size listings / status reports! Expected results: Correct size listings and correctly downloaded files. Additional info:
Comment 1 C.H. 2004-10-14 10:58:35 UTC
From the original poster now using FC3T2 x86_64 stock distribution: uname -a Linux 2.6.8-1.541 #1 Wed Sep 1 18:01:46 EDT 2004 x86_64 x86_64 x86_64 GNU/Linux WGET -- wget --mirror 'ftp://download.fedora.redhat.com//pub/fedora/linux/core/test/2.92/x86_64/iso/FC3-test3-x86_64-DVD.iso' --19:34:37-- ftp://download.fedora.redhat.com//pub/fedora/linux/core/test/2.92/x86_64/iso/FC3-test3-x86_64-DVD.iso => `download.fedora.redhat.com/pub/fedora/linux/core/test/2.92/x86_64/iso/FC3-test3-x86_64-DVD.iso' ==> CWD not required. ==> PASV ... done. ==> RETR FC3-test3-x86_64-DVD.iso ... done. Length: 2,630,025,216 *** N.B. the different size printed than the i386 FC3T2 wget! /USR/BIN/FTP -- /usr/bin/ftp download.fedora.redhat.com ftp> cd /pub/fedora/linux/core/test/2.92/i386/iso/ ftp> get FC3-test3-i386-DVD.iso 150 Opening BINARY mode data connection for FC3-test3-i386-DVD.iso (2427422720 bytes). /USR/BIN/FTP -- ftp> cd /pub/fedora/linux/core/test/2.91/x86_64/iso ftp> dir 227 Entering Passive Mode (66,187,224,20,43,127) 150 Here comes the directory listing. -rw-r--r-- 1 ftp ftp 4889448448 Sep 16 05:04 FC3-test2-x86_64-DVD.iso ftp> get FC3-test2-x86_64-DVD.iso local: FC3-test2-x86_64-DVD.iso remote: FC3-test2-x86_64-DVD.iso 227 Entering Passive Mode (66,187,224,20,54,76) 150 Opening BINARY mode data connection for FC3-test2-x86_64-DVD.iso (4889448448 bytes). WGET AGAIN -- wget --mirror 'ftp://download.fedora.redhat.com/pub/fedora/linux/core/test/2.91/x86_64/iso/FC3-test2-x86_64-DVD.iso' --19:41:41-- ==> PASV ... done. ==> RETR FC3-test2-x86_64-DVD.iso ... done. Length: 4,889,448,448 CURL -- curl -v -O 'ftp://download.fedora.redhat.com/pub/fedora/linux/core/test/2.91/x86_64/iso/FC3-test2-x86_64-DVD.iso' < 200 Switching to Binary mode. > SIZE FC3-test2-x86_64-DVD.iso < 213 4889448448 > RETR FC3-test2-x86_64-DVD.iso < 150 Opening BINARY mode data connection for FC3-test2-x86_64-DVD.iso (4889448448 bytes). * Getting file with size: 4889448448 /USR/KERBEROS/BIN/FTP -- /usr/kerberos/bin/ftp download.fedora.redhat.com ftp> cd /pub/fedora/linux/core/test/2.91/x86_64/iso ftp> get FC3-test2-x86_64-DVD.iso local: FC3-test2-x86_64-DVD.iso remote: FC3-test2-x86_64-DVD.iso 227 Entering Passive Mode (209,132,176,220,46,218) 150 Opening BINARY mode data connection for FC3-test2-x86_64-DVD.iso (4889448448 bytes). Ok from this it looks like maybe: (a) the sizes of the FC3T3 DVD's may *REALLY* be 2.x GBy now, unlike all past Fedora Core / Core Test releases. Perhaps I was extrapolating from past misbehaviour (errors in server directory listings and errors in i386 client tools with truncations and bad size reports) and assuming that the sizes of the FC3T3 DVD's were ALSO 4.x GBy... a reasonable but seemingly wrong presumption. All this underscores the needs to publish a text file listing the SIZES of the files along with the ISO images and their MD5s so the user can more easily tell if they downloaded properly. (b) the X86_64 64 bit FC3T2 client tools in general might be working (lftp, and web browsers weren't checked) after all. It's reassuring that they seem to list the true 4.x GBY size of the FC3T2 X86_64 DVD. (c) I believe there still IS a genuine problem with some i386 Fedora distributed (as of FC3T2) WWW browsers, and WGET and possibly other tools, because I and others (on fedora-test-list) did have the above sorts of truncation and size report errors downloading the 4.X GBy FC3T2 DVD images with those tools. As you see above, the i386 FC3T2 wget *IS* broken listing a NEGATIVE download size for even the smaller FC3T3 DVD image, unlike the x86_64 FC3T2 wget. So although the 'test cases' and extrapolations I originally listed may be a bit incorrect, I think there is an issue here still deserving test / repair. Here's something clearly broken that I noticed previously on FC3T2 x86_64 and now have a log of: $ ls -al -r--r--r-- 1 2588073784 Oct 12 22:42 FC3-test3-x86_64-DVD.iso -r--r--r-- 1 59 Oct 13 20:02 FC3-test3-x86_64-DVD.iso.md5 (the ISO file was from a previous transfer attempt whuich 'finished' [on the same X86_64 box] and was kept to try a restart subsequently) $ cat FC3-test3-x86_64-DVD.iso.md5 6c717d6b03685151742ab77cbc3076b4 *FC3-test3-x86_64-DVD.iso /usr/kerberos/bin/ftp limestone.uoregon.edu dir 227 Entering Passive Mode (128,223,162,20,218,15). 150 Opening ASCII mode data connection for file list drwxr-xr-x 2 263 263 496 Oct 6 17:08 . drwxr-xr-x 5 263 263 152 Oct 6 16:58 .. -rw-r--r-- 1 263 263 2630025216 Oct 6 02:22 FC3-test3-x86_64-DVD.iso -rw-r--r-- 1 263 263 586057728 Oct 6 02:13 FC3-test3-x86_64-SRPMS-disc1.iso -rw-r--r-- 1 263 263 586061824 Oct 6 02:15 FC3-test3-x86_64-SRPMS-disc2.iso -rw-r--r-- 1 263 263 586049536 Oct 6 02:16 FC3-test3-x86_64-SRPMS-disc3.iso -rw-r--r-- 1 263 263 586055680 Oct 6 02:17 FC3-test3-x86_64-SRPMS-disc4.iso -rw-r--r-- 1 263 263 662339584 Oct 6 02:08 FC3-test3-x86_64-disc1.iso -rw-r--r-- 1 263 263 653533184 Oct 6 02:10 FC3-test3-x86_64-disc2.iso -rw-r--r-- 1 263 263 669048832 Oct 6 02:11 FC3-test3-x86_64-disc3.iso -rw-r--r-- 1 263 263 646377472 Oct 6 02:12 FC3-test3-x86_64-disc4.iso -rw-r--r-- 1 263 263 807 Oct 6 17:08 MD5SUM 226 Transfer complete. ftp> bin 200 Type set to I ftp> pas Passive mode off. ftp> pas Passive mode on. ftp> restart 2500000000 restarting at 2500000000. execute get, put or append to initiate transfer ftp> get FC3-test3-x86_64-DVD.iso local: FC3-test3-x86_64-DVD.iso remote: FC3-test3-x86_64-DVD.iso 227 Entering Passive Mode (128,223,162,20,218,21). 350 Restarting at 2147483647. Send STORE or RETRIEVE to initiate transfer 150 Opening BINARY mode data connection for FC3-test3-x86_64-DVD.iso (482541569 bytes) So N.B. (a) that in the 'ftp dir' the file size is listed as 2630025216 Oct 6 02:22 FC3-test3-x86_64-DVD.iso (b) restart 2500000000 restarting at 2500000000. execute get, put or append to initiate transfer seems to accept the restart request, and specifies the given offset to resume. Note from above log that the DISK SIZE of the file being restarted before FTP was run is: 2588073784 Oct 12 22:42 FC3-test3-x86_64-DVD.iso (d) Notice the 4.X GBY size listed in the 'get' command's logging (in restart mode), which is VERY different than the 'ftp dir' command output! What does this mean?? (e) Notice that the GET command says it is restarting at some totally different location than was requested / indicated above when the restart command was given and its response received. ftp> get FC3-test3-x86_64-DVD.iso local: FC3-test3-x86_64-DVD.iso remote: FC3-test3-x86_64-DVD.iso 227 Entering Passive Mode (128,223,162,20,218,21). 350 Restarting at 2147483647. Send STORE or RETRIEVE to initiate transfer 150 Opening BINARY mode data connection for FC3-test3-x86_64-DVD.iso (482541569 bytes) 226 Transfer complete. 482541569 bytes received in 3.3e+03 seconds (1.4e+02 Kbytes/s) ...After this transfer finished -- $ ls -l FC3-test3-x86_64-DVD.iso -rw------- 1 2982541569 Oct 13 21:35 FC3-test3-x86_64-DVD.iso (a) The '(482541569 bytes)' and '482541569 bytes received' outputs from the ftp program are just totally wrong and seem to have no bearing on any reality. It's both the historical truth of the DVD images really being 4.X GB as well as this 4.X GBy size reported at 'ftp restart' that had me convinced the FC3T3 DVDs were 4.X GBy and that ftp/wget was breaking, and as it turns out only the latter two seems to be the case. :) (b) So the ftp restart caused the disk file to grow from 2588073784 bytes to 2982541569 bytes, an addition of 394,467,785 bytes. This behaviour seems inexplicable and quite random. Anyway it'd be useful to go back and look at the i386 FC3T2 utilities' behaviours with the FC3T2 X86_64 DVD image which is truly 4.X GBy and I suspect that beyond the demonstrated WGET bug there will be other problems with other transfer utilities.
Comment 2 Ed Bailey 2004-10-15 20:54:00 UTC
*** This bug has been marked as a duplicate of 134508 ***
Comment 3 Red Hat Bugzilla 2006-02-21 19:06:18 UTC
Changed to 'CLOSED' state since 'RESOLVED' has been deprecated.