Bug 213687 - /usr/bin/ftp REGET fails if local file larger 2GiB
/usr/bin/ftp REGET fails if local file larger 2GiB
Status: CLOSED INSUFFICIENT_DATA
Product: Fedora
Classification: Fedora
Component: ftp (Show other bugs)
5
i386 Linux
medium Severity medium
: ---
: ---
Assigned To: Marcela Mašláňová
Ben Levenson
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2006-11-02 11:02 EST by Eric Lee
Modified: 2010-09-26 17:06 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2007-07-31 06:57:29 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
log of ftp connection (2.25 KB, application/octet-stream)
2006-11-06 13:16 EST, Eric Lee
no flags Details

  None (edit)
Description Eric Lee 2006-11-02 11:02:17 EST
Description of problem:     
REGET operation fails if partially downloaded file larger than 2GB     
     
Version-Release number of selected component (if applicable):     
/usr/bin/ftp  
ftp-0.17-33.fc5  
     
How reproducible:     
100%     
     
Steps to Reproduce:     
1. Download more than 2GB of a large file.    
     
2. Abandon the transfer before completion.    
  ftp> !ls -l FC-6-i386-DVD.iso  
-rw-r--r--  1 eric rats 2834486610 Nov  2 10:59 FC-6-i386-DVD.iso  
ftp> ls FC-6-i386-DVD.iso  
227 Entering Passive Mode (193,190,198,20,253,103).  
150 Opening ASCII mode data connection for file list  
-rw-r-----   1 ftp      ftp      3525195776 Oct 18 00:49 FC-6-i386-DVD.iso  
  
3. Issue REGET command to continue download   
 ftp> reget FC-6-i386-DVD.iso  
       
Actual results:     
 local: FC-6-i386-DVD.iso remote: FC-6-i386-DVD.iso  
227 Entering Passive Mode (193,190,198,20,253,150).  
501 REST requires a value greater than or equal to 0  
  
(tcpdump snippet:)  
        0x0030:  628b 8ec5 5245 5354 202d 3134 3630 3438  b...REST.-146048  
        0x0040:  3036 3836 0d0a                           0686..  
  
Expected results:     
   File transfer should resume 
     
Additional info:
Comment 1 Marcela Mašláňová 2006-11-06 07:40:39 EST
I can't reproduce it. I aborted mget file has 2.1G and then reget. File was
transfered ok. 
I have the same version ftp.
Which ftp-server do you use?
Comment 2 Eric Lee 2006-11-06 13:16:04 EST
Created attachment 140492 [details]
log of ftp connection

Maybe you weren't quite far enough past 2GiB (2147483648 bytes).
Here's some transcript of me using ftp.mirrorservice.org.
BTW, the source code appears littered with signed 32 bit vars.
Comment 3 Marcela Mašláňová 2006-11-14 09:15:40 EST
I was under 2G. Problem could be on the server side. Investigate...
Comment 4 Maros Barabas 2007-01-18 12:45:13 EST
I tried this bug reproduce on ftp-0.17-33 on vsftpd-2.0.5-9 but whithout errors.
This seems to be a bug on server's side. Please reassign this bug to your server
package. Thanks.
Comment 5 Marcela Mašláňová 2007-07-31 06:57:29 EDT
Looks like problem on server side. I don't know, which server cause this problem
(comment#4).
Comment 6 Neville Dempsey 2010-09-14 01:15:25 EDT
Basically when a file is very large >= 2147483649 bytes (eg a .iso), and the ftp client "reget" command issues a "REST" command to the server with a negative number.

The "reget" command is used when a ftp download has failed, and the user want to continue the download from the point where the download was interrupted.

I have only checked this with an old version of Centos (exact details follow), hence maybe the bug has been fixed.

Here are some logs demonstrating the issue: 

[nevilled@zod Downloads]$ ftp mirror.primusdatacentre.com.au
Connected to mirror.primusdatacentre.com.au.
220 iPrimus Public Mirror
500 AUTH not understood
500 AUTH not understood
KERBEROS_V4 rejected as an authentication type
Name (mirror.primusdatacentre.com.au:nevilled): anonymous
331 Anonymous login ok, send your complete email address as your password
Password:
230 Anonymous access granted, restrictions apply
Remote system type is UNIX.
Using binary mode to transfer files.
ftp>  cd centos/5.5/isos/i386
250 CWD command successful
ftp> reget CentOS-5.5-i386-bin-DVD.iso
local: CentOS-5.5-i386-bin-DVD.iso remote: CentOS-5.5-i386-bin-DVD.iso
227 Entering Passive Mode (210,50,4,6,185,94).
501 REST requires a value greater than or equal to 0
ftp> !ls -l CentOS-5.5-i386-bin-DVD.iso
-rw-rw-r-- 1 nevilled nevilled 3206618288 Sep 14 11:57 CentOS-5.5-i386-bin-DVD.iso
ftp> quit
221 Goodbye.


USER anonymous
331 Anonymous login ok, send your complete email address as your password
PASS a@b.c
230 Anonymous access granted, restrictions apply
SYST
215 UNIX Type: L8
CWD centos/5.5/isos/i386
250 CWD command successful
TYPE I
200 Type set to I
PASV
227 Entering Passive Mode (210,50,4,6,195,110).
REST -1088349008
501 REST requires a value greater than or equal to 0
QUIT
221 Goodbye.

[nevilled@zod Downloads]$ ls -ltr *.iso
-rw-rw-r-- 1 nevilled nevilled 3206618288 Sep 14 11:57 CentOS-5.5-i386-bin-DVD.iso

[nevilled@zod Downloads]$ uname -a
Linux zod.sgr-a.net 2.6.18-164.11.1.el5 #1 SMP Wed Jan 20 07:39:04 EST 2010 i686 i686 i386 GNU/Linux


[nevilled@zod Downloads]$ rpm -qi ftp
Name        : ftp                          Relocations: (not relocatable)
Version     : 0.17                              Vendor: CentOS
Release     : 35.el5                        Build Date: Fri 27 Feb 2009 08:15:00 AM EST
Install Date: Thu 04 Jun 2009 10:34:16 PM EST      Build Host: chamkaur.karan.org
Group       : Applications/Internet         Source RPM: ftp-0.17-35.el5.src.rpm
Size        : 86109                            License: BSD
Signature   : DSA/SHA1, Mon 09 Mar 2009 11:45:28 AM EST, Key ID a8a447dce8562897
Packager    : Karanbir Singh <kbsingh@karan.org>
Summary     : The standard UNIX FTP (File Transfer Protocol) client.
Description :
The ftp package provides the standard UNIX command-line FTP (File
Transfer Protocol) client.  FTP is a widely used protocol for
transferring files over the Internet and for archiving files.

If your system is on a network, you should install ftp in order to do
file transfers.
Comment 7 Neville Dempsey 2010-09-14 01:32:55 EDT
Here is a patch that solved my problem.  I don't have a test suit, so I cannot take it any further.

There remains a bug in ftp, this will manifest itself when a very large downloads where byte size is ≥ 4294967296 bytes - eg when a Blu-ray Disc download - is restarted/reget

Cheers
NevilleD

[nevilled@zod ftp_bug_fix]$ diff -Naur vanilla/BUILD/netkit-ftp-0.17/ftp/ftp.c reget_fix/BUILD/netkit-ftp-0.17/ftp/ftp.c
--- vanilla/BUILD/netkit-ftp-0.17/ftp/ftp.c     2010-09-14 13:40:20.000000000 +1000
+++ reget_fix/BUILD/netkit-ftp-0.17/ftp/ftp.c   2010-09-14 13:29:27.000000000 +1000
@@ -675,7 +675,7 @@
 
        if (restart_point &&
            (strcmp(cmd, "STOR") == 0 || strcmp(cmd, "APPE") == 0)) {
-               if (fseek(fin, (long) restart_point, 0) < 0) {
+               if (fseek(fin, (unsigned long) restart_point, 0) < 0) {
                        fprintf(stderr, "local: %s: %s\n", local,
                                strerror(errno));
                        restart_point = 0;
@@ -683,7 +683,7 @@
                                (*closefunc)(fin);
                        return;
                }
-               if (command("REST %ld", (long) restart_point)
+               if (command("REST %lu", (unsigned long) restart_point)
                        != CONTINUE) {
                        restart_point = 0;
                        if (closefunc != NULL)
@@ -997,7 +997,7 @@
        if (sigsetjmp(recvabort, 1))
                goto abort;
        if (is_retr && restart_point &&
-           command("REST %ld", (long) restart_point) != CONTINUE)
+           command("REST %lu", (unsigned long) restart_point) != CONTINUE)
                return;
        if (remote) {
                if (command("%s %s", cmd, remote) != PRELIM) {
@@ -1053,7 +1053,7 @@
        case TYPE_I:
        case TYPE_L:
                if (restart_point &&
-                   lseek(fileno(fout), (long) restart_point, L_SET) < 0) {
+                   lseek(fileno(fout), (unsigned long) restart_point, L_SET) < 0) {
                        fprintf(stderr, "local: %s: %s\n", local,
                                strerror(errno));
                        if (closefunc != NULL)

END PATCH

To fix the bug with filesize ≥ 4294967296 bytes - (eg when a Blu-ray Disc download fails) the LSET options needs to be replaced with L_END, or multiple L_CURR.

NOTE: the above STRICTLY details a ftp client bug.  The Server eventually I tested (ftp.swin.edu.au) seemed not to have any problem starting at offset 3206618288.  (I suspect that some ftp servers will encounter problems at filesize ≥ 4294967297 bytes, worth checking)
Comment 8 Neville Dempsey 2010-09-14 01:44:12 EDT
Bug still exists in Fedora 13 and RHEL6:

In particular: http://ftp.redhat.com/redhat/rhel/beta/6/source/SRPMS/ftp-0.17-51.1.el6.src.rpm

Also note that - according to the man pages - a value of (off_t)-1 is returned by lseek and errno is set to indicate the error.

netkit-ftp-0.17 needs to be re-factored to use the type "off_t" to removed misc other bugs.

Good luck
NevilleDNZ

Note You need to log in before you can comment on or make changes to this bug.