Bug 140387 - yum header FTP download fails with: 501 REST not compatible with server configuration
yum header FTP download fails with: 501 REST not compatible with server confi...
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: yum (Show other bugs)
3
All Linux
medium Severity medium
: ---
: ---
Assigned To: Jeremy Katz
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2004-11-22 13:53 EST by Deron Meranda
Modified: 2014-01-21 17:50 EST (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2005-06-20 15:59:56 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Deron Meranda 2004-11-22 13:53:28 EST
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.3)
Gecko/20041020

Description of problem:
When trying to do 'yum update' in FC3 against an FTP mirror of the
updates, it fails with an ftp error "501" on a REST directive:

---> Downloading header for openoffice.org-libs to pack into
transaction set.
ftp://ftp.XXXX.com/pub/linux/fedora/core3/updates/openoffice.org-libs-1.1.2-11.5.fc3.i386.rpm:
[Errno 4] IOError: [Errno ftp error] [Errno ftp error] 501 REST not
compatible with server configuration


The FTP server is running ProFTP <http://www.proftpd.org/>.  Under
certain configurations it is unable to service the byte range request
(FTP REST directive).  It thus returns a 501 error.  This limitation
is discussed in the online FAQ on their website, at
<http://www.proftpd.org/docs/faq/linked/faq-ch4.html#AEN413>

This byte range retrieval code is in the urlgrabber module
$PYTHONLIB/site-packages/urlgrabber/byterange.py in the function
FTPRangeHandler().


This appears to be a change in the way yum works in FC3, by trying to
extract the header from the RPM package itself using the byte offsets
present in the repodata XML summary; rather than retrieve the separate
pre-extracted header (*.hdr) file from the server.

Ideally, the urlgrabber should detect FTP servers which refuse the
REST directive, and either simulate it; or fall back to attempting to
download the header from a separate header file rather than extracting
it from a byterange of the RPM file.


Version-Release number of selected component (if applicable):
yum-2.1.11-3

How reproducible:
Always

Steps to Reproduce:
1. Need to put yum repository on a ProFTPd server and configure
appropriately so it refuses REST directives
2. Run "yum update"
    

Actual Results:
---> Downloading header for openoffice.org-libs to pack into
transaction set.
ftp://ftp.XXXX.com/pub/linux/fedora/core3/updates/openoffice.org-libs-1.1.2-11.5.fc3.i386.rpm:
[Errno 4] IOError: [Errno ftp error] [Errno ftp error] 501 REST not
compatible with server configuration

Expected Results:  RPM header would be downloaded.

This is yum-2.1.11-3
Comment 1 Ryan Tomayko 2004-12-09 01:26:51 EST
We have a couple of options at the urlgrabber level which were noted in the original 
report:

1. Fake it. We can act like we have REST support by RETRing the file starting at byte 0 like 
normal, read and disgard up to the first byte of the range, read the range, and then close 
the connection. At first I thought this would be too time consuming but in the case of yum 
it may be viable if the RPM header is fairly close to the beginning of the file.

2. WONTFIX in urlgrabber. Raise this exception up to yum and let yum try to fallback on 
the header file.

Seth, is the header usually pretty close to the beginning of the file? I had originally planned 
to emulate byte-ranges for servers that don't support it by reading-and-disgarding up to 
the range offset but I wasn't sure it would ever really be used and the logic will be a little 
on the complex side.
Comment 2 Deron Meranda 2004-12-09 03:09:48 EST
I looked throught the repo xml files for all of the RPMs in FC3, and
the header data for all packages always starts at byte offset 440.

So, I would think that reading from byte 0 and discarding would seem
to be a pretty reasonable thing to do.  I don't know about all RPMs
though.  Perhaps you could just define some limit (10k?), beyond which
you refused to do the read-and-discard routine?

About ProFTPd in particular; I'm convinced not accepting REST is their
problem, a misinterpretation of the FTP standards.  It has a feature
where uploaded files are saved with a random temporary "hidden" name
until the upload completes, at which point the file is renamed to it's
intended name.  This is meant to prevent people seeing partial files.
 That feature is quite obviously incompatible with REST when used with
uploading.  However there is no reason not to allow REST when
downloading, which is what yum uses it for.  I've started raising the
issue with them but don't know if it will ever be fixed.  ... However,
I still think that the yum urlgrabber problem should be fixed anyway.
Comment 3 Seth Vidal 2004-12-09 17:58:50 EST
comment #1: It's close but not extremely close - it varies based on
the number of signatures on the rpm. Shortest distance is 120bytes, iirc.
I say fake it - if you can't open it - go from 0 until the closest
byte. I think my readheaderFromFile function will handle it. If not,
we'll find out. :)


Comment #2: it varies based on number of sigs on the package.
Comment 4 Ryan Tomayko 2004-12-12 00:34:00 EST
This is fixed in urlgrabber CVS. When urlgrabber runs into a server that returns 501 to a 
REST request, it will send a normal RETR, and gobble up any bytes leading up to the 
beginning range.

I've tested against ProFTPD 1.2.10 (stable) with the HiddenStor option enabled. 

When this will be included in Yum is up to Seth and Jeremy.

Note You need to log in before you can comment on or make changes to this bug.