From Bugzilla Helper: User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.3) Gecko/20041020 Description of problem: When trying to do 'yum update' in FC3 against an FTP mirror of the updates, it fails with an ftp error "501" on a REST directive: ---> Downloading header for openoffice.org-libs to pack into transaction set. ftp://ftp.XXXX.com/pub/linux/fedora/core3/updates/openoffice.org-libs-1.1.2-11.5.fc3.i386.rpm: [Errno 4] IOError: [Errno ftp error] [Errno ftp error] 501 REST not compatible with server configuration The FTP server is running ProFTP <http://www.proftpd.org/>. Under certain configurations it is unable to service the byte range request (FTP REST directive). It thus returns a 501 error. This limitation is discussed in the online FAQ on their website, at <http://www.proftpd.org/docs/faq/linked/faq-ch4.html#AEN413> This byte range retrieval code is in the urlgrabber module $PYTHONLIB/site-packages/urlgrabber/byterange.py in the function FTPRangeHandler(). This appears to be a change in the way yum works in FC3, by trying to extract the header from the RPM package itself using the byte offsets present in the repodata XML summary; rather than retrieve the separate pre-extracted header (*.hdr) file from the server. Ideally, the urlgrabber should detect FTP servers which refuse the REST directive, and either simulate it; or fall back to attempting to download the header from a separate header file rather than extracting it from a byterange of the RPM file. Version-Release number of selected component (if applicable): yum-2.1.11-3 How reproducible: Always Steps to Reproduce: 1. Need to put yum repository on a ProFTPd server and configure appropriately so it refuses REST directives 2. Run "yum update" Actual Results: ---> Downloading header for openoffice.org-libs to pack into transaction set. ftp://ftp.XXXX.com/pub/linux/fedora/core3/updates/openoffice.org-libs-1.1.2-11.5.fc3.i386.rpm: [Errno 4] IOError: [Errno ftp error] [Errno ftp error] 501 REST not compatible with server configuration Expected Results: RPM header would be downloaded. This is yum-2.1.11-3
We have a couple of options at the urlgrabber level which were noted in the original report: 1. Fake it. We can act like we have REST support by RETRing the file starting at byte 0 like normal, read and disgard up to the first byte of the range, read the range, and then close the connection. At first I thought this would be too time consuming but in the case of yum it may be viable if the RPM header is fairly close to the beginning of the file. 2. WONTFIX in urlgrabber. Raise this exception up to yum and let yum try to fallback on the header file. Seth, is the header usually pretty close to the beginning of the file? I had originally planned to emulate byte-ranges for servers that don't support it by reading-and-disgarding up to the range offset but I wasn't sure it would ever really be used and the logic will be a little on the complex side.
I looked throught the repo xml files for all of the RPMs in FC3, and the header data for all packages always starts at byte offset 440. So, I would think that reading from byte 0 and discarding would seem to be a pretty reasonable thing to do. I don't know about all RPMs though. Perhaps you could just define some limit (10k?), beyond which you refused to do the read-and-discard routine? About ProFTPd in particular; I'm convinced not accepting REST is their problem, a misinterpretation of the FTP standards. It has a feature where uploaded files are saved with a random temporary "hidden" name until the upload completes, at which point the file is renamed to it's intended name. This is meant to prevent people seeing partial files. That feature is quite obviously incompatible with REST when used with uploading. However there is no reason not to allow REST when downloading, which is what yum uses it for. I've started raising the issue with them but don't know if it will ever be fixed. ... However, I still think that the yum urlgrabber problem should be fixed anyway.
comment #1: It's close but not extremely close - it varies based on the number of signatures on the rpm. Shortest distance is 120bytes, iirc. I say fake it - if you can't open it - go from 0 until the closest byte. I think my readheaderFromFile function will handle it. If not, we'll find out. :) Comment #2: it varies based on number of sigs on the package.
This is fixed in urlgrabber CVS. When urlgrabber runs into a server that returns 501 to a REST request, it will send a normal RETR, and gobble up any bytes leading up to the beginning range. I've tested against ProFTPD 1.2.10 (stable) with the HiddenStor option enabled. When this will be included in Yum is up to Seth and Jeremy.