Bug 481592

Summary: autodownloader gives me grief
Product: [Fedora] Fedora Reporter: Leszek Matok <lam>
Component: autodownloaderAssignee: Hans de Goede <hdegoede>
Status: CLOSED NEXTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: low    
Version: 10CC: hdegoede
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-01-29 23:05:46 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Leszek Matok 2009-01-26 16:33:07 UTC
As the summary says: AutoDL should randomize the mirror list for the file it's trying to download. Right now it always downloads from the first mirror in the autodlrc file.

Maybe it's always using some tricky GeoIP magic similar to yum to choose always the same, slowest server that can upload 10 KB/s tops and breaks the transfer after 200 MiB of Urban Terror. Probably yes, it's Fedora, after all. OK, sarcasm stops here.

Randomizing the list can spread the load the users put on servers hosting binary data.

This change is trivial and has the potential to actually make AutoDL work for me :) Other solutions, like allowing continuing broken downloads or making the "Next mirror" button work, can prove to require more work from your side.

Thanks!

Comment 1 Hans de Goede 2009-01-26 18:44:40 UTC
Erm, no, the best (most reliable / fastest) mirrors are put in front. Some mirrors are quite low bandwidth, so randomization is not an option.

Also it should fallback to the next mirror (after a timeout) automatically, and the next mirror button *does* work.

autodl is not threaded so if its hanging on some network timeout it could be that it is waiting /frozen for a pretty long time. Try giving it some time (like 10 minutes).

If that does not help, can you please describe what you are seeing more detailed and how you think randomizing would fix it ?

Comment 2 Leszek Matok 2009-01-26 19:28:04 UTC
Like I've said, I'm trying to download Urban Terror.

So I get two short notices about licences to accept, then it starts downloading from "Mirror 1 / 6". This mirror is probably using some kind of traffic shaping, so I was trying to use another by pressing "Next mirror". Every single time it gives me MD5 sum error, with one option - Exit.

I'm trying to download this for 5 hours now. Being an Ordinary User unable to change mirror, I let it use the slow one. Once it was interrupted after like 200 MiB (server's fault), second time after 420 MiB (my link's fault). I'm in my third try now. Every run straight from the start.

I wanted to compare mirror speed, so I've resorted to using wget on all the URLs from autodlrc and probably found the problem causing MD5 errors being reported on "Next mirror" - all of the URLs give me 404, 403 or 30x redirecting to main page of a given site.

This last situation (redirection) causes wget download site index (in HTML), I guess autodownloader blindly does the same thing and thinks it's downloaded the file. No wonder the MD5 doesn't match.

OK, so it looks the first mirror truly is the best one, because it's the only one working at all. So that's good news in a way.

Problems:
- of course the package maintainer doesn't maintain the URL list (the package in Rawhide is still "fc9" built almost a year ago), that's something you can't fix yourself, obviously (but it is a problem with autodownloader philosophy, we knew there'll be no control over other people's resources),
- autodownloader shouldn't follow redirections (or at least redirections to text/* documents), instead it should treat it just like 40x errors,
- when something interrupts my download after couple of hours, it'd be really nice to give me a choice to retry instead of offering only one button to quit the application,
- better yet, it should retry automatically after a minute of my inactivity - I know for sure I'm going to leave Urban Terror downloading when I go to sleep tonight,
- after an interruption, no matter if I retry or quit, the next run should try to continue from the point it got interrupted (I've checked, wget -c works on this single working server),
- and not every time you get an EOF from the remote server, you get the whole file (actually, the autodlrc could contain that and be compared with the Content-Length header or however it's called, besides I'm sure that header contained length different that the one up to EOF-from-server, you should retry and not count MD5 sum in that case anyhow).


And yes, now that I understand the problem with the package, I admit it can't be solved with randomization. I thought the button simply doesn't work at all. Better error handling (catching _and_ reporting) would save me some time debugging this and spare me looking silly as well :)

Comment 3 Leszek Matok 2009-01-26 19:54:40 UTC
I've just had another link outage (at 280 MiB). When the link is down, it tries to run through the mirror list, sees "network unreachable" on all of them (instead of downloading garbage) and spits "File UrbanTerror_41_FULL.zip not found on server(s)." with one obvious option: Exit. Nice. Guess what a restart gives me. Compare that to wget -c -w 600, which I'd like autodownloader to mimic.

Comment 4 Hans de Goede 2009-01-28 16:07:27 UTC
I agree that having better handling of broken downloads and supporting resume of dowbloads would be good to have, but that is an RFE not a bug, and an RFE I'm afraid I do not have the time to implement (patches welcome).

In the mean time I've fixed the mirror lists of all .autodlrc files in the quake3 subpackages, an update is on its way to the repos.

Comment 5 Fedora Update System 2009-01-29 23:05:43 UTC
quake3-1.34-0.10.rc4.fc10 has been pushed to the Fedora 10 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 6 Fedora Update System 2009-01-29 23:11:06 UTC
quake3-1.34-0.10.rc4.fc9 has been pushed to the Fedora 9 stable repository.  If problems still persist, please make note of it in this bug report.