Bug 169214

so you'd rather yum NOT traverse all mirrors? How would it know when there have
been 'enough' failures and to abort instead of looking through all the mirrors?
What criteria would be appropriate - I mean ' a few failures' seems a bit narrow.

I thought the whole point of the mirrors was so people wouldn't get stuck behind
a few failing mirror repositories either overloaded or broken. 

(In reply to comment #1)
> so you'd rather yum NOT traverse all mirrors?
Yes.

> How would it know when there have been 'enough' failures and to abort
> instead of looking through all the mirrors?
The most simple approach would be a user-configurable counter ("abort after N
broken mirrors"). 

Many other, more intelligent approaches are thinkable, e.g. a ranking/priority
based mirror preference scheduling, based on statistics on bandwith,
reachability and etc.

More abstract: From my perspective, this a special case of "dynamically
assigning resources", "error handling" and "scheduling". Topics, many
intelligent people have written many papers and thesises on, e.g. in operations
research, industrial automation and computer science.

Probably any such approach is better than the current situation
(non-interruptable round robin scheduling without error handling)

> What criteria would be appropriate - I mean ' a few failures' seems a bit
> narrow.
Yes, it would be a primitive heuristics.
 
> I thought the whole point of the mirrors was so people wouldn't get 
> stuck behind a few failing mirror repositories either overloaded or broken.
Agreed, but letting yum iterate through all mirrors, and let it download 2.8MB
from all of them also isn't nice. How many mirrors does FC have 20 .. 40 -> 50
.. 100MB, makes half a day on low bandwith connection.

What makes things worse (IMO inacceptable) is yum not having been interruptable
when this had happened.

the uninterruptability is a bug and it is the process of being fixed, but the
fix requires some changes in rpm, too.

When the interruptability bug gets fixed then I think the above complaint goes
away and adding an option for the above just seems silly.

What exactly do you find "silly" in preventing a nightly yum update from
downloading broken crap from mirrors too often?

I find it silly to add a config option and a bunch of code that someone would
need to configure in order to deter a situation that is:
1. not all that common
2. more easily and correctly dealt with by:
   a. fixing the mirrors
   b. configuring your own mirror list
   c. not using the mirrorlist

(In reply to comment #5)
> I find it silly to add a config option and a bunch of code that someone would
> need to configure in order to deter a situation that is:
> 1. not all that common
> 2. more easily and correctly dealt with by:
>    a. fixing the mirrors
>    b. configuring your own mirror list
>    c. not using the mirrorlist

Well, seems to me as if you still haven't comprehended that implementing at
minimum a functional error handling is a must. In programming mission critical
SW, this often by way exceeds the amount of effort to implement "nominal
operation". 

Admitted, yum isn't necessarily mission critical, but a very important core
component of Fedora, so not wanting to look into it isn't necessarily a clever idea.

> Well, seems to me as if you still haven't comprehended that implementing at
> minimum a functional error handling is a must. In programming mission critical
> SW, this often by way exceeds the amount of effort to implement "nominal
> operation". 

What isn't functional error handling. When yum encounters an error on one mirror
it moves onto the next one until it either:
 1. runs out of mirrors
 2. finds a mirror that solves the problem

what's your problem with that?

(In reply to comment #7)

> What isn't functional error handling. When yum encounters an error on one mirror
> it moves onto the next one until it either:
>  1. runs out of mirrors
... several dozens of mirrors 

>  2. finds a mirror that solves the problem
... if it doesn't find any, because the master is corrupted?
 
> what's your problem with that?

3. It isn't interruptable

I see a lot of problems with this:
a) 3. to me qualifies as candidate FC5 blocker
A non-interruptable program is crap and must not be shipped.

b) You are ignoring:
- time (slow/fast mirrors)
- amount (size of downloads required)
- cost (a product of time and amount of downloads)
- HW failures/timeouts
- Quality of mirrors (reachability, some mirrors seem to be broken or
unreachable most of the time)
- User customizability.

Finally, ... should such incidents occur often, you should be aware that people
will switch off mirroring and resort to hard-coding mirrors.

IIRC, it was you who repeatedly recommended people to do so ;)

Do you need more evidence to understand that mirroring in yum lacks a lot?

Ralf as has been explained repeatedly in this bug, there is an issue with rpm
causing the uninterruptable error, we are working on this.  Calling things crap
is hostile and not a good incentive towards co-operative working - can we be
civil here.

Yum supplies a plugin architecture which enables you to define some policy on
this - have you tried the fastestmirror plug in in yum-utils cvs?  

We also intend to ship a repository configuration tool in fc5 which will enable
mirrorselection addition/removal.

This problem is hugely multiplied on a dial-up connection.

What happens is something like this (NOT the _actual_ output, but very similar):

Downloading primary.xml
Socket error!
Trying another mirror!
Downloading primary.xml
Socket error!
Trying another mirror!
Downloading primary.xml
Socket error!
Trying another mirror!
Downloading primary.xml
Socket error!
Trying another mirror!
Downloading primary.xml
Socket error!
Trying another mirror!
Downloading primary.xml
Socket error!
Trying another mirror!
Downloading primary.xml
Socket error!
Trying another mirror!
Downloading primary.xml
Socket error!
Trying another mirror!
Downloading primary.xml
Socket error!
Trying another mirror!
Out of mirrors!

I believe that the cause of this is the timeout being a few seconds too short. 
If the timeout were extended by even 3 seconds, I might actually be able to
update my Fedora Core 4 installation!  And then there will be one less insecure
computer in the world!

the timeout is an adjustable value in the config

under [main] put:
timeout=300

see if it helps

That seemed to work!  That should be made the default in future versions of yum.

It happened again:

# yum update
...
primary.xml.gz            100% |=========================| 264 kB    01:12
http://ftp.nara.wide.ad.jp/pub/Linux/fedora/core/updates/4/i386/repodata/primary.xml.gz:
[Errno 4] Socket Error: timed out
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://ftp.uni-bayreuth.de/linux/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://ftp.ndlug.nd.edu/pub/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:09
http://fedora.cat.pdx.edu/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://zeniv.linux.org.uk/pub/distributions/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 352 kB    00:10
http://less.cogeco.net/pub/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:11
http://ftp.funet.fi/pub/mirrors/ftp.redhat.com/pub/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:10
http://ftp.iasi.roedu.net/mirrors/fedora.redhat.com/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://mirror.web-ster.com/fedora/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://mirror.switch.ch/ftp/mirror/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 347 kB    00:09
http://mirror2.etf.bg.ac.yu/distributions/fedora/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:07
http://ftp-stud.fht-esslingen.de/pub/Mirrors/fedora.redhat.com/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://www.fedora.is/fedora/core/updates/4/i386/repodata/primary.xml.gz: [Errno
-1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://mirror.linux.duke.edu/pub/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://ftp.ussg.iu.edu/linux/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://ftp.tugraz.at/mirror/redhat-linux/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://ftp.upjs.sk/pub/linux/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
http://mirror.netglobalis.net/pub/fedora/core/updates/4/i386/repodata/primary.xml.gz:
[Errno 4] IOError: HTTP Error 404: Date: Sat, 19 Nov 2005 10:02:07 GMT
Server:
Content-Length: 319
Connection: close
Content-Type: text/html; charset=iso-8859-1
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://dl.atrpms.net/mirrors/fedoracore/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 352 kB    00:08
http://sunsite.icm.edu.pl/pub/Linux/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://ftp.udl.es/pub/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://fedora.arcticnetwork.ca/updates/4/i386/repodata/primary.xml.gz: [Errno
-1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://www.las.ic.unicamp.br/pub/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:26
http://www.muug.mb.ca/pub/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://ftp.riken.jp/Linux/fedora/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://sunsite.mff.cuni.cz/pub/fedora/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://fr2.rpmfind.net/linux/fedora/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://ftp.ale.org/pub/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 346 kB    00:08
http://mirror.phy.olemiss.edu/mirror/fedora/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://ftp1.skynet.cz/pub/linux/fedora/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:09
http://planetmirror.com/pub/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 352 kB    00:14
http://mirrors.tummy.com/mirrors/fedora.redhat.com/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 352 kB    00:08
http://mirror.stanford.edu/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://mirror.newnanutilities.org/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:07
http://wftp.tu-chemnitz.de/pub/linux/fedora-core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://mirror.hiwaay.net/redhat/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksumTrying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://fedora.mirrors.tds.net/pub/fedora-core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://ftp.esat.net/mirrors/download.fedora.redhat.com/pub/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://download.fedora.redhat.com/pub/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 352 kB    00:08
http://mirrors.bevc.net/fedora/updates/4/i386/repodata/primary.xml.gz: [Errno
-1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://fedora.ngi.it/updates/4/i386/repodata/primary.xml.gz: [Errno -1] Metadata
file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://fr.rpmfind.net/linux/fedora/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://ftp.ps.pl/pub/linux/fedora-core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://srl.cs.jhu.edu/YUM/fedora-core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================|    0 B    00:09
http://ftp.tecnoera.com/pub/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno 4] Socket Error: timed out
Trying other mirror.
http://mirror.averse.net/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno 4] IOError: HTTP Error 404: Date: Sat, 19 Nov 2005 10:07:25 GMT
Server: Apache
Transfer-Encoding: chunked
Content-Type: text/html; charset=iso-8859-1
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://www-ftp.lip6.fr/pub/linux/distributions/fedora/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://ftp.lug.ro/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 343 kB    00:07
http://mirror.usu.edu/mirrors/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
http://linux.admin.uillinois.edu/pub/linux/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno 4] IOError: HTTP Error 403: Date: Sat, 19 Nov 2005 10:08:01 GMT
Server: Apache/2.0.46 (Red Hat)
Content-Length: 362
Connection: close
Content-Type: text/html; charset=iso-8859-1
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://mirrors.playboy.com/fedora/updates/4/i386/repodata/primary.xml.gz: [Errno
-1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://ftp.isu.edu.tw/pub/Linux/Fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksumTrying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:11
http://mirror.cs.wisc.edu/pub/mirrors/linux/download.fedora.redhat.com/pub/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://ftp.heanet.ie/pub/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
http://klid.dk/homeftp/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno 4] IOError: HTTP Error 403: Date: Sat, 19 Nov 2005 10:08:45 GMT
Server: Apache-AdvancedExtranetServer/1.3.28 (Mandrake Linux/3.4.92mdk) PHP/4.3.3
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html; charset=iso-8859-1
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://ftp.crihan.fr/mirrors/fedora.redhat.com/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://mirror.etf.bg.ac.yu/distributions/fedora/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
http://ftp.chg.ru/pub/Linux/fedora/core/updates/4/i386/repodata/primary.xml.gz:
[Errno 4] IOError: HTTP Error 404: Date: Sat, 19 Nov 2005 10:09:17 GMT
Server: Apache/1.3.33 (Unix)
Transfer-Encoding: chunked
Content-Type: text/html; charset=iso-8859-1
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://limestone.uoregon.edu/ftp/fedora/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 352 kB    00:08
http://mirrors.csumb.edu/pub/fedora/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://redhat.linux.ee/pub/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
http://ftp.ipv6.funet.fi/pub/mirrors/ftp.redhat.com/pub/fedora/linux/core/updates/4/i386/repodata/primary.xml.gz:
[Errno 4] IOError: <urlopen error (101, 'Network is unreachable')>
Trying other mirror.
primary.xml.gz            100% |=========================| 354 kB    00:08
http://mirrors.kernel.org/fedora/core/updates/4/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
Error: failure: repodata/primary.xml.gz from updates-released: [Errno 256] No
more mirrors to try.

This is a mix of bug 139459 and the timeout stuff which has since been fixed