Bug 851178 - yum doesn't prefer private top priority repository
yum doesn't prefer private top priority repository
Product: Fedora
Classification: Fedora
Component: yum (Show other bugs)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Fedora Packaging Toolset Team
Fedora Extras Quality Assurance
Depends On:
  Show dependency treegraph
Reported: 2012-08-23 08:20 EDT by Kamil Páral
Modified: 2014-01-21 18:23 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2012-08-31 02:45:02 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
yum output (4.33 KB, text/plain)
2012-08-23 11:44 EDT, Kamil Páral
no flags Details
yum interactive mode with makecache (48.09 KB, image/png)
2012-08-29 09:46 EDT, Kamil Páral
no flags Details
yum output log with makecache (29.99 KB, text/plain)
2012-08-29 09:48 EDT, Kamil Páral
no flags Details

  None (edit)
Description Kamil Páral 2012-08-23 08:20:57 EDT
Description of problem:
In our office we have a local Fedora repository mirror we use. Because we have a very slow Internet connection and a very fast LAN connection, it is essential that this mirror is automatically selected by default. We use MirrorManager [1] to do that.

I verified that both
return our private mirror as the top one (in the metalink file it is marked as preference="100" and mm0:private="True").

However, yum doesn't use it (most of the time in the least).

$ yum clean all
$ yum makecache
and the metadata are downloaded with 100kB/s speed (instead of 100MB/s). Yum doesn't display which mirror it uses (even with --debuglevel=10), but I used network introspection tools to confirm it uses third-party repositories instead of our private one.

I checked repomd.xml in our private repo and it is up-to-date (compared with repomd.xml from master mirror dl.fedoraproject.org).

If I use baseurl= instead of mirrorlist= in the .repo files and provide our local mirror URL, the metadata are downloaded blazingly fast.

Sometimes, after some time of using yum, it finally downloads the metadata from our local mirror. Maybe it realized it is the fastest mirror available. I can easily revert back to original behavior by running:

$ yum clean all
$ rm /var/cache/yum/* -rf

That erases 'timedhosts' file and our local mirror is not used again.

There were certainly good intentions to make yum automatically select fastest mirror in the long run. But we, as Fedora QA, perform dozens of yum operations per day, and we need to use our local mirror all of the time (unless it's outdated). We can't afford Anaconda/preupgrade/"yum update" to use an Internet mirror instead of our local one. That seriously hampers our testing speed. We can't also modify repo files for every virtual machine we create. That's why we have MirrorManager.

Please make sure private top priority repositories are respected and prefered - i.e. if metalink indicates the top preferred repository is private, don't download from other repositories, because it is reasonable to assume that the private repository will have much faster access than the rest of them.

Also, if possible, please show currently used repositories with --debuglevel=10 (or lower). That will help debugging similar issues much easier.

[1] http://fedoraproject.org/wiki/Infrastructure/Mirroring#MirrorManager:_the_Fedora_Mirror_Management_system

Version-Release number of selected component (if applicable):

How reproducible:
very often

Steps to Reproduce:
1. you must have a private repository registered in mirror manager
2. $ yum clean all
3. $ rm /var/cache/yum/* -rf
3. $ yum makecache
4. see that the download speed is too slow (Internet mirror being used instead of LAN mirror).

Additional info:
We would really appreciate some solution soon, because this issue makes Fedora 18 testing painful for us.
Comment 1 Zdeněk Pavlas 2012-08-23 08:43:49 EDT
When there's no cached information, the 1st mirror should be used, and if it's faster than the "default_speed" option (1MBps is the default) and does not fail, no other mirror should be ever tried.

Could you try adding "failovermethod=priority" to yum.conf?

The default value is "roundrobin", and this shuffles mirrors on each run.  Normally it does not matter, since the ordering is mostly ignored, but this seems to be a special case.
Comment 2 James Antill 2012-08-23 10:46:09 EDT
You can use:

yum repolist -v fedora updates

...and it will show the first host it will hit. Unless you have the "fastestmirror" plugin installed (or something similar) it should be the highest priority item in the metalink/mirror list.
Comment 3 Kamil Páral 2012-08-23 11:44:37 EDT
Created attachment 606640 [details]
yum output

When I do that, it reports

Repo-baseurl : http://download.eng.brq.redhat.com/pub/fedora/linux/development/18/i386/os/ (22 more)

which is correct, but which is definitely not the mirror that was used for downloading metadata (because it was too slow). Also from the error messages in the output it's clear that other mirrors were used.

It is a stock F18 installation, no additional yum plugins used.
Comment 4 Zdeněk Pavlas 2012-08-27 02:57:54 EDT
Ok, found what's really going on.

1) failovermethod=priority, and the fast local mirror is at the beginning of the mirror list (so everything is set up correctly)

2) repomd.xml is fetched, and the first mirror gets used.  The file size is 4306b, and it takes anything from 5ms to 120ms to download.  Don't know exactly what's causing the lag (it's certainly not the fork/exec/import overhead, probably something on the network layer).  Maybe the HTTP server itself does some session tracking, and the 1st request has extra latency, who knows.

3) Normally, small files are assumed to give a poor speed estimates, and don't alter the cached speed much.  But when there's *no* prior measurement, this mechanism fails, and the mirror speed is set to what we calculate (718kBps in this case).

4) The default speed of all other mirrors is 1MBps, so they are tried first, before we hit the "fast" one again.

I think handling the "first measurement" case specially should fix this- just don't update the speed if it's the 1st measurement, and the file was smaller than 1MB.
Comment 5 Fedora Update System 2012-08-27 06:59:36 EDT
python-urlgrabber-3.9.1-19.fc18 has been submitted as an update for Fedora 18.
Comment 6 Fedora Update System 2012-08-27 12:29:28 EDT
Package python-urlgrabber-3.9.1-19.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing python-urlgrabber-3.9.1-19.fc18'
as soon as you are able to.
Please go to the following url:
then log in and leave karma (feedback).
Comment 7 Kamil Páral 2012-08-29 09:45:10 EDT
Zdenek, thank you very much, this update improves the behavior substantially.

I tried:
$ yum clean all
$ rm /var/cache/yum/* -rf
$ yum repolist enabled
$ yum search asdfasdf
$ yum provides asdfasdf
$ yum group list
$ yum update kernel --disableplugin=presto

Everything is downloaded extremely fast, therefore our local mirror is used.

However, there is still some glitch for this use case:

$ yum clean all
$ rm /var/cache/yum/* -rf
$ yum makecache

Some metadata are downloaded very fast, but some of them are downloaded from different mirrors. According to the interactive console output it seems to be a problem of fedora/group_gz and fedora/other_db (see screenshot). I tried to log the output, but with an output redirection some key information are missing, so the log is not much useful.

Please note that that the first use case have no such problems (I executed a lot of search etc commands to make sure all the databases are downloaded). It just happens with yum makecache.
Comment 8 Kamil Páral 2012-08-29 09:46:06 EDT
Created attachment 607906 [details]
yum interactive mode with makecache
Comment 9 Kamil Páral 2012-08-29 09:48:18 EDT
Created attachment 607908 [details]
yum output log with makecache

Our local mirror is mentioned on lines 8 and 24.
Comment 10 Kamil Páral 2012-08-29 09:51:23 EDT
Proposing as F18 Alpha NTH. We need to get this into Alpha TCs/RCs in order to be able to test effectively.
Comment 11 Fedora Update System 2012-08-29 11:53:24 EDT
yum-3.4.3-38.fc18 has been submitted as an update for Fedora 18.
Comment 12 Fedora Update System 2012-08-29 11:55:24 EDT
python-urlgrabber-3.9.1-20.fc18 has been submitted as an update for Fedora 18.
Comment 13 Adam Williamson 2012-08-29 12:58:42 EDT
Discussed at 2012-08-29 NTH review meeting. We agreed this isn't really serious enough to rate a post-freeze change to a critical package like yum, and is workaroundable for the single group of users affected.
Comment 14 Fedora Update System 2012-08-31 02:45:02 EDT
yum-3.4.3-40.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 15 Fedora Update System 2012-08-31 02:54:13 EDT
python-urlgrabber-3.9.1-20.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.