Bug 851178

Summary: yum doesn't prefer private top priority repository
Product: [Fedora] Fedora Reporter: Kamil Páral <kparal>
Component: yumAssignee: Fedora Packaging Toolset Team <packaging-team>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 18CC: awilliam, ffesti, james.antill, maxamillion, packaging-team, tim.lauridsen, zpavlas
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: RejectedNTH
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-08-31 06:45:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
yum output
none
yum interactive mode with makecache
none
yum output log with makecache none

Description Kamil Páral 2012-08-23 12:20:57 UTC
Description of problem:
In our office we have a local Fedora repository mirror we use. Because we have a very slow Internet connection and a very fast LAN connection, it is essential that this mirror is automatically selected by default. We use MirrorManager [1] to do that.

I verified that both
http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-18&arch=i386
and
http://mirrors.fedoraproject.org/metalink?repo=fedora-18&arch=i386
return our private mirror as the top one (in the metalink file it is marked as preference="100" and mm0:private="True").

However, yum doesn't use it (most of the time in the least).

Reproducer:
$ yum clean all
$ yum makecache
and the metadata are downloaded with 100kB/s speed (instead of 100MB/s). Yum doesn't display which mirror it uses (even with --debuglevel=10), but I used network introspection tools to confirm it uses third-party repositories instead of our private one.

I checked repomd.xml in our private repo and it is up-to-date (compared with repomd.xml from master mirror dl.fedoraproject.org).

If I use baseurl= instead of mirrorlist= in the .repo files and provide our local mirror URL, the metadata are downloaded blazingly fast.

Sometimes, after some time of using yum, it finally downloads the metadata from our local mirror. Maybe it realized it is the fastest mirror available. I can easily revert back to original behavior by running:

$ yum clean all
$ rm /var/cache/yum/* -rf

That erases 'timedhosts' file and our local mirror is not used again.

There were certainly good intentions to make yum automatically select fastest mirror in the long run. But we, as Fedora QA, perform dozens of yum operations per day, and we need to use our local mirror all of the time (unless it's outdated). We can't afford Anaconda/preupgrade/"yum update" to use an Internet mirror instead of our local one. That seriously hampers our testing speed. We can't also modify repo files for every virtual machine we create. That's why we have MirrorManager.

Remedy:
Please make sure private top priority repositories are respected and prefered - i.e. if metalink indicates the top preferred repository is private, don't download from other repositories, because it is reasonable to assume that the private repository will have much faster access than the rest of them.

Also, if possible, please show currently used repositories with --debuglevel=10 (or lower). That will help debugging similar issues much easier.

[1] http://fedoraproject.org/wiki/Infrastructure/Mirroring#MirrorManager:_the_Fedora_Mirror_Management_system

Version-Release number of selected component (if applicable):
yum-3.4.3-33.fc18

How reproducible:
very often

Steps to Reproduce:
1. you must have a private repository registered in mirror manager
2. $ yum clean all
3. $ rm /var/cache/yum/* -rf
3. $ yum makecache
4. see that the download speed is too slow (Internet mirror being used instead of LAN mirror).

Additional info:
We would really appreciate some solution soon, because this issue makes Fedora 18 testing painful for us.

Comment 1 Zdeněk Pavlas 2012-08-23 12:43:49 UTC
When there's no cached information, the 1st mirror should be used, and if it's faster than the "default_speed" option (1MBps is the default) and does not fail, no other mirror should be ever tried.

Could you try adding "failovermethod=priority" to yum.conf?

The default value is "roundrobin", and this shuffles mirrors on each run.  Normally it does not matter, since the ordering is mostly ignored, but this seems to be a special case.

Comment 2 James Antill 2012-08-23 14:46:09 UTC
You can use:

yum repolist -v fedora updates

...and it will show the first host it will hit. Unless you have the "fastestmirror" plugin installed (or something similar) it should be the highest priority item in the metalink/mirror list.

Comment 3 Kamil Páral 2012-08-23 15:44:37 UTC
Created attachment 606640 [details]
yum output

When I do that, it reports

Repo-baseurl : http://download.eng.brq.redhat.com/pub/fedora/linux/development/18/i386/os/ (22 more)

which is correct, but which is definitely not the mirror that was used for downloading metadata (because it was too slow). Also from the error messages in the output it's clear that other mirrors were used.

It is a stock F18 installation, no additional yum plugins used.

Comment 4 Zdeněk Pavlas 2012-08-27 06:57:54 UTC
Ok, found what's really going on.

1) failovermethod=priority, and the fast local mirror is at the beginning of the mirror list (so everything is set up correctly)

2) repomd.xml is fetched, and the first mirror gets used.  The file size is 4306b, and it takes anything from 5ms to 120ms to download.  Don't know exactly what's causing the lag (it's certainly not the fork/exec/import overhead, probably something on the network layer).  Maybe the HTTP server itself does some session tracking, and the 1st request has extra latency, who knows.

3) Normally, small files are assumed to give a poor speed estimates, and don't alter the cached speed much.  But when there's *no* prior measurement, this mechanism fails, and the mirror speed is set to what we calculate (718kBps in this case).

4) The default speed of all other mirrors is 1MBps, so they are tried first, before we hit the "fast" one again.

I think handling the "first measurement" case specially should fix this- just don't update the speed if it's the 1st measurement, and the file was smaller than 1MB.

Comment 5 Fedora Update System 2012-08-27 10:59:36 UTC
python-urlgrabber-3.9.1-19.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/python-urlgrabber-3.9.1-19.fc18

Comment 6 Fedora Update System 2012-08-27 16:29:28 UTC
Package python-urlgrabber-3.9.1-19.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing python-urlgrabber-3.9.1-19.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-12749/python-urlgrabber-3.9.1-19.fc18
then log in and leave karma (feedback).

Comment 7 Kamil Páral 2012-08-29 13:45:10 UTC
Zdenek, thank you very much, this update improves the behavior substantially.

I tried:
$ yum clean all
$ rm /var/cache/yum/* -rf
$ yum repolist enabled
$ yum search asdfasdf
$ yum provides asdfasdf
$ yum group list
$ yum update kernel --disableplugin=presto

Everything is downloaded extremely fast, therefore our local mirror is used.


However, there is still some glitch for this use case:

$ yum clean all
$ rm /var/cache/yum/* -rf
$ yum makecache

Some metadata are downloaded very fast, but some of them are downloaded from different mirrors. According to the interactive console output it seems to be a problem of fedora/group_gz and fedora/other_db (see screenshot). I tried to log the output, but with an output redirection some key information are missing, so the log is not much useful.

Please note that that the first use case have no such problems (I executed a lot of search etc commands to make sure all the databases are downloaded). It just happens with yum makecache.

Comment 8 Kamil Páral 2012-08-29 13:46:06 UTC
Created attachment 607906 [details]
yum interactive mode with makecache

Comment 9 Kamil Páral 2012-08-29 13:48:18 UTC
Created attachment 607908 [details]
yum output log with makecache

Our local mirror is mentioned on lines 8 and 24.

Comment 10 Kamil Páral 2012-08-29 13:51:23 UTC
Proposing as F18 Alpha NTH. We need to get this into Alpha TCs/RCs in order to be able to test effectively.

Comment 11 Fedora Update System 2012-08-29 15:53:24 UTC
yum-3.4.3-38.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/yum-3.4.3-38.fc18

Comment 12 Fedora Update System 2012-08-29 15:55:24 UTC
python-urlgrabber-3.9.1-20.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/python-urlgrabber-3.9.1-20.fc18

Comment 13 Adam Williamson 2012-08-29 16:58:42 UTC
Discussed at 2012-08-29 NTH review meeting. We agreed this isn't really serious enough to rate a post-freeze change to a critical package like yum, and is workaroundable for the single group of users affected.

Comment 14 Fedora Update System 2012-08-31 06:45:02 UTC
yum-3.4.3-40.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 15 Fedora Update System 2012-08-31 06:54:13 UTC
python-urlgrabber-3.9.1-20.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.