Bug 1018686 - fastest mirror detection slow
fastest mirror detection slow
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: librepo (Show other bugs)
20
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Tomas Mlcoch
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-14 03:52 EDT by Ales Kozumplik
Modified: 2014-09-30 19:41 EDT (History)
2 users (show)

See Also:
Fixed In Version: librepo-1.4.0-1.fc20
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-11-24 18:45:52 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Ales Kozumplik 2013-10-14 03:52:45 EDT
testing out some package downloading features in DNF I measured that detecting the fastestmirror takes about 5s, i.e. DNF saves in average 5 seconds per the download phase if I disable fastestmirror in Repo.get_package_target():

            self._handle.setopt(librepo.LRO_FASTESTMIRROR, False)

Can this be sped up?

It takes as much time even if single package has to be downloaded, and even if (as it turns out eventually) it is already downloaded.

If this can not be improved please assign back to DNF. We will have to implement some strategy then to only use LRO_FASTESTMIRROR selectively.
Comment 1 Tomas Mlcoch 2013-10-14 06:50:54 EDT
Fastest mirror detection is done once per handle before first download. Currently it takes about 3-4 sec at most, it could be less if less number of mirrors is present and all the mirrors are fast.

Librepo itself doesn't consider number of packages that should be downloaded and do this detection, if enabled, at the time when the internal mirrorlist is builded (right after the metalink/mirrorlist is parsed). So if you know that only one single (and small) package will be downloaded, then disable the LRO_FASTESTMIRROR option could be suitable.

The "single and already downloaded package trigger the fastest mirror detection" issue is fixed by the commit https://github.com/Tojaj/librepo/commit/5688dae9413523bb5c64a910a6aa64a0697860ab

A measure that should leads to better performance could be a cache for the fastestmirror results. I will implement a cache mechanism soon.
Comment 2 Ales Kozumplik 2013-10-14 08:37:16 EDT
(In reply to Tomas Mlcoch from comment #1)
> The "single and already downloaded package trigger the fastest mirror
> detection" issue is fixed by the commit
> https://github.com/Tojaj/librepo/commit/
> 5688dae9413523bb5c64a910a6aa64a0697860ab

Thanks!

> A measure that should leads to better performance could be a cache for the
> fastestmirror results. I will implement a cache mechanism soon.

Or maybe have a hard limit on total download size (10MB?) below which fastestmirror is not triggered (e.g. change LRO_FASTESTMIRROR not to take a boolean but an integer for MB).
Comment 3 Tomas Mlcoch 2013-10-15 04:10:22 EDT
Your idea is not bad, but I would prefer the approach with the cache.

Pros of cache:
- All downloads, not just the big one, are done from the fastest mirror(s).
- The mentioned 5sec overhead comes only when the cache is expired or doesn't exist, not for every bigger download.

I will implement a prototype of cache and we'll see.
Comment 4 Ales Kozumplik 2013-10-15 08:34:01 EDT
> Pros of cache:
> - All downloads, not just the big one, are done from the fastest mirror(s).
> - The mentioned 5sec overhead comes only when the cache is expired or
> doesn't exist, not for every bigger download.
> 
> I will implement a prototype of cache and we'll see.

Good idea, thanks.
Comment 5 Tomas Mlcoch 2013-10-24 05:01:30 EDT
New options for caching FASTESTMIRROR results are available:

LRO_FASTESTMIRRORCACHE - Path to the cache file
LRO_FASTESTMIRRORMAXAGE - Maximum age of a record in cache (older records won't be used)

https://github.com/Tojaj/librepo/commit/b8a063763ccd8a84b8ec21a643461eaace9b9c08

Example of usage:
https://github.com/Tojaj/librepo/commit/b819d08c5df0063e691f7b675937ef2d9d7d23bc

Acceptance test that cover this behaviour:
https://github.com/Tojaj/librepo/commit/bc85698e87f19b759091e37cdd4a135ea6ab0196
Comment 6 Fedora Update System 2013-11-19 03:52:19 EST
librepo-1.4.0-1.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/librepo-1.4.0-1.fc20
Comment 7 Fedora Update System 2013-11-19 16:50:39 EST
Package librepo-1.4.0-1.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing librepo-1.4.0-1.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-21715/librepo-1.4.0-1.fc20
then log in and leave karma (feedback).
Comment 8 Fedora Update System 2013-11-24 18:45:52 EST
librepo-1.4.0-1.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.