Bug 484381 - yum-fastestmirror should not rely on a ping response time when choosing a mirror to download
yum-fastestmirror should not rely on a ping response time when choosing a mir...
Status: NEW
Product: Fedora
Classification: Fedora
Component: yum-utils (Show other bugs)
25
All Linux
low Severity low
: ---
: ---
Assigned To: packaging-team-maint
Fedora Extras Quality Assurance
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-02-06 10:00 EST by Artem S. Tashkinov
Modified: 2016-11-24 06:29 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-02-18 06:58:30 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
wireshark log showing fastestmirror testing mirrors.kernel.org (1012 bytes, text/plain)
2011-05-10 05:18 EDT, Steve Tyler
no flags Details
wireshark capture while downloading repomd.xml from mirrors.kernel.org (2.31 KB, text/plain)
2011-05-10 09:24 EDT, Steve Tyler
no flags Details
Incomplete patch for fastestmirror.py to use httping instead of tcp connect (1.84 KB, patch)
2013-04-19 13:38 EDT, Penelope Fudd
no flags Details | Diff

  None (edit)
Description Artem S. Tashkinov 2009-02-06 10:00:50 EST
Many servers listed as fedora mirrors have an excellent ping time but at the same time they offer a very low download speed.

So, instead of relying on a ping round-trip time, yum-fastestmirror *should* fetch a small file (say 100-300KB would be the best) from every mirror and only according to this information it should decide which mirror to use.

Thank you.
Comment 1 James Antill 2009-02-06 10:31:40 EST
 This is a bad idea, fastestmirror should finish very quickly.
Comment 2 Artem S. Tashkinov 2009-02-06 11:02:42 EST
I absolutely don't get you.

yum-fastestmirror often makes more much harm than helps and you say it just "should finish quickly".

OK, I wash my hands of this issue.
Comment 3 Steve Tyler 2011-05-09 21:34:30 EDT
Artem, you have a good suggestion, but very bad manners ...
The yum-guys would love to make a good thing even better ...
Comment 4 Artem S. Tashkinov 2011-05-10 01:18:47 EDT
(In reply to comment #3)
> Artem, you have a good suggestion, but very bad manners ...
> The yum-guys would love to make a good thing even better ...

OK, let's be polite. :)

However this bug is already closed (regardless its validity) so I don't think anything can be changed. I can reopen it if you think it's worth it.
Comment 5 Steve Tyler 2011-05-10 05:18:33 EDT
Created attachment 497976 [details]
wireshark log showing fastestmirror testing mirrors.kernel.org

Now that we are being polite ... :-)

With wireshark, I found that fastestmirror appears to open and immediately close a TCP connection with each mirror. The attached log is an example with mirrors.kernel.org (149.20.4.71).

$ grep kernel.org /var/cache/yum/x86_64/14/timedhosts.txt
mirrors.kernel.org 0.0580010414124

>>> 43.085103+0.0580010414124
43.1431040414124

This is slightly before the [FIN, ACK] from the mirror, as captured by wireshark.

Procedure:
Start wireshark
$ sudo yum clean all
$ sudo yum repolist

yum-3.2.28-5.fc14.noarch
yum-plugin-fastestmirror-1.1.28-1.fc14.noarch
Comment 6 Steve Tyler 2011-05-10 05:35:10 EDT
BTW, the only reason I am using fastestmirror is to exclude a mirror that is frequently stale.
Comment 7 Steve Tyler 2011-05-10 09:24:22 EDT
Created attachment 498002 [details]
wireshark capture while downloading repomd.xml from mirrors.kernel.org

A quick test shows that downloading repomd.xml from 63 mirrors could take about 7.8 seconds.

$ wc -l /var/cache/yum/x86_64/14/timedhosts.txt 
63 /var/cache/yum/x86_64/14/timedhosts.txt

>>> 0.149429-0.026102
0.123327
>>> 0.123327*63
7.769601000000001

Command was:
$ curl 'http://mirrors.kernel.org/fedora/releases/14/Everything/x86_64/os/repodata/repomd.xml' > /dev/null
Comment 8 Steve Tyler 2011-05-10 10:05:36 EDT
The metalink.xml files have URLs for repomd.xml files on mirrors. With the following command, curl reports their download speed in bytes per second. In a test run, the curl command took about 15 seconds.

$ pwd
/var/cache/yum/x86_64/14/fedora

$ grep http metalink.xml | grep location | sed -e 's/^[^>]*>//' -e 's/<.*$//' | xargs /usr/bin/time curl -L > /dev/null

$ grep http metalink.xml | grep location | wc -l
57
Comment 9 Steve Tyler 2011-05-10 10:39:27 EDT
Curl download statistics can be formatted on the command line:

$ grep http metalink.xml | grep location | sed -e 's/^[^>]*>//' -e 's/<.*$//' | xargs curl -L -s -w 'DATA: %{time_total} %{url_effective}\n' | grep 'DATA:'

Example output line:

DATA: 0.122 http://mirrors.kernel.org/fedora/releases/14/Everything/x86_64/os/repodata/repomd.xml
Comment 10 Steve Tyler 2011-05-10 11:08:49 EDT
(In reply to comment #9)
> Curl download statistics can be formatted on the command line:
> 
> $ grep http metalink.xml | grep location | sed -e 's/^[^>]*>//' -e 's/<.*$//' |
> xargs curl -L -s -w 'DATA: %{time_total} %{url_effective}\n' | grep 'DATA:'
> 
> Example output line:
> 
> DATA: 0.122
> http://mirrors.kernel.org/fedora/releases/14/Everything/x86_64/os/repodata/repomd.xml

If you sort the output on field two, and separately sort timedhosts.txt on field two, you can get an idea of how well the two methods correlate.

In a test, the host that has the shortest repomd.xml download time has one of the longest TCP ping times.

So you are right, Artem. :-)
Comment 11 Steve Tyler 2011-05-11 08:06:43 EDT
(In reply to comment #1)
>  This is a bad idea, fastestmirror should finish very quickly.

To put this in perspective, I ran several tests and found that yum can download repodata for two repos in 15-20 seconds from an unknown mirror over my fiber optic connection. As the above tests show, the time to download repomd.xml from many mirrors could significantly increase that time:

1. Estimate of 7.8 seconds with 63 mirrors (Comment 7)
2. Measurement of about 15 seconds with 57 mirrors (Comment 8)

Procedure:
$ sudo yum clean all
$ /usr/bin/time sudo yum repolist --noplugins

Example (13+3=16 seconds; 22 seconds elapsed):
$ /usr/bin/time sudo yum repolist --noplugins
fedora/metalink                                          | 9.8 kB     00:00     
fedora                                                   | 4.2 kB     00:00     
fedora/primary_db                                        |  14 MB     00:13     
updates-testing/metalink                                 |  12 kB     00:00     
updates-testing                                          | 4.7 kB     00:00     
updates-testing/primary_db                               | 3.5 MB     00:03     
repo id                     repo name                                     status
fedora                      Fedora 15 - x86_64                            24,086
*updates-testing            Fedora 15 - x86_64 - Test Updates              4,715
repolist: 28,801
5.55user 0.87system 0:22.27elapsed 28%CPU (0avgtext+0avgdata 111264maxresident)k
0inputs+195584outputs (0major+11062minor)pagefaults 0swaps

yum-3.2.29-4.fc15.noarch
Comment 12 Steve Tyler 2011-05-11 09:38:44 EDT
Artem, the component is yum-utils.

$ rpm -qi yum-plugin-fastestmirror | grep Source
Source RPM  : yum-utils-1.1.30-2.fc15.src.rpm
Comment 13 Fedora End Of Life 2013-04-03 16:15:54 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 19 development cycle.
Changing version to '19'.

(As we did not run this process for some time, it could affect also pre-Fedora 19 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 19 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora19
Comment 14 Penelope Fudd 2013-04-19 13:38:03 EDT
Created attachment 737706 [details]
Incomplete patch for fastestmirror.py to use httping instead of tcp connect
Comment 15 Penelope Fudd 2013-04-19 13:39:42 EDT
Hi, I attempted to switch fastestmirror to use 'httping'.  My patch is included.

However, due to the threading model used by fastestmirror, the plugin hangs after running httping 13 times.  I don't know enough about python to change the threading configuration to a multiprocess solution.

strace of yum when hung reports: 
  futex(0x149c910, FUTEX_WAIT_PRIVATE, 0, NULL

Thanks
Comment 16 Fedora End Of Life 2015-01-09 16:38:23 EST
This message is a notice that Fedora 19 is now at end of life. Fedora 
has stopped maintaining and issuing updates for Fedora 19. It is 
Fedora's policy to close all bug reports from releases that are no 
longer maintained. Approximately 4 (four) weeks from now this bug will
be closed as EOL if it remains open with a Fedora 'version' of '19'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 19 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.
Comment 17 Fedora End Of Life 2015-02-18 06:58:30 EST
Fedora 19 changed to end-of-life (EOL) status on 2015-01-06. Fedora 19 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.
Comment 18 Jan Kurik 2015-07-15 11:22:17 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 23 development cycle.
Changing version to '23'.

(As we did not run this process for some time, it could affect also pre-Fedora 23 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 23 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora23
Comment 19 Fedora End Of Life 2016-11-24 05:23:25 EST
This message is a reminder that Fedora 23 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 23. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '23'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 23 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Note You need to log in before you can comment on or make changes to this bug.