Bug 484381
| Summary: | yum-fastestmirror should not rely on a ping response time when choosing a mirror to download | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Artem S. Tashkinov <aros> |
| Component: | yum-utils | Assignee: | Packaging Maintenance Team <packaging-team-maint> |
| Status: | CLOSED WONTFIX | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | low | Docs Contact: | |
| Priority: | low | ||
| Version: | 27 | CC: | ffesti, james.antill, maxamillion, stephent98, tim.lauridsen |
| Target Milestone: | --- | Keywords: | Reopened |
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-07-18 21:30:22 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Attachments: | |||
|
Description
Artem S. Tashkinov
2009-02-06 15:00:50 UTC
This is a bad idea, fastestmirror should finish very quickly. I absolutely don't get you. yum-fastestmirror often makes more much harm than helps and you say it just "should finish quickly". OK, I wash my hands of this issue. Artem, you have a good suggestion, but very bad manners ... The yum-guys would love to make a good thing even better ... (In reply to comment #3) > Artem, you have a good suggestion, but very bad manners ... > The yum-guys would love to make a good thing even better ... OK, let's be polite. :) However this bug is already closed (regardless its validity) so I don't think anything can be changed. I can reopen it if you think it's worth it. Created attachment 497976 [details] wireshark log showing fastestmirror testing mirrors.kernel.org Now that we are being polite ... :-) With wireshark, I found that fastestmirror appears to open and immediately close a TCP connection with each mirror. The attached log is an example with mirrors.kernel.org (149.20.4.71). $ grep kernel.org /var/cache/yum/x86_64/14/timedhosts.txt mirrors.kernel.org 0.0580010414124 >>> 43.085103+0.0580010414124 43.1431040414124 This is slightly before the [FIN, ACK] from the mirror, as captured by wireshark. Procedure: Start wireshark $ sudo yum clean all $ sudo yum repolist yum-3.2.28-5.fc14.noarch yum-plugin-fastestmirror-1.1.28-1.fc14.noarch BTW, the only reason I am using fastestmirror is to exclude a mirror that is frequently stale. Created attachment 498002 [details] wireshark capture while downloading repomd.xml from mirrors.kernel.org A quick test shows that downloading repomd.xml from 63 mirrors could take about 7.8 seconds. $ wc -l /var/cache/yum/x86_64/14/timedhosts.txt 63 /var/cache/yum/x86_64/14/timedhosts.txt >>> 0.149429-0.026102 0.123327 >>> 0.123327*63 7.769601000000001 Command was: $ curl 'http://mirrors.kernel.org/fedora/releases/14/Everything/x86_64/os/repodata/repomd.xml' > /dev/null The metalink.xml files have URLs for repomd.xml files on mirrors. With the following command, curl reports their download speed in bytes per second. In a test run, the curl command took about 15 seconds. $ pwd /var/cache/yum/x86_64/14/fedora $ grep http metalink.xml | grep location | sed -e 's/^[^>]*>//' -e 's/<.*$//' | xargs /usr/bin/time curl -L > /dev/null $ grep http metalink.xml | grep location | wc -l 57 Curl download statistics can be formatted on the command line:
$ grep http metalink.xml | grep location | sed -e 's/^[^>]*>//' -e 's/<.*$//' | xargs curl -L -s -w 'DATA: %{time_total} %{url_effective}\n' | grep 'DATA:'
Example output line:
DATA: 0.122 http://mirrors.kernel.org/fedora/releases/14/Everything/x86_64/os/repodata/repomd.xml
(In reply to comment #9) > Curl download statistics can be formatted on the command line: > > $ grep http metalink.xml | grep location | sed -e 's/^[^>]*>//' -e 's/<.*$//' | > xargs curl -L -s -w 'DATA: %{time_total} %{url_effective}\n' | grep 'DATA:' > > Example output line: > > DATA: 0.122 > http://mirrors.kernel.org/fedora/releases/14/Everything/x86_64/os/repodata/repomd.xml If you sort the output on field two, and separately sort timedhosts.txt on field two, you can get an idea of how well the two methods correlate. In a test, the host that has the shortest repomd.xml download time has one of the longest TCP ping times. So you are right, Artem. :-) (In reply to comment #1) > This is a bad idea, fastestmirror should finish very quickly. To put this in perspective, I ran several tests and found that yum can download repodata for two repos in 15-20 seconds from an unknown mirror over my fiber optic connection. As the above tests show, the time to download repomd.xml from many mirrors could significantly increase that time: 1. Estimate of 7.8 seconds with 63 mirrors (Comment 7) 2. Measurement of about 15 seconds with 57 mirrors (Comment 8) Procedure: $ sudo yum clean all $ /usr/bin/time sudo yum repolist --noplugins Example (13+3=16 seconds; 22 seconds elapsed): $ /usr/bin/time sudo yum repolist --noplugins fedora/metalink | 9.8 kB 00:00 fedora | 4.2 kB 00:00 fedora/primary_db | 14 MB 00:13 updates-testing/metalink | 12 kB 00:00 updates-testing | 4.7 kB 00:00 updates-testing/primary_db | 3.5 MB 00:03 repo id repo name status fedora Fedora 15 - x86_64 24,086 *updates-testing Fedora 15 - x86_64 - Test Updates 4,715 repolist: 28,801 5.55user 0.87system 0:22.27elapsed 28%CPU (0avgtext+0avgdata 111264maxresident)k 0inputs+195584outputs (0major+11062minor)pagefaults 0swaps yum-3.2.29-4.fc15.noarch Artem, the component is yum-utils. $ rpm -qi yum-plugin-fastestmirror | grep Source Source RPM : yum-utils-1.1.30-2.fc15.src.rpm This bug appears to have been reported against 'rawhide' during the Fedora 19 development cycle. Changing version to '19'. (As we did not run this process for some time, it could affect also pre-Fedora 19 development cycle bugs. We are very sorry. It will help us with cleanup during Fedora 19 End Of Life. Thank you.) More information and reason for this action is here: https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora19 Created attachment 737706 [details]
Incomplete patch for fastestmirror.py to use httping instead of tcp connect
Hi, I attempted to switch fastestmirror to use 'httping'. My patch is included. However, due to the threading model used by fastestmirror, the plugin hangs after running httping 13 times. I don't know enough about python to change the threading configuration to a multiprocess solution. strace of yum when hung reports: futex(0x149c910, FUTEX_WAIT_PRIVATE, 0, NULL Thanks This message is a notice that Fedora 19 is now at end of life. Fedora has stopped maintaining and issuing updates for Fedora 19. It is Fedora's policy to close all bug reports from releases that are no longer maintained. Approximately 4 (four) weeks from now this bug will be closed as EOL if it remains open with a Fedora 'version' of '19'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 19 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. Fedora 19 changed to end-of-life (EOL) status on 2015-01-06. Fedora 19 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed. This bug appears to have been reported against 'rawhide' during the Fedora 23 development cycle. Changing version to '23'. (As we did not run this process for some time, it could affect also pre-Fedora 23 development cycle bugs. We are very sorry. It will help us with cleanup during Fedora 23 End Of Life. Thank you.) More information and reason for this action is here: https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora23 This message is a reminder that Fedora 23 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 23. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '23'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 23 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. This message is a reminder that Fedora 25 is nearing its end of life. Approximately 4 (four) weeks from now Fedora will stop maintaining and issuing updates for Fedora 25. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '25'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 25 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. I'm not sure if this bug affects Fedora 27. It needs to be rechecked. yum and related packages are no longer actively developed. They are being replaced with dnf, dnf-utils, etc. I'm closing this bug because it's most likely never going to be fixed. If you still consider your bug report important, reopen it, please. yum and related packages are no longer actively developed. They are being replaced with dnf, dnf-utils, etc. I'm closing this bug because it's most likely never going to be fixed. If you still consider your bug report important, reopen it, please. |