Bug 1043177 - decrease the urlgrabber default_speed value
Summary: decrease the urlgrabber default_speed value
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: yum
Version: 20
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Packaging Maintenance Team
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-12-14 17:00 UTC by Fabrice Bellet
Modified: 2014-06-04 07:54 UTC (History)
5 users (show)

Fixed In Version: python-urlgrabber-3.10.1-0.fc20
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-06-04 07:54:07 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Fabrice Bellet 2013-12-14 17:00:26 UTC
Hi!

When running yum in a situation where there's a lot of data churn on the mirrors, I often encounter cases where all mirrors don't have the same set of metadata stuff available to download, and it causes yum download errors. 

I tried to make yum specifically use the public mirror from the metalink that has the highest priority. I know this mirror is up to date, I maintain it :) So there's no reason for my boxes not to use it... I verified that the "failover" option is correctly set to "priority".

But while running a "yum update", I noticed that yum still tried to download metadata urls from other mirrors, that were not uptodate, which caused download errors. I searched again, and found that urlgrabber launches by default up to five connections in parallel. This value can be controlled by the max_connections option in yum.conf, so I decreased this value to "1".

And again, I noticed that yum still tried to use other mirrors. I found that urlgrabber also uses internally a "default_speed" value to discard the current mirror if the estimated download speed is below the "default_speed". This mirror shuffling occurs _even_ if the current download did succeed. This default_speed is relatively high (1e6 # 1Mbit, line 1002 in grabber.py).

According to my speed tests, I think this value expresses bytes rather than bits, so the default_speed is rather closer to 10Mbit instead. With a download speed of 380kB/s (kBytes) displayed by the yum progress bar, the estimated speed for my mirror around line 2291 of grabber.py is 510454, which is below the default_speed, and causes another mirror to be selected.

Also, when the box is on a slow network, below this default_speed, no mirror can achieve the download speed expected by urlgrabber, and that certainly may cause many "mirror rotations", useless in this case.

I suggest to fix the default_speed value in urlgrabber (125e4 for 1Mbit), and ideally, this value could be added as an option in the yum config file (for people on permanently slow networks) ?

Comment 1 Fabrice Bellet 2013-12-14 17:09:20 UTC
doh, 125e3 for 1Mbit

Comment 2 Zdeněk Pavlas 2013-12-16 11:24:30 UTC
Right, it's 1MBps, not 1Mbit. default_speed is used only as the first estimate for never-tried-before mirrors, but when there's a lot of them it might still cause problems.. 

OTOh when set too low, Yum uses only first few mirrors even if there are faster ones available.  And while the 1st condition fixes automatically once all mirrors have been tried (and found to be inferior), the 2nd one is permanent... that's why I set the default speed relatively high.  Anyway, it's on the top of the list of urlgrabber options that should be exposed in yum.conf.  Eg:

default_mirror_speed

This is the speed estimate for mirrors Yum has not used yet. Setting this slightly below the speed of your favorite mirror makes Yum much less promisuous. The default is 1MBps.

Comment 3 Fedora Update System 2014-05-30 09:28:39 UTC
python-urlgrabber-3.10.1-0.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/python-urlgrabber-3.10.1-0.fc20

Comment 4 Fedora Update System 2014-05-30 23:53:19 UTC
Package python-urlgrabber-3.10.1-0.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing python-urlgrabber-3.10.1-0.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-6918/python-urlgrabber-3.10.1-0.fc20
then log in and leave karma (feedback).

Comment 5 Fedora Update System 2014-06-04 07:54:07 UTC
python-urlgrabber-3.10.1-0.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.