Bug 909744 - [rfe] support parallel downloads
Summary: [rfe] support parallel downloads
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: dnf   
(Show other bugs)
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Zdeněk Pavlas
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Keywords:
Depends On:
Blocks: 871892
TreeView+ depends on / blocked
 
Reported: 2013-02-10 20:32 UTC by Rahul Sundaram
Modified: 2014-02-02 22:29 UTC (History)
5 users (show)

Fixed In Version: dnf-0.4.2-1.gitc1716d7.fc20
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-10-09 14:20:38 UTC
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

Description Rahul Sundaram 2013-02-10 20:32:23 UTC
Description of problem:

yum (after the dnf fork) has added support for parallel downloads and other enhancements, bug fixes.  merge those changes.

Comment 1 Rahul Sundaram 2013-02-28 16:04:50 UTC
While looking into this, you might want to incorporate some improvements from

http://repos.fedorapeople.org/repos/hedayat/yum-fast-downloader/

It downloads aggressively from multiple mirrors by using aria2 as a backend

Comment 2 Zdeněk Pavlas 2013-02-28 16:18:02 UTC
There was a considerable backlash against using too many connections (downloading 2 files from the same mirror in parallel, or downloading 1 file from 2+ mirrors with range request).  People that run the mirrors were seriously concerned (and the parallel downloading code in Yum is not aggressive by any means)

http://lists.fedoraproject.org/pipermail/devel/2012-May/167122.html

Comment 3 Rahul Sundaram 2013-02-28 17:58:19 UTC
From the thread, there isn't really much of a backlash.  There are some mirror admins who were concerned that yum is using more than one connection to the same mirror but yum fast downloader distributes this load.  Either way, yes, dnf developer would need  to talk to the mirror admins and figure out what is the optimal way to handle this.

Comment 4 Ales Kozumplik 2013-04-29 07:33:32 UTC
The DNF version in F19 and beyond uses librepo for the repodata and package download backend. By default librepo downloads different metadata files per repo in parallel. It currently however does not download repositories in parallel, nor packages in parallel. 

Careful extension of librepo API will have to happen first, leaving this open until then.

Comment 5 Ales Kozumplik 2013-06-27 14:12:07 UTC
(In reply to Ales Kozumplik from comment #4)
> Careful extension of librepo API will have to happen first, leaving this
> open until then.

Tomas, can librepo already do parallel download of packages?

Comment 6 Ales Kozumplik 2013-08-13 14:58:34 UTC
Moving over to Zdenek who implemented parallel downloads in Yum and is in touch with tmlcoch about how to do this with librepo.

Comment 7 Tomas Mlcoch 2013-09-17 12:48:59 UTC
Librepo since 1.0.0 supports parallel downloads out-of-box.

Comment 8 Fedora Update System 2013-10-01 13:51:56 UTC
dnf-0.4.2-1.gitc1716d7.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/dnf-0.4.2-1.gitc1716d7.fc20

Comment 9 Fedora Update System 2013-10-02 06:49:20 UTC
Package dnf-0.4.2-1.gitc1716d7.fc20, librepo-1.2.0-1.fc20, hawkey-0.4.2-1.git4c51f65.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing dnf-0.4.2-1.gitc1716d7.fc20 librepo-1.2.0-1.fc20 hawkey-0.4.2-1.git4c51f65.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-18123/librepo-1.2.0-1.fc20,hawkey-0.4.2-1.git4c51f65.fc20,dnf-0.4.2-1.gitc1716d7.fc20
then log in and leave karma (feedback).

Comment 10 Fedora Update System 2013-10-09 14:20:38 UTC
dnf-0.4.2-1.gitc1716d7.fc20, librepo-1.2.0-1.fc20, hawkey-0.4.2-1.git4c51f65.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.