Bug 1332157 - reposync downloads all the packages before filtering by 'includes'
Summary: reposync downloads all the packages before filtering by 'includes'
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: dnf-plugins-core
Version: 23
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
Assignee: Michael Mráka
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-05-02 11:30 UTC by David Caro
Modified: 2016-07-21 14:08 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-07-21 14:08:47 UTC
Type: Bug


Attachments (Terms of Use)

Description David Caro 2016-05-02 11:30:38 UTC
Description of problem:
When using dnf reposync, if you have an 'includes' tag on the repo config to get only some of the packages, reposync will download all the packages and then filter them out using a lot of bandwidth and using a lot of space for big repos (like epel) when all that can be done once you have the metadata, and download only the needed rpms

Version-Release number of selected component (if applicable):
0.1.20-1.fc23

How reproducible:
You can use a yum config file like:

[main]
reposdir=/etc/reposync.repos.d
retries=3
debuglevel=10
rpmverbosity=debug
[epel-el7]
name=EPEL repo
baseurl=https://dl.fedoraproject.org/pub/epel/7/x86_64/
enabled=1
gpgcheck=0
includepkgs=python-ioproces

Then run:
  dnf reposync  --repoid epel-el7 -c test.repo --download-path=/tmp/reposync

Steps to Reproduce:
1.
2.
3.

Actual results:
You get all the packages downloaded, and then removed all but the ones in the 'include' tag.

Expected results:
Only the included rpms get downloaded

Additional info:

Comment 1 Igor Gnatenko 2016-05-09 11:36:57 UTC
> includepkgs=python-ioproces

I think it should be include=python-ioproces..

Comment 2 Honza Silhan 2016-05-09 11:38:25 UTC
Can you confirm whether tip from comment 1 helped?

Comment 3 David Caro 2016-05-09 11:47:21 UTC
Nop, same behavior, it first downloads everything, and then it removes all but the included one:

$ dnf reposync  --repoid epel-el7 -c test.repo --download-path=/tmp/reposync
timer: config: 5 ms
cachedir: /var/tmp/dnf-dcaro-AkaMbN
Loaded plugins: copr, reposync, needs-restarting, generate_completion_cache, noroot, Query, debuginfo-install, system-upgrade, protected_packages, config-manager, download, playground, builddep
DNF version: 1.1.8
Command: dnf reposync --repoid epel-el7 -c test.repo --download-path=/tmp/reposync 
Installroot: /
Releasever: 23
Base command: reposync
Extra commands: ['--repoid', 'epel-el7', '--download-path=/tmp/reposync']
repo: downloading from remote: epel-el7, _Handle: metalnk: None, mlist: None, urls ['https://dl.fedoraproject.org/pub/epel/7/x86_64/'].
EPEL repo                                      457 kB/s |  11 MB     00:24    
not found deltainfo for: EPEL repo
timer: sack setup: 27926 ms
epel-el7: using metadata from Sun May  8 20:47:47 2016.
Last metadata expiration check: 0:00:05 ago on Mon May  9 13:44:41 2016.
Completion plugin: Generating completion cache...
Completion plugin: Can't write completion cache: attempt to write a readonly database
(1/10034): 2048-cli-nocurses-0.9.1-1.el7.x86_6  14 kB/s |  16 kB     00:01    
(2/10034): 2048-cli-0.9.1-1.el7.x86_64.rpm      14 kB/s |  17 kB     00:01    
(3/10034): 2ping-3.2.1-2.el7.noarch.rpm        132 kB/s |  62 kB     00:00    
(4/10034): 389-admin-console-1.1.10-1.el7.noar 206 kB/s | 203 kB     00:00    
(5/10034): 389-admin-console-doc-1.1.10-1.el7. 148 kB/s |  45 kB     00:00    
(6/10034): 389-adminutil-1.1.21-2.el7.x86_64.r 202 kB/s |  73 kB     00:00    
(7/10034): 389-admin-1.1.38-1.el7.x86_64.rpm   169 kB/s | 387 kB     00:02    
(8/10034): 389-adminutil-devel-1.1.21-2.el7.x8  91 kB/s |  27 kB     00:00    
(9/10034): 389-console-1.1.9-1.el7.noarch.rpm  108 kB/s |  75 kB     00:00    
(10/10034): 389-ds-console-doc-1.2.12-1.el7.no 124 kB/s |  56 kB     00:00    
(11/10034): 3proxy-0.7-1.el7.x86_64.rpm        134 kB/s | 170 kB     00:01    
(12/10034): 3proxy-sysvinit-0.7-1.el7.x86_64.r  24 kB/s | 8.3 kB     00:00    
(13/10034): Agda-2.3.2.2-3.el7.x86_64.rpm       23 kB/s | 6.6 kB     00:00    
(14/10034): AntTweakBar-1.16-2.el7.x86_64.rpm  188 kB/s | 189 kB     00:01    
(15/10034): AntTweakBar-devel-1.16-2.el7.x86_6  96 kB/s |  26 kB     00:00    
(16/10034): aalib-1.4.0-0.22.rc5.el7.x86_64.rp  72 kB/s |  20 kB     00:00    
(17/10034): aalib-devel-1.4.0-0.22.
...

Comment 4 Honza Silhan 2016-05-16 11:18:39 UTC
IMO it should work in DNF 2.0 stack where `--repo` is DNF option.

Comment 5 Michael Mráka 2016-06-22 14:32:05 UTC
Fixed as a part of
https://github.com/rpm-software-management/dnf/pull/518

Comment 7 Igor Gnatenko 2016-07-21 14:08:47 UTC
PR has been merged for DNF 2.0 and will be available at some point.


Note You need to log in before you can comment on or make changes to this bug.