Bug 1332157

Summary: reposync downloads all the packages before filtering by 'includes'
Product: [Fedora] Fedora Reporter: David Caro <dcaroest>
Component: dnf-plugins-coreAssignee: Michael Mráka <mmraka>
Status: CLOSED NEXTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: medium    
Version: 23CC: eedri, ignatenko, jsilhan, mluscon, packaging-team-maint, pnemade, RadekHolyPublic, vmukhame
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-07-21 14:08:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description David Caro 2016-05-02 11:30:38 UTC
Description of problem:
When using dnf reposync, if you have an 'includes' tag on the repo config to get only some of the packages, reposync will download all the packages and then filter them out using a lot of bandwidth and using a lot of space for big repos (like epel) when all that can be done once you have the metadata, and download only the needed rpms

Version-Release number of selected component (if applicable):
0.1.20-1.fc23

How reproducible:
You can use a yum config file like:

[main]
reposdir=/etc/reposync.repos.d
retries=3
debuglevel=10
rpmverbosity=debug
[epel-el7]
name=EPEL repo
baseurl=https://dl.fedoraproject.org/pub/epel/7/x86_64/
enabled=1
gpgcheck=0
includepkgs=python-ioproces

Then run:
  dnf reposync  --repoid epel-el7 -c test.repo --download-path=/tmp/reposync

Steps to Reproduce:
1.
2.
3.

Actual results:
You get all the packages downloaded, and then removed all but the ones in the 'include' tag.

Expected results:
Only the included rpms get downloaded

Additional info:

Comment 1 Igor Gnatenko 2016-05-09 11:36:57 UTC
> includepkgs=python-ioproces

I think it should be include=python-ioproces..

Comment 2 Honza Silhan 2016-05-09 11:38:25 UTC
Can you confirm whether tip from comment 1 helped?

Comment 3 David Caro 2016-05-09 11:47:21 UTC
Nop, same behavior, it first downloads everything, and then it removes all but the included one:

$ dnf reposync  --repoid epel-el7 -c test.repo --download-path=/tmp/reposync
timer: config: 5 ms
cachedir: /var/tmp/dnf-dcaro-AkaMbN
Loaded plugins: copr, reposync, needs-restarting, generate_completion_cache, noroot, Query, debuginfo-install, system-upgrade, protected_packages, config-manager, download, playground, builddep
DNF version: 1.1.8
Command: dnf reposync --repoid epel-el7 -c test.repo --download-path=/tmp/reposync 
Installroot: /
Releasever: 23
Base command: reposync
Extra commands: ['--repoid', 'epel-el7', '--download-path=/tmp/reposync']
repo: downloading from remote: epel-el7, _Handle: metalnk: None, mlist: None, urls ['https://dl.fedoraproject.org/pub/epel/7/x86_64/'].
EPEL repo                                      457 kB/s |  11 MB     00:24    
not found deltainfo for: EPEL repo
timer: sack setup: 27926 ms
epel-el7: using metadata from Sun May  8 20:47:47 2016.
Last metadata expiration check: 0:00:05 ago on Mon May  9 13:44:41 2016.
Completion plugin: Generating completion cache...
Completion plugin: Can't write completion cache: attempt to write a readonly database
(1/10034): 2048-cli-nocurses-0.9.1-1.el7.x86_6  14 kB/s |  16 kB     00:01    
(2/10034): 2048-cli-0.9.1-1.el7.x86_64.rpm      14 kB/s |  17 kB     00:01    
(3/10034): 2ping-3.2.1-2.el7.noarch.rpm        132 kB/s |  62 kB     00:00    
(4/10034): 389-admin-console-1.1.10-1.el7.noar 206 kB/s | 203 kB     00:00    
(5/10034): 389-admin-console-doc-1.1.10-1.el7. 148 kB/s |  45 kB     00:00    
(6/10034): 389-adminutil-1.1.21-2.el7.x86_64.r 202 kB/s |  73 kB     00:00    
(7/10034): 389-admin-1.1.38-1.el7.x86_64.rpm   169 kB/s | 387 kB     00:02    
(8/10034): 389-adminutil-devel-1.1.21-2.el7.x8  91 kB/s |  27 kB     00:00    
(9/10034): 389-console-1.1.9-1.el7.noarch.rpm  108 kB/s |  75 kB     00:00    
(10/10034): 389-ds-console-doc-1.2.12-1.el7.no 124 kB/s |  56 kB     00:00    
(11/10034): 3proxy-0.7-1.el7.x86_64.rpm        134 kB/s | 170 kB     00:01    
(12/10034): 3proxy-sysvinit-0.7-1.el7.x86_64.r  24 kB/s | 8.3 kB     00:00    
(13/10034): Agda-2.3.2.2-3.el7.x86_64.rpm       23 kB/s | 6.6 kB     00:00    
(14/10034): AntTweakBar-1.16-2.el7.x86_64.rpm  188 kB/s | 189 kB     00:01    
(15/10034): AntTweakBar-devel-1.16-2.el7.x86_6  96 kB/s |  26 kB     00:00    
(16/10034): aalib-1.4.0-0.22.rc5.el7.x86_64.rp  72 kB/s |  20 kB     00:00    
(17/10034): aalib-devel-1.4.0-0.22.
...

Comment 4 Honza Silhan 2016-05-16 11:18:39 UTC
IMO it should work in DNF 2.0 stack where `--repo` is DNF option.

Comment 5 Michael Mráka 2016-06-22 14:32:05 UTC
Fixed as a part of
https://github.com/rpm-software-management/dnf/pull/518

Comment 7 Igor Gnatenko 2016-07-21 14:08:47 UTC
PR has been merged for DNF 2.0 and will be available at some point.