Bug 1750273

Summary: dnf reposync ignoring includepkgs directive (regression)
Product: [Fedora] Fedora Reporter: Paul Howarth <paul>
Component: dnf-plugins-coreAssignee: Marek Blaha <mblaha>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 30CC: jmracek, mblaha, packaging-team-maint, pkratoch, praiskup, rpm-software-management, vmukhame
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-01 08:34:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Paul Howarth 2019-09-09 08:13:35 UTC
Description of problem:
I have a reposync job that is intended to pull just a few packages from an upstream repo to create a local subset mirror. The packages to be downloaded are  specified using "includepkgs=list-of-packages" in the repo configuration. This has been working for years. Recently, dnf has started ignoring this directive and actually appears to try to sync all packages from the remote repo until it falls over having filled up the disk.

Version-Release number of selected component (if applicable):
dnf-plugins-core-4.0.9-1.fc30

How reproducible:
Every time.

Steps to Reproduce:

Repo configuration:

[branched-buildmirror-x86_64]
name=Fedora Branched - x86_64 - Mirror of Build Requirement Packages
baseurl=http://dl.fedoraproject.org/pub/fedora/linux/development/31/Everything/x86_64/os
        http://mirror.bytemark.co.uk/fedora/linux/development/31/Everything/x86_64/os
        http://sunsite.mff.cuni.cz/MIRRORS/fedora.redhat.com/fedora/development/31/Everything/x86_64/os
includepkgs=aajohan-comfortaa-fonts,abattis-cantarell-fonts,accountsservice,accountsservice-libs,acl
enabled=0
gpgcheck=0
metadata_expire=600

Verify:
$ dnf repolist -v branched-buildmirror-x86_64
...
Completion plugin: Generating completion cache...
Completion plugin: Can't write completion cache: attempt to write a readonly database
Excludes from versionlock plugin were not applied

Repo-id      : branched-buildmirror-x86_64
Repo-name    : Fedora Branched - x86_64 - Mirror of Build Requirement Packages
Repo-status  : disabled
Repo-baseurl : http://dl.fedoraproject.org/pub/fedora/linux/development/31/Everything/x86_64/os,
             : http://mirror.bytemark.co.uk/fedora/linux/development/31/Everything/x86_64/os,
             : http://sunsite.mff.cuni.cz/MIRRORS/fedora.redhat.com/fedora/development/31/Everything/x86_64/os
Repo-expire  : 600 second(s) (last: unknown)
Repo-include : aajohan-comfortaa-fonts, abattis-cantarell-fonts, accountsservice, accountsservice-libs, acl
Repo-filename: /etc/yum.repos.d/fedora-branched-mirror.repo
Total packages: 0

(total packages is 0 because I deleted all of the originally downloaded rpms to free up disk space).

Actual results:
 dnf reposync --repoid branched-buildmirror-x86_64 --remote-time -p /srv/nb/distros/fc31 
Fedora Branched - x86_64 - Mirror of Build Requirement Packages                                                                     17 kB/s | 6.1 kB     00:00    
(1/56077): 0xFFFF-0.8-2.fc31.x86_64.rpm                                                                                             69 kB/s |  72 kB     00:01    
(2/56077): 2048-cli-0.9.1-8.fc31.x86_64.rpm                                                                                        101 kB/s |  21 kB     00:00    
(3/56077): 2048-cli-nocurses-0.9.1-8.fc31.x86_64.rpm                                                                                84 kB/s |  20 kB     00:00    
(4/56077): 2048-cli-sdl-0.9.1-8.fc31.x86_64.rpm                                                                                    125 kB/s |  22 kB     00:00    
(5/56077): 2ping-4.3-3.fc31.noarch.rpm                                                                                             166 kB/s |  71 kB     00:00    
(6/56077): 0ad-0.0.23b-8.fc31.x86_64.rpm                                                                                           1.9 MB/s | 3.9 MB     00:02    
(7/56077): 389-ds-base-1.4.1.6-1.fc31.1.x86_64.rpm                                                                                 2.4 MB/s | 1.7 MB     00:00    
(8/56077): 389-ds-base-devel-1.4.1.6-1.fc31.1.x86_64.rpm                                                                           236 kB/s | 192 kB     00:00    
(9/56077): 389-ds-base-legacy-tools-1.4.1.6-1.fc31.1.x86_64.rpm                                                                    1.5 MB/s | 309 kB     00:00    
(10/56077): 389-ds-base-snmp-1.4.1.6-1.fc31.1.x86_64.rpm                                                                           583 kB/s |  93 kB     00:00    
(11/56077): 389-ds-base-libs-1.4.1.6-1.fc31.1.x86_64.rpm                                                                           343 kB/s | 875 kB     00:02    
(12/56077): 3Depict-0.0.22-3.fc31.x86_64.rpm                                                                                       2.4 MB/s | 5.7 MB     00:02    
(13/56077): 3dprinter-udev-rules-0.2.2-2.fc31.noarch.rpm                                                                            81 kB/s |  12 kB     00:00    
(14/56077): 3proxy-0.8.12-3.fc31.x86_64.rpm                                                                                        987 kB/s | 170 kB     00:00    
(15/56077): 4diac-forte-1.9.0-5.fc31.x86_64.rpm                                                                                    1.4 MB/s | 405 kB     00:00    
(16/56077): 4th-3.62.5-4.fc31.i686.rpm                                                                                             1.7 MB/s | 341 kB     00:00    
(17/56077): 4th-3.62.5-4.fc31.x86_64.rpm                                                                                           719 kB/s | 343 kB     00:00    
(18/56077): 4th-devel-3.62.5-4.fc31.i686.rpm                                                                                        78 kB/s |  17 kB     00:00    
(19/56077): 4th-devel-3.62.5-4.fc31.x86_64.rpm                                                                                      78 kB/s |  17 kB     00:00    
(20/56077): 4ti2-1.6.9-3.fc31.x86_64.rpm                                                                                           1.3 MB/s | 424 kB     00:00    
(21/56077): 4ti2-devel-1.6.9-3.fc31.i686.rpm                                                                                       492 kB/s |  78 kB     00:00    
(22/56077): 4ti2-devel-1.6.9-3.fc31.x86_64.rpm                                                                                     466 kB/s |  78 kB     00:00    
(23/56077): 4ti2-libs-1.6.9-3.fc31.i686.rpm                                                                                        1.9 MB/s | 649 kB     00:00    
(24/56077): 4ti2-libs-1.6.9-3.fc31.x86_64.rpm                                                                                      1.5 MB/s | 588 kB     00:00    
(25/56077): 4Pane-5.0-8.fc31.x86_64.rpm                                                                                            544 kB/s | 1.6 MB     00:02    
(26/56077): 5minute-0.2.32-2.fc31.noarch.rpm                                                                                       103 kB/s |  47 kB     00:00    
(27/56077): 64tass-1.54.1900-2.fc31.x86_64.rpm                                                                                     709 kB/s | 316 kB     00:00    
(28/56077): 7kaa-2.14.7-6.fc31.x86_64.rpm                                                                                          1.7 MB/s | 827 kB     00:00    
(29/56077): 7kaa-music-2.14.7-6.fc31.noarch.rpm                                                                                     30 kB/s | 9.5 kB     00:00    
(30/56077): 8sync-0.4.2-8.fc31.x86_64.rpm                                                                                          1.4 MB/s | 317 kB     00:00    
(31/56077): 90-Second-Portraits-1.01b-7.fc31.noarch.rpm                                                                            2.6 MB/s | 5.0 MB     00:01    
(32/56077): 99soft-oss-parent-10-12.fc30.noarch.rpm                                                                                118 kB/s |  18 kB     00:00    
(33/56077): 9pfs-1.0.9-17.fc31.noarch.rpm                                                                                           83 kB/s |  13 kB     00:00    
(34/56077): 9wm-1.4.1-4.fc31.x86_64.rpm                                                                                            226 kB/s |  36 kB     00:00    
(35/56077): AGReader-1.2-23.fc31.x86_64.rpm                                                                                        322 kB/s |  50 kB     00:00    
(36/56077): AcetoneISO-6.7-28.fc31.x86_64.rpm                                                                                      1.4 MB/s | 287 kB     00:00    
(37/56077): Add64-1.2.2-18.fc31.x86_64.rpm                                                                                         687 kB/s | 119 kB     00:00    
(38/56077): Agda-2.6.0.1-19.fc31.x86_64.rpm                                                                                        448 kB/s |  73 kB     00:00    
(39/56077): Agda-common-2.6.0.1-19.fc31.noarch.rpm                                                                                 1.4 MB/s | 514 kB     00:00    
(40/56077): 7kaa-data-2.14.7-6.fc31.noarch.rpm                                                                                     2.6 MB/s |  31 MB     00:11    
^C(41/56077): Agda-stdlib-docs-1.1-1.fc31.noarch.rpm                                                                                 1.7 MB/s | 1.9 MB     00:01    

Interrupted at this point before it could fill my disk up again.

Expected results:
Download only the configured set of files.

Additional info:

As this has only just started happening, I suspect it may be an unwanted side effect of the fix for Bug #1714788 (Reposync should sync the entire repository to include module information. reposync should download the packages regardless of whether a module is enabled or disabled).

Comment 1 Paul Howarth 2019-09-09 08:34:08 UTC
Downgrading to dnf-plugins-core-4.0.6-1.fc30 enables me to work around this problem for now.

Comment 2 Marek Blaha 2019-11-08 10:08:39 UTC
PR https://github.com/rpm-software-management/dnf-plugins-core/pull/368 should fix the issue.

Comment 3 Paul Howarth 2019-11-08 14:10:47 UTC
That helps a lot. Nothing to sync at the moment but not trying to pull in the whole repo so that's a good start. Will see over the next few days if there are any issues with regular updates.

Comment 4 Fedora Update System 2019-12-02 14:39:09 UTC
FEDORA-2019-0d122cc67a has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-0d122cc67a

Comment 5 Fedora Update System 2019-12-03 00:53:50 UTC
dnf-4.2.17-1.fc31, dnf-plugins-core-4.0.12-1.fc31, libcomps-0.1.12-1.fc31, libdnf-0.39.1-1.fc31, microdnf-3.3.0-1.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-1635fee8f5

Comment 6 Fedora Update System 2019-12-03 01:33:42 UTC
dnf-4.2.17-1.fc30, dnf-plugins-core-4.0.12-1.fc30, libcomps-0.1.12-1.fc30, libdnf-0.39.1-1.fc30, microdnf-3.3.0-1.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-0d122cc67a

Comment 7 Fedora Update System 2019-12-04 07:52:46 UTC
FEDORA-2019-1635fee8f5 has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2019-1635fee8f5

Comment 8 Fedora Update System 2019-12-04 08:04:13 UTC
FEDORA-2019-0d122cc67a has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-0d122cc67a

Comment 9 Fedora Update System 2019-12-05 01:23:25 UTC
dnf-4.2.17-1.fc31, dnf-plugins-core-4.0.12-1.fc31, libcomps-0.1.14-1.fc31, libdnf-0.39.1-1.fc31, microdnf-3.3.0-1.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-1635fee8f5

Comment 10 Fedora Update System 2019-12-05 02:00:34 UTC
dnf-4.2.17-1.fc30, dnf-plugins-core-4.0.12-1.fc30, libcomps-0.1.14-1.fc30, libdnf-0.39.1-1.fc30, microdnf-3.3.0-1.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-0d122cc67a

Comment 11 Fedora Update System 2019-12-06 05:43:59 UTC
dnf-4.2.17-1.fc31, dnf-plugins-core-4.0.12-1.fc31, libcomps-0.1.14-1.fc31, libdnf-0.39.1-1.fc31, microdnf-3.3.0-1.fc31 has been pushed to the Fedora 31 stable repository. If problems still persist, please make note of it in this bug report.

Comment 12 Fedora Update System 2020-01-07 01:51:21 UTC
dnf-4.2.17-1.fc30, dnf-plugins-core-4.0.12-1.fc30, libcomps-0.1.14-1.fc30, libdnf-0.39.1-1.fc30, microdnf-3.3.0-1.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.

Comment 13 Ben Cotton 2020-04-30 20:25:56 UTC
This message is a reminder that Fedora 30 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 30 on 2020-05-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '30'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 30 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.