Bug 1568654

Summary: debmirror not syncing required components
Product: [Fedora] Fedora EPEL Reporter: Jeff Wilkins <Jeffrey.Wilkins>
Component: debmirrorAssignee: Sergio Basto <sergio>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: epel7CC: puiterwijk, ruben, sergio
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: debmirror-2.29-1.fc28 debmirror-2.29-1.fc26 debmirror-2.29-1.fc27 debmirror-2.29-1.el7 debmirror-2.29-1.el6 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-05-03 15:25:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jeff Wilkins 2018-04-18 01:32:34 UTC 
Description of problem:
We are running debmirror to create repos for Ubuntu Xenial & Trusty.
The sync process is failing to pull down required parts of each structure.
The Trusty tree is missing the Contents-<arch>.gz files, though only from the trusty (main) branch, they are coming down fine for trusty-security etc.

ie:
# tree -L 3
.
├── dists
│   ├── trusty
│   │   ├── main
│   │   ├── multiverse
│   │   ├── Release
│   │   ├── Release.gpg
│   │   ├── restricted
│   │   └── universe
│   ├── trusty-security
│   │   ├── Contents-amd64.gz
│   │   ├── Contents-i386.gz
│   │   ├── main
│   │   ├── multiverse
│   │   ├── Release
│   │   ├── Release.gpg
│   │   ├── restricted
│   │   └── universe
│   ├── trusty-updates
│   │   ├── Contents-amd64.gz
│   │   ├── Contents-i386.gz
│   │   ├── main
│   │   ├── multiverse
│   │   ├── Release
│   │   ├── Release.gpg
│   │   ├── restricted
│   │   └── universe

Xenial syncs are not pulling down the dep11 structure, thus the resultant repo is unusable.

There are numerous reports of this bug circa 2016 in the Ubuntu stream that were apparently resolved, however this does not seem to have filtered across to the current version available for both RHEL 6 & RHEL 7 in the EPEL repo.

We are seeing these same failures across both the RHEL 6 & RHEL 7 debmirror packages.

Version-Release number of selected component (if applicable):
debmirror-2.14-2.el6.noarch
debmirror-2.16-4.el7.noarch

Manual rsync has verified that the required structures are most certainly available at the target repos (alternate repos were attempted in any case, with the same results).

Equally, though of less import, the sync of the i18n directory (translations) also fails.  The directory is created, but no content is populated.

How reproducible:
100% - Install debmirror and perform a sync.  As explained above, we are seeing the same failure across the two different package streams upon two servers running them.

Steps to Reproduce:
1.Install debmirror
2.Configure to sync required dists
3.sync - note missing pieces

Actual results:
debmirror sync reports completed sync but required elements are missing

Expected results:
debmirror completely mirrors all required components

Additional info:
Pertinent /etc/debmirror.conf entries:
@dists="trusty,trusty-security,trusty-updates,xenial,xenial-security,xenial-updates";
@sections="main,multiverse,restricted,universe";
@arches="amd64,i386";
# @ignores="";
# @excludes="";
# @includes="";
# @excludes_deb_section="";
# @limit_priority="";
$omit_suite_symlinks=0;
$skippackages=0;
@rsync_extra="doc,tools,dep11";
$i18n=1;
$getcontents=1;
$do_source=1;
$max_batch=0;

@di_dists="trusty,xenial";
@di_archs="amd64,i386";
Comment 1 Fedora Update System 2018-04-20 02:00:25 UTC 
debmirror-2.26-1.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-43610fda37
Comment 2 Fedora Update System 2018-04-20 02:02:49 UTC 
debmirror-2.26-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-54884f9daf
Comment 3 Fedora Update System 2018-04-20 02:06:31 UTC 
debmirror-2.26-1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2018-e1059929c9
Comment 4 Fedora Update System 2018-04-20 02:07:46 UTC 
debmirror-2.26-1.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-351c131a75
Comment 5 Fedora Update System 2018-04-20 03:20:01 UTC 
debmirror-2.26-2.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-3d8f96d379
Comment 6 Fedora Update System 2018-04-20 03:20:23 UTC 
debmirror-2.26-2.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2018-3be58a1b5f
Comment 7 Fedora Update System 2018-04-20 03:20:50 UTC 
debmirror-2.26-2.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-20b9a3e8a3
Comment 8 Fedora Update System 2018-04-20 03:21:10 UTC 
debmirror-2.26-2.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-f773499066
Comment 9 Fedora Update System 2018-04-20 03:21:47 UTC 
debmirror-2.26-2.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-42b2c52b84
Comment 10 Fedora Update System 2018-04-20 04:01:54 UTC 
debmirror-2.26-1.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-43610fda37
Comment 11 Jeff Wilkins 2018-04-20 04:23:11 UTC 
debmirror-2.26-2.el6
debmirror-2.26-2.el7

I have tested the above two files, and they are now able, in both cases, to pull down the previously missing ../xenial/main/dep11/* and /i18n/* structures.

The dists/trusty/Contents-<arch>.gz files are still missing, an additional manual rsync of the two wayward files are filling this gap.

Thanks for the updates - its significantly more functional than previously.
Comment 12 Sergio Basto 2018-04-20 14:15:58 UTC 
Hi should I update to debmirror-2.29 ?
Comment 13 Fedora Update System 2018-04-20 14:45:33 UTC 
debmirror-2.26-2.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-3d8f96d379
Comment 14 Fedora Update System 2018-04-21 04:17:06 UTC 
debmirror-2.26-2.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-42b2c52b84
Comment 15 Fedora Update System 2018-04-21 04:37:57 UTC 
debmirror-2.26-2.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-3be58a1b5f
Comment 16 Fedora Update System 2018-04-21 05:02:43 UTC 
debmirror-2.26-2.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-20b9a3e8a3
Comment 17 Fedora Update System 2018-04-21 06:19:13 UTC 
debmirror-2.26-2.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-f773499066
Comment 18 Jeff Wilkins 2018-04-22 22:56:44 UTC 
(In reply to Sergio Monteiro Basto from comment #12)
> Hi should I update to debmirror-2.29 ?

While I do not see the one remaining issue as being resolved in the changelog for 2.29, I can see some other bugs squashed in 2.27/2.28 that would be better resolved.
If it its not too much trouble, then the further update would indeed be appreciated.

Thanks.
Comment 19 Sergio Basto 2018-04-23 01:59:42 UTC 
OK thanks for the report ,
Comment 20 Fedora Update System 2018-04-23 03:21:59 UTC 
debmirror-2.29-1.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-b71da2def1
Comment 21 Fedora Update System 2018-04-23 03:22:20 UTC 
debmirror-2.29-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-23dca3e8e7
Comment 22 Fedora Update System 2018-04-23 03:22:43 UTC 
debmirror-2.29-1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2018-d2dec2af47
Comment 23 Fedora Update System 2018-04-23 03:23:25 UTC 
debmirror-2.29-1.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-1c99004668
Comment 24 Fedora Update System 2018-04-23 03:23:49 UTC 
debmirror-2.29-1.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-d139147f5d
Comment 25 Fedora Update System 2018-04-23 22:52:41 UTC 
debmirror-2.29-1.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-b71da2def1
Comment 26 Fedora Update System 2018-04-24 04:46:14 UTC 
debmirror-2.29-1.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-1c99004668
Comment 27 Fedora Update System 2018-04-24 05:03:31 UTC 
debmirror-2.29-1.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-d2dec2af47
Comment 28 Fedora Update System 2018-04-24 05:37:52 UTC 
debmirror-2.29-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-23dca3e8e7
Comment 29 Fedora Update System 2018-04-24 07:52:03 UTC 
debmirror-2.29-1.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-d139147f5d
Comment 30 Fedora Update System 2018-05-03 15:25:32 UTC 
debmirror-2.29-1.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.
Comment 31 Fedora Update System 2018-05-05 21:23:58 UTC 
debmirror-2.29-1.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.
Comment 32 Fedora Update System 2018-05-05 22:26:09 UTC 
debmirror-2.29-1.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.
Comment 33 Fedora Update System 2018-05-11 01:32:38 UTC 
debmirror-2.29-1.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.
Comment 34 Fedora Update System 2018-05-20 19:32:41 UTC 
debmirror-2.29-1.el6 has been pushed to the Fedora EPEL 6 stable repository. If problems still persist, please make note of it in this bug report.