Description of problem: Capsule sync fails with 'some distro file downloads failed' message in /va/l/foreman/production.log file Apr 12 01:02:20 katello1 pulp: pulp_rpm.plugins.importers.yum.parse.treeinfo:ERROR: some distro file downloads failed Apr 12 01:02:21 katello1 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (4010-66624) Apr 12 01:02:21 katello1 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (4010-66624) Traceback (most recent call last): Apr 12 01:02:21 katello1 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (4010-66624) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 228, in run Apr 12 01:02:21 katello1 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (4010-66624) dist_sync.run() Apr 12 01:02:21 katello1 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (4010-66624) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/parse/treeinfo.py", line 125, in run Apr 12 01:02:21 katello1 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (4010-66624) self._run(tmp_dir) Apr 12 01:02:21 katello1 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (4010-66624) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/parse/treeinfo.py", line 189, in _run Apr 12 01:02:21 katello1 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (4010-66624) for destination, location in downloaded: Apr 12 01:02:21 katello1 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (4010-66624) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/parse/treeinfo.py", line 274, in download_files Apr 12 01:02:21 katello1 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (4010-66624) raise DownloadFailed() Apr 12 01:02:21 katello1 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: (4010-66624) Version-Release number of selected component (if applicable): sat62-snap7.1 How reproducible: Steps to Reproduce: 1. Enable the following repos, rhel6-ks, rhel7-ks, rhel6, rhel7 and sync 2. Publish and promote to Dev lifecycle env 3. Add Dev env to capsule and sync content, Actual results: Capsule Sync fails Expected results: Capsule Sync should be successful. Additional info:
Apr 12 01:02:20 katello1 pulp: pulp_rpm.plugins.importers.yum.parse.treeinfo:ERROR: some distro file downloads failed ---------------------------------------------------------------------- Tried provisioning both RHEL7 and RHEL6 repos, RHEL7: provisioning was successful ( boot files could be fetched from isolated capsule by the HOST). RHEL6 : It was unable to fetch the boot files { vmlinuz and initrd ) May be the above "ERROR: some distro file downloads failed" is being referred to RHEL6 distro files.
Looking into this with Justin, we found that the kickstart repo structure is different between 6.7 and 7.2 rhel kickstart repos on the cdn. On the capsule sync, the 7.2 kickstart repo succeeded, while the 6.7 kickstart repo failed. The rhel 6.7 kickstart repo contains a PULP_DISTRIBUTION file that points to a repodata/productid file.[1] The 7.2 kickstart repo does not contain this same file [2] Pulp is presumably publishing the distribution into the publish directory including the ‘/repodata/productid’ file. It then is generating the yum metadata for the repository wiping out the ‘/repodata/productid’ file. This results in a missing distribution file that leads to the issue. I'll attach the pulp error from dynflow where you can see that its looking for productid [1]http://cdn.rcm-internal.redhat.com/content/dist/rhel/server/6/6.7/x86_64/kickstart/PULP_DISTRIBUTION.xml [2]http://cdn.rcm-internal.redhat.com/content/dist/rhel/server/7/7.2/x86_64/kickstart/
Created attachment 1146669 [details] pulp error from dynflow
This is caused by these two lines, which specifically exclude any distribution file that starts with "repodata/" https://github.com/pulp/pulp_rpm/blob/pulp-rpm-2.8.2-1/plugins/pulp_rpm/plugins/distributors/yum/publish.py#L796-L797 Barnaby, do you recall why that change was made?
Michael, Those lines were added to prevent a distribution from overwriting any of the generated metadata files. The productid file should be in the repomd.xml and be included separately from the distribution. However, if it would be more beneficial to blacklist the files the pulp_distribution does not get to include at a more fine-grained level then go for it.
Looking at this, my guess is Pulp won't be able to sync from other Pulp servers if the original repository contained a PULP_DISTRIBTION.xml file _and_ that PULP_DISTRIBUTION.xml file references anything in 'repodata/'. The reason is the files are stripped out at publish time (which seems inappropriate to me) as mhrivnak noticed. Pulp does not create PULP_DISTRIBUTION.xml: it just republishes whatever it found in the upstream repository[0]. Therefore, PULP_DISTRIBUTION.xml references files Pulp intentionally doesn't publish and the Pulp that tries to sync from that published repository fails just like it does here. Therefore, in my opinion, this is a bug with Pulp. It would be possibly (I think) to work around this issue on this particular repository by modifying the CDN's version of PULP_DISTRIBUTION.xml, but Pulp should not be getting itself into this state at all. [0] https://github.com/pulp/pulp_rpm/blob/master/plugins/pulp_rpm/plugins/importers/yum/parse/treeinfo.py#L437-L441
I've filed an issue on the upstream Pulp bug tracker: https://pulp.plan.io/issues/1843
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.
VERIFIED With sat62-snap9.0 Capsule sync now successful with ks-rhel{6, 7} and rhel{6, 7} repos.
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.
The Pulp upstream bug status is at VERIFIED. Updating the external tracker on this bug.
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1501