Bug 1318339 - [lazy sync] Some packages not downloaded when needed
Summary: [lazy sync] Some packages not downloaded when needed
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.2.0
Hardware: Unspecified
OS: Unspecified
urgent vote
Target Milestone: Unspecified
Assignee: Mike McCune
QA Contact: jcallaha
Depends On:
TreeView+ depends on / blocked
Reported: 2016-03-16 14:50 UTC by jcallaha
Modified: 2019-09-25 21:17 UTC (History)
15 users (show)

Fixed In Version: pulp-
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2016-10-26 12:25:09 UTC
Target Upstream Version:

Attachments (Terms of Use)
katello-agent install failure (300.90 KB, image/png)
2016-03-16 14:50 UTC, jcallaha
no flags Details
section of repo after request (314.06 KB, image/png)
2016-03-16 14:51 UTC, jcallaha
no flags Details
foreman-debug (624.03 KB, application/x-xz)
2016-03-22 14:30 UTC, jcallaha
no flags Details
500 Error (288.76 KB, image/png)
2016-04-04 15:02 UTC, jcallaha
no flags Details
foreman-debug 2 (370.62 KB, application/x-xz)
2016-04-04 15:03 UTC, jcallaha
no flags Details

System ID Priority Status Summary Last Updated
Pulp Redmine 1771 High CLOSED - CURRENTRELEASE requests or urllib3 can't read a file which causes Nectar to fail mysteriously 2016-06-08 16:00:51 UTC
Red Hat Product Errata RHBA-2016:2109 normal SHIPPED_LIVE Satellite 6.2.3 Tools Async Release 2016-10-26 16:21:13 UTC

Description jcallaha 2016-03-16 14:50:20 UTC
Created attachment 1137068 [details]
katello-agent install failure

Description of problem:
When a repository is set to on_demand and packages are requested from a host, some packages might not be provided.

Version-Release number of selected component (if applicable):
Satellite 6.2 Beta Snap 3

How reproducible:

Steps to Reproduce:
1. Create/Enable a repository with a download policy of on_demand
2. Sync the repo
3. Consume content in that repo from a host (install katello-agent)

Actual results:
Some required packages are not available when needed. An error may be seen like "Package does not match intended download."

Expected results:
All packages are downloaded/streamed to the host when requested.

Additional info:
See attached images

Comment 1 jcallaha 2016-03-16 14:51:08 UTC
Created attachment 1137069 [details]
section of repo after request

Comment 4 David Davis 2016-03-18 15:41:41 UTC
Can you please attach a foreman-debug output?

Comment 5 Michael Hrivnak 2016-03-18 15:50:40 UTC
We'll need to see pulp logs to make any sense of why this happened. If you can add them, please needsinfo me.

It's worth noting that one of the risks of on-demand content retrieval is that content might not be available when you want it. If pulp can't get it from the remote source at that moment, there's nothing else that can be done. I'm not sure if that's what happened here, but it's worth being aware of.

Comment 6 jcallaha 2016-03-18 19:06:23 UTC
Which logs would you like specifically? 

Pulp was able to provide a good number of packages requested, but the others didn't make it.

Comment 7 David Davis 2016-03-20 17:27:47 UTC
jcalla, if you provide the output from foreman-debug, it should have everything we need. Here's how to run it:


Comment 8 David Davis 2016-03-20 17:28:25 UTC
Alternatively, /var/log/messages from the Satellite server should work.

Comment 9 jcallaha 2016-03-22 14:30:20 UTC
I've reproduced the results, but with a slightly different error. The packages in question did not download, but others did. Interestingly, this occurred only in this specific repo. I had previously installed over 200 packages from the rhel-7-server and rh-common repos. I will attach the debug output.

Comment 10 jcallaha 2016-03-22 14:30:49 UTC
Created attachment 1139082 [details]

Comment 11 Michael Hrivnak 2016-03-22 22:25:20 UTC
Jeremy, does this look like one of the streamer bugs we're already working on?

Comment 12 Jeremy Cline 2016-03-23 14:45:18 UTC
This could well be https://pulp.plan.io/issues/1788

Did the specific repository in question have a different entitlement certificate than the one you downloaded from before? If you restart the pulp_streamer service and try to download from the repository that was failing, does it work?

There's another concerning issue in the logs:

ibm-x3550m3-09 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: ibm-x3550m3-09.lab.eng.brq.redhat.com

shows up a lot. I think this is occurring when Pulp tries to download the cached content via the 'download_deferred' task (because it's resetting the dropped connection to the same host it's on).  This makes me think there's potentially an Apache configuration problem.

Comment 13 Mike McCune 2016-03-31 23:28:31 UTC
This is https://pulp.plan.io/issues/1771 which is being tracked in:


Comment 15 jcallaha 2016-04-04 15:02:31 UTC
Failed QA on Satellite 6.2 Beta Snap 6 Compose 1.
I'm now seeing the following error for all packages attempting to be downloaded on_demand.

katello-agent-1.5.3-7.el7sat.n FAILED                                          
https://rhsm-qe-1.rhq.lab.eng.bos.redhat.com/pulp/repos/Default_Organization/Library/rhel7/content/dist/rhel/server/7/7Server/x86_64/rh-common/os/katello-agent-1.5.3-7.el7sat.noarch.rpm: [Errno 14] HTTPS Error 500 - Internal Server Error
Trying other mirror.

See attached image and foreman-debug output.

Comment 16 jcallaha 2016-04-04 15:02:58 UTC
Created attachment 1143373 [details]
500 Error

Comment 17 jcallaha 2016-04-04 15:03:36 UTC
Created attachment 1143375 [details]
foreman-debug 2

Comment 18 Jeremy Cline 2016-04-04 15:07:56 UTC
The failure is due to SELinux denials to `/var/lib/pulp/importers/.../pki/ca.crt` which is a known issue and being worked on: see https://pulp.plan.io/issues/1771#note-28

Comment 20 Brian Bouterse 2016-04-05 13:14:28 UTC
I have to remove this link to upstream Pulp issue 1771 because 1771 is already associated with Bugzilla 1317047. A current limitation is that each Pulp bug can only correspond with 1 bugzilla Bug and vice-versa. That limitation will be overcome soon, but it's not there yet.

Comment 24 pulp-infra@redhat.com 2016-06-08 16:00:52 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 25 pulp-infra@redhat.com 2016-06-08 16:00:56 UTC
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.

Comment 26 Brad Buckingham 2016-07-25 13:20:15 UTC
Moving to POST as upstream fix is available.

Comment 29 jcallaha 2016-10-25 18:59:49 UTC
Verified in Satellite 6.2.3 Snap 4.

Successfully streamed over 2,500 packages to multiple hosts at once. No errors seen on either Satellite or Host ends.

Comment 31 errata-xmlrpc 2016-10-26 12:25:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.