Bug 1318339
| Summary: | [lazy sync] Some packages not downloaded when needed | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | jcallaha | ||||||||||||
| Component: | Pulp | Assignee: | Mike McCune <mmccune> | ||||||||||||
| Status: | CLOSED ERRATA | QA Contact: | jcallaha | ||||||||||||
| Severity: | urgent | Docs Contact: | |||||||||||||
| Priority: | unspecified | ||||||||||||||
| Version: | 6.2.0 | CC: | bbuckingham, bkearney, bmbouter, cwelton, daviddavis, dkliban, ehelms, ggainey, ipanova, jcallaha, jcline, jeremy, mhrivnak, mmccune, pcreech, rchan, ttereshc, zhunting | ||||||||||||
| Target Milestone: | Unspecified | Keywords: | Triaged | ||||||||||||
| Target Release: | Unused | ||||||||||||||
| Hardware: | Unspecified | ||||||||||||||
| OS: | Unspecified | ||||||||||||||
| Whiteboard: | |||||||||||||||
| Fixed In Version: | pulp-2.8.7.1-1 | Doc Type: | Bug Fix | ||||||||||||
| Doc Text: | Story Points: | --- | |||||||||||||
| Clone Of: | Environment: | ||||||||||||||
| Last Closed: | 2016-10-26 12:25:09 UTC | Type: | Bug | ||||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||||
| Documentation: | --- | CRM: | |||||||||||||
| Verified Versions: | Category: | --- | |||||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||
| Embargoed: | |||||||||||||||
| Attachments: |
|
||||||||||||||
Created attachment 1137069 [details]
section of repo after request
Can you please attach a foreman-debug output? We'll need to see pulp logs to make any sense of why this happened. If you can add them, please needsinfo me. It's worth noting that one of the risks of on-demand content retrieval is that content might not be available when you want it. If pulp can't get it from the remote source at that moment, there's nothing else that can be done. I'm not sure if that's what happened here, but it's worth being aware of. Which logs would you like specifically? Pulp was able to provide a good number of packages requested, but the others didn't make it. jcalla, if you provide the output from foreman-debug, it should have everything we need. Here's how to run it: https://access.redhat.com/solutions/1177823 Alternatively, /var/log/messages from the Satellite server should work. I've reproduced the results, but with a slightly different error. The packages in question did not download, but others did. Interestingly, this occurred only in this specific repo. I had previously installed over 200 packages from the rhel-7-server and rh-common repos. I will attach the debug output. Created attachment 1139082 [details]
foreman-debug
Jeremy, does this look like one of the streamer bugs we're already working on? This could well be https://pulp.plan.io/issues/1788 Did the specific repository in question have a different entitlement certificate than the one you downloaded from before? If you restart the pulp_streamer service and try to download from the repository that was failing, does it work? There's another concerning issue in the logs: ibm-x3550m3-09 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: ibm-x3550m3-09.lab.eng.brq.redhat.com shows up a lot. I think this is occurring when Pulp tries to download the cached content via the 'download_deferred' task (because it's resetting the dropped connection to the same host it's on). This makes me think there's potentially an Apache configuration problem. This is https://pulp.plan.io/issues/1771 which is being tracked in: https://bugzilla.redhat.com/show_bug.cgi?id=1317047 Failed QA on Satellite 6.2 Beta Snap 6 Compose 1. I'm now seeing the following error for all packages attempting to be downloaded on_demand. katello-agent-1.5.3-7.el7sat.n FAILED https://rhsm-qe-1.rhq.lab.eng.bos.redhat.com/pulp/repos/Default_Organization/Library/rhel7/content/dist/rhel/server/7/7Server/x86_64/rh-common/os/katello-agent-1.5.3-7.el7sat.noarch.rpm: [Errno 14] HTTPS Error 500 - Internal Server Error Trying other mirror. See attached image and foreman-debug output. Created attachment 1143373 [details]
500 Error
Created attachment 1143375 [details]
foreman-debug 2
The failure is due to SELinux denials to `/var/lib/pulp/importers/.../pki/ca.crt` which is a known issue and being worked on: see https://pulp.plan.io/issues/1771#note-28 I have to remove this link to upstream Pulp issue 1771 because 1771 is already associated with Bugzilla 1317047. A current limitation is that each Pulp bug can only correspond with 1 bugzilla Bug and vice-versa. That limitation will be overcome soon, but it's not there yet. The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug. The Pulp upstream bug priority is at High. Updating the external tracker on this bug. Moving to POST as upstream fix is available. Verified in Satellite 6.2.3 Snap 4. Successfully streamed over 2,500 packages to multiple hosts at once. No errors seen on either Satellite or Host ends. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:2109 |
Created attachment 1137068 [details] katello-agent install failure Description of problem: When a repository is set to on_demand and packages are requested from a host, some packages might not be provided. Version-Release number of selected component (if applicable): Satellite 6.2 Beta Snap 3 How reproducible: Consistently Steps to Reproduce: 1. Create/Enable a repository with a download policy of on_demand 2. Sync the repo 3. Consume content in that repo from a host (install katello-agent) Actual results: Some required packages are not available when needed. An error may be seen like "Package does not match intended download." Expected results: All packages are downloaded/streamed to the host when requested. Additional info: See attached images