| Summary: | Some packages do not download with download policy of background | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | jcallaha | ||||||||||
| Component: | Pulp | Assignee: | David Davis <daviddavis> | ||||||||||
| Status: | CLOSED ERRATA | QA Contact: | jcallaha | ||||||||||
| Severity: | high | Docs Contact: | |||||||||||
| Priority: | unspecified | ||||||||||||
| Version: | 6.2.0 | CC: | bbuckingham, bkearney, bmbouter, cwelton, daviddavis, dkliban, ehelms, ggainey, greartes, ipanova, jcallaha, jmatthew, mburgerh, mhrivnak, mmccune, pcreech, rchan, ttereshc | ||||||||||
| Target Milestone: | Unspecified | Keywords: | Triaged | ||||||||||
| Target Release: | Unused | ||||||||||||
| Hardware: | Unspecified | ||||||||||||
| OS: | Unspecified | ||||||||||||
| Whiteboard: | |||||||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||||||
| Doc Text: | Story Points: | --- | |||||||||||
| Clone Of: | Environment: | ||||||||||||
| Last Closed: | 2018-02-21 17:02:00 UTC | Type: | Bug | ||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||
| Documentation: | --- | CRM: | |||||||||||
| Verified Versions: | Category: | --- | |||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
| Attachments: |
|
||||||||||||
|
Description
jcallaha
2016-03-15 21:01:52 UTC
jcallaha, can you attach a foreman-debug output? Also, it would be helpful to have the output of "pulp-admin -u admin -p adminpw tasks list -a". Nevermind the foreman-debug output. I reproduced the bug. Actually I did not reproduce the bug. I was using immediate and ran into this bug https://bugzilla.redhat.com/show_bug.cgi?id=1315752. Going to try with a larger repository. Still waiting on a foreman-debug. Created attachment 1138807 [details]
foreman-debug
Created attachment 1138808 [details]
missing packages
reproduced with Red Hat Enterprise Linux 7 Server - RH Common RPMs x86_64 7Server. I've attached the foreman-debug output as well as a screenshot of the links. Looking at /var/log/messages, I see a couple items of interest: Mar 21 21:53:58 ibm-x3550m3-09 pulp: pulp.server.controllers.repository:INFO: Download of /var/lib/pulp/content/units/rpm/b8/affb52aed62ae809299aea8692d7271f1bc563143d330b96af9355a532e37b/glusterfs-3.7.5-19.el7.x86_64.rpm failed: Proxy Error. For the powerbar gem: Mar 21 21:55:52 ibm-x3550m3-09 pulp_streamer: [-] 127.0.0.1 - - [21/Mar/2016:20:55:51 +0000] "GET /var/lib/pulp/content/units/rpm/3d/e504e44d621124d0699f9bfce3902c078e14efb70cd9e902abd469a07fc159/rubygem-powerbar-1.0.11-8.el7sat.noarch.rpm HTTP/1.1" 404 - "-" "python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-327.el7.x86_64" Mar 21 21:55:52 ibm-x3550m3-09 pulp: pulp.server.controllers.repository:INFO: Download of /var/lib/pulp/content/units/rpm/3d/e504e44d621124d0699f9bfce3902c078e14efb70cd9e902abd469a07fc159/rubygem-powerbar-1.0.11-8.el7sat.noarch.rpm failed: Not Found. Also, I see a bunch of lines like this: Mar 21 21:55:55 ibm-x3550m3-09 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: ibm-x3550m3-09.lab.eng.brq.redhat.com Created attachment 1139998 [details]
pulp streamer
Looking at the pulp_streamer logs, the main error I see is this one: https://pulp.plan.io/issues/1734 Although I also see this one as well: https://pulp.plan.io/issues/1771 The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug. The Pulp upstream bug priority is at High. Updating the external tracker on this bug. fwiw, an on-demand repo I have shows this exact same behaviour: not downloading certain packages with 404 errors. I haven't switched from on_demand to background at all. Not sure if relevant. Updating bug title to reflect that this doesn't require a switch from on_demand to background. Failed QA in Satellite 6.2 Beta Snap 6 Compose 2. No packages download when using the background download policy. The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug. This is in Pulp 2.8.1, moving ON_QA I am currently seeing this in 6.2.3. Packages are till not being downloaded via deferred download. However, the streamer was able to supply the packages when needed. Interestingly, the streamed packages never populated the satellite-side content. Additionally, it seems to be fairly consistent in what doesn't get downloaded. For example, libreoffice paclages are missing from RHEL6/7 and openoffice from RHEL5. Jake, I tried to reproduce but was unable to. Here were the steps I went through: 1. Spun up Satellite 6.2.3 box 2. Uploaded manifest. Enabled Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server repo. 3. Changed download policy for the repo to background and synced it. 4. On the filesystem, cd into /var/lib/pulp/published/yum/https/repos/Default_Organization/Library/content/dist/rhel/server/7/7Server/x86_64/os and ran 'find . -xtype l' Here were the satellite and pulp-server packages on my system: satellite-6.2.3-1.0.el7sat.noarch pulp-server-2.8.7.1-2.el7sat.noarch I noticed that the packages mentioned though (libreoffice, etc) aren't part of the RHEL 7 sub I am using. Is the box still up? Do you see anything in /var/log/messages? Or better yet, can I maybe connect to the box? Was able to reproduce. I synced Red Hat Enterprise Linux 7 Server - RH Common RPMs x86_64 7Server and got tons of dead symlinks in /var/lib/pulp/published/yum/https/repos/Default_Organization/Library/content/dist/rhel/server/7/7Server/x86_64/rh-common/os. About 134 in all. In /var/log/messages, I am seeing lots of these errors for the missing packages: Oct 21 14:46:03 localhost pulp: pulp.server.controllers.repository:INFO: Download of /var/lib/pulp/content/units/rpm/c8/3079c6b2a88ead802d8800639c0eaa2e47b4d41ad8bef649d0357ca705b62c/saslwrapper-0.22-5.el7sat. x86_64.rpm failed: Internal Server Error. According to Jeff, this might be the same cause as https://bugzilla.redhat.com/show_bug.cgi?id=1331863. Retested with the following RPMs: satellite-6.2.3-1.0.el7sat.noarch python-pulp-repoauth-2.8.7.2-1.el7sat.noarch pulp-client-1.0-1.noarch pulp-docker-plugins-2.0.3-1.el7sat.noarch pulp-selinux-2.8.7.2-1.el7sat.noarch python-pulp-streamer-2.8.7.2-1.el7sat.noarch python-pulp-docker-common-2.0.3-1.el7sat.noarch pulp-katello-1.0.1-1.el7sat.noarch pulp-puppet-tools-2.8.7.1-1.el7sat.noarch python-pulp-oid_validation-2.8.7.2-1.el7sat.noarch python-pulp-rpm-common-2.8.7.3-1.el7sat.noarch pulp-puppet-plugins-2.8.7.1-1.el7sat.noarch rubygem-smart_proxy_pulp-1.2.2-1.el7sat.noarch pulp-rpm-plugins-2.8.7.3-1.el7sat.noarch python-pulp-puppet-common-2.8.7.1-1.el7sat.noarch python-pulp-common-2.8.7.2-1.el7sat.noarch pulp-server-2.8.7.2-1.el7sat.noarch I synced the RH Common RPMs sub down and there are no dead symlinks. I think the fix for https://bugzilla.redhat.com/show_bug.cgi?id=1331863 fixed this as well. Setting to ON_QA for QE to test. Going to reopen this as jcalla hit this again. He synced 'Red Hat Enterprise Linux 6 Server RPMs x86_64 6Server' and had about 18290 packages. One of them was a dead symlink: # pwd /var/lib/pulp/published/yum/https/repos/Default_Organization/Library/rhel6/content/dist/rhel/server/6/6Server/x86_64/os # find . -xtype l ./freetype-devel-2.3.11-17.el6.x86_64.rpm Attaching the relevant lines from /var/log/messages in a separate file. Created attachment 1213556 [details]
/var/log/messages snippet
Setting back to ON_QA. Looks like the package was eventually downloaded as the dead symlink is no longer there. I've opened a separate RFE to address the issue that errors that may happen during the background download of packages aren't being reported: https://bugzilla.redhat.com/show_bug.cgi?id=1388652 Verified in Satellite 6.3 Snap 16. Unable to reproduce this issue while testing across 20 repositories. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:0336 |