Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1318048 - Some packages do not download with download policy of background
Summary: Some packages do not download with download policy of background
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: Unspecified
Assignee: David Davis
QA Contact: jcallaha
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-03-15 21:01 UTC by jcallaha
Modified: 2021-04-06 17:59 UTC (History)
18 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-02-21 17:02:00 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
foreman-debug (413.39 KB, application/x-xz)
2016-03-21 21:05 UTC, jcallaha 		no flags Details
missing packages (377.49 KB, image/png)
2016-03-21 21:06 UTC, jcallaha 		no flags Details
pulp streamer (10.93 MB, text/plain)
2016-03-24 14:31 UTC, jcallaha 		no flags Details
/var/log/messages snippet (1.85 KB, text/plain)
2016-10-24 20:23 UTC, David Davis 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Pulp Redmine 1734 0 High CLOSED - CURRENTRELEASE The Pulp streamer logs a traceback when a ConnectionError occurs. 2016-04-05 20:33:51 UTC

Description jcallaha 2016-03-15 21:01:52 UTC 
Version-Release number of selected component (if applicable):
6.2 Beta Snap 3

How reproducible:
Occasional

Steps to Reproduce:
1. Enable/Create a repository with an on_demand download policy.
2. Sync the repository
3. Set the download policy to background
4. Re-sync the repository
5. Monitor the repo's local file system for downloaded packages

Actual results:
After a reasonable amount of time (3-5 times what it took to download the bulk of the packages) there are still packages that have not been downloaded.

Expected results:
All packages have been downloaded.

Additional info:
I've reproduced with:
   Red Hat Satellite 6.1 for RHEL 7 Server RPMs x86_64
   Red Hat Satellite Capsule 6.1 for RHEL 7 Server RPMs x86_64
   Red Hat Enterprise Linux 7 Server Kickstart x86_64 7.2
   Red Hat Enterprise Linux 7 Server - RH Common RPMs x86_64 7Server
Comment 3 David Davis 2016-03-16 14:27:04 UTC 
jcallaha, can you attach a foreman-debug output?

Also, it would be helpful to have the output of "pulp-admin -u admin -p adminpw tasks list -a".
Comment 4 David Davis 2016-03-16 14:30:37 UTC 
Related to https://bugzilla.redhat.com/show_bug.cgi?id=1315752
Comment 5 David Davis 2016-03-16 15:34:52 UTC 
Nevermind the foreman-debug output. I reproduced the bug.
Comment 6 David Davis 2016-03-16 15:57:40 UTC 
Actually I did not reproduce the bug. I was using immediate and ran into this bug https://bugzilla.redhat.com/show_bug.cgi?id=1315752. Going to try with a larger repository.
Comment 7 David Davis 2016-03-18 16:35:44 UTC 
Still waiting on a foreman-debug.
Comment 9 jcallaha 2016-03-21 21:05:58 UTC 
Created attachment 1138807 [details]
foreman-debug
Comment 10 jcallaha 2016-03-21 21:06:27 UTC 
Created attachment 1138808 [details]
missing packages
Comment 11 jcallaha 2016-03-21 21:07:43 UTC 
reproduced with Red Hat Enterprise Linux 7 Server - RH Common RPMs x86_64 7Server. I've attached the foreman-debug output as well as a screenshot of the links.
Comment 12 David Davis 2016-03-23 19:20:40 UTC 
Looking at /var/log/messages, I see a couple items of interest:

Mar 21 21:53:58 ibm-x3550m3-09 pulp: pulp.server.controllers.repository:INFO: Download of /var/lib/pulp/content/units/rpm/b8/affb52aed62ae809299aea8692d7271f1bc563143d330b96af9355a532e37b/glusterfs-3.7.5-19.el7.x86_64.rpm failed: Proxy Error.


For the powerbar gem:

Mar 21 21:55:52 ibm-x3550m3-09 pulp_streamer: [-] 127.0.0.1 - - [21/Mar/2016:20:55:51 +0000] "GET /var/lib/pulp/content/units/rpm/3d/e504e44d621124d0699f9bfce3902c078e14efb70cd9e902abd469a07fc159/rubygem-powerbar-1.0.11-8.el7sat.noarch.rpm HTTP/1.1" 404 - "-" "python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-327.el7.x86_64"
Mar 21 21:55:52 ibm-x3550m3-09 pulp: pulp.server.controllers.repository:INFO: Download of /var/lib/pulp/content/units/rpm/3d/e504e44d621124d0699f9bfce3902c078e14efb70cd9e902abd469a07fc159/rubygem-powerbar-1.0.11-8.el7sat.noarch.rpm failed: Not Found.


Also, I see a bunch of lines like this:

Mar 21 21:55:55 ibm-x3550m3-09 pulp: requests.packages.urllib3.connectionpool:INFO: Resetting dropped connection: ibm-x3550m3-09.lab.eng.brq.redhat.com
Comment 13 jcallaha 2016-03-24 14:31:08 UTC 
Created attachment 1139998 [details]
pulp streamer
Comment 14 David Davis 2016-03-24 14:59:49 UTC 
Looking at the pulp_streamer logs, the main error I see is this one:

https://pulp.plan.io/issues/1734

Although I also see this one as well:

https://pulp.plan.io/issues/1771
Comment 15 pulp-infra@redhat.com 2016-03-24 19:11:23 UTC 
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.
Comment 16 pulp-infra@redhat.com 2016-03-24 19:11:25 UTC 
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
Comment 17 Maxim Burgerhout 2016-03-24 22:54:46 UTC 
fwiw, an on-demand repo I have shows this exact same behaviour: not downloading certain packages with 404 errors. I haven't switched from on_demand to background at all. 

Not sure if relevant.
Comment 18 David Davis 2016-03-28 16:03:45 UTC 
Updating bug title to reflect that this doesn't require a switch from on_demand to background.
Comment 19 jcallaha 2016-04-04 15:29:26 UTC 
Failed QA in Satellite 6.2 Beta Snap 6 Compose 2. 

No packages download when using the background download policy.
Comment 21 pulp-infra@redhat.com 2016-04-05 20:33:52 UTC 
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.
Comment 22 Mike McCune 2016-05-26 19:51:05 UTC 
This is in Pulp 2.8.1, moving ON_QA
Comment 24 jcallaha 2016-10-17 19:40:01 UTC 
I am currently seeing this in 6.2.3. Packages are till not being downloaded via deferred download. However, the streamer was able to supply the packages when needed. Interestingly, the streamed packages never populated the satellite-side content.
Comment 25 jcallaha 2016-10-17 19:48:35 UTC 
Additionally, it seems to be fairly consistent in what doesn't get downloaded. For example, libreoffice paclages are missing from RHEL6/7 and openoffice from RHEL5.
Comment 26 David Davis 2016-10-20 22:15:51 UTC 
Jake,

I tried to reproduce but was unable to. Here were the steps I went through:

1. Spun up Satellite 6.2.3 box
2. Uploaded manifest. Enabled Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server repo.
3. Changed download policy for the repo to background and synced it.
4. On the filesystem, cd into /var/lib/pulp/published/yum/https/repos/Default_Organization/Library/content/dist/rhel/server/7/7Server/x86_64/os and ran 'find . -xtype l'

Here were the satellite and pulp-server packages on my system:

satellite-6.2.3-1.0.el7sat.noarch
pulp-server-2.8.7.1-2.el7sat.noarch

I noticed that the packages mentioned though (libreoffice, etc)  aren't part of the RHEL 7 sub I am using.

Is the box still up? Do you see anything in /var/log/messages? Or better yet, can I maybe connect to the box?
Comment 27 David Davis 2016-10-21 15:06:48 UTC 
Was able to reproduce. I synced Red Hat Enterprise Linux 7 Server - RH Common RPMs x86_64 7Server and got tons of dead symlinks in /var/lib/pulp/published/yum/https/repos/Default_Organization/Library/content/dist/rhel/server/7/7Server/x86_64/rh-common/os. About 134 in all. In /var/log/messages, I am seeing lots of these errors for the missing packages:

Oct 21 14:46:03 localhost pulp: pulp.server.controllers.repository:INFO: Download of /var/lib/pulp/content/units/rpm/c8/3079c6b2a88ead802d8800639c0eaa2e47b4d41ad8bef649d0357ca705b62c/saslwrapper-0.22-5.el7sat.
x86_64.rpm failed: Internal Server Error.
Comment 28 David Davis 2016-10-21 16:04:48 UTC 
According to Jeff, this might be the same cause as https://bugzilla.redhat.com/show_bug.cgi?id=1331863.
Comment 29 David Davis 2016-10-24 18:35:59 UTC 
Retested with the following RPMs:

satellite-6.2.3-1.0.el7sat.noarch
python-pulp-repoauth-2.8.7.2-1.el7sat.noarch
pulp-client-1.0-1.noarch
pulp-docker-plugins-2.0.3-1.el7sat.noarch
pulp-selinux-2.8.7.2-1.el7sat.noarch
python-pulp-streamer-2.8.7.2-1.el7sat.noarch
python-pulp-docker-common-2.0.3-1.el7sat.noarch
pulp-katello-1.0.1-1.el7sat.noarch
pulp-puppet-tools-2.8.7.1-1.el7sat.noarch
python-pulp-oid_validation-2.8.7.2-1.el7sat.noarch
python-pulp-rpm-common-2.8.7.3-1.el7sat.noarch
pulp-puppet-plugins-2.8.7.1-1.el7sat.noarch
rubygem-smart_proxy_pulp-1.2.2-1.el7sat.noarch
pulp-rpm-plugins-2.8.7.3-1.el7sat.noarch
python-pulp-puppet-common-2.8.7.1-1.el7sat.noarch
python-pulp-common-2.8.7.2-1.el7sat.noarch
pulp-server-2.8.7.2-1.el7sat.noarch

I synced the RH Common RPMs sub down and there are no dead symlinks. I think the fix for https://bugzilla.redhat.com/show_bug.cgi?id=1331863 fixed this as well. Setting to ON_QA for QE to test.
Comment 30 David Davis 2016-10-24 20:23:04 UTC 
Going to reopen this as jcalla hit this again. He synced 'Red Hat Enterprise Linux 6 Server RPMs x86_64 6Server' and had about 18290 packages. One of them was a dead symlink: 

# pwd
/var/lib/pulp/published/yum/https/repos/Default_Organization/Library/rhel6/content/dist/rhel/server/6/6Server/x86_64/os
# find . -xtype l
./freetype-devel-2.3.11-17.el6.x86_64.rpm

Attaching the relevant lines from /var/log/messages in a separate file.
Comment 31 David Davis 2016-10-24 20:23:44 UTC 
Created attachment 1213556 [details]
/var/log/messages snippet
Comment 32 David Davis 2016-10-25 20:44:39 UTC 
Setting back to ON_QA. Looks like the package was eventually downloaded as the dead symlink is no longer there. I've opened a separate RFE to address the issue that errors that may happen during the background download of packages aren't being reported:

https://bugzilla.redhat.com/show_bug.cgi?id=1388652
Comment 33 jcallaha 2017-09-20 19:10:31 UTC 
Verified in Satellite 6.3 Snap 16.

Unable to reproduce this issue while testing across 20 repositories.
Comment 34 Bryan Kearney 2018-02-21 17:02:00 UTC 
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.

For information on the advisory, and where to find the updated files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:0336
Note You need to log in before you can comment on or make changes to this bug.