Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1288855 - Custom product content is not syncing to capsule
Summary: Custom product content is not syncing to capsule
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.1.3
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: jcallaha
URL:
Whiteboard:
Depends On:
Blocks: 1315263 1338516
TreeView+ depends on / blocked
 
Reported: 2015-12-06 16:43 UTC by Alexander Braverman
Modified: 2022-07-09 07:40 UTC (History)
18 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1315263 (view as bug list)
Environment:
Last Closed: 2016-05-04 20:13:50 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
foreman-debug (15.70 MB, application/x-xz)
2015-12-06 16:43 UTC, Alexander Braverman 		no flags Details
capsule-debug (432.91 KB, application/x-xz)
2015-12-07 11:27 UTC, Alexander Braverman 		no flags Details
capsule-debug (432.91 KB, application/x-xz)
2015-12-07 11:28 UTC, Alexander Braverman 		no flags Details
incorrect_contexts (1.35 KB, text/plain)
2015-12-07 11:32 UTC, Alexander Braverman 		no flags Details
capsule-messages (1.50 MB, application/x-gzip)
2015-12-08 13:33 UTC, Alexander Braverman 		no flags Details
satellite-messages (1.50 MB, application/x-gzip)
2015-12-08 13:34 UTC, Alexander Braverman 		no flags Details
task-export (7.35 MB, application/x-gzip)
2015-12-08 13:50 UTC, Alexander Braverman 		no flags Details
/var/log/messages from capsule (2.54 MB, application/x-gzip)
2015-12-08 14:54 UTC, Pavel Moravec 		no flags Details
capsule foreman-debug after update to 6.1.7 (875.14 KB, application/x-xz)
2016-02-23 13:50 UTC, Alexander Braverman 		no flags Details
satellite foreman-debug after update to 6.1.7 (18.74 MB, application/x-xz)
2016-02-23 14:30 UTC, Alexander Braverman 		no flags Details
Show Obsolete (2) View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2016:0174 0 normal SHIPPED_LIVE Moderate: Satellite 6.1.7 security, bug and enhancement fix update 2016-02-15 20:50:32 UTC

Description Alexander Braverman 2015-12-06 16:43:39 UTC 
Created attachment 1102821 [details]
foreman-debug

Description of problem:
Custom product content is not syncing to capsule.
The product content on Satellite main server (satellite6-ops.rhev-ci-vms.eng.rdu2.redhat.com/pulp/repos/RHEVM-QE-INFRA/Library/custom/RHEL_7_2_download_eng_tlv/RHEL_7_2_DEVEL) is different then on the capsule (http://capsule-ops.eng.lab.tlv.redhat.com/pulp/repos/RHEVM-QE-INFRA/Library/custom/RHEL_7_2_download_eng_tlv/RHEL_7_2_DEVEL/).

Tried to restart service on capsule with 'katello-service restart' and manually trigger the sync on Satellite with 'hammer capsule content synchronize --id 10'.
Sync the product didn't help either. Both actions completed without a problem.

Actual results:
different packages

Expected results:
same packages

Additional info:
Before was an issue syncing that product on capsule; it gave an error:
"RuntimeError: Will not create a symlink to a non-existent source"
The issue was resolved using the workaround from solution https://access.redhat.com/solutions/2069243
Comment 1 Lukas Zapletal 2015-12-07 11:24:52 UTC 
I see many denials in the logs in the debug log. Please issue:

restorecon -FvvRn /var/lib/pulp > incorrect_contexts.log

Putting the server into permissive might fix the issue as a quick and dirty workaround.
Comment 2 Alexander Braverman 2015-12-07 11:27:33 UTC 
Created attachment 1103175 [details]
capsule-debug

running foreman-debug on capsule
Comment 3 Alexander Braverman 2015-12-07 11:28:50 UTC 
Created attachment 1103177 [details]
capsule-debug
Comment 4 Alexander Braverman 2015-12-07 11:32:12 UTC 
Created attachment 1103180 [details]
incorrect_contexts
Comment 6 Alexander Braverman 2015-12-07 15:57:57 UTC 
set SELinux to permissive fixed the denials but the issue persists.
Comment 8 Lukas Zapletal 2015-12-08 13:05:05 UTC 
Can you dig from the shell history what commands you issued as a workaround? Thanks.

Looks like duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1285830 but let's keep it opened for now until resolution is found.
Comment 9 Alexander Braverman 2015-12-08 13:32:50 UTC 
commands issued as a workaround(as described in the solution):
$ grep "create a symlink to a non-existent source" /var/log/messages | grep "productid.gz" | cut -d\[ -f2 | sort -u
$ cd /var/lib/pulp/content/yum_repo_metadata_file/RHEVM-QE-INFRA-Library-RHEl_7_2-RHEL_7_2_download_eng_tlv-RHEL_7_2_DEVEL/
$ wget http://satellite6-ops.rhev-ci-vms.eng.rdu2.redhat.com/pulp/repos/RHEVM-QE-INFRA/Library/custom/RHEL_7_2_download_eng_tlv/RHEL_7_2_DEVEL/repodata/e67a1b710afb8343fb2c1fef5c670e25b46e6aa25b736c2db657bce6cd356a4d-productid.gz
$ chown apache:apache e67a1b710afb8343fb2c1fef5c670e25b46e6aa25b736c2db657bce6cd356a4d-productid.gz
$ cp e67a1b710afb8343fb2c1fef5c670e25b46e6aa25b736c2db657bce6cd356a4d-productid.gz /var/lib/pulp/content/yum_repo_metadata_file/RHEVM-QE-INFRA-Library-vdsm_3_6_jenkins_host_rhel_7_2-RHEL_7_2_download_eng_tlv-RHEL_7_2_DEVEL/e67a1b710afb8343fb2c1fef5c670e25b46e6aa25b736c2db657bce6cd356a4d-productid.gz
$ chown apache:apache /var/lib/pulp/content/yum_repo_metadata_file/RHEVM-QE-INFRA-Library-vdsm_3_6_jenkins_host_rhel_7_2-RHEL_7_2_download_eng_tlv-RHEL_7_2_DEVEL/e67a1b710afb8343fb2c1fef5c670e25b46e6aa25b736c2db657bce6cd356a4d-productid.gz
$ cp e67a1b710afb8343fb2c1fef5c670e25b46e6aa25b736c2db657bce6cd356a4d-productid.gz /var/lib/pulp/content/yum_repo_metadata_file/RHEVM-QE-INFRA-RHEL_7_2_download_eng_tlv-RHEL_7_2_DEVEL/e67a1b710afb8343fb2c1fef5c670e25b46e6aa25b736c2db657bce6cd356a4d-productid.gz
$ chown apache:apache /var/lib/pulp/content/yum_repo_metadata_file/RHEVM-QE-INFRA-RHEL_7_2_download_eng_tlv-RHEL_7_2_DEVEL/e67a1b710afb8343fb2c1fef5c670e25b46e6aa25b736c2db657bce6cd356a4d-productid.gz
$ for i in pulp_resource_manager pulp_workers pulp_celerybeat; do service $i restart; done
Comment 10 Alexander Braverman 2015-12-08 13:33:41 UTC 
Created attachment 1103606 [details]
capsule-messages
Comment 11 Alexander Braverman 2015-12-08 13:34:17 UTC 
Created attachment 1103607 [details]
satellite-messages
Comment 12 Alexander Braverman 2015-12-08 13:50:31 UTC 
Created attachment 1103609 [details]
task-export
Comment 13 Pavel Moravec 2015-12-08 14:54:05 UTC 
Created attachment 1103633 [details]
/var/log/messages from capsule

(previous messages from capsule were duplicate file of satellite's messages)
Comment 14 Pavel Moravec 2015-12-09 10:43:36 UTC 
If the cause is same like in bz1288560, then worth taking some extra logging when the issue reoccurs:

https://bugzilla.redhat.com/show_bug.cgi?id=1288560#c4
Comment 15 Jeff Ortel 2015-12-17 14:18:41 UTC 
I have not been able to reproduce this.  Also, after review of the node sync logic, it's unclear how this could happen.  The sync logic only creates the content unit if the file is successfully downloaded.  Specifically to prevent this situation.  This raises the possibility that something unrelated to node sync is deleting the productid files.  In order to proceed with troubleshooting this, I really need a repeatable reproducer.  Or steps to reproduce.
Comment 18 pulp-infra@redhat.com 2015-12-23 22:30:31 UTC 
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.
Comment 19 pulp-infra@redhat.com 2015-12-23 22:30:35 UTC 
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.
Comment 21 pulp-infra@redhat.com 2016-01-08 16:00:29 UTC 
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.
Comment 22 pulp-infra@redhat.com 2016-01-08 16:00:35 UTC 
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
Comment 23 Jeff Ortel 2016-01-18 20:27:41 UTC 
Was this upgraded from 6.0 -> 6.1.
Comment 24 Jeff Ortel 2016-01-18 23:15:19 UTC 
I have reproduced this and determined the root cause.  The node sync logic needs to be updated to properly handle changed content units with associated files (such as productid and prestodata).
Comment 25 Alexander Braverman 2016-01-19 08:55:21 UTC 
Yes, it was upgrade from 6.0 to 6.1
Comment 26 pulp-infra@redhat.com 2016-01-20 17:00:24 UTC 
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.
Comment 28 Bryan Kearney 2016-01-21 15:02:47 UTC 
This is same fix as https://bugzilla.redhat.com/show_bug.cgi?id=1276911. Please plan to verify both together.
Comment 29 pulp-infra@redhat.com 2016-01-22 20:30:22 UTC 
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.
Comment 33 jcallaha 2016-02-08 20:37:52 UTC 
Verified in Satellite 6.1.7 compose. 

All custom content was successfully sync'd to the capsule.
Comment 34 pulp-infra@redhat.com 2016-02-11 21:00:24 UTC 
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.
Comment 36 errata-xmlrpc 2016-02-15 15:52:23 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2016:0174
Comment 37 Alexander Braverman 2016-02-23 13:46:05 UTC 
The issue persists after update of Satellite and Capsule to 6.1.7 with the erreta RHSA-2016:0174. Syncing the custom product repository didn't help and neither full Capsule sync using hammer.
Comment 38 Alexander Braverman 2016-02-23 13:50:24 UTC 
Created attachment 1129766 [details]
capsule foreman-debug after update to 6.1.7
Comment 39 Alexander Braverman 2016-02-23 14:30:43 UTC 
Created attachment 1129771 [details]
satellite foreman-debug after update to 6.1.7
Comment 40 Og Maciel 2016-03-02 07:33:09 UTC 
Short version for QE:

The problem here is that we have a Satellite Server (RHEL 6) and a Capsule (RHEL 7) with several custom YUM repositories pointing to internal repositories.

At one point, content from this custom repository was synchronized to the Satellite (and added to content view, published, etc) and successfully synchronized to the external capsule.

Later on, a re-sync of the same custom repository brought in newer versions of packages but apparently one or more of these new packages never got synchronized to the capsule.

Today we manually walked through the same steps, to see if we could get packages synced to the capsule, but we still don't see the new packages in the capsule.

There is a possibility that the sync process to the capsule was terminated before completion and I wonder if the metadata in the capsule is "up to date" and somehow the sync process 'thinks' that it doesn't need to do anything?
Comment 41 Brian Bouterse 2016-03-07 23:18:50 UTC 
Removing the upstream Pulp bug. It is the correct upstream bug but the code is not being backported to 6.1.z.
Comment 42 Og Maciel 2016-03-15 13:39:54 UTC 
Comment for consideration: since the fix for this issue requires a newer version of Pulp (see https://pulp.plan.io/issues/1463), I don't think that we can handle this for a z-stream release. I vote for closing it and commenting that the fix will be available for 6.2 only.
Comment 43 Kedar Bidarkar 2016-03-21 20:04:03 UTC 
If this bug gets moved for sat62, this has been verified here already for sat62. 
https://bugzilla.redhat.com/show_bug.cgi?id=1315263#c9
Comment 45 Mike McCune 2016-03-28 23:32:32 UTC 
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions
Comment 46 Eyal Edri 2016-04-12 08:45:18 UTC 
Hi Eric,
Any ETA on this patch? its something that will help us (RHEV) move to SAT6 from foreman.
Comment 47 Mike McCune 2016-05-04 20:13:50 UTC 
This bug is fixed, tested and verified as resolved in Satellite 6.2 and is unable to be backported into 6.1.z. You can track the bug against Satellite 6.2 here:


https://bugzilla.redhat.com/show_bug.cgi?id=1315263
Note You need to log in before you can comment on or make changes to this bug.