Bug 1288855 - Custom product content is not syncing to capsule
Custom product content is not syncing to capsule
Status: CLOSED NEXTRELEASE
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Pulp (Show other bugs)
6.1.3
x86_64 Linux
unspecified Severity urgent (vote)
: 6.1.9
: --
Assigned To: satellite6-bugs
jcallaha
: PrioBumpQA, Regression, Reopened, Triaged
Depends On:
Blocks: 1338516 1315263
  Show dependency treegraph
 
Reported: 2015-12-06 11:43 EST by Alexander Braverman
Modified: 2017-07-26 15:44 EDT (History)
18 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1315263 (view as bug list)
Environment:
Last Closed: 2016-05-04 16:13:50 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
foreman-debug (15.70 MB, application/x-xz)
2015-12-06 11:43 EST, Alexander Braverman
no flags Details
capsule-debug (432.91 KB, application/x-xz)
2015-12-07 06:27 EST, Alexander Braverman
no flags Details
capsule-debug (432.91 KB, application/x-xz)
2015-12-07 06:28 EST, Alexander Braverman
no flags Details
incorrect_contexts (1.35 KB, text/plain)
2015-12-07 06:32 EST, Alexander Braverman
no flags Details
capsule-messages (1.50 MB, application/x-gzip)
2015-12-08 08:33 EST, Alexander Braverman
no flags Details
satellite-messages (1.50 MB, application/x-gzip)
2015-12-08 08:34 EST, Alexander Braverman
no flags Details
task-export (7.35 MB, application/x-gzip)
2015-12-08 08:50 EST, Alexander Braverman
no flags Details
/var/log/messages from capsule (2.54 MB, application/x-gzip)
2015-12-08 09:54 EST, Pavel Moravec
no flags Details
capsule foreman-debug after update to 6.1.7 (875.14 KB, application/x-xz)
2016-02-23 08:50 EST, Alexander Braverman
no flags Details
satellite foreman-debug after update to 6.1.7 (18.74 MB, application/x-xz)
2016-02-23 09:30 EST, Alexander Braverman
no flags Details

  None (edit)
Description Alexander Braverman 2015-12-06 11:43:39 EST
Created attachment 1102821 [details]
foreman-debug

Description of problem:
Custom product content is not syncing to capsule.
The product content on Satellite main server (satellite6-ops.rhev-ci-vms.eng.rdu2.redhat.com/pulp/repos/RHEVM-QE-INFRA/Library/custom/RHEL_7_2_download_eng_tlv/RHEL_7_2_DEVEL) is different then on the capsule (http://capsule-ops.eng.lab.tlv.redhat.com/pulp/repos/RHEVM-QE-INFRA/Library/custom/RHEL_7_2_download_eng_tlv/RHEL_7_2_DEVEL/).

Tried to restart service on capsule with 'katello-service restart' and manually trigger the sync on Satellite with 'hammer capsule content synchronize --id 10'.
Sync the product didn't help either. Both actions completed without a problem.

Actual results:
different packages

Expected results:
same packages

Additional info:
Before was an issue syncing that product on capsule; it gave an error:
"RuntimeError: Will not create a symlink to a non-existent source"
The issue was resolved using the workaround from solution https://access.redhat.com/solutions/2069243
Comment 1 Lukas Zapletal 2015-12-07 06:24:52 EST
I see many denials in the logs in the debug log. Please issue:

restorecon -FvvRn /var/lib/pulp > incorrect_contexts.log

Putting the server into permissive might fix the issue as a quick and dirty workaround.
Comment 2 Alexander Braverman 2015-12-07 06:27 EST
Created attachment 1103175 [details]
capsule-debug

running foreman-debug on capsule
Comment 3 Alexander Braverman 2015-12-07 06:28 EST
Created attachment 1103177 [details]
capsule-debug
Comment 4 Alexander Braverman 2015-12-07 06:32 EST
Created attachment 1103180 [details]
incorrect_contexts
Comment 6 Alexander Braverman 2015-12-07 10:57:57 EST
set SELinux to permissive fixed the denials but the issue persists.
Comment 8 Lukas Zapletal 2015-12-08 08:05:05 EST
Can you dig from the shell history what commands you issued as a workaround? Thanks.

Looks like duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1285830 but let's keep it opened for now until resolution is found.
Comment 9 Alexander Braverman 2015-12-08 08:32:50 EST
commands issued as a workaround(as described in the solution):
$ grep "create a symlink to a non-existent source" /var/log/messages | grep "productid.gz" | cut -d\[ -f2 | sort -u
$ cd /var/lib/pulp/content/yum_repo_metadata_file/RHEVM-QE-INFRA-Library-RHEl_7_2-RHEL_7_2_download_eng_tlv-RHEL_7_2_DEVEL/
$ wget http://satellite6-ops.rhev-ci-vms.eng.rdu2.redhat.com/pulp/repos/RHEVM-QE-INFRA/Library/custom/RHEL_7_2_download_eng_tlv/RHEL_7_2_DEVEL/repodata/e67a1b710afb8343fb2c1fef5c670e25b46e6aa25b736c2db657bce6cd356a4d-productid.gz
$ chown apache:apache e67a1b710afb8343fb2c1fef5c670e25b46e6aa25b736c2db657bce6cd356a4d-productid.gz
$ cp e67a1b710afb8343fb2c1fef5c670e25b46e6aa25b736c2db657bce6cd356a4d-productid.gz /var/lib/pulp/content/yum_repo_metadata_file/RHEVM-QE-INFRA-Library-vdsm_3_6_jenkins_host_rhel_7_2-RHEL_7_2_download_eng_tlv-RHEL_7_2_DEVEL/e67a1b710afb8343fb2c1fef5c670e25b46e6aa25b736c2db657bce6cd356a4d-productid.gz
$ chown apache:apache /var/lib/pulp/content/yum_repo_metadata_file/RHEVM-QE-INFRA-Library-vdsm_3_6_jenkins_host_rhel_7_2-RHEL_7_2_download_eng_tlv-RHEL_7_2_DEVEL/e67a1b710afb8343fb2c1fef5c670e25b46e6aa25b736c2db657bce6cd356a4d-productid.gz
$ cp e67a1b710afb8343fb2c1fef5c670e25b46e6aa25b736c2db657bce6cd356a4d-productid.gz /var/lib/pulp/content/yum_repo_metadata_file/RHEVM-QE-INFRA-RHEL_7_2_download_eng_tlv-RHEL_7_2_DEVEL/e67a1b710afb8343fb2c1fef5c670e25b46e6aa25b736c2db657bce6cd356a4d-productid.gz
$ chown apache:apache /var/lib/pulp/content/yum_repo_metadata_file/RHEVM-QE-INFRA-RHEL_7_2_download_eng_tlv-RHEL_7_2_DEVEL/e67a1b710afb8343fb2c1fef5c670e25b46e6aa25b736c2db657bce6cd356a4d-productid.gz
$ for i in pulp_resource_manager pulp_workers pulp_celerybeat; do service $i restart; done
Comment 10 Alexander Braverman 2015-12-08 08:33 EST
Created attachment 1103606 [details]
capsule-messages
Comment 11 Alexander Braverman 2015-12-08 08:34 EST
Created attachment 1103607 [details]
satellite-messages
Comment 12 Alexander Braverman 2015-12-08 08:50 EST
Created attachment 1103609 [details]
task-export
Comment 13 Pavel Moravec 2015-12-08 09:54 EST
Created attachment 1103633 [details]
/var/log/messages from capsule

(previous messages from capsule were duplicate file of satellite's messages)
Comment 14 Pavel Moravec 2015-12-09 05:43:36 EST
If the cause is same like in bz1288560, then worth taking some extra logging when the issue reoccurs:

https://bugzilla.redhat.com/show_bug.cgi?id=1288560#c4
Comment 15 Jeff Ortel 2015-12-17 09:18:41 EST
I have not been able to reproduce this.  Also, after review of the node sync logic, it's unclear how this could happen.  The sync logic only creates the content unit if the file is successfully downloaded.  Specifically to prevent this situation.  This raises the possibility that something unrelated to node sync is deleting the productid files.  In order to proceed with troubleshooting this, I really need a repeatable reproducer.  Or steps to reproduce.
Comment 18 pulp-infra@redhat.com 2015-12-23 17:30:31 EST
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.
Comment 19 pulp-infra@redhat.com 2015-12-23 17:30:35 EST
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.
Comment 21 pulp-infra@redhat.com 2016-01-08 11:00:29 EST
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.
Comment 22 pulp-infra@redhat.com 2016-01-08 11:00:35 EST
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
Comment 23 Jeff Ortel 2016-01-18 15:27:41 EST
Was this upgraded from 6.0 -> 6.1.
Comment 24 Jeff Ortel 2016-01-18 18:15:19 EST
I have reproduced this and determined the root cause.  The node sync logic needs to be updated to properly handle changed content units with associated files (such as productid and prestodata).
Comment 25 Alexander Braverman 2016-01-19 03:55:21 EST
Yes, it was upgrade from 6.0 to 6.1
Comment 26 pulp-infra@redhat.com 2016-01-20 12:00:24 EST
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.
Comment 28 Bryan Kearney 2016-01-21 10:02:47 EST
This is same fix as https://bugzilla.redhat.com/show_bug.cgi?id=1276911. Please plan to verify both together.
Comment 29 pulp-infra@redhat.com 2016-01-22 15:30:22 EST
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.
Comment 33 jcallaha 2016-02-08 15:37:52 EST
Verified in Satellite 6.1.7 compose. 

All custom content was successfully sync'd to the capsule.
Comment 34 pulp-infra@redhat.com 2016-02-11 16:00:24 EST
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.
Comment 36 errata-xmlrpc 2016-02-15 10:52:23 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2016:0174
Comment 37 Alexander Braverman 2016-02-23 08:46:05 EST
The issue persists after update of Satellite and Capsule to 6.1.7 with the erreta RHSA-2016:0174. Syncing the custom product repository didn't help and neither full Capsule sync using hammer.
Comment 38 Alexander Braverman 2016-02-23 08:50 EST
Created attachment 1129766 [details]
capsule foreman-debug after update to 6.1.7
Comment 39 Alexander Braverman 2016-02-23 09:30 EST
Created attachment 1129771 [details]
satellite foreman-debug after update to 6.1.7
Comment 40 Og Maciel 2016-03-02 02:33:09 EST
Short version for QE:

The problem here is that we have a Satellite Server (RHEL 6) and a Capsule (RHEL 7) with several custom YUM repositories pointing to internal repositories.

At one point, content from this custom repository was synchronized to the Satellite (and added to content view, published, etc) and successfully synchronized to the external capsule.

Later on, a re-sync of the same custom repository brought in newer versions of packages but apparently one or more of these new packages never got synchronized to the capsule.

Today we manually walked through the same steps, to see if we could get packages synced to the capsule, but we still don't see the new packages in the capsule.

There is a possibility that the sync process to the capsule was terminated before completion and I wonder if the metadata in the capsule is "up to date" and somehow the sync process 'thinks' that it doesn't need to do anything?
Comment 41 Brian Bouterse 2016-03-07 18:18:50 EST
Removing the upstream Pulp bug. It is the correct upstream bug but the code is not being backported to 6.1.z.
Comment 42 Og Maciel 2016-03-15 09:39:54 EDT
Comment for consideration: since the fix for this issue requires a newer version of Pulp (see https://pulp.plan.io/issues/1463), I don't think that we can handle this for a z-stream release. I vote for closing it and commenting that the fix will be available for 6.2 only.
Comment 43 Kedar Bidarkar 2016-03-21 16:04:03 EDT
If this bug gets moved for sat62, this has been verified here already for sat62. 
https://bugzilla.redhat.com/show_bug.cgi?id=1315263#c9
Comment 45 Mike McCune 2016-03-28 19:32:32 EDT
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune@redhat.com with any questions
Comment 46 Eyal Edri 2016-04-12 04:45:18 EDT
Hi Eric,
Any ETA on this patch? its something that will help us (RHEV) move to SAT6 from foreman.
Comment 47 Mike McCune 2016-05-04 16:13:50 EDT
This bug is fixed, tested and verified as resolved in Satellite 6.2 and is unable to be backported into 6.1.z. You can track the bug against Satellite 6.2 here:


https://bugzilla.redhat.com/show_bug.cgi?id=1315263

Note You need to log in before you can comment on or make changes to this bug.