Red Hat Bugzilla – Bug 1288855
Custom product content is not syncing to capsule
Last modified: 2017-07-26 15:44:31 EDT
Created attachment 1102821 [details] foreman-debug Description of problem: Custom product content is not syncing to capsule. The product content on Satellite main server (satellite6-ops.rhev-ci-vms.eng.rdu2.redhat.com/pulp/repos/RHEVM-QE-INFRA/Library/custom/RHEL_7_2_download_eng_tlv/RHEL_7_2_DEVEL) is different then on the capsule (http://capsule-ops.eng.lab.tlv.redhat.com/pulp/repos/RHEVM-QE-INFRA/Library/custom/RHEL_7_2_download_eng_tlv/RHEL_7_2_DEVEL/). Tried to restart service on capsule with 'katello-service restart' and manually trigger the sync on Satellite with 'hammer capsule content synchronize --id 10'. Sync the product didn't help either. Both actions completed without a problem. Actual results: different packages Expected results: same packages Additional info: Before was an issue syncing that product on capsule; it gave an error: "RuntimeError: Will not create a symlink to a non-existent source" The issue was resolved using the workaround from solution https://access.redhat.com/solutions/2069243
I see many denials in the logs in the debug log. Please issue: restorecon -FvvRn /var/lib/pulp > incorrect_contexts.log Putting the server into permissive might fix the issue as a quick and dirty workaround.
Created attachment 1103175 [details] capsule-debug running foreman-debug on capsule
Created attachment 1103177 [details] capsule-debug
Created attachment 1103180 [details] incorrect_contexts
set SELinux to permissive fixed the denials but the issue persists.
Can you dig from the shell history what commands you issued as a workaround? Thanks. Looks like duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1285830 but let's keep it opened for now until resolution is found.
commands issued as a workaround(as described in the solution): $ grep "create a symlink to a non-existent source" /var/log/messages | grep "productid.gz" | cut -d\[ -f2 | sort -u $ cd /var/lib/pulp/content/yum_repo_metadata_file/RHEVM-QE-INFRA-Library-RHEl_7_2-RHEL_7_2_download_eng_tlv-RHEL_7_2_DEVEL/ $ wget http://satellite6-ops.rhev-ci-vms.eng.rdu2.redhat.com/pulp/repos/RHEVM-QE-INFRA/Library/custom/RHEL_7_2_download_eng_tlv/RHEL_7_2_DEVEL/repodata/e67a1b710afb8343fb2c1fef5c670e25b46e6aa25b736c2db657bce6cd356a4d-productid.gz $ chown apache:apache e67a1b710afb8343fb2c1fef5c670e25b46e6aa25b736c2db657bce6cd356a4d-productid.gz $ cp e67a1b710afb8343fb2c1fef5c670e25b46e6aa25b736c2db657bce6cd356a4d-productid.gz /var/lib/pulp/content/yum_repo_metadata_file/RHEVM-QE-INFRA-Library-vdsm_3_6_jenkins_host_rhel_7_2-RHEL_7_2_download_eng_tlv-RHEL_7_2_DEVEL/e67a1b710afb8343fb2c1fef5c670e25b46e6aa25b736c2db657bce6cd356a4d-productid.gz $ chown apache:apache /var/lib/pulp/content/yum_repo_metadata_file/RHEVM-QE-INFRA-Library-vdsm_3_6_jenkins_host_rhel_7_2-RHEL_7_2_download_eng_tlv-RHEL_7_2_DEVEL/e67a1b710afb8343fb2c1fef5c670e25b46e6aa25b736c2db657bce6cd356a4d-productid.gz $ cp e67a1b710afb8343fb2c1fef5c670e25b46e6aa25b736c2db657bce6cd356a4d-productid.gz /var/lib/pulp/content/yum_repo_metadata_file/RHEVM-QE-INFRA-RHEL_7_2_download_eng_tlv-RHEL_7_2_DEVEL/e67a1b710afb8343fb2c1fef5c670e25b46e6aa25b736c2db657bce6cd356a4d-productid.gz $ chown apache:apache /var/lib/pulp/content/yum_repo_metadata_file/RHEVM-QE-INFRA-RHEL_7_2_download_eng_tlv-RHEL_7_2_DEVEL/e67a1b710afb8343fb2c1fef5c670e25b46e6aa25b736c2db657bce6cd356a4d-productid.gz $ for i in pulp_resource_manager pulp_workers pulp_celerybeat; do service $i restart; done
Created attachment 1103606 [details] capsule-messages
Created attachment 1103607 [details] satellite-messages
Created attachment 1103609 [details] task-export
Created attachment 1103633 [details] /var/log/messages from capsule (previous messages from capsule were duplicate file of satellite's messages)
If the cause is same like in bz1288560, then worth taking some extra logging when the issue reoccurs: https://bugzilla.redhat.com/show_bug.cgi?id=1288560#c4
I have not been able to reproduce this. Also, after review of the node sync logic, it's unclear how this could happen. The sync logic only creates the content unit if the file is successfully downloaded. Specifically to prevent this situation. This raises the possibility that something unrelated to node sync is deleting the productid files. In order to proceed with troubleshooting this, I really need a repeatable reproducer. Or steps to reproduce.
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.
Was this upgraded from 6.0 -> 6.1.
I have reproduced this and determined the root cause. The node sync logic needs to be updated to properly handle changed content units with associated files (such as productid and prestodata).
Yes, it was upgrade from 6.0 to 6.1
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.
This is same fix as https://bugzilla.redhat.com/show_bug.cgi?id=1276911. Please plan to verify both together.
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.
Verified in Satellite 6.1.7 compose. All custom content was successfully sync'd to the capsule.
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2016:0174
The issue persists after update of Satellite and Capsule to 6.1.7 with the erreta RHSA-2016:0174. Syncing the custom product repository didn't help and neither full Capsule sync using hammer.
Created attachment 1129766 [details] capsule foreman-debug after update to 6.1.7
Created attachment 1129771 [details] satellite foreman-debug after update to 6.1.7
Short version for QE: The problem here is that we have a Satellite Server (RHEL 6) and a Capsule (RHEL 7) with several custom YUM repositories pointing to internal repositories. At one point, content from this custom repository was synchronized to the Satellite (and added to content view, published, etc) and successfully synchronized to the external capsule. Later on, a re-sync of the same custom repository brought in newer versions of packages but apparently one or more of these new packages never got synchronized to the capsule. Today we manually walked through the same steps, to see if we could get packages synced to the capsule, but we still don't see the new packages in the capsule. There is a possibility that the sync process to the capsule was terminated before completion and I wonder if the metadata in the capsule is "up to date" and somehow the sync process 'thinks' that it doesn't need to do anything?
Removing the upstream Pulp bug. It is the correct upstream bug but the code is not being backported to 6.1.z.
Comment for consideration: since the fix for this issue requires a newer version of Pulp (see https://pulp.plan.io/issues/1463), I don't think that we can handle this for a z-stream release. I vote for closing it and commenting that the fix will be available for 6.2 only.
If this bug gets moved for sat62, this has been verified here already for sat62. https://bugzilla.redhat.com/show_bug.cgi?id=1315263#c9
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune@redhat.com with any questions
Hi Eric, Any ETA on this patch? its something that will help us (RHEV) move to SAT6 from foreman.
This bug is fixed, tested and verified as resolved in Satellite 6.2 and is unable to be backported into 6.1.z. You can track the bug against Satellite 6.2 here: https://bugzilla.redhat.com/show_bug.cgi?id=1315263