Bug 2098271

> Once they are re-synced, it will work fine.

A force-sync on the capsule, given that as mentioned the capsules might be optimizing out the syncs.

The reason repositories in Library are not getting synced on Capsules, after the very first sync of that repository on Satellite, is that the smart_proxy_sync_history used by the 'optimized' sync method is not getting cleared like it should:

# sudo -u postgres psql -d foreman -c 'select * from katello_smart_proxy_sync_history where repository_id = 1 and smart_proxy_id = 2;'
could not change directory to "/root": Permission denied
 id | smart_proxy_id | repository_id |         started_at         |        finished_at         
----+----------------+---------------+----------------------------+----------------------------
 23 |              2 |             1 | 2022-06-21 19:52:04.032127 | 2022-06-21 20:07:31.965945
(1 row)

The reason the smart_proxy_sync_history doesn't get cleared like it should, is that this only happens when Actions::Pulp3::Orchestration::Repository::GenerateMetadata is called with `contents_changed => true`. For Repository syncs in Library (unlike in CV publish/promote), the value of contents_changed should come from the output of Actions::Pulp3::Repository::SaveVersion, but there is a bug on Satellite 6.10 where regardless of the actual output value, Actions::Pulp3::Repository::SaveVersion always passes contents_changed => nil to the GenerateMetadata step. A hotfix[1] is available for this bug.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=2089580#c12

However, installing the hotfix will not necessarily cause the next repository sync on Satellite to immediately trigger repository syncs on Capsules, unless the sync from the upstream repository downloaded new packages/metadata, because passing (correctly) `contents_changed => false` will not clear the smart_proxy_sync_history. Depending on the urgency, the following approaches may be taken.

I. PASSIVE APPROACH

  1. install the hotfix linked above.
  2. wait for a repository sync from the upstream repository that includes some changes.
  3. when the sync on Satellite detects new changes on the upstream repository, it will trigger syncs on Capsules (that have the Library environment assigned).

II. ACTIVE APPROACH (Specific Repos Only)

  1. install the hotfix linked above (to prevent the issue from occurring when the repositories are synced with new changes in the future).
  2. use `hammer repository list --organization-id 1` (substitute the ID for your organization on Satellite if other than 1) and get a list of the IDs for repositories which need to be synced in the next optimized sync on each Capsule
  3. download the script to reset smart proxy sync histories:

    # wget -O reset_sync_histories.rb https://gist.githubusercontent.com/wbclark/489781bc4d146e554c1103c72e934c2a/raw/c0bc82b432e64daf2272d32db77f0d14ee677129/clear_smart_proxy_sync_histories.rb

  4. edit the list of repository_ids on line 26 to match your list from step II.2 above
  5. run the script using the foreman-rake console:

    # cat reset_sync_histories.rb |foreman-rake console

  6. run 'Optimized' Capsule syncs for each of the affected Capsules, and notice the repositories which had their smart_proxy_sync_histories cleared are included in this optimized sync

III. ACTIVE APPROACH (Blanket all Repositories in the Library Environment, which is still faster than a 'Complete' sync that includes other LCEs than Library)

  1. install the hotfix linked above (to prevent the issue from occurring when the repositories are synced with new changes in the future).
  2. clear smart proxy sync histories for all Library repos with this command:

    # echo "Katello::RootRepository.all.map { |repo| repo.id }.each { |repo_id| Katello::Repository.find(repo_id).clear_smart_proxy_sync_histories }" |foreman-rake console

  3. run 'Optimized' syncs of the affected Capsules



The third approach (III) is the recommended approach to assure immediate correctness of all Library repos on all Capsules.

Amendment to step III.2. from comment #15 above:

The correct script should instead be this instead, which does not rely on the faulty assumption of a correspondence between Katello::Repository and Katello::RootRepository ids:

III. ACTIVE APPROACH (Blanket all Repositories in the Library Environment, which is still faster than a 'Complete' sync that includes other LCEs than Library)

  1. install the hotfix linked above (to prevent the issue from occurring when the repositories are synced with new changes in the future).
  2. clear smart proxy sync histories for all Library repos with this command:

    # echo "Katello::Repository.where(content_view_version_id: 1).each(&:clear_smart_proxy_sync_histories)" |foreman-rake console

  3. run 'Optimized' syncs of the affected Capsules