Description of problem: If a repository's publication is somehow missing in Pulp, syncing a repository that has "Mirror on Sync" set to true can fail with the above error. This is due to the fact the pulp_rpm 3.14 does not recreate publications at sync time if a new repository version is not generated. This is not an issue for Satellite versions running pulp_rpm 3.16. There's an issue in Pulp to fix this, but the backport is reportedly very difficult (and hence risky): https://github.com/pulp/pulp_rpm/issues/2399 A workaround in Satellite exists: 1) Set the repository's "Mirror on Sync" option to false 2) There will be an error when changing this, so skip the failing RefreshDistribution step in the Dynflow task for updating the repository. 3) Run a complete sync on the repository after the update task finishes. The repository should now be fixed. The Mirror on Sync option can be turned back on if necessary. Version-Release number of selected component (if applicable): Satellote 6.10.2 How reproducible: Every time Steps to Reproduce: 1. Create and sync a repo 2. Destroy the repo's publication via the Pulp API 3. Try to sync the repository Actual results: The sync fails with the error in the title. Expected results: The sync should succeed. Additional info: Granted that there's a workaround, and publications should not be disappearing out of nowhere, I would consider this a low-priority.
This should not be a duplicated of this similar BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1982542
This should not be a duplicate of this similar BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1982542
I hit something similar but more generic: - either repo sync or CV publish/promote failing with 400 Bad request - deeper infestigation found out that either publication or distribution was gone (or both) To identify such repos and to manually create the missing publication/distribution, I wrote https://access.redhat.com/solutions/6892661 . It is more generic than this BZ, so not sure how much relevant. Just in case somebody hitting the more generic problem would find this bug..
To figure out the root cause of the problem, for anyone who hits this issue, look in the system logs for an API request to Pulp to delete the missing publication. The missing publication can be determined in the foreman console by running `Repository.find(<bad repo ID>).publication_href`. There may be logs stating that the publication was deleted. If that is found, we should be able to see what in Katello requested that. I suppose it's also possible that the publications are getting deleted within Pulp itself. In that case there would be no API call.
Regarding my comment above, I just chatted with Pulp folks, and they reminded me that publications will be destroyed if the related repository version is destroyed. That should also be checked for on machines that are having this issue.
(In reply to Ian Ballou from comment #7) > Regarding my comment above, I just chatted with Pulp folks, and they > reminded me that publications will be destroyed if the related repository > version is destroyed. That should also be checked for on machines that are > having this issue. Thinking loudly (and maybe wrongly): can't some _failed_ task (repo sync, CV publish/promote) trigger this? Failed in some specific step such that old pulp repo version is deleted but new is not created (due to some failure)? (but I *think* katello does not delete pulp repo versions, or does it do so..?)
(In reply to Pavel Moravec from comment #8) > Thinking loudly (and maybe wrongly): can't some _failed_ task (repo sync, CV > publish/promote) trigger this? Failed in some specific step such that old > pulp repo version is deleted but new is not created (due to some failure)? > (but I *think* katello does not delete pulp repo versions, or does it do > so..?) Here are all the places where we destroy repo versions: 1) Orphan cleanup (DeleteOrphanRepositoryVersions task) -> Retrieves a list of all repository versions from Pulp and deletes those that are not indexed into Katello repository records. 2) Repository deletion when destroying content view versions. -> If `repository.environment.nil?`, which signifies that the repository is the "archived instance". -> See Actions::Pulp3::Orchestration::Repository::Delete for the context. I suppose from this it would be good to check if orphan cleanup was run recently, or if content view versions were deleted recently.
Question for the folks who've seen this issue: when the missing publication error pops up during CV publish, are filters ever involved? I'm going to guess no because when filters are involved we create a new publication anyway at publish time.
FYI in the 03200628 case, where multiple publications have gone, foreman-ssl_access_ssl.log* show just "POST /pulp/api/v3/publications/rpm/rpm/" and no other activity on publications "endpoint". This suggests some internal pulp activity removed the publications..?
Destroying a repo (or just a repository version) will also destroy any linked publications, which sounds like may be the case. It's likely that this is noticeable only for 'mirror complete' repositories because, on 6.10 at least, Pulp will not always auto-create a new publication.
Created redmine issue https://projects.theforeman.org/issues/34880 from this bug
Upstream bug assigned to iballou
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/34880 has been resolved.
@bbuckingham Do we have this bug cloned already for 6.11.z since I see the flags are set for 6.11.z? The SD is looking at QEs to verify and confirm this it works for 6.11.z and 6.10.z !
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Satellite 6.12 Release), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:8506