This should not be a duplicated of this similar BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1982542

This should not be a duplicate of this similar BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1982542

I hit something similar but more generic:

- either repo sync or CV publish/promote failing with 400 Bad request
- deeper infestigation found out that either publication or distribution was gone (or both)

To identify such repos and to manually create the missing publication/distribution, I wrote https://access.redhat.com/solutions/6892661 . It is more generic than this BZ, so not sure how much relevant. Just in case somebody hitting the more generic problem would find this bug..

To figure out the root cause of the problem, for anyone who hits this issue, look in the system logs for an API request to Pulp to delete the missing publication. The missing publication can be determined in the foreman console by running `Repository.find(<bad repo ID>).publication_href`.  There may be logs stating that the publication was deleted. If that is found, we should be able to see what in Katello requested that.

I suppose it's also possible that the publications are getting deleted within Pulp itself. In that case there would be no API call.

Regarding my comment above, I just chatted with Pulp folks, and they reminded me that publications will be destroyed if the related repository version is destroyed.  That should also be checked for on machines that are having this issue.

(In reply to Ian Ballou from comment #7)
> Regarding my comment above, I just chatted with Pulp folks, and they
> reminded me that publications will be destroyed if the related repository
> version is destroyed.  That should also be checked for on machines that are
> having this issue.

Thinking loudly (and maybe wrongly): can't some _failed_ task (repo sync, CV publish/promote) trigger this? Failed in some specific step such that old pulp repo version is deleted but new is not created (due to some failure)? (but I *think* katello does not delete pulp repo versions, or does it do so..?)

(In reply to Pavel Moravec from comment #8)
> Thinking loudly (and maybe wrongly): can't some _failed_ task (repo sync, CV
> publish/promote) trigger this? Failed in some specific step such that old
> pulp repo version is deleted but new is not created (due to some failure)?
> (but I *think* katello does not delete pulp repo versions, or does it do
> so..?)

Here are all the places where we destroy repo versions:

1) Orphan cleanup (DeleteOrphanRepositoryVersions task)
  -> Retrieves a list of all repository versions from Pulp and deletes those that are not indexed into Katello repository records.
2) Repository deletion when destroying content view versions.
  -> If `repository.environment.nil?`, which signifies that the repository is the "archived instance".
  -> See Actions::Pulp3::Orchestration::Repository::Delete for the context.

I suppose from this it would be good to check if orphan cleanup was run recently, or if content view versions were deleted recently.

Question for the folks who've seen this issue: when the missing publication error pops up during CV publish, are filters ever involved? I'm going to guess no because when filters are involved we create a new publication anyway at publish time.

FYI in the 03200628 case, where multiple publications have gone, foreman-ssl_access_ssl.log* show just "POST /pulp/api/v3/publications/rpm/rpm/" and no other activity on publications "endpoint".

This suggests some internal pulp activity removed the publications..?

Destroying a repo (or just a repository version) will also destroy any linked publications, which sounds like may be the case.

It's likely that this is noticeable only for 'mirror complete' repositories because, on 6.10 at least, Pulp will not always auto-create a new publication.

Created redmine issue https://projects.theforeman.org/issues/34880 from this bug

Upstream bug assigned to iballou

Upstream bug assigned to iballou

Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/34880 has been resolved.

@bbuckingham Do we have this bug cloned already for 6.11.z since I see the flags are set for 6.11.z?

The SD is looking at QEs to verify and confirm this it works for 6.11.z and 6.10.z !

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.12 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8506