Sayan,

Are the repositories that your content view version is referring to using "complete mirroring" by chance? We cannot regenerate metadata for "complete mirroring" distributions in Pulp, so those get skipped. If you publish a content view with complete mirroring repos and don't use filters, the content view version repositories will also use the complete mirroring distribution. Regenerating the distribution may lose certain repodata, like signatures (?).

If the complete mirroring repodata is corrupt, then I think that would mean the repository itself has corrupt repodata? Since it's literally taken from the upstream repo as far as I'm aware.

I'm making some assumptions here, so let me know if it's not a complete mirroring case.

Hello,

Of course, it's a "Complete Mirroring" scenario as all repos I used are RHEL repos.

About,
~~
Regenerating the distribution may lose certain repodata, like signatures (?).
~~

The only case I am aware of would be if upstream of any repo has the signed-repo data bits. But no RH repo ships that and hardly any customer syncs any Custom repos with Complete Mirroring set. 


Why I filled this BZ because:

A) We often need to use "hammer content-view version republish-repositories" for some disconnected import-related troubleshooting. (Of course, we can say that nothing like that should be expected if everything happens correctly, but not always everything happens correctly ).

B) If you consider https://bugzilla.redhat.com/show_bug.cgi?id=2184278 or some corner cases where pulp artifacts have got corrupted for some reason, The only way we can get the repodata back is by republishing the same.

   Since CV version metadata republishing would be blocked, If we go for a whole new CV version published, That pulls in new content as well and it may not be desirable by the end user since they have already some locked contents in existing CV versions for their patching cycles. 

C) And the behavior has changed only since 6.14 


I can disregard point A ( as that may indicate some issues within the import process itself ) but for me, point B is more concerning. If you have any recommendations for the same, Then let me know  and based on that feel free to close the BZ as well

We can re-allow regenerating the metadata. The only caveat will be that, for users that do regenerate their metadata, they may have seemingly arbitrary kinds of content mirroring for their CVs.  RH repositories will end up using Pulp-generated metadata.

If that's okay, here's my proposed solution:

- Re-allow metadata regenerating for all sorts of content view repositories
- Add a warning that tells users that the repodata will change an signatures will be missing if the Library repo uses complete mirroring

That sounds good to me. If the change would be made, then "5.16. Republishing Repository Metadata" and "5.17. Republishing Content View Metadata" from "Managing Content" guide of Sat 6.14 would need a little bit of modification as well.

But before you implement the change, I would like to know the opinion of some other folks from the support team. I might be over-concerned about Point B of comment 3 and hence it's best to get few other folks to share their point of views. 

@hyu @pmoravec @jpasqual , If you can kindly review and share your opinion about the discussion happened so far, That would be great.

I had a discussion with Grant today and there was some concern that, even though signed metadata isn't used so much right now, it could become popular in the near future with trends towards having more secure environments.

It might be better to have a new Pulp RFE to allow complete mirroring repositories to be refetched from the upstream by Pulp rather than having them regenerated as "content only" Pulp metadata.

I'm a bit indifferent because we would introduce a strong warning in Katello telling users that regenerating metadata will break repodata signatures (and thus their clients) if they're in use.

Maybe we could allow that for now and also have the RFE filed at the same time so that Support can continue fixing environments in the short term and have a longer term fix in the pipeline.

@ggainey how would you feel about that?

We should maybe add documentation or an explanation somewhere that complete_mirroring repos can not have metadata locally republished in pulp to avoid these problems. You can always republish metadata on those repositories after you update your repository to not be mirror_complete anymore which is accurate since the repo doesn't completely mirror the repo metadata.

(In reply to Samir Jha from comment #7)
> We should maybe add documentation or an explanation somewhere that
> complete_mirroring repos can not have metadata locally republished in pulp
> to avoid these problems. 

From a user (or support engineer) perspective, when having any scenario described by Sayan above, what would be the recovery procedure, then? Only changing repo to not mirror_complete and republish metadata? Then we would end up with repos having locally / pulp generated metadata as well. Technically correct (repo type allows it), but still breaking the reason why customer wanted to have mirror_complete.

Having a repo with mirror_complete and regenerating its metadata by either way - once the repo is synced from upstream where a change happened since the repodata regenerate, the repo metadata will be "restored" to mirror_complete feature / restored to be aligned with upstream again - am I right? If so, the scope of the subject/problem is smaller (limited in time until new real-op sync).

I agree with Sayan that we need *some* way in regenerating metadata, that procedure is imho used on daily basis, global wide. *Ideally*, the metadata should not be generated by local pulp, but fetched from the upstream - as Ian suggests the RFE (+1 for it, that feature makes sense alone, when one corrupts artifact files in /var/lib/pulp, e.g.).

It might not be possible to reliably refetch the repo metadata from the upstream repository because upstream repo can update its metadata at any time. We could possibly refetch the metadata like primary, updateinfo and filelist xml of some repos as they may keep a few versions of the old metadata but not the repomd.xml.

I could be wrong about this. Based on my understanding, what customers concern about is whether or not Satellite will verify the repomd.xml with its signature to ensure the integrity repo before syncing. Thus, whether or not the synced local repository has the exactly the same metadata with the upstream repository might not be too concern by them.

In Satellite perspective, I think complete_mirroring is more useful to ensure all capsules are having consistent repository metadata when they are behind a LB.

Hao brings up a good point that, for an old content view version, refetching the upstream metadata doesn't really make sense since it will have changed. If the mirror_complete metadata has broken for a content view version, the only choice is to regenerate it with Pulp as a "content only" mirror.

As such, my thoughts in Comment 4 remain relevant still to my stance.  A user can regenerate it, we just need to warn them that their CVV metadata will not be exactly the same afterwards.

Created redmine issue https://projects.theforeman.org/issues/36638 from this bug

Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/36638 has been resolved.

Verified on: 6.14 - 10

Steps to Verify:
1. Create a CV with a RedHat Repo
2. Sync the repos, and publish the CV
3. Attempt to use the Republish Metadata action through the UI on Repo
4. Attempt to use the Republish Metadata action through the UI on the CV containing the Repo
5. Using hammer commands for both with the '--force true' flag set, attempt to republish the metadata for both

Expected Results:
The UI disallows republishing of Complete Mirroring repos, and content views that contain them. However, you can use the '--force true' flag in the cli to override this.

Actual Results:
The UI disallows republishing of Complete Mirroring repos, and content views that contain them. However, you can use the '--force true' flag in the cli to override this.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.14 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6818