Bug 1975545

Summary: [MCO] Remove stale cruft installed by CVO in earlier releases
Product: OpenShift Container Platform Reporter: Jack Ottofaro <jack.ottofaro>
Component: Machine Config OperatorAssignee: MCO Bug Bot <mco-triage>
Machine Config Operator sub component: Machine Config Operator QA Contact: Rio Liu <rioliu>
Status: CLOSED WONTFIX Docs Contact:
Severity: unspecified    
Priority: medium CC: mfojtik, mkrejci, rioliu, skumari, sttts, xxia, yanyang
Version: 4.9   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1975533 Environment:
Last Closed: 2023-03-06 20:54:40 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Spreadsheet containing leaked resources. none

Description Jack Ottofaro 2021-06-23 21:41:26 UTC 
Created attachment 1793641 [details]
Spreadsheet containing leaked resources.

+++ This bug was initially created as a clone of Bug #1975533 +++

This "stale cruft" is created as a result of the following scenario. Release A had manifest M that lead the CVO to reconcile resource R. But then the component maintainers decided they didn't need R any longer, so they dropped manifest M in release B. The new CVO will no longer reconcile R, but clusters updating from A to B will still have resource R in-cluster, as an unmaintained orphan.

Now that https://issues.redhat.com/browse/OTA-222 has been implemented teams can go back through and create deletion manifests for these leaked resources.

The attachment delete-candidates.csv contains a list of leaked resources as compared to a freshly installed 4.9 cluster. Use this list to find your component's resources and use the manifest delete annotation (https://github.com/openshift/cluster-version-operator/pull/438) to remove them.

Note also that in the case of a cluster-scoped resource it may not need to be removed but simply be modified to remove namespace.
Comment 1 Yu Qi Zhang 2021-06-23 22:35:20 UTC 
To check my understanding, reading the doc it says:

Developers can remove any of the currently managed CVO objects by modifying an existing manifest and adding the delete annotation `.metadata.annotations["release.openshift.io/delete"]="true"`. This manifest annotation is a request for the CVO to delete the in-cluster object instead of creating/updating it.

What if the manifest is no longer being shipped at all? Do I create a dummy manifest that reintroduces it just to delete it?
Comment 2 Jack Ottofaro 2021-06-24 13:53:24 UTC 
(In reply to Yu Qi Zhang from comment #1)
> To check my understanding, reading the doc it says:
> 
> Developers can remove any of the currently managed CVO objects by modifying
> an existing manifest and adding the delete annotation
> `.metadata.annotations["release.openshift.io/delete"]="true"`. This manifest
> annotation is a request for the CVO to delete the in-cluster object instead
> of creating/updating it.

Correct. But deletes only occur during an upgrade.

> What if the manifest is no longer being shipped at all? Do I create a dummy
> manifest that reintroduces it just to delete it?
Yes.
Comment 6 Sinny Kumari 2023-03-06 20:54:40 UTC 
Thank you for reporting the issue. Due to other higher priority work, We couldn't prioritize this issue. Closing this issue, please open a new bug if this issue leads to undesired cluster behavior.