Created attachment 1793637 [details] Spreadsheet containing leaked resources. +++ This bug was initially created as a clone of Bug #1975533 +++ This "stale cruft" is created as a result of the following scenario. Release A had manifest M that lead the CVO to reconcile resource R. But then the component maintainers decided they didn't need R any longer, so they dropped manifest M in release B. The new CVO will no longer reconcile R, but clusters updating from A to B will still have resource R in-cluster, as an unmaintained orphan. Now that https://issues.redhat.com/browse/OTA-222 has been implemented teams can go back through and create deletion manifests for these leaked resources. The attachment delete-candidates.csv contains a list of leaked resources as compared to a freshly installed 4.9 cluster. Use this list to find your component's resources and use the manifest delete annotation (https://github.com/openshift/cluster-version-operator/pull/438) to remove them. Note also that in the case of a cluster-scoped resource it may not need to be removed but simply be modified to remove namespace.
> So if I read > https://github.com/openshift/cluster-version-operator/blob/master/docs/dev/ > object-deletion.md correctly I need to reinsert that imagestream yaml but > add the delete annotaion. > Correct
Verifying with 4.9.0-0.nightly-2021-07-07-021823 Fresh install a cluster with 4.9 # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.0-0.nightly-2021-07-07-021823 True False 38m Cluster version is 4.9.0-0.nightly-2021-07-07-021823 # oc get is hello-openshift -n openshift Error from server (NotFound): imagestreams.image.openshift.io "hello-openshift" not found Imagestream hello-openshift is not present in the fresh installed cluster. Will continue to verify on cluster upgrade ========================================= Verifying with cluster upgrade Install a cluster with 4.7 # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.19 True False 28m Cluster version is 4.7.19 [root@preserve-yangyangmerrn-1 build]# oc get is hello-openshift -n openshift NAME IMAGE REPOSITORY TAGS UPDATED hello-openshift image-registry.openshift-image-registry.svc:5000/openshift/hello-openshift latest 37 minutes ago Imagestream hello-openshift is present, upgrade to 4.8 # oc adm upgrade --to-image=quay.io/openshift-release-dev/ocp-release@sha256:af238ba15d5dcb839e4f0adff5e87d36f4bae096d31f0f7192c726e3a35c203b --allow-explicit-upgrade warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade to the update to proceed anyway Updating to release image quay.io/openshift-release-dev/ocp-release@sha256:af238ba15d5dcb839e4f0adff5e87d36f4bae096d31f0f7192c726e3a35c203b # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.8.0-rc.3 True False 86m Cluster version is 4.8.0-rc.3 [root@preserve-yangyangmerrn-1 build]# oc get is hello-openshift -n openshift NAME IMAGE REPOSITORY TAGS UPDATED hello-openshift image-registry.openshift-image-registry.svc:5000/openshift/hello-openshift latest 3 hours ago Upgrade to 4.9 # oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release@sha256:4e6a1b011084ef13cc344af0fb219cc30855f8fb36797581373e8ef1fc0f0e4e --allow-explicit-upgrade --force warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade to the update to proceed anyway warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures. Updating to release image registry.ci.openshift.org/ocp/release@sha256:4e6a1b011084ef13cc344af0fb219cc30855f8fb36797581373e8ef1fc0f0e4e # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.0-0.nightly-2021-07-07-021823 True False 20m Cluster version is 4.9.0-0.nightly-2021-07-07-021823 # oc get is hello-openshift -n openshift Error from server (NotFound): imagestreams.image.openshift.io "hello-openshift" not found # grep hello /tmp/cvo1.log I0708 13:07:39.086880 1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:07:39.276843 1 request.go:600] Waited for 189.424131ms due to client-side throttling, not priority and fairness, request: GET:https://api-int.yangyang47.qe.gcp.devcluster.openshift.com:6443/apis/image.openshift.io/v1/namespaces/openshift/imagestreams/hello-openshift W0708 13:07:39.285196 1 helper.go:97] imagestream "openshift/hello-openshift" has already been removed. I0708 13:07:39.285234 1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:10:50.814553 1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:10:50.815126 1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:14:22.591794 1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:14:22.592248 1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:17:50.173125 1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:17:50.173599 1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:21:22.050516 1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:21:22.051112 1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:24:49.621019 1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:24:49.621409 1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:28:21.395653 1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:28:21.396133 1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680) Imagestream hello-openshift is deleted after upgrading to 4.9. Moving it to verified state.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759