Bug 1975539 - [ImageStreams] Remove stale cruft installed by CVO in earlier releases
Summary: [ImageStreams] Remove stale cruft installed by CVO in earlier releases
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Samples
Version: 4.9
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.9.0
Assignee: Gabe Montero
QA Contact: Yang Yang
URL:
Whiteboard:
Depends On:
Blocks: 2001974
TreeView+ depends on / blocked
 
Reported: 2021-06-23 21:31 UTC by Jack Ottofaro
Modified: 2021-10-18 17:36 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1975533
: 2001974 (view as bug list)
Environment:
Last Closed: 2021-10-18 17:36:28 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Spreadsheet containing leaked resources. (11.48 KB, text/plain)
2021-06-23 21:31 UTC, Jack Ottofaro
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-samples-operator pull 380 0 None open Bug 1975539: delete hello-openshift in payload imagestream via CVO annotation 2021-06-24 14:09:55 UTC
Red Hat Product Errata RHSA-2021:3759 0 None None None 2021-10-18 17:36:42 UTC

Description Jack Ottofaro 2021-06-23 21:31:32 UTC
Created attachment 1793637 [details]
Spreadsheet containing leaked resources.

+++ This bug was initially created as a clone of Bug #1975533 +++

This "stale cruft" is created as a result of the following scenario. Release A had manifest M that lead the CVO to reconcile resource R. But then the component maintainers decided they didn't need R any longer, so they dropped manifest M in release B. The new CVO will no longer reconcile R, but clusters updating from A to B will still have resource R in-cluster, as an unmaintained orphan.

Now that https://issues.redhat.com/browse/OTA-222 has been implemented teams can go back through and create deletion manifests for these leaked resources.

The attachment delete-candidates.csv contains a list of leaked resources as compared to a freshly installed 4.9 cluster. Use this list to find your component's resources and use the manifest delete annotation (https://github.com/openshift/cluster-version-operator/pull/438) to remove them.

Note also that in the case of a cluster-scoped resource it may not need to be removed but simply be modified to remove namespace.

Comment 3 Jack Ottofaro 2021-06-24 14:05:05 UTC
> So if I read
> https://github.com/openshift/cluster-version-operator/blob/master/docs/dev/
> object-deletion.md correctly I need to reinsert that imagestream yaml but
> add the delete annotaion.
> 
Correct

Comment 6 Yang Yang 2021-07-09 01:59:00 UTC
Verifying with 4.9.0-0.nightly-2021-07-07-021823

Fresh install a cluster with 4.9

# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0-0.nightly-2021-07-07-021823   True        False         38m     Cluster version is 4.9.0-0.nightly-2021-07-07-021823


# oc get  is hello-openshift -n openshift
Error from server (NotFound): imagestreams.image.openshift.io "hello-openshift" not found

Imagestream hello-openshift is not present in the fresh installed cluster. Will continue to verify on cluster upgrade

=========================================

Verifying with cluster upgrade

Install a cluster with 4.7
# oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.19    True        False         28m     Cluster version is 4.7.19

[root@preserve-yangyangmerrn-1 build]# oc get is hello-openshift -n openshift
NAME              IMAGE REPOSITORY                                                             TAGS     UPDATED
hello-openshift   image-registry.openshift-image-registry.svc:5000/openshift/hello-openshift   latest   37 minutes ago

Imagestream hello-openshift is present, upgrade to 4.8

# oc adm upgrade --to-image=quay.io/openshift-release-dev/ocp-release@sha256:af238ba15d5dcb839e4f0adff5e87d36f4bae096d31f0f7192c726e3a35c203b --allow-explicit-upgrade 
warning: The requested upgrade image is not one of the available updates.  You have used --allow-explicit-upgrade to the update to proceed anyway
Updating to release image quay.io/openshift-release-dev/ocp-release@sha256:af238ba15d5dcb839e4f0adff5e87d36f4bae096d31f0f7192c726e3a35c203b

# oc get clusterversion
NAME      VERSION      AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-rc.3   True        False         86m     Cluster version is 4.8.0-rc.3

[root@preserve-yangyangmerrn-1 build]# oc get is hello-openshift -n openshift
NAME              IMAGE REPOSITORY                                                             TAGS     UPDATED
hello-openshift   image-registry.openshift-image-registry.svc:5000/openshift/hello-openshift   latest   3 hours ago

Upgrade to 4.9

# oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release@sha256:4e6a1b011084ef13cc344af0fb219cc30855f8fb36797581373e8ef1fc0f0e4e --allow-explicit-upgrade --force
warning: The requested upgrade image is not one of the available updates.  You have used --allow-explicit-upgrade to the update to proceed anyway
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
Updating to release image registry.ci.openshift.org/ocp/release@sha256:4e6a1b011084ef13cc344af0fb219cc30855f8fb36797581373e8ef1fc0f0e4e

# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0-0.nightly-2021-07-07-021823   True        False         20m     Cluster version is 4.9.0-0.nightly-2021-07-07-021823

# oc get is hello-openshift -n openshift
Error from server (NotFound): imagestreams.image.openshift.io "hello-openshift" not found

# grep hello /tmp/cvo1.log
I0708 13:07:39.086880       1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680)
I0708 13:07:39.276843       1 request.go:600] Waited for 189.424131ms due to client-side throttling, not priority and fairness, request: GET:https://api-int.yangyang47.qe.gcp.devcluster.openshift.com:6443/apis/image.openshift.io/v1/namespaces/openshift/imagestreams/hello-openshift
W0708 13:07:39.285196       1 helper.go:97] imagestream "openshift/hello-openshift" has already been removed.
I0708 13:07:39.285234       1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680)
I0708 13:10:50.814553       1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680)
I0708 13:10:50.815126       1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680)
I0708 13:14:22.591794       1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680)
I0708 13:14:22.592248       1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680)
I0708 13:17:50.173125       1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680)
I0708 13:17:50.173599       1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680)
I0708 13:21:22.050516       1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680)
I0708 13:21:22.051112       1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680)
I0708 13:24:49.621019       1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680)
I0708 13:24:49.621409       1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680)
I0708 13:28:21.395653       1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680)
I0708 13:28:21.396133       1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680)

Imagestream hello-openshift is deleted after upgrading to 4.9. 

Moving it to verified state.

Comment 11 errata-xmlrpc 2021-10-18 17:36:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759


Note You need to log in before you can comment on or make changes to this bug.