Bug 1975539

Summary:

[ImageStreams] Remove stale cruft installed by CVO in earlier releases

Product:

OpenShift Container Platform

Reporter:

Jack Ottofaro <jack.ottofaro>

Component:

Samples

Assignee:

Gabe Montero <gmontero>

Status:

CLOSED ERRATA

QA Contact:

Yang Yang <yanyang>

Severity:

medium

Docs Contact:

Priority:

medium

Version:

4.9

CC:

aos-bugs, jitsingh, jokerman, mfojtik, sttts, xiuwang, xxia, yanyang

Target Milestone:

---

Keywords:

NeedsTestCase

Target Release:

4.9.0

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

1975533

Clones:

2001974 (view as bug list)

Environment:

Last Closed:

2021-10-18 17:36:28 UTC

Type:

---

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

Bug Blocks:

2001974

Attachments:

Description	Flags
Spreadsheet containing leaked resources.	none

Description Jack Ottofaro 2021-06-23 21:31:32 UTC

Created attachment 1793637 [details]
Spreadsheet containing leaked resources.

+++ This bug was initially created as a clone of Bug #1975533 +++

This "stale cruft" is created as a result of the following scenario. Release A had manifest M that lead the CVO to reconcile resource R. But then the component maintainers decided they didn't need R any longer, so they dropped manifest M in release B. The new CVO will no longer reconcile R, but clusters updating from A to B will still have resource R in-cluster, as an unmaintained orphan.

Now that https://issues.redhat.com/browse/OTA-222 has been implemented teams can go back through and create deletion manifests for these leaked resources.

The attachment delete-candidates.csv contains a list of leaked resources as compared to a freshly installed 4.9 cluster. Use this list to find your component's resources and use the manifest delete annotation (https://github.com/openshift/cluster-version-operator/pull/438) to remove them.

Note also that in the case of a cluster-scoped resource it may not need to be removed but simply be modified to remove namespace.

Comment 3 Jack Ottofaro 2021-06-24 14:05:05 UTC

> So if I read
> https://github.com/openshift/cluster-version-operator/blob/master/docs/dev/
> object-deletion.md correctly I need to reinsert that imagestream yaml but
> add the delete annotaion.
> 
Correct

Comment 6 Yang Yang 2021-07-09 01:59:00 UTC

Verifying with 4.9.0-0.nightly-2021-07-07-021823

Fresh install a cluster with 4.9

# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0-0.nightly-2021-07-07-021823   True        False         38m     Cluster version is 4.9.0-0.nightly-2021-07-07-021823


# oc get  is hello-openshift -n openshift
Error from server (NotFound): imagestreams.image.openshift.io "hello-openshift" not found

Imagestream hello-openshift is not present in the fresh installed cluster. Will continue to verify on cluster upgrade

=========================================

Verifying with cluster upgrade

Install a cluster with 4.7
# oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.19    True        False         28m     Cluster version is 4.7.19

[root@preserve-yangyangmerrn-1 build]# oc get is hello-openshift -n openshift
NAME              IMAGE REPOSITORY                                                             TAGS     UPDATED
hello-openshift   image-registry.openshift-image-registry.svc:5000/openshift/hello-openshift   latest   37 minutes ago

Imagestream hello-openshift is present, upgrade to 4.8

# oc adm upgrade --to-image=quay.io/openshift-release-dev/ocp-release@sha256:af238ba15d5dcb839e4f0adff5e87d36f4bae096d31f0f7192c726e3a35c203b --allow-explicit-upgrade 
warning: The requested upgrade image is not one of the available updates.  You have used --allow-explicit-upgrade to the update to proceed anyway
Updating to release image quay.io/openshift-release-dev/ocp-release@sha256:af238ba15d5dcb839e4f0adff5e87d36f4bae096d31f0f7192c726e3a35c203b

# oc get clusterversion
NAME      VERSION      AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-rc.3   True        False         86m     Cluster version is 4.8.0-rc.3

[root@preserve-yangyangmerrn-1 build]# oc get is hello-openshift -n openshift
NAME              IMAGE REPOSITORY                                                             TAGS     UPDATED
hello-openshift   image-registry.openshift-image-registry.svc:5000/openshift/hello-openshift   latest   3 hours ago

Upgrade to 4.9

# oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release@sha256:4e6a1b011084ef13cc344af0fb219cc30855f8fb36797581373e8ef1fc0f0e4e --allow-explicit-upgrade --force
warning: The requested upgrade image is not one of the available updates.  You have used --allow-explicit-upgrade to the update to proceed anyway
warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures.
Updating to release image registry.ci.openshift.org/ocp/release@sha256:4e6a1b011084ef13cc344af0fb219cc30855f8fb36797581373e8ef1fc0f0e4e

# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0-0.nightly-2021-07-07-021823   True        False         20m     Cluster version is 4.9.0-0.nightly-2021-07-07-021823

# oc get is hello-openshift -n openshift
Error from server (NotFound): imagestreams.image.openshift.io "hello-openshift" not found

# grep hello /tmp/cvo1.log
I0708 13:07:39.086880       1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680)
I0708 13:07:39.276843       1 request.go:600] Waited for 189.424131ms due to client-side throttling, not priority and fairness, request: GET:https://api-int.yangyang47.qe.gcp.devcluster.openshift.com:6443/apis/image.openshift.io/v1/namespaces/openshift/imagestreams/hello-openshift
W0708 13:07:39.285196       1 helper.go:97] imagestream "openshift/hello-openshift" has already been removed.
I0708 13:07:39.285234       1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680)
I0708 13:10:50.814553       1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680)
I0708 13:10:50.815126       1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680)
I0708 13:14:22.591794       1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680)
I0708 13:14:22.592248       1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680)
I0708 13:17:50.173125       1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680)
I0708 13:17:50.173599       1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680)
I0708 13:21:22.050516       1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680)
I0708 13:21:22.051112       1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680)
I0708 13:24:49.621019       1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680)
I0708 13:24:49.621409       1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680)
I0708 13:28:21.395653       1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680)
I0708 13:28:21.396133       1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680)

Imagestream hello-openshift is deleted after upgrading to 4.9. 

Moving it to verified state.

Comment 11 errata-xmlrpc 2021-10-18 17:36:28 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759