When the delete annotation is added to an ImageStream manifest the ImageStream is created during installation since the check is missing. With the check the manifest is ignored except during upgrade at which time the ImageStream is deleted. However ImageStreams were recently added to resourcebuilder - they were previously handled by the generic handler - so ImageStream delete logic must also be added to resourcebuilder.
Verifying with 4.9.0-0.nightly-2021-07-07-021823 Fresh install a cluster with 4.9 # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.0-0.nightly-2021-07-07-021823 True False 38m Cluster version is 4.9.0-0.nightly-2021-07-07-021823 # oc get is hello-openshift -n openshift Error from server (NotFound): imagestreams.image.openshift.io "hello-openshift" not found Imagestream hello-openshift is not present in the fresh installed cluster. Will continue to verify on cluster upgrade
Verifying with cluster upgrade Install a cluster with 4.7 # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.19 True False 28m Cluster version is 4.7.19 [root@preserve-yangyangmerrn-1 build]# oc get is hello-openshift -n openshift NAME IMAGE REPOSITORY TAGS UPDATED hello-openshift image-registry.openshift-image-registry.svc:5000/openshift/hello-openshift latest 37 minutes ago Imagestream hello-openshift is present, upgrade to 4.8 # oc adm upgrade --to-image=quay.io/openshift-release-dev/ocp-release@sha256:af238ba15d5dcb839e4f0adff5e87d36f4bae096d31f0f7192c726e3a35c203b --allow-explicit-upgrade warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade to the update to proceed anyway Updating to release image quay.io/openshift-release-dev/ocp-release@sha256:af238ba15d5dcb839e4f0adff5e87d36f4bae096d31f0f7192c726e3a35c203b # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.8.0-rc.3 True False 86m Cluster version is 4.8.0-rc.3 [root@preserve-yangyangmerrn-1 build]# oc get is hello-openshift -n openshift NAME IMAGE REPOSITORY TAGS UPDATED hello-openshift image-registry.openshift-image-registry.svc:5000/openshift/hello-openshift latest 3 hours ago Upgrade to 4.9 # oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release@sha256:4e6a1b011084ef13cc344af0fb219cc30855f8fb36797581373e8ef1fc0f0e4e --allow-explicit-upgrade --force warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade to the update to proceed anyway warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures. Updating to release image registry.ci.openshift.org/ocp/release@sha256:4e6a1b011084ef13cc344af0fb219cc30855f8fb36797581373e8ef1fc0f0e4e # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.0-0.nightly-2021-07-07-021823 True False 20m Cluster version is 4.9.0-0.nightly-2021-07-07-021823 # oc get is hello-openshift -n openshift Error from server (NotFound): imagestreams.image.openshift.io "hello-openshift" not found # grep hello /tmp/cvo1.log I0708 13:07:39.086880 1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:07:39.276843 1 request.go:600] Waited for 189.424131ms due to client-side throttling, not priority and fairness, request: GET:https://api-int.yangyang47.qe.gcp.devcluster.openshift.com:6443/apis/image.openshift.io/v1/namespaces/openshift/imagestreams/hello-openshift W0708 13:07:39.285196 1 helper.go:97] imagestream "openshift/hello-openshift" has already been removed. I0708 13:07:39.285234 1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:10:50.814553 1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:10:50.815126 1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:14:22.591794 1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:14:22.592248 1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:17:50.173125 1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:17:50.173599 1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:21:22.050516 1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:21:22.051112 1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:24:49.621019 1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:24:49.621409 1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:28:21.395653 1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:28:21.396133 1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680) Imagestream hello-openshift is deleted after upgrading to 4.9.
Collecting logs for resource deletion Staring tail CVO logs # while sleep 1; do POD="$(oc -n openshift-cluster-version get -o jsonpath='{.items[].metadata.name}{"\n"}' pods)" && oc -n openshift-cluster-version logs -f "${POD}" > "$(date --utc --iso=s)-${POD}.log"; done Upgrading cluster in a separate terminal # oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release@sha256:4e6a1b011084ef13cc344af0fb219cc30855f8fb36797581373e8ef1fc0f0e4e --allow-explicit-upgrade --force warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade to the update to proceed anyway warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures. Updating to release image registry.ci.openshift.org/ocp/release@sha256:4e6a1b011084ef13cc344af0fb219cc30855f8fb36797581373e8ef1fc0f0e4e # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.8.0-rc.3 True True 3m55s Working towards 4.9.0-0.nightly-2021-07-07-021823: 69 of 680 done (10% complete) Grepping logs after upgrade # grep hello * | grep -v sync 2021-07-09T09:20:26+0000-cluster-version-operator-589c75ff89-lbsdv.log:I0709 08:58:12.828152 1 helper.go:65] Delete requested for imagestream "openshift/hello-openshift". 2021-07-09T09:20:26+0000-cluster-version-operator-589c75ff89-lbsdv.log:I0709 09:01:06.521522 1 upgradeable.go:229] Resource deletions in progress; resources=imagestream "openshift/hello-openshift" 2021-07-09T09:20:26+0000-cluster-version-operator-589c75ff89-lbsdv.log:I0709 09:04:05.859680 1 request.go:600] Waited for 139.773637ms due to client-side throttling, not priority and fairness, request: GET:https://api-int.yangyang47.qe.gcp.devcluster.openshift.com:6443/apis/image.openshift.io/v1/namespaces/openshift/imagestreams/hello-openshift 2021-07-09T09:20:26+0000-cluster-version-operator-589c75ff89-lbsdv.log:I0709 09:04:05.870223 1 helper.go:76] Delete of imagestream "openshift/hello-openshift" completed. 2021-07-09T09:20:26+0000-cluster-version-operator-589c75ff89-lbsdv.log:I0709 09:13:12.065457 1 request.go:600] Waited for 179.487575ms due to client-side throttling, not priority and fairness, request: GET:https://api-int.yangyang47.qe.gcp.devcluster.openshift.com:6443/apis/image.openshift.io/v1/namespaces/openshift/imagestreams/hello-openshift 2021-07-09T09:27:22+0000-cluster-version-operator-589c75ff89-c44br.log:I0709 09:38:39.857940 1 request.go:600] Waited for 190.9657ms due to client-side throttling, not priority and fairness, request: GET:https://api-int.yangyang47.qe.gcp.devcluster.openshift.com:6443/apis/image.openshift.io/v1/namespaces/openshift/imagestreams/hello-openshift 2021-07-09T09:27:22+0000-cluster-version-operator-589c75ff89-c44br.log:W0709 09:38:39.867300 1 helper.go:97] imagestream "openshift/hello-openshift" has already been removed. Log messages for deletion are found.
Verifying with minor update to from 4.9 to 4.9 # oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release@sha256:b00684aac99c85e603b940a7fdcdcc2797a23fd2912c2b38cd1d125baa39225a --allow-explicit-upgrade --force warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade to the update to proceed anyway warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures. Updating to release image registry.ci.openshift.org/ocp/release@sha256:b00684aac99c85e603b940a7fdcdcc2797a23fd2912c2b38cd1d125baa39225a # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.0-0.nightly-2021-07-09-002446 True False 7m Cluster version is 4.9.0-0.nightly-2021-07-09-002446 # oc get is hello-openshift -n openshift Error from server (NotFound): imagestreams.image.openshift.io "hello-openshift" not found Imagestream hello-openshift is no present.
Based on above 4 comments, moving it to verified state.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759