Bug 1976307
Summary: | CVO missing ImageStreams manifest delete annotation logic | ||
---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Jack Ottofaro <jack.ottofaro> |
Component: | Cluster Version Operator | Assignee: | Jack Ottofaro <jack.ottofaro> |
Status: | CLOSED ERRATA | QA Contact: | Yang Yang <yanyang> |
Severity: | medium | Docs Contact: | |
Priority: | unspecified | ||
Version: | 4.9 | CC: | aos-bugs, gmontero, jokerman |
Target Milestone: | --- | ||
Target Release: | 4.9.0 | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2021-10-18 17:36:53 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Jack Ottofaro
2021-06-25 17:55:38 UTC
Verifying with 4.9.0-0.nightly-2021-07-07-021823 Fresh install a cluster with 4.9 # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.0-0.nightly-2021-07-07-021823 True False 38m Cluster version is 4.9.0-0.nightly-2021-07-07-021823 # oc get is hello-openshift -n openshift Error from server (NotFound): imagestreams.image.openshift.io "hello-openshift" not found Imagestream hello-openshift is not present in the fresh installed cluster. Will continue to verify on cluster upgrade Verifying with cluster upgrade Install a cluster with 4.7 # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.19 True False 28m Cluster version is 4.7.19 [root@preserve-yangyangmerrn-1 build]# oc get is hello-openshift -n openshift NAME IMAGE REPOSITORY TAGS UPDATED hello-openshift image-registry.openshift-image-registry.svc:5000/openshift/hello-openshift latest 37 minutes ago Imagestream hello-openshift is present, upgrade to 4.8 # oc adm upgrade --to-image=quay.io/openshift-release-dev/ocp-release@sha256:af238ba15d5dcb839e4f0adff5e87d36f4bae096d31f0f7192c726e3a35c203b --allow-explicit-upgrade warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade to the update to proceed anyway Updating to release image quay.io/openshift-release-dev/ocp-release@sha256:af238ba15d5dcb839e4f0adff5e87d36f4bae096d31f0f7192c726e3a35c203b # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.8.0-rc.3 True False 86m Cluster version is 4.8.0-rc.3 [root@preserve-yangyangmerrn-1 build]# oc get is hello-openshift -n openshift NAME IMAGE REPOSITORY TAGS UPDATED hello-openshift image-registry.openshift-image-registry.svc:5000/openshift/hello-openshift latest 3 hours ago Upgrade to 4.9 # oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release@sha256:4e6a1b011084ef13cc344af0fb219cc30855f8fb36797581373e8ef1fc0f0e4e --allow-explicit-upgrade --force warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade to the update to proceed anyway warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures. Updating to release image registry.ci.openshift.org/ocp/release@sha256:4e6a1b011084ef13cc344af0fb219cc30855f8fb36797581373e8ef1fc0f0e4e # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.0-0.nightly-2021-07-07-021823 True False 20m Cluster version is 4.9.0-0.nightly-2021-07-07-021823 # oc get is hello-openshift -n openshift Error from server (NotFound): imagestreams.image.openshift.io "hello-openshift" not found # grep hello /tmp/cvo1.log I0708 13:07:39.086880 1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:07:39.276843 1 request.go:600] Waited for 189.424131ms due to client-side throttling, not priority and fairness, request: GET:https://api-int.yangyang47.qe.gcp.devcluster.openshift.com:6443/apis/image.openshift.io/v1/namespaces/openshift/imagestreams/hello-openshift W0708 13:07:39.285196 1 helper.go:97] imagestream "openshift/hello-openshift" has already been removed. I0708 13:07:39.285234 1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:10:50.814553 1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:10:50.815126 1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:14:22.591794 1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:14:22.592248 1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:17:50.173125 1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:17:50.173599 1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:21:22.050516 1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:21:22.051112 1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:24:49.621019 1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:24:49.621409 1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:28:21.395653 1 sync_worker.go:753] Running sync for imagestream "openshift/hello-openshift" (394 of 680) I0708 13:28:21.396133 1 sync_worker.go:765] Done syncing for imagestream "openshift/hello-openshift" (394 of 680) Imagestream hello-openshift is deleted after upgrading to 4.9. Collecting logs for resource deletion Staring tail CVO logs # while sleep 1; do POD="$(oc -n openshift-cluster-version get -o jsonpath='{.items[].metadata.name}{"\n"}' pods)" && oc -n openshift-cluster-version logs -f "${POD}" > "$(date --utc --iso=s)-${POD}.log"; done Upgrading cluster in a separate terminal # oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release@sha256:4e6a1b011084ef13cc344af0fb219cc30855f8fb36797581373e8ef1fc0f0e4e --allow-explicit-upgrade --force warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade to the update to proceed anyway warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures. Updating to release image registry.ci.openshift.org/ocp/release@sha256:4e6a1b011084ef13cc344af0fb219cc30855f8fb36797581373e8ef1fc0f0e4e # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.8.0-rc.3 True True 3m55s Working towards 4.9.0-0.nightly-2021-07-07-021823: 69 of 680 done (10% complete) Grepping logs after upgrade # grep hello * | grep -v sync 2021-07-09T09:20:26+0000-cluster-version-operator-589c75ff89-lbsdv.log:I0709 08:58:12.828152 1 helper.go:65] Delete requested for imagestream "openshift/hello-openshift". 2021-07-09T09:20:26+0000-cluster-version-operator-589c75ff89-lbsdv.log:I0709 09:01:06.521522 1 upgradeable.go:229] Resource deletions in progress; resources=imagestream "openshift/hello-openshift" 2021-07-09T09:20:26+0000-cluster-version-operator-589c75ff89-lbsdv.log:I0709 09:04:05.859680 1 request.go:600] Waited for 139.773637ms due to client-side throttling, not priority and fairness, request: GET:https://api-int.yangyang47.qe.gcp.devcluster.openshift.com:6443/apis/image.openshift.io/v1/namespaces/openshift/imagestreams/hello-openshift 2021-07-09T09:20:26+0000-cluster-version-operator-589c75ff89-lbsdv.log:I0709 09:04:05.870223 1 helper.go:76] Delete of imagestream "openshift/hello-openshift" completed. 2021-07-09T09:20:26+0000-cluster-version-operator-589c75ff89-lbsdv.log:I0709 09:13:12.065457 1 request.go:600] Waited for 179.487575ms due to client-side throttling, not priority and fairness, request: GET:https://api-int.yangyang47.qe.gcp.devcluster.openshift.com:6443/apis/image.openshift.io/v1/namespaces/openshift/imagestreams/hello-openshift 2021-07-09T09:27:22+0000-cluster-version-operator-589c75ff89-c44br.log:I0709 09:38:39.857940 1 request.go:600] Waited for 190.9657ms due to client-side throttling, not priority and fairness, request: GET:https://api-int.yangyang47.qe.gcp.devcluster.openshift.com:6443/apis/image.openshift.io/v1/namespaces/openshift/imagestreams/hello-openshift 2021-07-09T09:27:22+0000-cluster-version-operator-589c75ff89-c44br.log:W0709 09:38:39.867300 1 helper.go:97] imagestream "openshift/hello-openshift" has already been removed. Log messages for deletion are found. Verifying with minor update to from 4.9 to 4.9 # oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release@sha256:b00684aac99c85e603b940a7fdcdcc2797a23fd2912c2b38cd1d125baa39225a --allow-explicit-upgrade --force warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade to the update to proceed anyway warning: --force overrides cluster verification of your supplied release image and waives any update precondition failures. Updating to release image registry.ci.openshift.org/ocp/release@sha256:b00684aac99c85e603b940a7fdcdcc2797a23fd2912c2b38cd1d125baa39225a # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.0-0.nightly-2021-07-09-002446 True False 7m Cluster version is 4.9.0-0.nightly-2021-07-09-002446 # oc get is hello-openshift -n openshift Error from server (NotFound): imagestreams.image.openshift.io "hello-openshift" not found Imagestream hello-openshift is no present. Based on above 4 comments, moving it to verified state. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759 |