Bug 1982046
Summary: | CVO gets stuck on resource deletion progress after re-creating the deleted resource | ||
---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Yang Yang <yanyang> |
Component: | Cluster Version Operator | Assignee: | Jack Ottofaro <jack.ottofaro> |
Status: | CLOSED ERRATA | QA Contact: | Yang Yang <yanyang> |
Severity: | medium | Docs Contact: | |
Priority: | medium | ||
Version: | 4.9 | CC: | aos-bugs, jokerman |
Target Milestone: | --- | ||
Target Release: | 4.9.0 | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2021-10-18 17:39:52 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Yang Yang
2021-07-14 04:40:06 UTC
CVO logs are available online https://drive.google.com/file/d/1H8kq1H903_DEZB2XCUaLaF-1Cv72vLXO/view?usp=sharing. From the logs provided I see the delete being requested: I0713 08:34:15.088642 1 helper.go:65] Delete requested for imagestream "openshift/hello-openshift". Assuming this is the one that results after you have manually re-created the imagestream. The last delete related log I see is: I0714 02:34:07.896946 1 upgradeable.go:229] Resource deletions in progress; resources=imagestream "openshift/hello-openshift" So never see the "delete completed" log probably because we are probably no longer in UpdatingMode when the delete finally completes and therefore the delete logic, including checking for delete completion, never runs. That's a bug as it will never clear the Upgradeable=false. Good catch. Thanks! Verifying with 4.9.0-0.nightly-2021-07-26-220837 1. Regression testing on installation passed, imagestream hello-openshift is not created. 2. The issue described in description is fixed. $ oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release@sha256:174648ea613381880c5f75c63e5dac78e330473ad93a530e434866c49e690a56 --allow-explicit-upgrade --force $ oc get is hello-openshift -n openshift Error from server (NotFound): imagestreams.image.openshift.io "hello-openshift" not found $ oc create -f /tmp/hello-openshift.yaml -n openshift imagestream.image.openshift.io/hello-openshift created $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.8.0-rc.3 True True 34m Working towards 4.9.0-0.nightly-2021-07-26-220837: 561 of 716 done (78% complete) $ oc get is hello-openshift -n openshift NAME IMAGE REPOSITORY TAGS UPDATED hello-openshift image-registry.openshift-image-registry.svc:5000/openshift/hello-openshift latest 54 seconds ago $ oc get is hello-openshift -n openshift Error from server (NotFound): imagestreams.image.openshift.io "hello-openshift" not found $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.8.0-rc.3 True True 56m Working towards 4.9.0-0.nightly-2021-07-26-220837: 606 of 716 done (84% complete) $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.0-0.nightly-2021-07-26-220837 True False 2m19s Cluster version is 4.9.0-0.nightly-2021-07-26-220837 $ grep hello-openshift * | grep -v 'throt\|sync' 2021-07-27T12:06:44+0000-cluster-version-operator-686d9dffd9-ssl8b.log:I0727 12:21:16.283680 1 helper.go:65] Delete requested for imagestream "openshift/hello-openshift". 2021-07-27T12:06:44+0000-cluster-version-operator-686d9dffd9-ssl8b.log:I0727 12:22:48.736950 1 upgradeable.go:229] Resource deletions in progress; resources=ConsoleQuickStart "ocs-install-tour",imagestream "openshift/hello-openshift" 2021-07-27T12:06:44+0000-cluster-version-operator-686d9dffd9-ssl8b.log:I0727 12:26:01.297677 1 upgradeable.go:229] Resource deletions in progress; resources=imagestream "openshift/hello-openshift",ConsoleQuickStart "ocs-install-tour" 2021-07-27T12:06:44+0000-cluster-version-operator-686d9dffd9-ssl8b.log:I0727 12:28:26.508633 1 helper.go:116] Delete of imagestream "openshift/hello-openshift" has already been requested. 2021-07-27T12:06:44+0000-cluster-version-operator-686d9dffd9-ssl8b.log:I0727 12:29:13.856989 1 upgradeable.go:229] Resource deletions in progress; resources=imagestream "openshift/hello-openshift" 2021-07-27T12:06:44+0000-cluster-version-operator-686d9dffd9-ssl8b.log:I0727 12:32:26.416845 1 upgradeable.go:229] Resource deletions in progress; resources=imagestream "openshift/hello-openshift" 2021-07-27T12:06:44+0000-cluster-version-operator-686d9dffd9-ssl8b.log:I0727 12:35:38.976630 1 upgradeable.go:229] Resource deletions in progress; resources=imagestream "openshift/hello-openshift" 2021-07-27T12:06:44+0000-cluster-version-operator-686d9dffd9-ssl8b.log:I0727 12:37:16.506803 1 helper.go:116] Delete of imagestream "openshift/hello-openshift" has already been requested. 2021-07-27T12:06:44+0000-cluster-version-operator-686d9dffd9-ssl8b.log:I0727 12:38:51.536174 1 upgradeable.go:229] Resource deletions in progress; resources=imagestream "openshift/hello-openshift" 2021-07-27T12:06:44+0000-cluster-version-operator-686d9dffd9-ssl8b.log:I0727 12:42:04.096542 1 upgradeable.go:229] Resource deletions in progress; resources=imagestream "openshift/hello-openshift" 2021-07-27T12:44:19+0000-cluster-version-operator-686d9dffd9-bq6xg.log:I0727 12:45:23.599269 1 helper.go:65] Delete requested for imagestream "openshift/hello-openshift". 2021-07-27T12:44:19+0000-cluster-version-operator-686d9dffd9-bq6xg.log:I0727 12:48:03.866345 1 upgradeable.go:229] Resource deletions in progress; resources=imagestream "openshift/hello-openshift" 2021-07-27T13:00:47+0000-cluster-version-operator-686d9dffd9-snxxc.log:W0727 13:04:15.126164 1 helper.go:97] imagestream "openshift/hello-openshift" has already been removed. $ oc get clusterversion -ojson | jq -r '.items[].status|select(.type == "Upgradeable")' null CVO logs "imagestream "openshift/hello-openshift" has already been removed" and upgradeable is not blocked by imagestream "openshift/hello-openshift. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759 |