Bug 2022813
| Summary: | CCO occasionally down, reporting networksecurity.googleapis.com API as disabled | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | OpenShift BugZilla Robot <openshift-bugzilla-robot> | |
| Component: | Cloud Credential Operator | Assignee: | Joel Diaz <jdiaz> | |
| Status: | CLOSED ERRATA | QA Contact: | wang lin <lwan> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 4.10 | CC: | amagrawa, cblecker, deads, dseals, jdiaz, jialiu, jiwei, jshu, lwan, oarribas, rcyriac, sdodson, travi, wking | |
| Target Milestone: | --- | Keywords: | FastFix, ServiceDeliveryImpact | |
| Target Release: | 4.9.z | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 2022838 (view as bug list) | Environment: | ||
| Last Closed: | 2021-11-22 21:47:23 UTC | Type: | --- | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 2021731, 2031858 | |||
| Bug Blocks: | 2022838 | |||
|
Comment 4
wang lin
2021-11-15 05:24:05 UTC
Verified upgrading cluster from 4.9.6 to 4.9.0-0.nightly-2021-11-12-222121 test steps: 1. enable "Network Security API" on tested gcp project 2. install a cluster with version 4.9.6 and wait for cluster installed, check installation succeed. 3. check co cco status is normal $ oc get co cloud-credential -w NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE cloud-credential 4.9.6 True False False 27m cloud-credential 4.9.6 True False False 31m 4. disable "Network Security API" on tested gcp project, then monitor and wait, cco will Degraded(needs about 1 hour) $ oc logs -f cloud-credential-operator-5546f788d8-59wn8 -n openshift-cloud-credential-operator -c cloud-credential-operator | grep "Detected required APIs that are disabled" time="2021-11-15T08:40:50Z" level=warning msg="Detected required APIs that are disabled: [networksecurity.googleapis.com]" actuator=gcp cr=openshift-cloud-credential-operator/openshift-machine-api-gcp $ oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE cloud-credential 4.9.6 True True True 81m 1 of 5 credentials requests are failing to sync. 5 run upgrade command and wait for upgrade finish $oc adm upgrade --to-image registry.ci.openshift.org/ocp/release:4.9.0-0.nightly-2021-11-12-222121 --allow-explicit-upgrade --force 6 Check upgrade successfully $ $ ./oc get clusterversion -w NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.6 True True 54m Working towards 4.9.0-0.nightly-2021-11-12-222121: 1 of 737 done (0% complete) version 4.9.6 True True 54m Working towards 4.9.0-0.nightly-2021-11-12-222121: 3 of 737 done (0% complete) version 4.9.6 True True 54m Working towards 4.9.0-0.nightly-2021-11-12-222121: 4 of 737 done (0% complete) version 4.9.6 True True 54m Working towards 4.9.0-0.nightly-2021-11-12-222121: 5 of 737 done (0% complete) version 4.9.6 True True 54m Working towards 4.9.0-0.nightly-2021-11-12-222121: 6 of 737 done (0% complete) version 4.9.6 True True 54m Working towards 4.9.0-0.nightly-2021-11-12-222121: 9 of 737 done (1% complete) version 4.9.6 True True 54m Working towards 4.9.0-0.nightly-2021-11-12-222121: 204 of 737 done (27% complete) $ ./oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.0-0.nightly-2021-11-12-222121 True False 47s Cluster version is 4.9.0-0.nightly-2021-11-12-222121 7 Check cco won't degraded $ oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE authentication 4.9.0-0.nightly-2021-11-12-222121 True False False 127m baremetal 4.9.0-0.nightly-2021-11-12-222121 True False False 139m cloud-controller-manager 4.9.0-0.nightly-2021-11-12-222121 True False False 143m cloud-credential 4.9.0-0.nightly-2021-11-12-222121 True False False 146m cluster-autoscaler 4.9.0-0.nightly-2021-11-12-222121 True False False 139m config-operator 4.9.0-0.nightly-2021-11-12-222121 True False False 141m console 4.9.0-0.nightly-2021-11-12-222121 True False False 22m csi-snapshot-controller 4.9.0-0.nightly-2021-11-12-222121 True False False 141m dns 4.9.0-0.nightly-2021-11-12-222121 True False False 140m etcd 4.9.0-0.nightly-2021-11-12-222121 True False False 139m image-registry 4.9.0-0.nightly-2021-11-12-222121 True False False 134m ingress 4.9.0-0.nightly-2021-11-12-222121 True False False 133m insights 4.9.0-0.nightly-2021-11-12-222121 True False False 135m kube-apiserver 4.9.0-0.nightly-2021-11-12-222121 True False False 137m kube-controller-manager 4.9.0-0.nightly-2021-11-12-222121 True False False 140m kube-scheduler 4.9.0-0.nightly-2021-11-12-222121 True False False 139m kube-storage-version-migrator 4.9.0-0.nightly-2021-11-12-222121 True False False 11m machine-api 4.9.0-0.nightly-2021-11-12-222121 True False False 137m machine-approver 4.9.0-0.nightly-2021-11-12-222121 True False False 141m machine-config 4.9.0-0.nightly-2021-11-12-222121 True False False 140m marketplace 4.9.0-0.nightly-2021-11-12-222121 True False False 140m monitoring 4.9.0-0.nightly-2021-11-12-222121 True False False 133m network 4.9.0-0.nightly-2021-11-12-222121 True False False 142m node-tuning 4.9.0-0.nightly-2021-11-12-222121 True False False 10m openshift-apiserver 4.9.0-0.nightly-2021-11-12-222121 True False False 135m openshift-controller-manager 4.9.0-0.nightly-2021-11-12-222121 True False False 36m openshift-samples 4.9.0-0.nightly-2021-11-12-222121 True False False 37m operator-lifecycle-manager 4.9.0-0.nightly-2021-11-12-222121 True False False 141m operator-lifecycle-manager-catalog 4.9.0-0.nightly-2021-11-12-222121 True False False 140m operator-lifecycle-manager-packageserver 4.9.0-0.nightly-2021-11-12-222121 True False False 135m service-ca 4.9.0-0.nightly-2021-11-12-222121 True False False 141m storage 4.9.0-0.nightly-2021-11-12-222121 True False False 140m $ oc get pods -n openshift-cloud-credential-operator NAME READY STATUS RESTARTS AGE cloud-credential-operator-6449f4c6b9-zhnng 2/2 Running 0 17m $ oc logs cloud-credential-operator-6449f4c6b9-zhnng -n openshift-cloud-credential-operator -c cloud-credential-operator | grep "Detected required APIs that are disabled" Nothing output Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.9.8 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:4712 |