Bug 2022839
| Summary: | CCO occasionally down, reporting networksecurity.googleapis.com API as disabled | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Michael McCune <mimccune> | |
| Component: | Cloud Credential Operator | Assignee: | Joel Diaz <jdiaz> | |
| Status: | CLOSED ERRATA | QA Contact: | Jianping SHu <jshu> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 4.10 | CC: | amagrawa, cblecker, deads, dseals, jdiaz, jialiu, jiwei, jshu, lwan, oarribas, openshift-bugzilla-robot, sdodson, travi, wking | |
| Target Milestone: | --- | Keywords: | FastFix, ServiceDeliveryImpact | |
| Target Release: | 4.7.z | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | 2022838 | |||
| : | 2022840 (view as bug list) | Environment: | ||
| Last Closed: | 2021-12-01 13:35:22 UTC | Type: | --- | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 2022838 | |||
| Bug Blocks: | 2022840 | |||
|
Comment 2
Jianping SHu
2021-11-15 06:32:37 UTC
Verified upgrading cluster from 4.6.49 to 4.7.0-0.nightly-2021-11-12-230709 test steps: 1. enable "Network Security API" on tested gcp project 2. install a cluster with version 4.6.49 and wait for cluster installed, check installation succeed. 3. check co cco status is normal $ oc get co cloud-credential -w NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE cloud-credential 4.6.49 True False False 29m 4. disable "Network Security API" on tested gcp project, then monitor and wait, cco will Degraded(needs about 1 hour) $ oc get co cloud-credential -w NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE cloud-credential 4.6.49 True False False 79m cloud-credential 4.6.49 True True True 83m $ oc logs cloud-credential-operator-5fd4b9d5cb-j6pzz -n openshift-cloud-credential-operator -c cloud-credential-operator | grep "Detected required APIs that are disabled" time="2021-11-15T08:41:31Z" level=warning msg="Detected required APIs that are disabled: [networksecurity.googleapis.com]" actuator=gcp cr=openshift-cloud-credential-operator/openshift-machine-api-gcp time="2021-11-15T08:41:33Z" level=warning msg="Detected required APIs that are disabled: [firebase.googleapis.com storage.googleapis.com]" actuator=gcp cr=openshift-cloud-credential-operator/openshift-image-registry-gcs time="2021-11-15T08:41:35Z" level=warning msg="Detected required APIs that are disabled: [networksecurity.googleapis.com]" actuator=gcp cr=openshift-cloud-credential-operator/openshift-machine-api-gcp time="2021-11-15T08:41:41Z" level=warning msg="Detected required APIs that are disabled: [networksecurity.googleapis.com]" actuator=gcp cr=openshift-cloud-credential-operator/openshift-machine-api-gcp time="2021-11-15T08:41:50Z" level=warning msg="Detected required APIs that are disabled: [networksecurity.googleapis.com]" actuator=gcp cr=openshift-cloud-credential-operator/openshift-machine-api-gcp 5 run upgrade command and wait for upgrade finish $oc adm upgrade --to-image registry.ci.openshift.org/ocp/release:4.7.0-0.nightly-2021-11-12-230709 --allow-explicit-upgrade --force 6 Check upgrade successfully $ ./oc get clusterversion -w NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.6.49 True True 64m Working towards 4.7.0-0.nightly-2021-11-12-230709: 173 of 668 done (25% complete) version 4.6.49 True True 67m Working towards 4.7.0-0.nightly-2021-11-12-230709: 320 of 668 done (47% complete) version 4.6.49 True True 67m Unable to apply 4.7.0-0.nightly-2021-11-12-230709: an unknown error has occurred: MultipleErrors version 4.6.49 True True 68m Working towards 4.7.0-0.nightly-2021-11-12-230709: 659 of 668 done (98% complete) version 4.7.0-0.nightly-2021-11-12-230709 True False 0s Cluster version is 4.7.0-0.nightly-2021-11-12-230709 $ ./oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.0-0.nightly-2021-11-12-230709 True False 57s Cluster version is 4.7.0-0.nightly-2021-11-12-230709 7 Check cco won't degraded ]$ oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.7.0-0.nightly-2021-11-12-230709 True False False 3m31s baremetal 4.7.0-0.nightly-2021-11-12-230709 True False False 40m cloud-credential 4.7.0-0.nightly-2021-11-12-230709 True False False 157m cluster-autoscaler 4.7.0-0.nightly-2021-11-12-230709 True False False 151m config-operator 4.7.0-0.nightly-2021-11-12-230709 True False False 152m console 4.7.0-0.nightly-2021-11-12-230709 True False False 8m39s csi-snapshot-controller 4.7.0-0.nightly-2021-11-12-230709 True False False 13m dns 4.7.0-0.nightly-2021-11-12-230709 True False False 151m etcd 4.7.0-0.nightly-2021-11-12-230709 True False False 150m image-registry 4.7.0-0.nightly-2021-11-12-230709 True False False 144m ingress 4.7.0-0.nightly-2021-11-12-230709 True False False 143m insights 4.7.0-0.nightly-2021-11-12-230709 True False False 153m kube-apiserver 4.7.0-0.nightly-2021-11-12-230709 True False False 150m kube-controller-manager 4.7.0-0.nightly-2021-11-12-230709 True False False 150m kube-scheduler 4.7.0-0.nightly-2021-11-12-230709 True False False 150m kube-storage-version-migrator 4.7.0-0.nightly-2021-11-12-230709 True False False 11m machine-api 4.7.0-0.nightly-2021-11-12-230709 True False False 141m machine-approver 4.7.0-0.nightly-2021-11-12-230709 True False False 152m machine-config 4.7.0-0.nightly-2021-11-12-230709 True False False 3m16s marketplace 4.7.0-0.nightly-2021-11-12-230709 True False False 13m monitoring 4.7.0-0.nightly-2021-11-12-230709 True False False 7m58s network 4.7.0-0.nightly-2021-11-12-230709 True False False 153m node-tuning 4.7.0-0.nightly-2021-11-12-230709 True False False 37m openshift-apiserver 4.7.0-0.nightly-2021-11-12-230709 True False False 3m34s openshift-controller-manager 4.7.0-0.nightly-2021-11-12-230709 True False False 143m openshift-samples 4.7.0-0.nightly-2021-11-12-230709 True False False 37m operator-lifecycle-manager 4.7.0-0.nightly-2021-11-12-230709 True False False 151m operator-lifecycle-manager-catalog 4.7.0-0.nightly-2021-11-12-230709 True False False 151m operator-lifecycle-manager-packageserver 4.7.0-0.nightly-2021-11-12-230709 True False False 5m46s service-ca 4.7.0-0.nightly-2021-11-12-230709 True False False 152m storage 4.7.0-0.nightly-2021-11-12-230709 True False False 13m $ oc logs cloud-credential-operator-78c849bf-7rq6z -n openshift-cloud-credential-operator -c cloud-credential-operator | grep "Detected required APIs that are disabled" Nothing output Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.7.38 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:4802 |