Bug 2022839 - CCO occasionally down, reporting networksecurity.googleapis.com API as disabled
Summary: CCO occasionally down, reporting networksecurity.googleapis.com API as disabled
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cloud Credential Operator
Version: 4.10
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.7.z
Assignee: Joel Diaz
QA Contact: Jianping SHu
URL:
Whiteboard:
Depends On: 2022838
Blocks: 2022840
TreeView+ depends on / blocked
 
Reported: 2021-11-12 17:41 UTC by Michael McCune
Modified: 2021-12-01 13:35 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 2022838
: 2022840 (view as bug list)
Environment:
Last Closed: 2021-12-01 13:35:22 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift machine-api-operator pull 954 0 None open Bug 2022839: GCP CI runs are complaining about APIs not being enabled 2021-11-12 18:10:22 UTC
Red Hat Product Errata RHBA-2021:4802 0 None None None 2021-12-01 13:35:43 UTC

Comment 2 Jianping SHu 2021-11-15 06:32:37 UTC
Verified on 4.7.0-0.nightly-2021-11-12-230709 include the fix.

1. Launch a basic gcp cluster
2. Monitor the installation process

the installaton can succeed and cco won't hit the issue about "Detected required APIs that are disabled: [networksecurity.googleapis.com]"

jianpingshu@jshu-mac 2022839_4.7 % oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.nightly-2021-11-12-230709   True        False         4m6s    Cluster version is 4.7.0-0.nightly-2021-11-12-230709

jianpingshu@jshu-mac 2022839_4.7 % oc logs cloud-credential-operator-78c849bf-w6hn7 -n openshift-cloud-credential-operator -c cloud-credential-operator | grep "Detected required APIs that are disabled"
Nothing output

############
The payload without the fix merged like 4.9.0-0.nightly-2021-11-11-155043 will fail to install, and cco Degraded because of [networksecurity.googleapis.com] disabled.

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version             False       True          53m     Unable to apply 4.9.0-0.nightly-2021-11-11-155043: some cluster operators have not yet rolled out

$ oc logs cloud-credential-operator-5b97f67944-qp6k2 -n openshift-cloud-credential-operator -c cloud-credential-operator | grep "Detected required APIs that are disabled"
time="2021-11-15T04:33:13Z" level=warning msg="Detected required APIs that are disabled: [networksecurity.googleapis.com]" actuator=gcp cr=openshift-cloud-credential-operator/openshift-machine-api-gcp
time="2021-11-15T04:33:17Z" level=warning msg="Detected required APIs that are disabled: [networksecurity.googleapis.com]" actuator=gcp cr=openshift-cloud-credential-operator/openshift-machine-api-gcp
time="2021-11-15T04:33:22Z" level=warning msg="Detected required APIs that are disabled: [networksecurity.googleapis.com]" actuator=gcp cr=openshift-cloud-credential-operator/openshift-machine-api-gcp
time="2021-11-15T04:33:31Z" level=warning msg="Detected required APIs that are disabled: [networksecurity.googleapis.com]" actuator=gcp cr=openshift-cloud-credential-operator/openshift-machine-api-gcp
time="2021-11-15T04:33:49Z" level=warning msg="Detected required APIs that are disabled: [networksecurity.googleapis.com]" actuator=gcp cr=openshift-cloud-credential-operator/openshift-machine-api-gcp

Comment 3 wang lin 2021-11-15 10:14:36 UTC
Verified upgrading cluster from 4.6.49 to 4.7.0-0.nightly-2021-11-12-230709

test steps:
1. enable "Network Security API" on tested gcp project

2. install a cluster with version 4.6.49 and wait for cluster installed, check installation succeed.

3. check co cco status is normal
$ oc get co cloud-credential -w
NAME               VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
cloud-credential   4.6.49    True        False         False      29m

4. disable "Network Security API" on tested gcp project, then monitor and wait, cco will Degraded(needs about 1 hour)
$ oc get co cloud-credential -w
NAME               VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
cloud-credential   4.6.49    True        False         False      79m
cloud-credential   4.6.49    True        True          True       83m
$ oc logs cloud-credential-operator-5fd4b9d5cb-j6pzz -n openshift-cloud-credential-operator  -c cloud-credential-operator | grep "Detected required APIs that are disabled"
time="2021-11-15T08:41:31Z" level=warning msg="Detected required APIs that are disabled: [networksecurity.googleapis.com]" actuator=gcp cr=openshift-cloud-credential-operator/openshift-machine-api-gcp
time="2021-11-15T08:41:33Z" level=warning msg="Detected required APIs that are disabled: [firebase.googleapis.com storage.googleapis.com]" actuator=gcp cr=openshift-cloud-credential-operator/openshift-image-registry-gcs
time="2021-11-15T08:41:35Z" level=warning msg="Detected required APIs that are disabled: [networksecurity.googleapis.com]" actuator=gcp cr=openshift-cloud-credential-operator/openshift-machine-api-gcp
time="2021-11-15T08:41:41Z" level=warning msg="Detected required APIs that are disabled: [networksecurity.googleapis.com]" actuator=gcp cr=openshift-cloud-credential-operator/openshift-machine-api-gcp
time="2021-11-15T08:41:50Z" level=warning msg="Detected required APIs that are disabled: [networksecurity.googleapis.com]" actuator=gcp cr=openshift-cloud-credential-operator/openshift-machine-api-gcp

5 run upgrade command and wait for upgrade finish
$oc adm upgrade --to-image registry.ci.openshift.org/ocp/release:4.7.0-0.nightly-2021-11-12-230709 --allow-explicit-upgrade --force

6 Check upgrade successfully
$ ./oc get clusterversion -w
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.49    True        True          64m     Working towards 4.7.0-0.nightly-2021-11-12-230709: 173 of 668 done (25% complete)
version   4.6.49    True        True          67m     Working towards 4.7.0-0.nightly-2021-11-12-230709: 320 of 668 done (47% complete)
version   4.6.49    True        True          67m     Unable to apply 4.7.0-0.nightly-2021-11-12-230709: an unknown error has occurred: MultipleErrors
version   4.6.49    True        True          68m     Working towards 4.7.0-0.nightly-2021-11-12-230709: 659 of 668 done (98% complete)
version   4.7.0-0.nightly-2021-11-12-230709   True        False         0s      Cluster version is 4.7.0-0.nightly-2021-11-12-230709
$ ./oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.nightly-2021-11-12-230709   True        False         57s     Cluster version is 4.7.0-0.nightly-2021-11-12-230709

7 Check cco won't degraded
]$ oc get co
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.7.0-0.nightly-2021-11-12-230709   True        False         False      3m31s
baremetal                                  4.7.0-0.nightly-2021-11-12-230709   True        False         False      40m
cloud-credential                           4.7.0-0.nightly-2021-11-12-230709   True        False         False      157m
cluster-autoscaler                         4.7.0-0.nightly-2021-11-12-230709   True        False         False      151m
config-operator                            4.7.0-0.nightly-2021-11-12-230709   True        False         False      152m
console                                    4.7.0-0.nightly-2021-11-12-230709   True        False         False      8m39s
csi-snapshot-controller                    4.7.0-0.nightly-2021-11-12-230709   True        False         False      13m
dns                                        4.7.0-0.nightly-2021-11-12-230709   True        False         False      151m
etcd                                       4.7.0-0.nightly-2021-11-12-230709   True        False         False      150m
image-registry                             4.7.0-0.nightly-2021-11-12-230709   True        False         False      144m
ingress                                    4.7.0-0.nightly-2021-11-12-230709   True        False         False      143m
insights                                   4.7.0-0.nightly-2021-11-12-230709   True        False         False      153m
kube-apiserver                             4.7.0-0.nightly-2021-11-12-230709   True        False         False      150m
kube-controller-manager                    4.7.0-0.nightly-2021-11-12-230709   True        False         False      150m
kube-scheduler                             4.7.0-0.nightly-2021-11-12-230709   True        False         False      150m
kube-storage-version-migrator              4.7.0-0.nightly-2021-11-12-230709   True        False         False      11m
machine-api                                4.7.0-0.nightly-2021-11-12-230709   True        False         False      141m
machine-approver                           4.7.0-0.nightly-2021-11-12-230709   True        False         False      152m
machine-config                             4.7.0-0.nightly-2021-11-12-230709   True        False         False      3m16s
marketplace                                4.7.0-0.nightly-2021-11-12-230709   True        False         False      13m
monitoring                                 4.7.0-0.nightly-2021-11-12-230709   True        False         False      7m58s
network                                    4.7.0-0.nightly-2021-11-12-230709   True        False         False      153m
node-tuning                                4.7.0-0.nightly-2021-11-12-230709   True        False         False      37m
openshift-apiserver                        4.7.0-0.nightly-2021-11-12-230709   True        False         False      3m34s
openshift-controller-manager               4.7.0-0.nightly-2021-11-12-230709   True        False         False      143m
openshift-samples                          4.7.0-0.nightly-2021-11-12-230709   True        False         False      37m
operator-lifecycle-manager                 4.7.0-0.nightly-2021-11-12-230709   True        False         False      151m
operator-lifecycle-manager-catalog         4.7.0-0.nightly-2021-11-12-230709   True        False         False      151m
operator-lifecycle-manager-packageserver   4.7.0-0.nightly-2021-11-12-230709   True        False         False      5m46s
service-ca                                 4.7.0-0.nightly-2021-11-12-230709   True        False         False      152m
storage                                    4.7.0-0.nightly-2021-11-12-230709   True        False         False      13m

$ oc logs cloud-credential-operator-78c849bf-7rq6z -n openshift-cloud-credential-operator -c cloud-credential-operator | grep "Detected required APIs that are disabled"
Nothing output

Comment 6 errata-xmlrpc 2021-12-01 13:35:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.7.38 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4802


Note You need to log in before you can comment on or make changes to this bug.