2022813 – CCO occasionally down, reporting networksecurity.googleapis.com API as disabled

Bug 2022813 - CCO occasionally down, reporting networksecurity.googleapis.com API as disabled

Summary: CCO occasionally down, reporting networksecurity.googleapis.com API as disabled

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Cloud Credential Operator
Sub Component:
Version:	4.10
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	4.9.z
Assignee:	Joel Diaz
QA Contact:	wang lin
Docs Contact:
URL:
Whiteboard:
Depends On:	2021731 2031858
Blocks:	2022838
TreeView+	depends on / blocked

Reported:	2021-11-12 16:46 UTC by OpenShift BugZilla Robot
Modified:	2021-12-13 15:27 UTC (History)
CC List:	14 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Clones:	2022838 (view as bug list)
Environment:
Last Closed:	2021-11-22 21:47:23 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	openshift machine-api-operator pull 952	0	None	open	Bug 2022813: GCP credentials reporting networksecurity.googleapis.com API disabled	2021-11-12 17:15:27 UTC
Red Hat Product Errata	RHBA-2021:4712	0	None	None	None	2021-11-22 21:47:28 UTC

Comment 4 wang lin 2021-11-15 05:24:05 UTC

Verified on 4.9.0-0.nightly-2021-11-12-222121 include the fix.

1. Launch a basic gcp cluster
2. Monitor the installation process

the installaton can succeed and cco won't hit the issue about "Detected required APIs that are disabled: [networksecurity.googleapis.com]"

$ $ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0-0.nightly-2021-11-12-222121   True        False         5m48s   Cluster version is 4.9.0-0.nightly-2021-11-12-222121

$ oc logs cloud-credential-operator-6449f4c6b9-hl4rw -n openshift-cloud-credential-operator -c cloud-credential-operator | grep "Detected required APIs that are disabled"
Nothing output

############
The payload without the fix merged like 4.9.0-0.nightly-2021-11-11-155043 will fail to install, and cco Degraded because of [networksecurity.googleapis.com] disabled.

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version             False       True          53m     Unable to apply 4.9.0-0.nightly-2021-11-11-155043: some cluster operators have not yet rolled out

$ oc logs cloud-credential-operator-5b97f67944-qp6k2 -n openshift-cloud-credential-operator -c cloud-credential-operator | grep "Detected required APIs that are disabled"
time="2021-11-15T04:33:13Z" level=warning msg="Detected required APIs that are disabled: [networksecurity.googleapis.com]" actuator=gcp cr=openshift-cloud-credential-operator/openshift-machine-api-gcp
time="2021-11-15T04:33:17Z" level=warning msg="Detected required APIs that are disabled: [networksecurity.googleapis.com]" actuator=gcp cr=openshift-cloud-credential-operator/openshift-machine-api-gcp
time="2021-11-15T04:33:22Z" level=warning msg="Detected required APIs that are disabled: [networksecurity.googleapis.com]" actuator=gcp cr=openshift-cloud-credential-operator/openshift-machine-api-gcp
time="2021-11-15T04:33:31Z" level=warning msg="Detected required APIs that are disabled: [networksecurity.googleapis.com]" actuator=gcp cr=openshift-cloud-credential-operator/openshift-machine-api-gcp
time="2021-11-15T04:33:49Z" level=warning msg="Detected required APIs that are disabled: [networksecurity.googleapis.com]" actuator=gcp cr=openshift-cloud-credential-operator/openshift-machine-api-gcp

Comment 5 wang lin 2021-11-15 10:23:23 UTC

Verified upgrading cluster from 4.9.6 to 4.9.0-0.nightly-2021-11-12-222121

test steps:
1. enable "Network Security API" on tested gcp project

2. install a cluster with version 4.9.6 and wait for cluster installed, check installation succeed.

3. check co cco status is normal
$ oc get co cloud-credential -w
NAME               VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
cloud-credential   4.9.6     True        False         False      27m     
cloud-credential   4.9.6     True        False         False      31m 

4. disable "Network Security API" on tested gcp project, then monitor and wait, cco will Degraded(needs about 1 hour)
$ oc logs -f cloud-credential-operator-5546f788d8-59wn8 -n openshift-cloud-credential-operator -c cloud-credential-operator | grep "Detected required APIs that are disabled"
time="2021-11-15T08:40:50Z" level=warning msg="Detected required APIs that are disabled: [networksecurity.googleapis.com]" actuator=gcp cr=openshift-cloud-credential-operator/openshift-machine-api-gcp

$ oc get co
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE   
cloud-credential                           4.9.6     True        True          True       81m     1 of 5 credentials requests are failing to sync.


5 run upgrade command and wait for upgrade finish
$oc adm upgrade --to-image registry.ci.openshift.org/ocp/release:4.9.0-0.nightly-2021-11-12-222121 --allow-explicit-upgrade --force

6 Check upgrade successfully
$ $ ./oc get clusterversion -w
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.6     True        True          54m     Working towards 4.9.0-0.nightly-2021-11-12-222121: 1 of 737 done (0% complete)
version   4.9.6     True        True          54m     Working towards 4.9.0-0.nightly-2021-11-12-222121: 3 of 737 done (0% complete)
version   4.9.6     True        True          54m     Working towards 4.9.0-0.nightly-2021-11-12-222121: 4 of 737 done (0% complete)
version   4.9.6     True        True          54m     Working towards 4.9.0-0.nightly-2021-11-12-222121: 5 of 737 done (0% complete)
version   4.9.6     True        True          54m     Working towards 4.9.0-0.nightly-2021-11-12-222121: 6 of 737 done (0% complete)
version   4.9.6     True        True          54m     Working towards 4.9.0-0.nightly-2021-11-12-222121: 9 of 737 done (1% complete)
version   4.9.6     True        True          54m     Working towards 4.9.0-0.nightly-2021-11-12-222121: 204 of 737 done (27% complete)

$ ./oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0-0.nightly-2021-11-12-222121   True        False         47s     Cluster version is 4.9.0-0.nightly-2021-11-12-222121


7 Check cco won't degraded
$ oc get co
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.9.0-0.nightly-2021-11-12-222121   True        False         False      127m    
baremetal                                  4.9.0-0.nightly-2021-11-12-222121   True        False         False      139m    
cloud-controller-manager                   4.9.0-0.nightly-2021-11-12-222121   True        False         False      143m    
cloud-credential                           4.9.0-0.nightly-2021-11-12-222121   True        False         False      146m    
cluster-autoscaler                         4.9.0-0.nightly-2021-11-12-222121   True        False         False      139m    
config-operator                            4.9.0-0.nightly-2021-11-12-222121   True        False         False      141m    
console                                    4.9.0-0.nightly-2021-11-12-222121   True        False         False      22m     
csi-snapshot-controller                    4.9.0-0.nightly-2021-11-12-222121   True        False         False      141m    
dns                                        4.9.0-0.nightly-2021-11-12-222121   True        False         False      140m    
etcd                                       4.9.0-0.nightly-2021-11-12-222121   True        False         False      139m    
image-registry                             4.9.0-0.nightly-2021-11-12-222121   True        False         False      134m    
ingress                                    4.9.0-0.nightly-2021-11-12-222121   True        False         False      133m    
insights                                   4.9.0-0.nightly-2021-11-12-222121   True        False         False      135m    
kube-apiserver                             4.9.0-0.nightly-2021-11-12-222121   True        False         False      137m    
kube-controller-manager                    4.9.0-0.nightly-2021-11-12-222121   True        False         False      140m    
kube-scheduler                             4.9.0-0.nightly-2021-11-12-222121   True        False         False      139m    
kube-storage-version-migrator              4.9.0-0.nightly-2021-11-12-222121   True        False         False      11m     
machine-api                                4.9.0-0.nightly-2021-11-12-222121   True        False         False      137m    
machine-approver                           4.9.0-0.nightly-2021-11-12-222121   True        False         False      141m    
machine-config                             4.9.0-0.nightly-2021-11-12-222121   True        False         False      140m    
marketplace                                4.9.0-0.nightly-2021-11-12-222121   True        False         False      140m    
monitoring                                 4.9.0-0.nightly-2021-11-12-222121   True        False         False      133m    
network                                    4.9.0-0.nightly-2021-11-12-222121   True        False         False      142m    
node-tuning                                4.9.0-0.nightly-2021-11-12-222121   True        False         False      10m     
openshift-apiserver                        4.9.0-0.nightly-2021-11-12-222121   True        False         False      135m    
openshift-controller-manager               4.9.0-0.nightly-2021-11-12-222121   True        False         False      36m     
openshift-samples                          4.9.0-0.nightly-2021-11-12-222121   True        False         False      37m     
operator-lifecycle-manager                 4.9.0-0.nightly-2021-11-12-222121   True        False         False      141m    
operator-lifecycle-manager-catalog         4.9.0-0.nightly-2021-11-12-222121   True        False         False      140m    
operator-lifecycle-manager-packageserver   4.9.0-0.nightly-2021-11-12-222121   True        False         False      135m    
service-ca                                 4.9.0-0.nightly-2021-11-12-222121   True        False         False      141m    
storage                                    4.9.0-0.nightly-2021-11-12-222121   True        False         False      140m    
$ oc get pods -n openshift-cloud-credential-operator
NAME                                         READY   STATUS    RESTARTS   AGE
cloud-credential-operator-6449f4c6b9-zhnng   2/2     Running   0          17m
$ oc logs cloud-credential-operator-6449f4c6b9-zhnng -n openshift-cloud-credential-operator -c cloud-credential-operator | grep "Detected required APIs that are disabled"
Nothing output

Comment 8 errata-xmlrpc 2021-11-22 21:47:23 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.9.8 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4712

Note You need to log in before you can comment on or make changes to this bug.