Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1781727

Summary:	Cluster operator console is not Available and is still Progressing: Cluster operator console has not yet reported success
Product:	OpenShift Container Platform	Reporter:	Jan Chaloupka <jchaloup>
Component:	Management Console	Assignee:	Samuel Padgett <spadgett>
Status:	CLOSED DUPLICATE	QA Contact:	Yadan Pei <yapei>
Severity:	high	Docs Contact:
Priority:	high
Version:	4.3.0	CC:	aos-bugs, jokerman
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-12-10 13:59:08 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Jan Chaloupka 2019-12-10 13:52:03 UTC

Description of problem:
release-openshift-ocp-installer-e2e-gcp-serial-4.3 is continuously failing with "Cluster operator console Progressing is True" and "Cluster operator console Available is False":

```
Installing from release registry.svc.ci.openshift.org/ocp/release:4.3.0-0.nightly-2019-12-10-014919
level=warning msg="Found override for release image. Please be warned, this is not advised"
level=info msg="Consuming Install Config from target directory"
level=info msg="Creating infrastructure resources..."
level=info msg="Waiting up to 30m0s for the Kubernetes API at https://api.ci-op-dqvci7gg-bcd4c.origin-ci-int-gce.dev.openshift.com:6443..."
level=info msg="API v1.16.2 up"
level=info msg="Waiting up to 30m0s for bootstrapping to complete..."
level=info msg="Destroying the bootstrap resources..."
level=info msg="Waiting up to 30m0s for the cluster at https://api.ci-op-dqvci7gg-bcd4c.origin-ci-int-gce.dev.openshift.com:6443 to initialize..."
level=info msg="Cluster operator console Progressing is True with SyncLoopRefreshProgressingInProgress: SyncLoopRefreshProgressing: Working toward version 4.3.0-0.nightly-2019-12-10-014919"
level=info msg="Cluster operator console Available is False with DeploymentAvailableFailedUpdate: DeploymentAvailable: 2 replicas ready at version 4.3.0-0.nightly-2019-12-10-014919"
level=info msg="Cluster operator insights Disabled is False with : "
level=fatal msg="failed to initialize the cluster: Cluster operator console has not yet reported success"
```

Happens every second run of the job on average:
- https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-gcp-serial-4.3/488
- https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-gcp-serial-4.3/486
- https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-gcp-serial-4.3/484
- https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-gcp-serial-4.3/483
- https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-gcp-serial-4.3/479

console operator logs at https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-gcp-serial-4.3/486/artifacts/e2e-gcp-serial/pods/openshift-console-operator_console-operator-7cdf546667-hjfk6_console-operator.log report the following lines until the installation is aborted:

```
E1210 02:34:18.935895       1 status.go:73] DeploymentAvailable FailedUpdate 2 replicas ready at version 4.3.0-0.nightly-2019-12-10-014919
E1210 02:34:19.281710       1 status.go:73] SyncLoopRefreshProgressing InProgress Working toward version 4.3.0-0.nightly-2019-12-10-014919
```


Version-Release number of selected component (if applicable):
4.3.0-0.nightly-2019-12-10-014919

How reproducible:
Every second time the job is run on average

Steps to Reproduce:
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-gcp-serial-4.3/488

Actual results:
Cluster operator console is not Available and is still Progressing

Expected results:
Cluster operator console is Available and is not Progressing

Additional info:

Comment 1 Samuel Padgett 2019-12-10 13:59:08 UTC


*** This bug has been marked as a duplicate of bug 1765280 ***

Comment 2 Samuel Padgett 2019-12-10 14:00:46 UTC

This is the same underlying problem as SDN bug 1765280.

https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-gcp-serial-4.3/488/artifacts/e2e-gcp-serial/pods/openshift-console_console-6bbb964677-6ndj8_console.log

```
2019/12/10 11:42:47 auth: error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.ci-op-fd6zgyry-bcd4c.origin-ci-int-gce.dev.openshift.com/oauth/token failed: Head https://oauth-openshift.apps.ci-op-fd6zgyry-bcd4c.origin-ci-int-gce.dev.openshift.com: dial tcp 34.73.14.62:443: connect: connection refused
```

Comment 3 Jan Chaloupka 2019-12-10 14:03:44 UTC

Thanks Samuel for the quick response.

Just for the record, some failures under https://prow.svc.ci.openshift.org/job-history/origin-ci-test/logs/release-openshift-ocp-installer-e2e-gcp-4.3 show the same error messages.