Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1766664

Summary: console fails to contact oauth server during upgrade
Product: OpenShift Container Platform Reporter: Oleg Bulatov <obulatov>
Component: NetworkingAssignee: Dan Mace <dmace>
Networking sub component: router QA Contact: Hongan Li <hongli>
Status: CLOSED DUPLICATE Docs Contact:
Severity: unspecified    
Priority: unspecified CC: aos-bugs, chancez, dmace, jokerman, mfojtik, spadgett, wking
Version: 4.3.0   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-11-01 13:10:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Oleg Bulatov 2019-10-29 15:38:10 UTC 
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-azure-upgrade-4.2/194

From https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-azure-upgrade-4.2/194/artifacts/e2e-azure-upgrade/clusteroperators.json

      {
        "lastTransitionTime": "2019-10-29T03:25:39Z",
        "message": "SyncLoopRefreshProgressing: Working toward version 4.2.0-0.ci-2019-10-26-043802",
        "reason": "SyncLoopRefreshProgressingInProgress",
        "status": "True",
        "type": "Progressing"
      },
      {
        "lastTransitionTime": "2019-10-29T03:30:03Z",
        "message": "DeploymentAvailable: 2 replicas ready at version 4.2.0-0.ci-2019-10-26-043802",
        "reason": "DeploymentAvailableFailedUpdate",
        "status": "False",
        "type": "Available"
      },

From https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-azure-upgrade-4.2/194/artifacts/e2e-azure-upgrade/must-gather/registry-svc-ci-openshift-org-ocp-4-2-2019-10-26-043802-sha256-72d3b1b48fd0c2b518b0ef7e681aad61429a7a27efec5b2faa48bc301cb65480/namespaces/openshift-console-operator/pods/console-operator-7d47f9cdb4-fwmgp/console-operator/console-operator/logs/current.log

2019-10-29T03:58:30.1465173Z E1029 03:58:30.146473       1 status.go:71] SyncLoopRefreshProgressing InProgress Working toward version 4.2.0-0.ci-2019-10-26-043802
2019-10-29T03:58:30.1465173Z E1029 03:58:30.146493       1 status.go:71] DeploymentAvailable FailedUpdate 2 replicas ready at version 4.2.0-0.ci-2019-10-26-043802
2019-10-29T03:59:37.1226093Z E1029 03:59:37.122570       1 status.go:71] SyncLoopRefreshProgressing InProgress Working toward version 4.2.0-0.ci-2019-10-26-043802
2019-10-29T03:59:37.1226093Z E1029 03:59:37.122592       1 status.go:71] DeploymentAvailable FailedUpdate 2 replicas ready at version 4.2.0-0.ci-2019-10-26-043802
2019-10-29T04:00:26.3196356Z W1029 04:00:26.319592       1 reflector.go:289] github.com/openshift/client-go/route/informers/externalversions/factory.go:101: watch of *v1.Route ended with: The resourceVersion for the provided watch is too old.
2019-10-29T04:00:27.3929302Z E1029 04:00:27.392887       1 status.go:71] SyncLoopRefreshProgressing InProgress Working toward version 4.2.0-0.ci-2019-10-26-043802
2019-10-29T04:00:27.3929302Z E1029 04:00:27.392909       1 status.go:71] DeploymentAvailable FailedUpdate 2 replicas ready at version 4.2.0-0.ci-2019-10-26-043802

From https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-azure-upgrade-4.2/194/artifacts/e2e-azure-upgrade/must-gather/registry-svc-ci-openshift-org-ocp-4-2-2019-10-26-043802-sha256-72d3b1b48fd0c2b518b0ef7e681aad61429a7a27efec5b2faa48bc301cb65480/namespaces/openshift-console/apps/replicasets.yaml

- apiVersion: apps/v1
  kind: ReplicaSet
  metadata:
    annotations:
      ...
      deployment.kubernetes.io/revision: "2"
      ...
    ...
    name: console-547ff6c548
    ...
  ...
  status:
    availableReplicas: 1
    fullyLabeledReplicas: 2
    observedGeneration: 2
    readyReplicas: 1
    replicas: 2

From https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-azure-upgrade-4.2/194/artifacts/e2e-azure-upgrade/must-gather/registry-svc-ci-openshift-org-ocp-4-2-2019-10-26-043802-sha256-72d3b1b48fd0c2b518b0ef7e681aad61429a7a27efec5b2faa48bc301cb65480/namespaces/openshift-console/pods/console-547ff6c548-2dkm9/console-547ff6c548-2dkm9.yaml

      terminated:
        containerID: cri-o://169bbe3df59e485656b33ba51e104699e48f87a6356b015e6e5273b1c7b7ddf5
        exitCode: 2
        finishedAt: 2019-10-29T03:57:01Z
        message: |
          ps://oauth-openshift.apps.ci-op-qmhhpbw8-43a0f.ci.azure.devcluster.openshift.com/oauth/token failed: Head https://oauth-openshift.apps.ci-op-qmhhpbw8-43a0f.ci.azure.devcluster.openshift.com: dial tcp 52.158.213.195:443: connect: connection refused
          2019/10/29 03:56:16 auth: error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.ci-op-qmhhpbw8-43a0f.ci.azure.devcluster.openshift.com/oauth/token failed: Head https://oauth-openshift.apps.ci-op-qmhhpbw8-43a0f.ci.azure.devcluster.openshift.com: dial tcp 52.158.213.195:443: connect: connection refused
Comment 1 Samuel Padgett 2019-10-29 15:44:17 UTC 
The connection is refused from the OAuth server. The URL looks correct. I don't believe this is a console issue. It might be either networking or OAuth server.

Reassigned to the Auth component for now to investigate further. If there is something the console should be doing differently, let us know.
Comment 2 Samuel Padgett 2019-10-30 19:13:07 UTC 
*** Bug 1767119 has been marked as a duplicate of this bug. ***
Comment 3 Samuel Padgett 2019-11-01 13:05:16 UTC 
I believe this is a networking problem since one of the console pods is healthy and auth is reporting healthy. It looks like it's only an issue for a specific pod on a specific node. Changing to Routing for investigation.
Comment 4 Dan Mace 2019-11-01 13:10:14 UTC 

*** This bug has been marked as a duplicate of bug 1765280 ***