Bug 1915798 - oauth connection errors for openshift console pods on an OVNKube OCP 4.7 cluster
Summary: oauth connection errors for openshift console pods on an OVNKube OCP 4.7 cluster
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.7
Hardware: ppc64le
OS: Linux
medium
high
Target Milestone: ---
: 4.7.0
Assignee: Peng Liu
QA Contact: huirwang
URL:
Whiteboard:
Depends On: 1899941
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-01-13 12:58 UTC by Archana Prabhakar
Modified: 2021-02-24 15:53 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-02-24 15:52:44 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:53:00 UTC

Description Archana Prabhakar 2021-01-13 12:58:55 UTC
Description of problem:
After migrating OCP 4.7 cluster's CNI to OVNKube, the console pods undergo multiple restarts and display errors contacting the auth provider.

Note - After multiple restarts, the console pods are in running state.

```
[root@arc-npv-ovn-bastion ~]# oc get pods -n openshift-console
NAME                         READY   STATUS    RESTARTS   AGE
console-9d84cdb69-kt5vg      1/1     Running   9          19h
console-9d84cdb69-mxmq5      1/1     Running   9          19h
downloads-6ffc7bf8d6-hqx9m   1/1     Running   0          19h
downloads-6ffc7bf8d6-zhmt2   1/1     Running   0          19h

[root@arc-npv-ovn-bastion ~]# oc describe pod console-9d84cdb69-kt5vg -n openshift-console
Name:                 console-9d84cdb69-kt5vg
Namespace:            openshift-console
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 master-2/9.114.98.128
Start Time:           Tue, 12 Jan 2021 12:46:38 -0500
Labels:               app=console
                      component=ui
                      pod-template-hash=9d84cdb69
Annotations:          console.openshift.io/console-config-version: 22221
                      console.openshift.io/default-ingress-cert-config-version: 18118
                      console.openshift.io/image:
                        quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88b51267f4db6b475260898d0087334997e9c6ae91f9579d72c787725b3e0ad6
                      console.openshift.io/oauth-secret-version: 18187
                      console.openshift.io/proxy-config-version: 535
                      console.openshift.io/service-ca-config-version: 18146
                      console.openshift.io/trusted-ca-config-version: 18156
                      k8s.ovn.org/pod-networks:
                        {"default":{"ip_addresses":["10.129.0.33/23"],"mac_address":"0a:58:0a:81:00:21","gateway_ips":["10.129.0.1"],"ip_address":"10.129.0.33/23"...
                      k8s.v1.cni.cncf.io/network-status:
                        [{
                            "name": "",
                            "interface": "eth0",
                            "ips": [
                                "10.129.0.33"
                            ],
                            "mac": "0a:58:0a:81:00:21",
                            "default": true,
                            "dns": {}
                        }]
                      k8s.v1.cni.cncf.io/networks-status:
                        [{
                            "name": "",
                            "interface": "eth0",
                            "ips": [
                                "10.129.0.33"
                            ],
                            "mac": "0a:58:0a:81:00:21",
                            "default": true,
                            "dns": {}
                        }]
                      openshift.io/scc: restricted
Status:               Running
IP:                   10.129.0.33
IPs:
  IP:           10.129.0.33
Controlled By:  ReplicaSet/console-9d84cdb69
Containers:
  console:
    Container ID:  cri-o://c5cf2c157772eeb94a3ad11996e995e5ae545b76e6ad6034e5eac660daa0fede
    Image:         quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88b51267f4db6b475260898d0087334997e9c6ae91f9579d72c787725b3e0ad6
    Image ID:      quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88b51267f4db6b475260898d0087334997e9c6ae91f9579d72c787725b3e0ad6
    Port:          8443/TCP
    Host Port:     0/TCP
    Command:
      /opt/bridge/bin/bridge
      --public-dir=/opt/bridge/static
      --config=/var/console-config/console-config.yaml
      --service-ca-file=/var/service-ca/service-ca.crt
      --v=2
    State:       Running
      Started:   Tue, 12 Jan 2021 13:21:45 -0500
    Last State:  Terminated
      Reason:    Error
      Message:    contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.arc-npv-ovn.redhat.com/oauth/token failed: Head "https://oauth-openshift.apps.arc-npv-ovn.redhat.com": EOF
E0112 18:20:40.012007       1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.arc-npv-ovn.redhat.com/oauth/token failed: Head "https://oauth-openshift.apps.arc-npv-ovn.redhat.com": EOF
E0112 18:20:50.019513       1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.arc-npv-ovn.redhat.com/oauth/token failed: Head "https://oauth-openshift.apps.arc-npv-ovn.redhat.com": EOF
E0112 18:21:00.025091       1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.arc-npv-ovn.redhat.com/oauth/token failed: Head "https://oauth-openshift.apps.arc-npv-ovn.redhat.com": EOF
E0112 18:21:10.033113       1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.arc-npv-ovn.redhat.com/oauth/token failed: Head "https://oauth-openshift.apps.arc-npv-ovn.redhat.com": EOF
E0112 18:21:20.039529       1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.arc-npv-ovn.redhat.com/oauth/token failed: Head "https://oauth-openshift.apps.arc-npv-ovn.redhat.com": EOF
E0112 18:21:30.046622       1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.arc-npv-ovn.redhat.com/oauth/token failed: Head "https://oauth-openshift.apps.arc-npv-ovn.redhat.com": EOF
E0112 18:21:40.058932       1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.arc-npv-ovn.redhat.com/oauth/token failed: Head "https://oauth-openshift.apps.arc-npv-ovn.redhat.com": EOF

      Exit Code:    2
      Started:      Tue, 12 Jan 2021 13:18:27 -0500
      Finished:     Tue, 12 Jan 2021 13:21:45 -0500
    Ready:          True
    Restart Count:  9
    Requests:
      cpu:        10m
      memory:     100Mi
    Liveness:     http-get https://:8443/health delay=150s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get https://:8443/health delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/pki/ca-trust/extracted/pem from trusted-ca-bundle (ro)
      /var/console-config from console-config (ro)
      /var/default-ingress-cert from default-ingress-cert (ro)
      /var/oauth-config from console-oauth-config (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from console-token-j9tnx (ro)
      /var/service-ca from service-ca (ro)
      /var/serving-cert from console-serving-cert (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  console-serving-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  console-serving-cert
    Optional:    false
  console-oauth-config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  console-oauth-config
    Optional:    false
  console-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      console-config
    Optional:  false
  service-ca:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      service-ca
    Optional:  false
  default-ingress-cert:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      default-ingress-cert
    Optional:  false
  trusted-ca-bundle:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      trusted-ca-bundle
    Optional:  false
  console-token-j9tnx:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  console-token-j9tnx
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  node-role.kubernetes.io/master=
Tolerations:     node-role.kubernetes.io/master:NoSchedule op=Exists
                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/not-reachable:NoExecute op=Exists for 120s
                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 120s
Events:          <none>

[root@arc-npv-ovn-bastion ~]# oc logs console-9d84cdb69-kt5vg -n openshift-console
W0112 18:21:46.034283       1 main.go:207] Flag inactivity-timeout is set to less then 300 seconds and will be ignored!
I0112 18:21:46.034392       1 main.go:274] cookies are secure!
E0112 18:21:46.159002       1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.arc-npv-ovn.redhat.com/oauth/token failed: Head "https://oauth-openshift.apps.arc-npv-ovn.redhat.com": EOF
E0112 18:21:56.175380       1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.arc-npv-ovn.redhat.com/oauth/token failed: Head "https://oauth-openshift.apps.arc-npv-ovn.redhat.com": EOF
E0112 18:22:06.193330       1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.arc-npv-ovn.redhat.com/oauth/token failed: Head "https://oauth-openshift.apps.arc-npv-ovn.redhat.com": EOF
E0112 18:22:16.216196       1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.arc-npv-ovn.redhat.com/oauth/token failed: Head "https://oauth-openshift.apps.arc-npv-ovn.redhat.com": EOF
E0112 18:22:26.243055       1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.arc-npv-ovn.redhat.com/oauth/token failed: Head "https://oauth-openshift.apps.arc-npv-ovn.redhat.com": EOF
E0112 18:22:36.261441       1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.arc-npv-ovn.redhat.com/oauth/token failed: Head "https://oauth-openshift.apps.arc-npv-ovn.redhat.com": EOF
W0112 18:22:46.309598       1 server.go:421] Failed to get cluster k8s version from api server Get "http://localhost/version?timeout=32s": dial tcp [::1]:80: connect: connection refused, falling back to env var KUBE_GIT_VERSION
I0112 18:22:46.309641       1 main.go:654] Binding to [::]:8443...
I0112 18:22:46.309661       1 main.go:656] using TLS

[root@arc-npv-ovn-bastion ~]# oc version
Client Version: 4.7.0-0.nightly-ppc64le-2021-01-11-043556
Server Version: 4.7.0-0.nightly-ppc64le-2021-01-11-043556
Kubernetes Version: v1.20.0+394a5a3

```


Version-Release number of selected component (if applicable):
OCP 4.7


How reproducible:
Every time a cluster CNI gets migrated to OVNKube.

Steps to Reproduce:
1. Deploy OCP 4.7 cluster on power.
2. Migrate the CNI plugin from openshift-sdn to ovnkube by following the procedure at https://docs.openshift.com/container-platform/4.6/networking/ovn_kubernetes_network_provider/migrate-from-openshift-sdn.html 
3. Check the console pods logs and describe it. The console pods go through multiple restarts and display errors.

Actual results:
After migrating OCP 4.7 cluster on power to OVNKube CNI, the console pods restart multiple times ( around 9 times ). The console pods show oauth connection errors after describing them.

Expected results:
The console pods should not go through multiple restarts or throw oauth connection errors.


Additional info:

Comment 1 Standa Laznicka 2021-01-13 14:37:46 UTC
You migrated your network, now you're getting network errors. Moving to Networking to investigate before us.

You'll probably want to get status of all operators for them, and must-gather.

Comment 2 Archana Prabhakar 2021-01-13 15:06:24 UTC
```
[root@arc-npv-ovn-bastion origin]# oc get co
NAME                                       VERSION                                     AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      20h
baremetal                                  4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      25h
cloud-credential                           4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      25h
cluster-autoscaler                         4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      25h
config-operator                            4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      25h
console                                    4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      20h
csi-snapshot-controller                    4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      21h
dns                                        4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      25h
etcd                                       4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      25h
image-registry                             4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      20h
ingress                                    4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      20h
insights                                   4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      25h
kube-apiserver                             4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      25h
kube-controller-manager                    4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      25h
kube-scheduler                             4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      25h
kube-storage-version-migrator              4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      20h
machine-api                                4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      25h
machine-approver                           4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      25h
machine-config                             4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      20h
marketplace                                4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      21h
monitoring                                 4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      20h
network                                    4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      25h
node-tuning                                4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      25h
openshift-apiserver                        4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      21h
openshift-controller-manager               4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      107m
openshift-samples                          4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      25h
operator-lifecycle-manager                 4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      25h
operator-lifecycle-manager-catalog         4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      25h
operator-lifecycle-manager-packageserver   4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      21h
service-ca                                 4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      25h
storage                                    4.7.0-0.nightly-ppc64le-2021-01-11-043556   True        False         False      25h
```

Comment 3 Peng Liu 2021-01-15 01:34:18 UTC
What is the platform the cluster running on?

Comment 4 Archana Prabhakar 2021-01-15 04:58:31 UTC
It is running on power platform.

Comment 5 Peng Liu 2021-01-15 06:05:01 UTC
Could you check the status of the router pods with 'oc get pod -n openshift-ingress -o wide'? Normally, it is caused by the router pods is not ready.

Comment 6 Peng Liu 2021-01-15 09:00:42 UTC
This could be caused by https://bugzilla.redhat.com/show_bug.cgi?id=1899941.

Comment 7 Archana Prabhakar 2021-01-19 10:59:26 UTC
Retried again with latest OCP build and did not notice the router pod restart or multiple console pod restart, but the same error is displayed once when I check the logs of the console pods.

```
[root@pravin-ovn-bastion ~]# oc get pods -n openshift-console -o wide
NAME                         READY   STATUS    RESTARTS   AGE    IP            NODE       NOMINATED NODE   READINESS GATES
console-66cf5dfb-ld46z       1/1     Running   0          3h     10.130.0.32   master-0   <none>           <none>
console-66cf5dfb-zk2k7       1/1     Running   0          174m   10.130.0.33   master-0   <none>           <none>
downloads-57748d68d4-77sff   1/1     Running   0          146m   10.129.0.38   worker-0   <none>           <none>
downloads-57748d68d4-flfvx   1/1     Running   0          146m   10.129.0.41   worker-0   <none>           <none>

[root@pravin-ovn-bastion ~]# oc get pods -n openshift-ingress -o wide
NAME                              READY   STATUS    RESTARTS   AGE    IP           NODE       NOMINATED NODE   READINESS GATES
router-default-5c76fb4b95-98gp4   1/1     Running   0          154m   9.47.89.74   worker-0   <none>           <none>
router-default-5c76fb4b95-n7q2j   1/1     Running   0          146m   9.47.89.3    worker-1   <none>           <none>

[root@pravin-ovn-bastion ~]# oc describe pod console-66cf5dfb-ld46z -n openshift-console
Name:                 console-66cf5dfb-ld46z
Namespace:            openshift-console
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 master-0/9.47.89.52
Start Time:           Tue, 19 Jan 2021 02:55:34 -0500
Labels:               app=console
                      component=ui
                      pod-template-hash=66cf5dfb
Annotations:          console.openshift.io/console-config-version: 61313
                      console.openshift.io/default-ingress-cert-config-version: 61289
                      console.openshift.io/image:
                        quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6bfcd517e043753e65856db4326525df5ed78175bfc0f667e22c5e40251b4609
                      console.openshift.io/oauth-secret-version: 412889
                      console.openshift.io/proxy-config-version: 533
                      console.openshift.io/service-ca-config-version: 61330
                      console.openshift.io/trusted-ca-config-version: 61362
                      k8s.ovn.org/pod-networks:
                        {"default":{"ip_addresses":["10.130.0.32/23"],"mac_address":"0a:58:0a:82:00:20","gateway_ips":["10.130.0.1"],"ip_address":"10.130.0.32/23"...
                      k8s.v1.cni.cncf.io/network-status:
                        [{
                            "name": "",
                            "interface": "eth0",
                            "ips": [
                                "10.130.0.32"
                            ],
                            "mac": "0a:58:0a:82:00:20",
                            "default": true,
                            "dns": {}
                        }]
                      k8s.v1.cni.cncf.io/networks-status:
                        [{
                            "name": "",
                            "interface": "eth0",
                            "ips": [
                                "10.130.0.32"
                            ],
                            "mac": "0a:58:0a:82:00:20",
                            "default": true,
                            "dns": {}
                        }]
                      openshift.io/scc: restricted
Status:               Running
IP:                   10.130.0.32
IPs:
  IP:           10.130.0.32
Controlled By:  ReplicaSet/console-66cf5dfb
Containers:
  console:
    Container ID:  cri-o://1ab979cb0fcb50b706ed337efdad8ea429215e5d049c8d4127be2c68c8c37c5f
    Image:         quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6bfcd517e043753e65856db4326525df5ed78175bfc0f667e22c5e40251b4609
    Image ID:      quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6bfcd517e043753e65856db4326525df5ed78175bfc0f667e22c5e40251b4609
    Port:          8443/TCP
    Host Port:     0/TCP
    Command:
      /opt/bridge/bin/bridge
      --public-dir=/opt/bridge/static
      --config=/var/console-config/console-config.yaml
      --service-ca-file=/var/service-ca/service-ca.crt
      --v=2
    State:          Running
      Started:      Tue, 19 Jan 2021 02:55:38 -0500
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        10m
      memory:     100Mi
    Liveness:     http-get https://:8443/health delay=150s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get https://:8443/health delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/pki/ca-trust/extracted/pem from trusted-ca-bundle (ro)
      /var/console-config from console-config (ro)
      /var/default-ingress-cert from default-ingress-cert (ro)
      /var/oauth-config from console-oauth-config (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from console-token-68znf (ro)
      /var/service-ca from service-ca (ro)
      /var/serving-cert from console-serving-cert (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  console-serving-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  console-serving-cert
    Optional:    false
  console-oauth-config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  console-oauth-config
    Optional:    false
  console-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      console-config
    Optional:  false
  service-ca:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      service-ca
    Optional:  false
  default-ingress-cert:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      default-ingress-cert
    Optional:  false
  trusted-ca-bundle:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      trusted-ca-bundle
    Optional:  false
  console-token-68znf:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  console-token-68znf
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  node-role.kubernetes.io/master=
Tolerations:     node-role.kubernetes.io/master:NoSchedule op=Exists
                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/not-reachable:NoExecute op=Exists for 120s
                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 120s
Events:
  Type     Reason          Age   From               Message
  ----     ------          ----  ----               -------
  Normal   Scheduled       3h1m  default-scheduler  Successfully assigned openshift-console/console-66cf5dfb-ld46z to master-0
  Normal   AddedInterface  3h1m  multus             Add eth0 [10.130.0.32/23]
  Normal   Pulled          3h1m  kubelet            Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6bfcd517e043753e65856db4326525df5ed78175bfc0f667e22c5e40251b4609" already present on machine
  Normal   Created         3h1m  kubelet            Created container console
  Normal   Started         3h1m  kubelet            Started container console
  Warning  Unhealthy       3h    kubelet            Readiness probe failed: Get "https://10.130.0.32:8443/health": dial tcp 10.130.0.32:8443: connect: connection refused
[root@pravin-ovn-bastion ~]# 

[root@pravin-ovn-bastion ~]# oc logs console-66cf5dfb-ld46z -n openshift-console
W0119 07:55:38.962101       1 main.go:211] Flag inactivity-timeout is set to less then 300 seconds and will be ignored!
I0119 07:55:38.962297       1 main.go:288] cookies are secure!
E0119 07:55:39.129502       1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.pravin-ovn.redhat.com/oauth/token failed: Head "https://oauth-openshift.apps.pravin-ovn.redhat.com": EOF
I0119 07:55:49.420427       1 main.go:670] Binding to [::]:8443...
I0119 07:55:49.420473       1 main.go:672] using TLS

[root@pravin-ovn-bastion ~]# oc  version
Client Version: 4.7.0-fc.3
Server Version: 4.7.0-fc.3
Kubernetes Version: v1.20.0+d9c52cc
```

Comment 8 Peng Liu 2021-01-19 14:38:25 UTC
The error is expected. During the SDN migration, the cluster network will be temporarily unavailable, which breaks the pod to service traffic. After the cluster network is up, the traffic shall be able to recover. I suggest we wait until BZ1899941 fixed, see if this issue can be still be reproduced.

Comment 10 Peng Liu 2021-01-29 12:39:08 UTC
Move this PR to ON_QA, as bz1899941's fix has been merged.

Comment 11 huirwang 2021-02-01 06:50:02 UTC
Verified in 4.7.0-0.nightly-2021-01-31-031653, did a couple of times migration from SDN->OVN, did not hit this issue.

Comment 12 Tania Kapoor 2021-02-03 06:53:51 UTC
Results:

# oc version
Client Version: 4.7.0-fc.5
Server Version: 4.7.0-fc.5
Kubernetes Version: v1.20.0+3b90e69

# oc get pods -n openshift-console
NAME                         READY   STATUS    RESTARTS   AGE
console-557686694b-9m9w2     1/1     Running   0          83m
console-557686694b-wn2tm     1/1     Running   0          83m
downloads-6d67bf48b7-bqjlk   1/1     Running   0          61m
downloads-6d67bf48b7-z96th   1/1     Running   0          61m




#  oc logs console-557686694b-9m9w2 -n openshift-console
W0201 09:03:10.157200       1 main.go:211] Flag inactivity-timeout is set to less then 300 seconds and will be ignored!
I0201 09:03:10.157668       1 main.go:288] cookies are secure!
I0201 09:03:11.242915       1 main.go:670] Binding to [::]:8443...
I0201 09:03:11.243086       1 main.go:672] using TLS



# oc get pods -n openshift-ingress -o wide
NAME                              READY   STATUS    RESTARTS   AGE   IP             NODE       NOMINATED NODE   READINESS GATES
router-default-669fb54899-kb7fk   1/1     Running   0          71m   9.114.99.74    worker-0   <none>           <none>
router-default-669fb54899-x9f8k   1/1     Running   0          63m   9.114.99.120   worker-1   <none>           <none>



# oc describe pod console-557686694b-9m9w2 -n openshift-console

Name:                 console-557686694b-9m9w2
Namespace:            openshift-console
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 master-1/9.114.99.111
Start Time:           Mon, 01 Feb 2021 04:02:57 -0500
Labels:               app=console
                      component=ui
                      pod-template-hash=557686694b
Annotations:          console.openshift.io/console-config-version: 19506
                      console.openshift.io/default-ingress-cert-config-version: 16602
                      console.openshift.io/image:
                        quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0a34c97c0c1676aa3a425f0f0ffb085eb2b5dfaad71f3deb4434e1f415723569
                      console.openshift.io/oauth-secret-version: 16650
                      console.openshift.io/proxy-config-version: 595
                      console.openshift.io/service-ca-config-version: 16757
                      console.openshift.io/trusted-ca-config-version: 16642
                      k8s.ovn.org/pod-networks:
                        {"default":{"ip_addresses":["10.130.0.24/23"],"mac_address":"0a:58:0a:82:00:18","gateway_ips":["10.130.0.1"],"ip_address":"10.130.0.24/23"...
                      k8s.v1.cni.cncf.io/network-status:
                        [{
                            "name": "",
                            "interface": "eth0",
                            "ips": [
                                "10.130.0.24"
                            ],
                            "mac": "0a:58:0a:82:00:18",
                            "default": true,
                            "dns": {}
                        }]
                      k8s.v1.cni.cncf.io/networks-status:
                        [{
                            "name": "",
                            "interface": "eth0",
                            "ips": [
                                "10.130.0.24"
                            ],
                            "mac": "0a:58:0a:82:00:18",
                            "default": true,
                            "dns": {}
                        }]
                      openshift.io/scc: restricted
Status:               Running
IP:                   10.130.0.24
IPs:
  IP:           10.130.0.24
Controlled By:  ReplicaSet/console-557686694b
Containers:
  console:
    Container ID:  cri-o://04f4bccc95e1bbc51cabc6ed4e50c32ef4a391e3eddd330dcec890739ec949db
    Image:         quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0a34c97c0c1676aa3a425f0f0ffb085eb2b5dfaad71f3deb4434e1f415723569
    Image ID:      quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0a34c97c0c1676aa3a425f0f0ffb085eb2b5dfaad71f3deb4434e1f415723569
    Port:          8443/TCP
    Host Port:     0/TCP
    Command:
      /opt/bridge/bin/bridge
      --public-dir=/opt/bridge/static
      --config=/var/console-config/console-config.yaml
      --service-ca-file=/var/service-ca/service-ca.crt
      --v=2
    State:          Running
      Started:      Mon, 01 Feb 2021 04:03:09 -0500
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        10m
      memory:     100Mi
    Liveness:     http-get https://:8443/health delay=150s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get https://:8443/health delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/pki/ca-trust/extracted/pem from trusted-ca-bundle (ro)
      /var/console-config from console-config (ro)
      /var/default-ingress-cert from default-ingress-cert (ro)
      /var/oauth-config from console-oauth-config (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from console-token-s4d9j (ro)
      /var/service-ca from service-ca (ro)
      /var/serving-cert from console-serving-cert (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  console-serving-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  console-serving-cert
    Optional:    false
  console-oauth-config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  console-oauth-config
    Optional:    false
  console-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      console-config
    Optional:  false
  service-ca:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      service-ca
    Optional:  false
  default-ingress-cert:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      default-ingress-cert
    Optional:  false
  trusted-ca-bundle:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      trusted-ca-bundle
    Optional:  false
  console-token-s4d9j:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  console-token-s4d9j
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  node-role.kubernetes.io/master=
Tolerations:     node-role.kubernetes.io/master:NoSchedule op=Exists
                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/not-reachable:NoExecute op=Exists for 120s
                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 120s
Events:
  Type    Reason          Age   From               Message
  ----    ------          ----  ----               -------
  Normal  Scheduled       84m   default-scheduler  Successfully assigned openshift-console/console-557686694b-9m9w2 to master-1
  Normal  AddedInterface  84m   multus             Add eth0 [10.130.0.24/23]
  Normal  Pulled          84m   kubelet            Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0a34c97c0c1676aa3a425f0f0ffb085eb2b5dfaad71f3deb4434e1f415723569" already present on machine
  Normal  Created         84m   kubelet            Created container console
  Normal  Started         84m   kubelet            Started container console

Comment 15 errata-xmlrpc 2021-02-24 15:52:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.