Bug 1915798
| Summary: | oauth connection errors for openshift console pods on an OVNKube OCP 4.7 cluster | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Archana Prabhakar <aprabhak> |
| Component: | Networking | Assignee: | Peng Liu <pliu> |
| Networking sub component: | ovn-kubernetes | QA Contact: | huirwang |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | high | ||
| Priority: | medium | CC: | aconstan, aos-bugs, bbennett, dosmith, huirwang, mfojtik, tkapoor |
| Version: | 4.7 | Keywords: | UpcomingSprint |
| Target Milestone: | --- | ||
| Target Release: | 4.7.0 | ||
| Hardware: | ppc64le | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-02-24 15:52:44 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1899941 | ||
| Bug Blocks: | |||
You migrated your network, now you're getting network errors. Moving to Networking to investigate before us. You'll probably want to get status of all operators for them, and must-gather. ``` [root@arc-npv-ovn-bastion origin]# oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 20h baremetal 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 25h cloud-credential 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 25h cluster-autoscaler 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 25h config-operator 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 25h console 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 20h csi-snapshot-controller 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 21h dns 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 25h etcd 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 25h image-registry 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 20h ingress 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 20h insights 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 25h kube-apiserver 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 25h kube-controller-manager 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 25h kube-scheduler 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 25h kube-storage-version-migrator 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 20h machine-api 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 25h machine-approver 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 25h machine-config 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 20h marketplace 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 21h monitoring 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 20h network 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 25h node-tuning 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 25h openshift-apiserver 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 21h openshift-controller-manager 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 107m openshift-samples 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 25h operator-lifecycle-manager 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 25h operator-lifecycle-manager-catalog 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 25h operator-lifecycle-manager-packageserver 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 21h service-ca 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 25h storage 4.7.0-0.nightly-ppc64le-2021-01-11-043556 True False False 25h ``` What is the platform the cluster running on? It is running on power platform. Could you check the status of the router pods with 'oc get pod -n openshift-ingress -o wide'? Normally, it is caused by the router pods is not ready. This could be caused by https://bugzilla.redhat.com/show_bug.cgi?id=1899941. Retried again with latest OCP build and did not notice the router pod restart or multiple console pod restart, but the same error is displayed once when I check the logs of the console pods.
```
[root@pravin-ovn-bastion ~]# oc get pods -n openshift-console -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
console-66cf5dfb-ld46z 1/1 Running 0 3h 10.130.0.32 master-0 <none> <none>
console-66cf5dfb-zk2k7 1/1 Running 0 174m 10.130.0.33 master-0 <none> <none>
downloads-57748d68d4-77sff 1/1 Running 0 146m 10.129.0.38 worker-0 <none> <none>
downloads-57748d68d4-flfvx 1/1 Running 0 146m 10.129.0.41 worker-0 <none> <none>
[root@pravin-ovn-bastion ~]# oc get pods -n openshift-ingress -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
router-default-5c76fb4b95-98gp4 1/1 Running 0 154m 9.47.89.74 worker-0 <none> <none>
router-default-5c76fb4b95-n7q2j 1/1 Running 0 146m 9.47.89.3 worker-1 <none> <none>
[root@pravin-ovn-bastion ~]# oc describe pod console-66cf5dfb-ld46z -n openshift-console
Name: console-66cf5dfb-ld46z
Namespace: openshift-console
Priority: 2000000000
Priority Class Name: system-cluster-critical
Node: master-0/9.47.89.52
Start Time: Tue, 19 Jan 2021 02:55:34 -0500
Labels: app=console
component=ui
pod-template-hash=66cf5dfb
Annotations: console.openshift.io/console-config-version: 61313
console.openshift.io/default-ingress-cert-config-version: 61289
console.openshift.io/image:
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6bfcd517e043753e65856db4326525df5ed78175bfc0f667e22c5e40251b4609
console.openshift.io/oauth-secret-version: 412889
console.openshift.io/proxy-config-version: 533
console.openshift.io/service-ca-config-version: 61330
console.openshift.io/trusted-ca-config-version: 61362
k8s.ovn.org/pod-networks:
{"default":{"ip_addresses":["10.130.0.32/23"],"mac_address":"0a:58:0a:82:00:20","gateway_ips":["10.130.0.1"],"ip_address":"10.130.0.32/23"...
k8s.v1.cni.cncf.io/network-status:
[{
"name": "",
"interface": "eth0",
"ips": [
"10.130.0.32"
],
"mac": "0a:58:0a:82:00:20",
"default": true,
"dns": {}
}]
k8s.v1.cni.cncf.io/networks-status:
[{
"name": "",
"interface": "eth0",
"ips": [
"10.130.0.32"
],
"mac": "0a:58:0a:82:00:20",
"default": true,
"dns": {}
}]
openshift.io/scc: restricted
Status: Running
IP: 10.130.0.32
IPs:
IP: 10.130.0.32
Controlled By: ReplicaSet/console-66cf5dfb
Containers:
console:
Container ID: cri-o://1ab979cb0fcb50b706ed337efdad8ea429215e5d049c8d4127be2c68c8c37c5f
Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6bfcd517e043753e65856db4326525df5ed78175bfc0f667e22c5e40251b4609
Image ID: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6bfcd517e043753e65856db4326525df5ed78175bfc0f667e22c5e40251b4609
Port: 8443/TCP
Host Port: 0/TCP
Command:
/opt/bridge/bin/bridge
--public-dir=/opt/bridge/static
--config=/var/console-config/console-config.yaml
--service-ca-file=/var/service-ca/service-ca.crt
--v=2
State: Running
Started: Tue, 19 Jan 2021 02:55:38 -0500
Ready: True
Restart Count: 0
Requests:
cpu: 10m
memory: 100Mi
Liveness: http-get https://:8443/health delay=150s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get https://:8443/health delay=0s timeout=1s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/etc/pki/ca-trust/extracted/pem from trusted-ca-bundle (ro)
/var/console-config from console-config (ro)
/var/default-ingress-cert from default-ingress-cert (ro)
/var/oauth-config from console-oauth-config (ro)
/var/run/secrets/kubernetes.io/serviceaccount from console-token-68znf (ro)
/var/service-ca from service-ca (ro)
/var/serving-cert from console-serving-cert (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
console-serving-cert:
Type: Secret (a volume populated by a Secret)
SecretName: console-serving-cert
Optional: false
console-oauth-config:
Type: Secret (a volume populated by a Secret)
SecretName: console-oauth-config
Optional: false
console-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: console-config
Optional: false
service-ca:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: service-ca
Optional: false
default-ingress-cert:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: default-ingress-cert
Optional: false
trusted-ca-bundle:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: trusted-ca-bundle
Optional: false
console-token-68znf:
Type: Secret (a volume populated by a Secret)
SecretName: console-token-68znf
Optional: false
QoS Class: Burstable
Node-Selectors: node-role.kubernetes.io/master=
Tolerations: node-role.kubernetes.io/master:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-reachable:NoExecute op=Exists for 120s
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 120s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3h1m default-scheduler Successfully assigned openshift-console/console-66cf5dfb-ld46z to master-0
Normal AddedInterface 3h1m multus Add eth0 [10.130.0.32/23]
Normal Pulled 3h1m kubelet Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6bfcd517e043753e65856db4326525df5ed78175bfc0f667e22c5e40251b4609" already present on machine
Normal Created 3h1m kubelet Created container console
Normal Started 3h1m kubelet Started container console
Warning Unhealthy 3h kubelet Readiness probe failed: Get "https://10.130.0.32:8443/health": dial tcp 10.130.0.32:8443: connect: connection refused
[root@pravin-ovn-bastion ~]#
[root@pravin-ovn-bastion ~]# oc logs console-66cf5dfb-ld46z -n openshift-console
W0119 07:55:38.962101 1 main.go:211] Flag inactivity-timeout is set to less then 300 seconds and will be ignored!
I0119 07:55:38.962297 1 main.go:288] cookies are secure!
E0119 07:55:39.129502 1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.pravin-ovn.redhat.com/oauth/token failed: Head "https://oauth-openshift.apps.pravin-ovn.redhat.com": EOF
I0119 07:55:49.420427 1 main.go:670] Binding to [::]:8443...
I0119 07:55:49.420473 1 main.go:672] using TLS
[root@pravin-ovn-bastion ~]# oc version
Client Version: 4.7.0-fc.3
Server Version: 4.7.0-fc.3
Kubernetes Version: v1.20.0+d9c52cc
```
The error is expected. During the SDN migration, the cluster network will be temporarily unavailable, which breaks the pod to service traffic. After the cluster network is up, the traffic shall be able to recover. I suggest we wait until BZ1899941 fixed, see if this issue can be still be reproduced. Move this PR to ON_QA, as bz1899941's fix has been merged. Verified in 4.7.0-0.nightly-2021-01-31-031653, did a couple of times migration from SDN->OVN, did not hit this issue. Results:
# oc version
Client Version: 4.7.0-fc.5
Server Version: 4.7.0-fc.5
Kubernetes Version: v1.20.0+3b90e69
# oc get pods -n openshift-console
NAME READY STATUS RESTARTS AGE
console-557686694b-9m9w2 1/1 Running 0 83m
console-557686694b-wn2tm 1/1 Running 0 83m
downloads-6d67bf48b7-bqjlk 1/1 Running 0 61m
downloads-6d67bf48b7-z96th 1/1 Running 0 61m
# oc logs console-557686694b-9m9w2 -n openshift-console
W0201 09:03:10.157200 1 main.go:211] Flag inactivity-timeout is set to less then 300 seconds and will be ignored!
I0201 09:03:10.157668 1 main.go:288] cookies are secure!
I0201 09:03:11.242915 1 main.go:670] Binding to [::]:8443...
I0201 09:03:11.243086 1 main.go:672] using TLS
# oc get pods -n openshift-ingress -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
router-default-669fb54899-kb7fk 1/1 Running 0 71m 9.114.99.74 worker-0 <none> <none>
router-default-669fb54899-x9f8k 1/1 Running 0 63m 9.114.99.120 worker-1 <none> <none>
# oc describe pod console-557686694b-9m9w2 -n openshift-console
Name: console-557686694b-9m9w2
Namespace: openshift-console
Priority: 2000000000
Priority Class Name: system-cluster-critical
Node: master-1/9.114.99.111
Start Time: Mon, 01 Feb 2021 04:02:57 -0500
Labels: app=console
component=ui
pod-template-hash=557686694b
Annotations: console.openshift.io/console-config-version: 19506
console.openshift.io/default-ingress-cert-config-version: 16602
console.openshift.io/image:
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0a34c97c0c1676aa3a425f0f0ffb085eb2b5dfaad71f3deb4434e1f415723569
console.openshift.io/oauth-secret-version: 16650
console.openshift.io/proxy-config-version: 595
console.openshift.io/service-ca-config-version: 16757
console.openshift.io/trusted-ca-config-version: 16642
k8s.ovn.org/pod-networks:
{"default":{"ip_addresses":["10.130.0.24/23"],"mac_address":"0a:58:0a:82:00:18","gateway_ips":["10.130.0.1"],"ip_address":"10.130.0.24/23"...
k8s.v1.cni.cncf.io/network-status:
[{
"name": "",
"interface": "eth0",
"ips": [
"10.130.0.24"
],
"mac": "0a:58:0a:82:00:18",
"default": true,
"dns": {}
}]
k8s.v1.cni.cncf.io/networks-status:
[{
"name": "",
"interface": "eth0",
"ips": [
"10.130.0.24"
],
"mac": "0a:58:0a:82:00:18",
"default": true,
"dns": {}
}]
openshift.io/scc: restricted
Status: Running
IP: 10.130.0.24
IPs:
IP: 10.130.0.24
Controlled By: ReplicaSet/console-557686694b
Containers:
console:
Container ID: cri-o://04f4bccc95e1bbc51cabc6ed4e50c32ef4a391e3eddd330dcec890739ec949db
Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0a34c97c0c1676aa3a425f0f0ffb085eb2b5dfaad71f3deb4434e1f415723569
Image ID: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0a34c97c0c1676aa3a425f0f0ffb085eb2b5dfaad71f3deb4434e1f415723569
Port: 8443/TCP
Host Port: 0/TCP
Command:
/opt/bridge/bin/bridge
--public-dir=/opt/bridge/static
--config=/var/console-config/console-config.yaml
--service-ca-file=/var/service-ca/service-ca.crt
--v=2
State: Running
Started: Mon, 01 Feb 2021 04:03:09 -0500
Ready: True
Restart Count: 0
Requests:
cpu: 10m
memory: 100Mi
Liveness: http-get https://:8443/health delay=150s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get https://:8443/health delay=0s timeout=1s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/etc/pki/ca-trust/extracted/pem from trusted-ca-bundle (ro)
/var/console-config from console-config (ro)
/var/default-ingress-cert from default-ingress-cert (ro)
/var/oauth-config from console-oauth-config (ro)
/var/run/secrets/kubernetes.io/serviceaccount from console-token-s4d9j (ro)
/var/service-ca from service-ca (ro)
/var/serving-cert from console-serving-cert (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
console-serving-cert:
Type: Secret (a volume populated by a Secret)
SecretName: console-serving-cert
Optional: false
console-oauth-config:
Type: Secret (a volume populated by a Secret)
SecretName: console-oauth-config
Optional: false
console-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: console-config
Optional: false
service-ca:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: service-ca
Optional: false
default-ingress-cert:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: default-ingress-cert
Optional: false
trusted-ca-bundle:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: trusted-ca-bundle
Optional: false
console-token-s4d9j:
Type: Secret (a volume populated by a Secret)
SecretName: console-token-s4d9j
Optional: false
QoS Class: Burstable
Node-Selectors: node-role.kubernetes.io/master=
Tolerations: node-role.kubernetes.io/master:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-reachable:NoExecute op=Exists for 120s
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 120s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 84m default-scheduler Successfully assigned openshift-console/console-557686694b-9m9w2 to master-1
Normal AddedInterface 84m multus Add eth0 [10.130.0.24/23]
Normal Pulled 84m kubelet Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0a34c97c0c1676aa3a425f0f0ffb085eb2b5dfaad71f3deb4434e1f415723569" already present on machine
Normal Created 84m kubelet Created container console
Normal Started 84m kubelet Started container console
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633 |
Description of problem: After migrating OCP 4.7 cluster's CNI to OVNKube, the console pods undergo multiple restarts and display errors contacting the auth provider. Note - After multiple restarts, the console pods are in running state. ``` [root@arc-npv-ovn-bastion ~]# oc get pods -n openshift-console NAME READY STATUS RESTARTS AGE console-9d84cdb69-kt5vg 1/1 Running 9 19h console-9d84cdb69-mxmq5 1/1 Running 9 19h downloads-6ffc7bf8d6-hqx9m 1/1 Running 0 19h downloads-6ffc7bf8d6-zhmt2 1/1 Running 0 19h [root@arc-npv-ovn-bastion ~]# oc describe pod console-9d84cdb69-kt5vg -n openshift-console Name: console-9d84cdb69-kt5vg Namespace: openshift-console Priority: 2000000000 Priority Class Name: system-cluster-critical Node: master-2/9.114.98.128 Start Time: Tue, 12 Jan 2021 12:46:38 -0500 Labels: app=console component=ui pod-template-hash=9d84cdb69 Annotations: console.openshift.io/console-config-version: 22221 console.openshift.io/default-ingress-cert-config-version: 18118 console.openshift.io/image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88b51267f4db6b475260898d0087334997e9c6ae91f9579d72c787725b3e0ad6 console.openshift.io/oauth-secret-version: 18187 console.openshift.io/proxy-config-version: 535 console.openshift.io/service-ca-config-version: 18146 console.openshift.io/trusted-ca-config-version: 18156 k8s.ovn.org/pod-networks: {"default":{"ip_addresses":["10.129.0.33/23"],"mac_address":"0a:58:0a:81:00:21","gateway_ips":["10.129.0.1"],"ip_address":"10.129.0.33/23"... k8s.v1.cni.cncf.io/network-status: [{ "name": "", "interface": "eth0", "ips": [ "10.129.0.33" ], "mac": "0a:58:0a:81:00:21", "default": true, "dns": {} }] k8s.v1.cni.cncf.io/networks-status: [{ "name": "", "interface": "eth0", "ips": [ "10.129.0.33" ], "mac": "0a:58:0a:81:00:21", "default": true, "dns": {} }] openshift.io/scc: restricted Status: Running IP: 10.129.0.33 IPs: IP: 10.129.0.33 Controlled By: ReplicaSet/console-9d84cdb69 Containers: console: Container ID: cri-o://c5cf2c157772eeb94a3ad11996e995e5ae545b76e6ad6034e5eac660daa0fede Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88b51267f4db6b475260898d0087334997e9c6ae91f9579d72c787725b3e0ad6 Image ID: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88b51267f4db6b475260898d0087334997e9c6ae91f9579d72c787725b3e0ad6 Port: 8443/TCP Host Port: 0/TCP Command: /opt/bridge/bin/bridge --public-dir=/opt/bridge/static --config=/var/console-config/console-config.yaml --service-ca-file=/var/service-ca/service-ca.crt --v=2 State: Running Started: Tue, 12 Jan 2021 13:21:45 -0500 Last State: Terminated Reason: Error Message: contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.arc-npv-ovn.redhat.com/oauth/token failed: Head "https://oauth-openshift.apps.arc-npv-ovn.redhat.com": EOF E0112 18:20:40.012007 1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.arc-npv-ovn.redhat.com/oauth/token failed: Head "https://oauth-openshift.apps.arc-npv-ovn.redhat.com": EOF E0112 18:20:50.019513 1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.arc-npv-ovn.redhat.com/oauth/token failed: Head "https://oauth-openshift.apps.arc-npv-ovn.redhat.com": EOF E0112 18:21:00.025091 1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.arc-npv-ovn.redhat.com/oauth/token failed: Head "https://oauth-openshift.apps.arc-npv-ovn.redhat.com": EOF E0112 18:21:10.033113 1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.arc-npv-ovn.redhat.com/oauth/token failed: Head "https://oauth-openshift.apps.arc-npv-ovn.redhat.com": EOF E0112 18:21:20.039529 1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.arc-npv-ovn.redhat.com/oauth/token failed: Head "https://oauth-openshift.apps.arc-npv-ovn.redhat.com": EOF E0112 18:21:30.046622 1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.arc-npv-ovn.redhat.com/oauth/token failed: Head "https://oauth-openshift.apps.arc-npv-ovn.redhat.com": EOF E0112 18:21:40.058932 1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.arc-npv-ovn.redhat.com/oauth/token failed: Head "https://oauth-openshift.apps.arc-npv-ovn.redhat.com": EOF Exit Code: 2 Started: Tue, 12 Jan 2021 13:18:27 -0500 Finished: Tue, 12 Jan 2021 13:21:45 -0500 Ready: True Restart Count: 9 Requests: cpu: 10m memory: 100Mi Liveness: http-get https://:8443/health delay=150s timeout=1s period=10s #success=1 #failure=3 Readiness: http-get https://:8443/health delay=0s timeout=1s period=10s #success=1 #failure=3 Environment: <none> Mounts: /etc/pki/ca-trust/extracted/pem from trusted-ca-bundle (ro) /var/console-config from console-config (ro) /var/default-ingress-cert from default-ingress-cert (ro) /var/oauth-config from console-oauth-config (ro) /var/run/secrets/kubernetes.io/serviceaccount from console-token-j9tnx (ro) /var/service-ca from service-ca (ro) /var/serving-cert from console-serving-cert (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: console-serving-cert: Type: Secret (a volume populated by a Secret) SecretName: console-serving-cert Optional: false console-oauth-config: Type: Secret (a volume populated by a Secret) SecretName: console-oauth-config Optional: false console-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: console-config Optional: false service-ca: Type: ConfigMap (a volume populated by a ConfigMap) Name: service-ca Optional: false default-ingress-cert: Type: ConfigMap (a volume populated by a ConfigMap) Name: default-ingress-cert Optional: false trusted-ca-bundle: Type: ConfigMap (a volume populated by a ConfigMap) Name: trusted-ca-bundle Optional: false console-token-j9tnx: Type: Secret (a volume populated by a Secret) SecretName: console-token-j9tnx Optional: false QoS Class: Burstable Node-Selectors: node-role.kubernetes.io/master= Tolerations: node-role.kubernetes.io/master:NoSchedule op=Exists node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/not-reachable:NoExecute op=Exists for 120s node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 120s Events: <none> [root@arc-npv-ovn-bastion ~]# oc logs console-9d84cdb69-kt5vg -n openshift-console W0112 18:21:46.034283 1 main.go:207] Flag inactivity-timeout is set to less then 300 seconds and will be ignored! I0112 18:21:46.034392 1 main.go:274] cookies are secure! E0112 18:21:46.159002 1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.arc-npv-ovn.redhat.com/oauth/token failed: Head "https://oauth-openshift.apps.arc-npv-ovn.redhat.com": EOF E0112 18:21:56.175380 1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.arc-npv-ovn.redhat.com/oauth/token failed: Head "https://oauth-openshift.apps.arc-npv-ovn.redhat.com": EOF E0112 18:22:06.193330 1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.arc-npv-ovn.redhat.com/oauth/token failed: Head "https://oauth-openshift.apps.arc-npv-ovn.redhat.com": EOF E0112 18:22:16.216196 1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.arc-npv-ovn.redhat.com/oauth/token failed: Head "https://oauth-openshift.apps.arc-npv-ovn.redhat.com": EOF E0112 18:22:26.243055 1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.arc-npv-ovn.redhat.com/oauth/token failed: Head "https://oauth-openshift.apps.arc-npv-ovn.redhat.com": EOF E0112 18:22:36.261441 1 auth.go:235] error contacting auth provider (retrying in 10s): request to OAuth issuer endpoint https://oauth-openshift.apps.arc-npv-ovn.redhat.com/oauth/token failed: Head "https://oauth-openshift.apps.arc-npv-ovn.redhat.com": EOF W0112 18:22:46.309598 1 server.go:421] Failed to get cluster k8s version from api server Get "http://localhost/version?timeout=32s": dial tcp [::1]:80: connect: connection refused, falling back to env var KUBE_GIT_VERSION I0112 18:22:46.309641 1 main.go:654] Binding to [::]:8443... I0112 18:22:46.309661 1 main.go:656] using TLS [root@arc-npv-ovn-bastion ~]# oc version Client Version: 4.7.0-0.nightly-ppc64le-2021-01-11-043556 Server Version: 4.7.0-0.nightly-ppc64le-2021-01-11-043556 Kubernetes Version: v1.20.0+394a5a3 ``` Version-Release number of selected component (if applicable): OCP 4.7 How reproducible: Every time a cluster CNI gets migrated to OVNKube. Steps to Reproduce: 1. Deploy OCP 4.7 cluster on power. 2. Migrate the CNI plugin from openshift-sdn to ovnkube by following the procedure at https://docs.openshift.com/container-platform/4.6/networking/ovn_kubernetes_network_provider/migrate-from-openshift-sdn.html 3. Check the console pods logs and describe it. The console pods go through multiple restarts and display errors. Actual results: After migrating OCP 4.7 cluster on power to OVNKube CNI, the console pods restart multiple times ( around 9 times ). The console pods show oauth connection errors after describing them. Expected results: The console pods should not go through multiple restarts or throw oauth connection errors. Additional info: