Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2016115

Summary: Authentication co is degraded: "OAuthServerDeploymentAvailable: no oauth-openshift.openshift-authentication pods available on any node"
Product: OpenShift Container Platform Reporter: Alexander Chuzhoy <sasha>
Component: NetworkingAssignee: Riccardo Ravaioli <rravaiol>
Networking sub component: ovn-kubernetes QA Contact: Anurag saxena <anusaxen>
Status: CLOSED DUPLICATE Docs Contact:
Severity: medium    
Priority: medium CC: akrzos, aos-bugs, bpickard, bzvonar, eglottma, imiller, mcornea, mfojtik, otuchfel, rfreiman, rravaiol, slaznick, surbania
Version: 4.9Keywords: Reopened
Target Milestone: ---Flags: rravaiol: needinfo-
rravaiol: needinfo-
rravaiol: needinfo-
rravaiol: needinfo-
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-11-02 16:01:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Alexander Chuzhoy 2021-10-20 18:40:46 UTC 
Version:
OCP 4.9.0
advanced-cluster-management.v2.4.0


oc --kubeconfig kubeconfig-sno00416 get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version             False       True          21h     Unable to apply 4.9.0: some cluster operators have not yet rolled out



oc --kubeconfig kubeconfig-sno00416 get co
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.9.0     False       False         True       20h     OAuthServerDeploymentAvailable: no oauth-openshift.openshift-authentication pods available on any node....
baremetal                                  4.9.0     True        False         False      20h     
cloud-controller-manager                   4.9.0     True        False         False      20h     
cloud-credential                           4.9.0     True        False         False      21h     
cluster-autoscaler                         4.9.0     True        False         False      20h     
config-operator                            4.9.0     True        False         False      21h     
console                                    4.9.0     False       True          False      20h     DeploymentAvailable: 0 replicas available for console deployment...
csi-snapshot-controller                    4.9.0     True        False         False      21h     
dns                                        4.9.0     True        False         False      20h     
etcd                                       4.9.0     True        False         False      20h     
image-registry                             4.9.0     True        False         False      20h     
ingress                                    4.9.0     True        False         False      20h     
insights                                   4.9.0     True        False         False      20h     
kube-apiserver                             4.9.0     True        False         False      20h     
kube-controller-manager                    4.9.0     True        False         False      20h     
kube-scheduler                             4.9.0     True        False         False      20h     
kube-storage-version-migrator              4.9.0     True        False         False      21h     
machine-api                                4.9.0     True        False         False      20h     
machine-approver                           4.9.0     True        False         False      20h     
machine-config                             4.9.0     True        False         False      20h     
marketplace                                4.9.0     True        False         False      20h     
monitoring                                 4.9.0     True        False         False      20h     
network                                    4.9.0     True        False         False      21h     
node-tuning                                4.9.0     True        False         False      20h     
openshift-apiserver                        4.9.0     True        False         False      122m    
openshift-controller-manager               4.9.0     True        False         False      20h     
openshift-samples                          4.9.0     True        False         False      20h     
operator-lifecycle-manager                 4.9.0     True        False         False      20h     
operator-lifecycle-manager-catalog         4.9.0     True        False         False      20h     
operator-lifecycle-manager-packageserver   4.9.0     True        False         False      20h     
service-ca                                 4.9.0     True        False         False      21h     
storage                                    4.9.0     True        False         False      20h  



oc --kubeconfig kubeconfig-sno00416 get pod -A|grep -v Run|grep -v Comple
NAMESPACE                                          NAME                                                         READY   STATUS             RESTARTS         AGE
openshift-authentication                           oauth-openshift-59cf9c7c74-rq7p2                             0/1     CrashLoopBackOff   248 (2m7s ago)   20h
openshift-console                                  console-77cb7b75f-bq8vs                                      0/1     CrashLoopBackOff   215 (18s ago)    20h





oc --kubeconfig kubeconfig-sno00416 describe pod -n openshift-authentication                           oauth-openshift-59cf9c7c74-rq7p2
Name:                 oauth-openshift-59cf9c7c74-rq7p2
Namespace:            openshift-authentication
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 sno00416/fc00:1001::283
Start Time:           Tue, 19 Oct 2021 21:51:25 +0000
Labels:               app=oauth-openshift
                      oauth-openshift-anti-affinity=true
                      pod-template-hash=59cf9c7c74
Annotations:          k8s.ovn.org/pod-networks:
                        {"default":{"ip_addresses":["fd01:0:0:1::60/64"],"mac_address":"0a:58:28:9b:49:24","gateway_ips":["fd01:0:0:1::1"],"ip_address":"fd01:0:0:...
                      k8s.v1.cni.cncf.io/network-status:
                        [{
                            "name": "ovn-kubernetes",
                            "interface": "eth0",
                            "ips": [
                                "fd01:0:0:1::60"
                            ],
                            "mac": "0a:58:28:9b:49:24",
                            "default": true,
                            "dns": {}
                        }]
                      k8s.v1.cni.cncf.io/networks-status:
                        [{
                            "name": "ovn-kubernetes",
                            "interface": "eth0",
                            "ips": [
                                "fd01:0:0:1::60"
                            ],
                            "mac": "0a:58:28:9b:49:24",
                            "default": true,
                            "dns": {}
                        }]
                      openshift.io/scc: anyuid
                      operator.openshift.io/bootstrap-user-exists: true
                      operator.openshift.io/rvs-hash: ipxrI3UikVvC8-XRwEHxdaJWek5cjCWtQsbxKsSoto42XUAs6NSYzSDnmOij2qAWYgHdFcDf684a7Bh-1OAitQ
                      workload.openshift.io/warning: the node "sno00416" does not have resource "management.workload.openshift.io/cores"
Status:               Running
IP:                   fd01:0:0:1::60
IPs:
  IP:           fd01:0:0:1::60
Controlled By:  ReplicaSet/oauth-openshift-59cf9c7c74
Containers:
  oauth-openshift:
    Container ID:  cri-o://1060c890f8de4ebbb2ed6c8b819b760c8afd35dcc00190be35ad0f3b5dbdc837
    Image:         quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9bf80e8f157528647578258c3bf334853f1d0d38681fff0d670febc45c75624
    Image ID:      quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9bf80e8f157528647578258c3bf334853f1d0d38681fff0d670febc45c75624
    Port:          6443/TCP
    Host Port:     0/TCP
    Command:
      /bin/bash
      -ec
    Args:
      if [ -s /var/config/system/configmaps/v4-0-config-system-trusted-ca-bundle/ca-bundle.crt ]; then
          echo "Copying system trust bundle"
          cp -f /var/config/system/configmaps/v4-0-config-system-trusted-ca-bundle/ca-bundle.crt /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
      fi
      exec oauth-server osinserver \
      --config=/var/config/system/configmaps/v4-0-config-system-cliconfig/v4-0-config-system-cliconfig \
      --v=2
      
    State:       Waiting
      Reason:    CrashLoopBackOff
    Last State:  Terminated
      Reason:    Error
      Message:   e.go:62 +0x154

goroutine 108 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000193680)
  k8s.io/client-go.0/util/workqueue/delaying_queue.go:231 +0x3df
created by k8s.io/client-go/util/workqueue.newDelayingQueue
  k8s.io/client-go.0/util/workqueue/delaying_queue.go:68 +0x185

goroutine 93 [chan receive]:
k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc0003d6ba0)
  k8s.io/client-go.0/util/workqueue/queue.go:204 +0xac
created by k8s.io/client-go/util/workqueue.newQueue
  k8s.io/client-go.0/util/workqueue/queue.go:62 +0x154

goroutine 94 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc0003d6d20)
  k8s.io/client-go.0/util/workqueue/delaying_queue.go:231 +0x3df
created by k8s.io/client-go/util/workqueue.newDelayingQueue
  k8s.io/client-go.0/util/workqueue/delaying_queue.go:68 +0x185

goroutine 95 [chan receive]:
k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc0003d6d80)
  k8s.io/client-go.0/util/workqueue/queue.go:204 +0xac
created by k8s.io/client-go/util/workqueue.newQueue
  k8s.io/client-go.0/util/workqueue/queue.go:62 +0x154

goroutine 96 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc0003d6ea0)
  k8s.io/client-go.0/util/workqueue/delaying_queue.go:231 +0x3df
created by k8s.io/client-go/util/workqueue.newDelayingQueue
  k8s.io/client-go.0/util/workqueue/delaying_queue.go:68 +0x185

goroutine 97 [chan receive]:
k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc0003d6f60)
  k8s.io/client-go.0/util/workqueue/queue.go:204 +0xac
created by k8s.io/client-go/util/workqueue.newQueue
  k8s.io/client-go.0/util/workqueue/queue.go:62 +0x154

goroutine 114 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc0003d70e0)
  k8s.io/client-go.0/util/workqueue/delaying_queue.go:231 +0x3df
created by k8s.io/client-go/util/workqueue.newDelayingQueue
  k8s.io/client-go.0/util/workqueue/delaying_queue.go:68 +0x185

      Exit Code:    255
      Started:      Wed, 20 Oct 2021 18:38:08 +0000
      Finished:     Wed, 20 Oct 2021 18:38:09 +0000
    Ready:          False
    Restart Count:  248
    Requests:
      cpu:        10m
      memory:     50Mi
    Liveness:     http-get https://:6443/healthz delay=30s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get https://:6443/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /var/config/system/configmaps/v4-0-config-system-cliconfig from v4-0-config-system-cliconfig (ro)
      /var/config/system/configmaps/v4-0-config-system-service-ca from v4-0-config-system-service-ca (ro)
      /var/config/system/configmaps/v4-0-config-system-trusted-ca-bundle from v4-0-config-system-trusted-ca-bundle (ro)
      /var/config/system/secrets/v4-0-config-system-ocp-branding-template from v4-0-config-system-ocp-branding-template (ro)
      /var/config/system/secrets/v4-0-config-system-router-certs from v4-0-config-system-router-certs (ro)
      /var/config/system/secrets/v4-0-config-system-serving-cert from v4-0-config-system-serving-cert (ro)
      /var/config/system/secrets/v4-0-config-system-session from v4-0-config-system-session (ro)
      /var/config/user/template/secret/v4-0-config-user-template-error from v4-0-config-user-template-error (ro)
      /var/config/user/template/secret/v4-0-config-user-template-login from v4-0-config-user-template-login (ro)
      /var/config/user/template/secret/v4-0-config-user-template-provider-selection from v4-0-config-user-template-provider-selection (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hjcjx (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  v4-0-config-system-session:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  v4-0-config-system-session
    Optional:    false
  v4-0-config-system-cliconfig:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      v4-0-config-system-cliconfig
    Optional:  false
  v4-0-config-system-serving-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  v4-0-config-system-serving-cert
    Optional:    false
  v4-0-config-system-service-ca:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      v4-0-config-system-service-ca
    Optional:  false
  v4-0-config-system-router-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  v4-0-config-system-router-certs
    Optional:    false
  v4-0-config-system-ocp-branding-template:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  v4-0-config-system-ocp-branding-template
    Optional:    false
  v4-0-config-user-template-login:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  v4-0-config-user-template-login
    Optional:    true
  v4-0-config-user-template-provider-selection:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  v4-0-config-user-template-provider-selection
    Optional:    true
  v4-0-config-user-template-error:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  v4-0-config-user-template-error
    Optional:    true
  v4-0-config-system-trusted-ca-bundle:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      v4-0-config-system-trusted-ca-bundle
    Optional:  true
  kube-api-access-hjcjx:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
    ConfigMapName:           openshift-service-ca.crt
    ConfigMapOptional:       <nil>
QoS Class:                   Burstable
Node-Selectors:              node-role.kubernetes.io/master=
Tolerations:                 node-role.kubernetes.io/master:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 120s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 120s
Events:
  Type     Reason   Age                     From     Message
  ----     ------   ----                    ----     -------
  Normal   Started  63m (x237 over 20h)     kubelet  Started container oauth-openshift
  Normal   Pulled   53m (x239 over 20h)     kubelet  Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9bf80e8f157528647578258c3bf334853f1d0d38681fff0d670febc45c75624" already present on machine
  Warning  BackOff  3m23s (x5969 over 20h)  kubelet  Back-off restarting failed container
Comment 1 Alexander Chuzhoy 2021-10-20 18:41:15 UTC 
Reproduction rate: 1% (reproduced 2 times on 200 deployments)
Comment 3 Standa Laznicka 2021-10-21 08:54:20 UTC 
The oauth-server logs:
```
2021-10-20T18:38:09.796734710Z F1020 18:38:09.796670       1 cmd.go:49] unable to load configmap based request-header-client-ca-file: Get "https://[fd02::1]:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication": dial tcp [fd02::1]:443: connect: no route to host
```

This appears to be SNO, moving to SNO component.
Comment 7 Sergiusz Urbaniak 2021-12-10 08:46:18 UTC 
Moving to networking, the failure "dial tcp [fd02::1]:443: connect: no route to host" indicates an issue with networking.
Comment 10 Alex Krzos 2022-02-14 20:48:40 UTC 
I had this reproduce on a smaller scale staging environment for testing ACM 2.5 ztp provisioning of 100 SNOs.

Current versions are
Hub OCP of 4.9.19, SNO OCP of 4.9.19
ACM Version of 2.5.0-DOWNSTREAM-2022-02-10-07-31-45
97 SNOs successfully provisioned out 100

2 of the failures seem to be exactly the issue as described. (sno00012 and sno00033)

sno00012

[root@f04-h17-b07-5039ms ~]# oc --kubeconfig=/root/acm-ztp/assisted-installer-batch-deploy-tool/clusters/sno00012/kubeconfig get co                                                                  
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE                                                                                            
authentication                             4.9.19    False       False         True       38h     OAuthServerDeploymentAvailable: no oauth-openshift.openshift-authentication pods available on any node.                                                                                                                                                                                                 
baremetal                                  4.9.19    True        False         False      38h                                                                                                        
cloud-controller-manager                   4.9.19    True        False         False      38h                                                                                                        
cloud-credential                           4.9.19    True        False         False      38h                                                                                                        
cluster-autoscaler                         4.9.19    True        False         False      38h                                                                                                        
config-operator                            4.9.19    True        False         False      38h                                                                                                        
console                                    4.9.19    False       True          False      38h     DeploymentAvailable: 0 replicas available for console deployment...                                
csi-snapshot-controller                    4.9.19    True        False         False      38h                                                                                                        
dns                                        4.9.19    True        False         False      38h                                                                                                        
etcd                                       4.9.19    True        False         False      38h                                                                                                        
image-registry                             4.9.19    True        False         False      38h                                                                                                        
ingress                                    4.9.19    True        False         False      38h                                                                                                        
insights                                   4.9.19    True        False         False      38h                                                                                                        
kube-apiserver                             4.9.19    True        False         False      38h                                                                                             
kube-controller-manager                    4.9.19    True        False         False      38h                                                                                                        
kube-scheduler                             4.9.19    True        False         False      38h                                                                                                        
kube-storage-version-migrator              4.9.19    True        False         False      38h                                                                                                        
machine-api                                4.9.19    True        False         False      38h                                                                                                        
machine-approver                           4.9.19    True        False         False      38h                                                                                                        
machine-config                             4.9.19    True        False         False      38h                                                                                                        
marketplace                                4.9.19    True        False         False      38h                                                                                                        
monitoring                                 4.9.19    True        False         False      38h                                                                                                        
network                                    4.9.19    True        False         False      38h                                                                                                       
node-tuning                                4.9.19    True        False         False      38h                                                                                                        
openshift-apiserver                        4.9.19    True        False         False      15h                                                                                             
openshift-controller-manager               4.9.19    True        False         False      19h
openshift-samples                          4.9.19    True        False         False      38h
operator-lifecycle-manager                 4.9.19    True        False         False      38h
operator-lifecycle-manager-catalog         4.9.19    True        False         False      38h
operator-lifecycle-manager-packageserver   4.9.19    True        False         False      38h
service-ca                                 4.9.19    True        False         False      38h
storage                                    4.9.19    True        False         False      38h



and on sno00033

[root@sno00033 ~]# oc --kubeconfig=/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/node-kubeconfigs/localhost.kubeconfig get no,clusterversion,co                                 
NAME            STATUS   ROLES           AGE   VERSION
node/sno00033   Ready    master,worker   36h   v1.22.3+2cb6068

NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
clusterversion.config.openshift.io/version             False       True          37h     Unable to apply 4.9.19: some cluster operators have not yet rolled out                                     

NAME                                                                           VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE                                                       
clusteroperator.config.openshift.io/authentication                             4.9.19    False       False         True       36h     OAuthServerDeploymentAvailable: no oauth-openshift.openshift-authentication pods available on any node....
clusteroperator.config.openshift.io/baremetal                                  4.9.19    True        False         False      36h                                                                   
clusteroperator.config.openshift.io/cloud-controller-manager                   4.9.19    True        False         False      36h                                                                   
clusteroperator.config.openshift.io/cloud-credential                           4.9.19    True        False         False      36h                                                                   
clusteroperator.config.openshift.io/cluster-autoscaler                         4.9.19    True        False         False      36h                                                                   
clusteroperator.config.openshift.io/config-operator                            4.9.19    True        False         False      36h                                                                   
clusteroperator.config.openshift.io/console                                    4.9.19    False       True          False      36h     DeploymentAvailable: 0 replicas available for console deployment...
clusteroperator.config.openshift.io/csi-snapshot-controller                    4.9.19    True        False         False      36h                                                                   
clusteroperator.config.openshift.io/dns                                        4.9.19    True        False         False      36h                                                                   
clusteroperator.config.openshift.io/etcd                                       4.9.19    True        False         False      36h                                                                   
clusteroperator.config.openshift.io/image-registry                             4.9.19    True        False         False      36h                                                                   
clusteroperator.config.openshift.io/ingress                                    4.9.19    True        False         False      36h                                                                   
clusteroperator.config.openshift.io/insights                                   4.9.19    True        False         False      36h                                                                   
clusteroperator.config.openshift.io/kube-apiserver                             4.9.19    True        False         False      36h                                                                   
clusteroperator.config.openshift.io/kube-controller-manager                    4.9.19    True        False         False      36h                                                                   
clusteroperator.config.openshift.io/kube-scheduler                             4.9.19    True        False         False      36h                                                                   
clusteroperator.config.openshift.io/kube-storage-version-migrator              4.9.19    True        False         False      36h                                                                   
clusteroperator.config.openshift.io/machine-api                                4.9.19    True        False         False      36h                                                                   
clusteroperator.config.openshift.io/machine-approver                           4.9.19    True        False         False      36h                                                                   
clusteroperator.config.openshift.io/machine-config                             4.9.19    True        False         False      36h                                                                   
clusteroperator.config.openshift.io/marketplace                                4.9.19    True        False         False      36h                                                                   
clusteroperator.config.openshift.io/monitoring                                 4.9.19    True        False         False      36h                                                                   
clusteroperator.config.openshift.io/network                                    4.9.19    True        False         False      36h                                                                   
clusteroperator.config.openshift.io/node-tuning                                4.9.19    True        False         False      36h                                                                   
clusteroperator.config.openshift.io/openshift-apiserver                        4.9.19    True        False         False      17h                                                                   
clusteroperator.config.openshift.io/openshift-controller-manager               4.9.19    True        False         False      17h                                                                   
clusteroperator.config.openshift.io/openshift-samples                          4.9.19    True        False         False      36h                                                                   
clusteroperator.config.openshift.io/operator-lifecycle-manager                 4.9.19    True        False         False      36h                                                                   
clusteroperator.config.openshift.io/operator-lifecycle-manager-catalog         4.9.19    True        False         False      36h                                                                   
clusteroperator.config.openshift.io/operator-lifecycle-manager-packageserver   4.9.19    True        False         False      36h                                                                   
clusteroperator.config.openshift.io/service-ca                                 4.9.19    True        False         False      36h                                                                   
clusteroperator.config.openshift.io/storage                                    4.9.19    True        False         False      36h



On sno00012, I saw the same crashlooping of the oauth and console pods:

# oc --kubeconfig=/root/acm-ztp/assisted-installer-batch-deploy-tool/clusters/sno00012/kubeconfig get po -A | grep -v Running | grep -v Complete                          
NAMESPACE                                          NAME                                                         READY   STATUS              RESTARTS          AGE                                    
openshift-authentication                           oauth-openshift-54df474554-9h7m6                             0/1     CrashLoopBackOff    453 (105s ago)    38h                                    
openshift-console                                  console-5bb9b6cfc9-x2z9r                                     0/1     CrashLoopBackOff    395 (94s ago)     38h                                    
openshift-marketplace                              certified-operators-5zx6r                                    0/1     ImagePullBackOff    0                 38h                                    
openshift-marketplace                              certified-operators-8d88z                                    0/1     ImagePullBackOff    0                 38h                                    
openshift-marketplace                              community-operators-kfhkx                                    0/1     ImagePullBackOff    0                 38h                                    
openshift-marketplace                              community-operators-t79q7                                    0/1     ImagePullBackOff    0                 38h                                    
openshift-marketplace                              redhat-marketplace-mhpn8                                     0/1     ImagePullBackOff    0                 38h                                    
openshift-marketplace                              redhat-marketplace-skf4f                                     0/1     ImagePullBackOff    0                 38h                         
openshift-marketplace                              redhat-operators-gsz82                                       0/1     ImagePullBackOff    0                 38h                                    
openshift-marketplace                              redhat-operators-wcg8w                                       0/1     ImagePullBackOff    0                 38h                                    
openshift-multus                                   ip-reconciler-27414510--1-bsplv                              0/1     ContainerCreating   0                 0s                                     
openshift-operator-lifecycle-manager               collect-profiles-27414510--1-gjw4v                           0/1     ContainerCreating   0                 0s 


I checked if simply deleting the pod would resolve the issue and it has:

[root@f04-h17-b07-5039ms ~]# oc --kubeconfig=/root/acm-ztp/assisted-installer-batch-deploy-tool/clusters/sno00012/kubeconfig delete po -n openshift-authentication oauth-openshift-54df474554-9h7m6  
pod "oauth-openshift-54df474554-9h7m6" deleted

# oc --kubeconfig=/root/acm-ztp/assisted-installer-batch-deploy-tool/clusters/sno00012/kubeconfig get clusterversion,co
NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
clusterversion.config.openshift.io/version   4.9.19    True        False         12m     Cluster version is 4.9.19

NAME                                                                           VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
clusteroperator.config.openshift.io/authentication                             4.9.19    True        False         False      14m     
clusteroperator.config.openshift.io/baremetal                                  4.9.19    True        False         False      38h     
clusteroperator.config.openshift.io/cloud-controller-manager                   4.9.19    True        False         False      38h     
clusteroperator.config.openshift.io/cloud-credential                           4.9.19    True        False         False      38h     
clusteroperator.config.openshift.io/cluster-autoscaler                         4.9.19    True        False         False      38h     
clusteroperator.config.openshift.io/config-operator                            4.9.19    True        False         False      39h     
clusteroperator.config.openshift.io/console                                    4.9.19    True        False         False      12m     
clusteroperator.config.openshift.io/csi-snapshot-controller                    4.9.19    True        False         False      38h     
clusteroperator.config.openshift.io/dns                                        4.9.19    True        False         False      38h     
clusteroperator.config.openshift.io/etcd                                       4.9.19    True        False         False      38h     
clusteroperator.config.openshift.io/image-registry                             4.9.19    True        False         False      38h     
clusteroperator.config.openshift.io/ingress                                    4.9.19    True        False         False      38h     
clusteroperator.config.openshift.io/insights                                   4.9.19    True        False         False      38h     
clusteroperator.config.openshift.io/kube-apiserver                             4.9.19    True        False         False      38h     
clusteroperator.config.openshift.io/kube-controller-manager                    4.9.19    True        False         False      38h     
clusteroperator.config.openshift.io/kube-scheduler                             4.9.19    True        False         False      38h     
clusteroperator.config.openshift.io/kube-storage-version-migrator              4.9.19    True        False         False      38h     
clusteroperator.config.openshift.io/machine-api                                4.9.19    True        False         False      38h     
clusteroperator.config.openshift.io/machine-approver                           4.9.19    True        False         False      38h     
clusteroperator.config.openshift.io/machine-config                             4.9.19    True        False         False      38h     
clusteroperator.config.openshift.io/marketplace                                4.9.19    True        False         False      38h     
clusteroperator.config.openshift.io/monitoring                                 4.9.19    True        False         False      38h     
clusteroperator.config.openshift.io/network                                    4.9.19    True        False         False      39h     
clusteroperator.config.openshift.io/node-tuning                                4.9.19    True        False         False      38h     
clusteroperator.config.openshift.io/openshift-apiserver                        4.9.19    True        False         False      15h     
clusteroperator.config.openshift.io/openshift-controller-manager               4.9.19    True        False         False      20h     
clusteroperator.config.openshift.io/openshift-samples                          4.9.19    True        False         False      38h     
clusteroperator.config.openshift.io/operator-lifecycle-manager                 4.9.19    True        False         False      38h     
clusteroperator.config.openshift.io/operator-lifecycle-manager-catalog         4.9.19    True        False         False      38h     
clusteroperator.config.openshift.io/operator-lifecycle-manager-packageserver   4.9.19    True        False         False      38h     
clusteroperator.config.openshift.io/service-ca                                 4.9.19    True        False         False      38h     
clusteroperator.config.openshift.io/storage                                    4.9.19    True        False         False      38h


I will post the journal logs of both clusters shortly
Comment 13 Omer Tuchfeld 2022-03-16 16:00:04 UTC 
We started defaulting OVN on SNO in Assisted Installer a few days ago and this exact bug began showing up in our CI:
https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-assisted-test-infra-master-e2e-metal-assisted-single-node-periodic/1503919724587651072
Comment 15 Riccardo Ravaioli 2022-03-18 17:11:32 UTC 
It's indeed the same issue described above. I am adding below a full analysis, from which this really looks like https://bugzilla.redhat.com/show_bug.cgi?id=2060534, which was fixed upstream and then downstream in the 4.11 branch only two weeks ago. I will work on the backport of that patch down to 4.9.

==========================================

So here is what happens from the logs.

***** The pod `oauth-openshift-58655c7994-ccvph` keeps restarting because it fails to connect to the API server:

2022-03-16T04:24:34.280454075Z F0316 04:24:34.280403       1 cmd.go:49] unable to load configmap based request-header-client-ca-file: Get "https://10.128.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication": dial tcp 10.128.0.1:443: connect: no route to host

$ omg get pods -A | grep oauth-openshift
openshift-authentication                          oauth-openshift-58655c7994-ccvph                               0/1    Running    15        54m

***** Now, nothing else is degraded apart from the console operator, which actually depends on authentication.
In OVN-K node, the CNI add for that particular pod was successful, but 15 seconds later the corresponding OVS port got deleted for no valid reason:

$ omg logs ovnkube-node-fd88n -n openshift-ovn-kubernetes -c ovnkube-node | grep 85b8d06134ceb50
2022-03-16T03:35:54.528269376Z I0316 03:35:54.528203    6661 cni.go:169] [openshift-authentication/oauth-openshift-58655c7994-ccvph 
 5b8d06134ceb503b4eb580e641aee072c448a827e23cc168a6d77b17ab17f18] ADD starting CNI request [openshift-authentication/oauth-openshift-58655c7994-ccvph 85b8d06134ceb503b4eb580e641aee072c448a827e23cc168a6d77b17ab17f18]

2022-03-16T03:35:56.429456526Z I0316 03:35:56.429330    6661 cni.go:190] [openshift-authentication/oauth-openshift-58655c7994-ccvph 85b8d06134ceb503b4eb580e641aee072c448a827e23cc168a6d77b17ab17f18] ADD finished CNI request [openshift-authentication/oauth-openshift-58655c7994-ccvph 85b8d06134ceb503b4eb580e641aee072c448a827e23cc168a6d77b17ab17f18], result "{\"interfaces\":[{\"name\":\"85b8d06134ceb50\",\"mac\":\"7a:9e:dd:d9:10:4f\"},{\"name\":\"eth0\",\"mac\":\"0a:58:ac:1e:00:53\",\"sandbox\":\"/var/run/netns/7822a7f3-e1a8-4b06-8894-0a42858aec8e\"}],\"ips\":[{\"version\":\"4\",\"interface\":1,\"address\":\"172.30.0.83/23\",\"gateway\":\"172.30.0.1\"}],\"dns\":{}}", err <nil>

2022-03-16T03:36:11.653368117Z W0316 03:36:11.653231    6661 healthcheck.go:196] Found stale OVS Interface, deleting OVS Port with interface 85b8d06134ceb50


*****  ovn-controller shows the corresponding OVN actions for this pod, with the last message showing that port for the logical port for this pod was released:

$ omg logs ovnkube-node-fd88n -n openshift-ovn-kubernetes -c ovn-controller | grep oauth-openshift-58655c7994-ccvph
2022-03-16T03:35:54.647605135Z 2022-03-16T03:35:54Z|00389|binding|INFO|Claiming lport openshift-authentication_oauth-openshift-58655c7994-ccvph for this chassis.
2022-03-16T03:35:54.647653382Z 2022-03-16T03:35:54Z|00390|binding|INFO|openshift-authentication_oauth-openshift-58655c7994-ccvph: Claiming 0a:58:ac:1e:00:53 172.30.0.83
2022-03-16T03:35:54.676139531Z 2022-03-16T03:35:54Z|00395|binding|INFO|Setting lport openshift-authentication_oauth-openshift-58655c7994-ccvph ovn-installed in OVS
2022-03-16T03:35:54.676160349Z 2022-03-16T03:35:54Z|00396|binding|INFO|Setting lport openshift-authentication_oauth-openshift-58655c7994-ccvph up in Southbound
2022-03-16T03:36:11.672236526Z 2022-03-16T03:36:11Z|00463|binding|INFO|Releasing lport openshift-authentication_oauth-openshift-58655c7994-ccvph from this chassis.

***** This happens of course all the way down to OVS:

host_service_logs/masters/ovs-vswitchd_service.log:324:Mar 16 03:35:54.644733 test-infra-cluster-495603bc-master-0 ovs-vswitchd[1253]: ovs|00306|bridge|INFO|bridge br-int: added interface 85b8d06134ceb50 on port 85
host_service_logs/masters/ovs-vswitchd_service.log:346:Mar 16 03:36:11.696045 test-infra-cluster-495603bc-master-0 ovs-vswitchd[1253]: ovs|00328|bridge|INFO|bridge br-int: deleted interface 85b8d06134ceb50 on port 85

***** So of course from this point onwards (after 03:36:11.696045), the pod has no connectivity, as kubelet reports:

host_service_logs/masters/kubelet_service.log:9761:Mar 16 03:35:57.372341 test-infra-cluster-495603bc-master-0 hyperkube[15412]: I0316 03:35:57.372326   15412 kubelet.go:2186] "SyncLoop (probe)" probe="readiness" status="" pod="openshift-authentication/oauth-openshift-58655c7994-ccvph"

host_service_logs/masters/kubelet_service.log:9774:Mar 16 03:35:58.027273 test-infra-cluster-495603bc-master-0 hyperkube[15412]: I0316 03:35:58.027237   15412 kubelet.go:2186] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-authentication/oauth-openshift-58655c7994-ccvph"

host_service_logs/masters/kubelet_service.log:9995:Mar 16 03:36:14.914303 test-infra-cluster-495603bc-master-0 hyperkube[15412]: I0316 03:36:14.914238   15412 patch_prober.go:29] interesting pod/oauth-openshift-58655c7994-ccvph container/oauth-openshift namespace/openshift-authentication: Readiness probe status=failure output="Get \"https://172.30.0.83:6443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=host_service_logs/masters/kubelet_service.log:9996:Mar 16 03:36:14.914541 test-infra-cluster-495603bc-master-0 hyperkube[15412]: I0316 03:36:14.914341   15412 prober.go:116] "Probe failed" probeType="Readiness" pod="openshift-authentication/oauth-openshift-58655c7994-ccvph" podUID=6219de8f-453a-4370-90ff-e1c41cc0a1d2 containerName="oauth-openshift" probeResult=failure output="Get \"https://172.30.0.83:6443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"

host_service_logs/masters/kubelet_service.log:10288:Mar 16 03:36:24.913514 test-infra-cluster-495603bc-master-0 hyperkube[15412]: I0316 03:36:24.913382   15412 patch_prober.go:29] interesting pod/oauth-openshift-58655c7994-ccvph container/oauth-openshift namespace/openshift-authentication: Readiness probe status=failure output="Get \"https://172.30.0.83:6443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=




*****  The bug is due to the VF cleanup in checkForStaleOVSRepresentorInterfaces not making sure it's /really/ considering only VFs...
https://github.com/openshift/ovn-kubernetes/blob/release-4.10/go-controller/pkg/node/healthcheck.go#L227

The same issue was reported here: https://bugzilla.redhat.com/show_bug.cgi?id=2060534

A fix was merged upstream: https://github.com/ovn-org/ovn-kubernetes/pull/2845 and to downstream master: https://github.com/openshift/ovn-kubernetes/commit/77678fbe280a701351decded0a16017f4e7a18aa

I will backport Tim's fix to 4.9.
Comment 16 Riccardo Ravaioli 2022-03-18 17:28:25 UTC 

*** This bug has been marked as a duplicate of bug 2060534 ***
Comment 17 Riccardo Ravaioli 2022-03-18 17:43:23 UTC 
Backport BZs and PRs:
** 4.10
https://bugzilla.redhat.com/show_bug.cgi?id=2065780
https://github.com/openshift/ovn-kubernetes/pull/1001

** 4.9
https://bugzilla.redhat.com/show_bug.cgi?id=2065782
https://github.com/openshift/ovn-kubernetes/pull/1002
Comment 18 Alexander Chuzhoy 2022-03-21 19:45:39 UTC 
@rravaiol I see this bug is closed as a duplicate. What's the needinfo for?
Comment 19 Riccardo Ravaioli 2022-03-22 08:59:01 UTC 
needinfo not needed at all, I actually wanted to remove it. My bad, sorry!
Comment 20 Alex Krzos 2022-06-02 14:27:04 UTC 
Re-opening this bug as we still witnessed this issue in the last large scale install of SNOs from ACM 2.5.

* 4.9.36 OCP
* 2296 SNOs attempted to be installed
* 2224 Successful, 72 failures
* 12 of those failures where of this bug

# cat run33-fail-sorted | grep AuthContainerCreating | awk '{print $2}' | xargs -I % oc --kubeconfig=/root/hv-sno/manifests/%/kubeconfig get co authentication 
NAME             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication   4.9.36    False       False         True       16h     OAuthServerDeploymentAvailable: no oauth-openshift.openshift-authentication pods available on any node....
NAME             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication   4.9.36    False       False         True       16h     OAuthServerDeploymentAvailable: no oauth-openshift.openshift-authentication pods available on any node....
NAME             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication   4.9.36    False       False         True       16h     OAuthServerDeploymentAvailable: no oauth-openshift.openshift-authentication pods available on any node....
NAME             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication   4.9.36    False       False         True       16h     APIServerDeploymentAvailable: no apiserver.openshift-oauth-apiserver pods available on any node....
NAME             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication   4.9.36    False       False         True       15h     OAuthServerDeploymentAvailable: no oauth-openshift.openshift-authentication pods available on any node....
NAME             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication   4.9.36    False       False         True       15h     OAuthServerDeploymentAvailable: no oauth-openshift.openshift-authentication pods available on any node....
NAME             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication   4.9.36    False       False         True       15h     OAuthServerDeploymentAvailable: no oauth-openshift.openshift-authentication pods available on any node....
NAME             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication   4.9.36    False       False         True       15h     OAuthServerDeploymentAvailable: no oauth-openshift.openshift-authentication pods available on any node....
NAME             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication   4.9.36    False       False         True       14h     OAuthServerDeploymentAvailable: no oauth-openshift.openshift-authentication pods available on any node....
NAME             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication   4.9.36    False       False         True       14h     OAuthServerDeploymentAvailable: no oauth-openshift.openshift-authentication pods available on any node....
NAME             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication   4.9.36    False       False         True       14h     OAuthServerDeploymentAvailable: no oauth-openshift.openshift-authentication pods available on any node....
NAME             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication   4.9.36    False       False         True       14h     OAuthServerDeploymentAvailable: no oauth-openshift.openshift-authentication pods available on any node....