Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2020705

Summary: Upgrade starting from 4.2.36 to 4.5.41-> 4.6.49 failing crashlooping container is waiting in apiserver-* pod
Product: OpenShift Container Platform Reporter: Paige Rubendall <prubenda>
Component: Machine Config OperatorAssignee: MCO Team <team-mco>
Machine Config Operator sub component: Machine Config Operator QA Contact: Rio Liu <rioliu>
Status: CLOSED EOL Docs Contact:
Severity: unspecified    
Priority: unspecified CC: aos-bugs, bpickard, jerzhang, jokerman, mfojtik, mkrejci, pmali, prubenda, sttts, surbania, vrutkovs, wking
Version: 4.6   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-11-29 16:58:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Paige Rubendall 2021-11-05 17:00:32 UTC 
Description of problem:
Upgrade form 4.2.36 all the way to 4.6.49 fails with the following message
APIServerDeployment_UnavailablePod APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-oauth-apiserver (crashlooping container is waiting in apiserver-97cf8f95b-kngz9 pod

Version-Release number of selected component (if applicable): 4.6.49


How reproducible: Unknown


Steps to Reproduce:
1. Create 4.2.36 cluster with IPI on GCP (FIPS off)
2. Upgrade to 4.3.40
3. Upgrade to 4.4.33 
4. Upgrade to 4.5.41
5. Upgrade to 4.6.49

Actual results:
Upgrade fails with degraded operators with main message being: APIServerDeployment_UnavailablePod APIServerDeploymentDegraded: 1 of 3 requested instances are unavailable for apiserver.openshift-oauth-apiserver (crashlooping container is waiting in apiserver-97cf8f95b-kngz9 pod

Expected results:
Upgrade passes with no degraded operators or notReady nodes

Additional info:


11-03 17:07:08.038  Post action: #oc get node: NAME                                             STATUS   ROLES    AGE     VERSION           INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                 CONTAINER-RUNTIME
11-03 17:07:08.038  ugd-18-l74kz-m-0.c.openshift-qe.internal         Ready    master   7h      v1.18.3+d8ef5ad   10.0.0.2                    Red Hat Enterprise Linux CoreOS 45.82.202106211530-0 (Ootpa)   4.18.0-193.56.1.el8_2.x86_64   cri-o://1.18.4-11.rhaos4.5.gitfa57051.el8
11-03 17:07:08.038  ugd-18-l74kz-m-1.c.openshift-qe.internal         Ready    master   7h      v1.18.3+d8ef5ad   10.0.0.3                    Red Hat Enterprise Linux CoreOS 45.82.202106211530-0 (Ootpa)   4.18.0-193.56.1.el8_2.x86_64   cri-o://1.18.4-11.rhaos4.5.gitfa57051.el8
11-03 17:07:08.038  ugd-18-l74kz-m-2.c.openshift-qe.internal         Ready    master   7h      v1.18.3+d8ef5ad   10.0.0.5                    Red Hat Enterprise Linux CoreOS 45.82.202106211530-0 (Ootpa)   4.18.0-193.56.1.el8_2.x86_64   cri-o://1.18.4-11.rhaos4.5.gitfa57051.el8
11-03 17:07:08.038  ugd-18-l74kz-w-a-jh4pj.c.openshift-qe.internal   Ready    worker   6h54m   v1.18.3+d8ef5ad   10.0.32.2                   Red Hat Enterprise Linux CoreOS 45.82.202106211530-0 (Ootpa)   4.18.0-193.56.1.el8_2.x86_64   cri-o://1.18.4-11.rhaos4.5.gitfa57051.el8
11-03 17:07:08.038  ugd-18-l74kz-w-b-8hwv4.c.openshift-qe.internal   Ready    worker   6h54m   v1.18.3+d8ef5ad   10.0.32.4                   Red Hat Enterprise Linux CoreOS 45.82.202106211530-0 (Ootpa)   4.18.0-193.56.1.el8_2.x86_64   cri-o://1.18.4-11.rhaos4.5.gitfa57051.el8
11-03 17:07:08.038  ugd-18-l74kz-w-c-fbhtz.c.openshift-qe.internal   Ready    worker   6h54m   v1.18.3+d8ef5ad   10.0.32.3                   Red Hat Enterprise Linux CoreOS 45.82.202106211530-0 (Ootpa)   4.18.0-193.56.1.el8_2.x86_64   cri-o://1.18.4-11.rhaos4.5.gitfa57051.el8
11-03 17:07:08.038  
11-03 17:07:08.038  
11-03 17:07:08.038  Post action: #oc get co:NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
11-03 17:07:08.038  authentication                             4.6.49    True        False         True       146m
11-03 17:07:08.038  cloud-credential                           4.6.49    True        False         False      6h58m
11-03 17:07:08.038  cluster-autoscaler                         4.6.49    True        False         False      6h47m
11-03 17:07:08.038  config-operator                            4.6.49    True        False         False      3h48m
11-03 17:07:08.038  console                                    4.6.49    False       False         False      147m
11-03 17:07:08.038  csi-snapshot-controller                    4.6.49    True        False         False      3h15m
11-03 17:07:08.038  dns                                        4.5.41    True        True          False      6h58m
11-03 17:07:08.038  etcd                                       4.6.49    True        False         False      4h40m
11-03 17:07:08.038  image-registry                             4.6.49    True        False         False      3h14m
11-03 17:07:08.039  ingress                                    4.6.49    True        False         False      157m
11-03 17:07:08.039  insights                                   4.6.49    True        False         False      6h58m
11-03 17:07:08.039  kube-apiserver                             4.6.49    True        False         False      4h38m
11-03 17:07:08.039  kube-controller-manager                    4.6.49    True        False         False      4h36m
11-03 17:07:08.039  kube-scheduler                             4.6.49    True        False         False      4h36m
11-03 17:07:08.039  kube-storage-version-migrator              4.6.49    True        False         False      3h13m
11-03 17:07:08.039  machine-api                                4.6.49    True        False         False      6h58m
11-03 17:07:08.039  machine-approver                           4.6.49    True        False         False      3h44m
11-03 17:07:08.039  machine-config                             4.5.41    True        False         False      178m
11-03 17:07:08.039  marketplace                                4.6.49    True        False         False      148m
11-03 17:07:08.039  monitoring                                 4.6.49    False       False         True       147m
11-03 17:07:08.039  network                                    4.6.49    True        False         False      6h57m
11-03 17:07:08.039  node-tuning                                4.6.49    True        False         False      158m
11-03 17:07:08.039  openshift-apiserver                        4.6.49    True        False         False      148m
11-03 17:07:08.039  openshift-controller-manager               4.6.49    True        False         False      148m
11-03 17:07:08.039  openshift-samples                          4.6.49    True        False         False      156m
11-03 17:07:08.039  operator-lifecycle-manager                 4.6.49    True        False         False      6h48m
11-03 17:07:08.039  operator-lifecycle-manager-catalog         4.6.49    True        False         False      6h48m
11-03 17:07:08.039  operator-lifecycle-manager-packageserver   4.6.49    True        False         False      157m
11-03 17:07:08.039  service-ca                                 4.6.49    True        True          False      6h58m
11-03 17:07:08.039  service-catalog-apiserver                  4.4.33    True        False         False      148m
11-03 17:07:08.039  service-catalog-controller-manager         4.4.33    True        False         False      4h11m
11-03 17:07:08.039  storage                                    4.6.49    True        False         False      158m


Will add must gather and describe of components in separate private comment
Comment 2 Stefan Schimanski 2021-11-08 08:45:58 UTC 
The oauth-apiserver pods reports:

  transport: Error while dialing dial tcp 10.0.0.3:2379: connect: no route to host
  Get "https://172.30.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication?timeout=10s": dial tcp 172.30.0.1:443: connect: no route to host 

This means SDN does not route our traffic.
Comment 5 Alexander Constantinescu 2021-11-11 14:17:04 UTC 
*** Bug 1940844 has been marked as a duplicate of this bug. ***