Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1997292

Summary:	openshift-apiserver degraded with "unable to handle the request" over OpenShift 4.7.24 VSphere UPI
Product:	OpenShift Container Platform	Reporter:	aygarg
Component:	openshift-apiserver	Assignee:	Stefan Schimanski <sttts>
Status:	CLOSED DUPLICATE	QA Contact:	Xingxing Xia <xxia>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	4.7	CC:	anr, aos-bugs, mfojtik, sreber
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-08-25 14:22:52 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description aygarg 2021-08-24 20:27:48 UTC

Description of problem:
Initially, the cluster upgrade from 4.7.23 to 4.7.24 was almost done but was incomplete because some of the operators weren't rolled out. The cluster was having exactly the same issue as per the following KCS.
--> https://access.redhat.com/solutions/5997331

We applied the ethtool workaround but nothing worked.
--> # ethtool -K <primary-interface> tx-udp_tnl-segmentation off
--> # ethtool -K <primary-interface> tx-udp_tnl-csum-segmentation off

So we ran the following command on all the nodes, which fixed the issue and the upgrade was completed.
--> # ethtool -K <primary-interface> tx-checksum-ip-generic off

Since, the workaround that we applied was not permanent so we manually rebooted the nodes just to check whether the cluster will again degrade or not. The cluster operators were degraded after node reboots i.e. the workaround gets removed.

Version-Release number of selected component (if applicable):
OpenShift 4.7.24 (VSphere UPI)

Actual results:
The openshift-apiserver operator is in a degraded state with the following errors.
~~~
APIServicesAvailable: "project.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request)
APIServicesAvailable: "quota.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request)
APIServicesAvailable: "route.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request)
APIServicesAvailable: "security.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request)
APIServicesAvailable: "template.openshift.io.v1" is not ready: 503 (the server is currently unable to handle the request))
~~~

Also, authentication operator degraded since the OAuth route over /healthz path isn't accessible because the router pods aren't serving traffic until we disable "tx-checksum-ip-generic" on the nodes where router pods are running.

Expected results:
A permanent fix for this issue as the "tx-checksum-ip-generic" workaround isn't feasible and the issue started coming up only in the 4.7.24 version.

Additional info:
- The issue seems to be similar to https://bugzilla.redhat.com/show_bug.cgi?id=1987108 Bugzilla.
- I will attach the must-gather.

Comment 2 aygarg 2021-08-24 20:36:07 UTC

This is how the cluster operators are without the workaround applied.

$ omg get co
NAME                                      VERSION  AVAILABLE  PROGRESSING  DEGRADED  SINCE
authentication                            4.7.24   False      True         True      6m34s
baremetal                                 4.7.24   True       False        False     118d
cloud-credential                          4.7.24   True       False        False     1d
cluster-autoscaler                        4.7.24   True       False        False     1d
config-operator                           4.7.24   True       False        False     371d
console                                   4.7.24   False      False        True      1m8s
csi-snapshot-controller                   4.7.24   True       False        False     7m2s
dns                                       4.7.24   True       False        False     6m1s
etcd                                      4.7.24   True       False        False     7m2s
image-registry                            4.7.24   True       False        False     23h
ingress                                   4.7.24   True       False        False     3m54s
insights                                  4.7.24   True       False        False     105d
kube-apiserver                            4.7.24   True       False        False     10m
kube-controller-manager                   4.7.24   True       False        False     1h50m
kube-scheduler                            4.7.24   True       False        False     1h42m
kube-storage-version-migrator             4.7.24   True       False        False     16m
machine-api                               4.7.24   True       False        False     1d
machine-approver                          4.7.24   True       False        False     371d
machine-config                            4.7.24   True       False        False     1d
marketplace                               4.7.24   True       False        False     1h40m
monitoring                                4.7.24   False      True         True      12s
network                                   4.7.24   True       False        False     5m42s
node-tuning                               4.7.24   True       False        False     1d
openshift-apiserver                       4.7.24   False      False        False     6m50s
openshift-controller-manager              4.7.24   True       False        False     1d
openshift-samples                         4.7.24   True       False        False     1d
operator-lifecycle-manager                4.7.24   True       False        False     1d
operator-lifecycle-manager-catalog        4.7.24   True       False        False     1d
operator-lifecycle-manager-packageserver  4.7.24   False      True         False     4m17s
service-ca                                4.7.24   True       False        False     9m37s
service-catalog-apiserver                 4.4.14   True       False        False     242d
service-catalog-controller-manager        4.4.14   True       False        False     243d
storage                                   4.7.24   True       False        False     13m


$ omg get co openshift-apiserver -oyaml
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: '2019-07-29T18:57:58Z'
  generation: 1
  name: openshift-apiserver
  resourceVersion: '1703592479'
  selfLink: /apis/config.openshift.io/v1/clusteroperators/openshift-apiserver
  uid: c7bc573c-b232-11e9-bf7f-005056a0d082
spec: {}
status:
  conditions:
  - lastTransitionTime: '2021-08-24T18:50:35Z'
    message: All is well
    reason: AsExpected
    status: 'False'
    type: Degraded
  - lastTransitionTime: '2021-08-16T13:10:04Z'
    message: All is well
    reason: AsExpected
    status: 'False'
    type: Progressing
  - lastTransitionTime: '2021-08-24T18:44:06Z'
    message: 'APIServicesAvailable: "authorization.openshift.io.v1" is not ready:
      503 (the server is currently unable to handle the request)

      APIServicesAvailable: "build.openshift.io.v1" is not ready: 503 (the server
      is currently unable to handle the request)

      APIServicesAvailable: "quota.openshift.io.v1" is not ready: 503 (the server
      is currently unable to handle the request)

      APIServicesAvailable: "security.openshift.io.v1" is not ready: 503 (the server
      is currently unable to handle the request)

      APIServicesAvailable: "template.openshift.io.v1" is not ready: 503 (the server
      is currently unable to handle the request)'
    reason: APIServices_Error
    status: 'False'
    type: Available
  - lastTransitionTime: '2019-07-29T18:57:59Z'
    message: All is well
    reason: AsExpected
    status: 'True'
    type: Upgradeable
  extension: null
  relatedObjects:
  - group: operator.openshift.io
    name: cluster
    resource: openshiftapiservers
  - group: ''
    name: openshift-config
    resource: namespaces
  - group: ''
    name: openshift-config-managed
    resource: namespaces
  - group: ''
    name: openshift-apiserver-operator
    resource: namespaces
  - group: ''
    name: openshift-apiserver
    resource: namespaces
  - group: ''
    name: openshift-etcd-operator
    resource: namespaces
  - group: ''
    name: host-etcd-2
    namespace: openshift-etcd
    resource: endpoints
  - group: controlplane.operator.openshift.io
    name: ''
    namespace: openshift-apiserver
    resource: podnetworkconnectivitychecks
  - group: apiregistration.k8s.io
    name: v1.apps.openshift.io
    resource: apiservices
  - group: apiregistration.k8s.io
    name: v1.authorization.openshift.io
    resource: apiservices
  - group: apiregistration.k8s.io
    name: v1.build.openshift.io
    resource: apiservices
  - group: apiregistration.k8s.io
    name: v1.image.openshift.io
    resource: apiservices
  - group: apiregistration.k8s.io
    name: v1.project.openshift.io
    resource: apiservices
  - group: apiregistration.k8s.io
    name: v1.quota.openshift.io
    resource: apiservices
  - group: apiregistration.k8s.io
    name: v1.route.openshift.io
    resource: apiservices
  - group: apiregistration.k8s.io
    name: v1.security.openshift.io
    resource: apiservices
  - group: apiregistration.k8s.io
    name: v1.template.openshift.io
    resource: apiservices
  versions:
  - name: operator
    version: 4.7.24
  - name: openshift-apiserver
    version: 4.7.24




$ omg get co authentication -oyaml
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
  creationTimestamp: '2019-07-29T19:01:40Z'
  generation: 1
  name: authentication
  resourceVersion: '1703592572'
  selfLink: /apis/config.openshift.io/v1/clusteroperators/authentication
  uid: 4c25d47e-b233-11e9-8688-005056a00703
spec: {}
status:
  conditions:
  - lastTransitionTime: '2021-08-24T18:46:37Z'
    message: 'OAuthServerDeploymentDegraded: Unable to get "openshift-browser-client"
      bootstrapped OAuth client: the server is currently unable to handle the request
      (post oauthclients.oauth.openshift.io)

      OAuthRouteCheckEndpointAccessibleControllerDegraded: Get "https://oauth-openshift.apps.ocp4.xxxxx.xxxxx/healthz":
      EOF

      RouteDegraded: Unable to get required route openshift-authentication/oauth-openshift:
      the server is currently unable to handle the request (get routes.route.openshift.io
      oauth-openshift)'
    reason: OAuthRouteCheckEndpointAccessibleController_SyncError::OAuthServerDeployment_GetFailed::Route_FailedGet
    status: 'True'
    type: Degraded
  - lastTransitionTime: '2021-08-24T18:50:51Z'
    message: 'OAuthVersionRouteProgressing: Request to "https://oauth-openshift.apps.ocp4.xxxxx.xxxxx/healthz"
      not successful yet'
    reason: OAuthVersionRoute_WaitingForRoute
    status: 'True'
    type: Progressing
  - lastTransitionTime: '2021-08-24T18:46:59Z'
    message: 'APIServicesAvailable: "user.openshift.io.v1" is not ready: 503 (the
      server is currently unable to handle the request)

      OAuthRouteCheckEndpointAccessibleControllerAvailable: Get "https://oauth-openshift.apps.ocp4.xxxxx.xxxxx/healthz":
      EOF

      OAuthVersionRouteAvailable: HTTP request to "https://oauth-openshift.apps.ocp4.xxxxx.xxxxx/healthz"
      failed: EOF'
    reason: APIServices_Error::OAuthRouteCheckEndpointAccessibleController_EndpointUnavailable::OAuthVersionRoute_RequestFailed
    status: 'False'
    type: Available
  - lastTransitionTime: '2019-08-09T14:23:23Z'
    message: All is well
    reason: AsExpected
    status: 'True'
    type: Upgradeable
  extension: null
  relatedObjects:
  - group: operator.openshift.io
    name: cluster
    resource: authentications
  - group: config.openshift.io
    name: cluster
    resource: authentications
  - group: config.openshift.io
    name: cluster
    resource: infrastructures
  - group: config.openshift.io
    name: cluster
    resource: oauths
  - group: route.openshift.io
    name: oauth-openshift
    namespace: openshift-authentication
    resource: routes
  - group: ''
    name: oauth-openshift
    namespace: openshift-authentication
    resource: services
  - group: ''
    name: openshift-config
    resource: namespaces
  - group: ''
    name: openshift-config-managed
    resource: namespaces
  - group: ''
    name: openshift-authentication
    resource: namespaces
  - group: ''
    name: openshift-authentication-operator
    resource: namespaces
  - group: ''
    name: openshift-ingress
    resource: namespaces
  - group: ''
    name: openshift-oauth-apiserver
    resource: namespaces
  versions:
  - name: operator
    version: 4.7.24
  - name: oauth-openshift
    version: 4.7.24_openshift
  - name: oauth-apiserver
    version: 4.7.24

Comment 3 Simon Reber 2021-08-25 14:22:52 UTC


*** This bug has been marked as a duplicate of bug 1987108 ***