Bug 1855500

Hello Wang Haoran, can you share both the kube-apiserver and openshift-apiserver logs around the time the incident occurred? Could be something related to invalid certificates.

Thanks for the logs!!! kube-apiserver-ip-10-0-187-163.ec2.internal.log and kube-apiserver-ip-10-0-252-80.ec2.internal.log contains the following panic:
```
E0806 09:33:24.563135       1 runtime.go:76] Observed a panic: No match; rd=RequestDigest{RequestInfo: &request.RequestInfo{IsResourceRequest:true, Path:"/apis/user.openshift.io/v1/users/afuc6@customdomain", Verb:"delete", APIPrefix:"apis", APIGroup:"user.openshift.io", APIVersion:"v1", Namespace:"", Resource:"users", Subresource:"", Name:"afuc6@customdomain", Parts:[]string{"users", "afuc6@customdomain"}}, User: &user.DefaultInfo{Name:"test@customdomain", UID:"", Groups:[]string{"dedicated-admins"}, Extra:map[string][]string{}}}, catchAll={"metadata":{"name":"catch-all","selfLink":"/apis/flowcontrol.apiserver.k8s.io/v1alpha1/flowschemas/catch-all","uid":"6ee022c7-65ef-4ee3-9547-5b804f76e9b5","resourceVersion":"87","generation":1,"creationTimestamp":"2020-08-06T01:09:58Z","managedFields":[{"manager":"api-priority-and-fairness-config-consumer-v1","operation":"Update","apiVersion":"flowcontrol.apiserver.k8s.io/v1alpha1","time":"2020-08-06T01:09:58Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{".":{},"k:{\"type\":\"Dangling\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}},{"manager":"api-priority-and-fairness-config-producer-v1","operation":"Update","apiVersion":"flowcontrol.apiserver.k8s.io/v1alpha1","time":"2020-08-06T01:09:58Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:distinguisherMethod":{".":{},"f:type":{}},"f:matchingPrecedence":{},"f:priorityLevelConfiguration":{"f:name":{}},"f:rules":{}}}}]},"spec":{"priorityLevelConfiguration":{"name":"catch-all"},"matchingPrecedence":10000,"distinguisherMethod":{"type":"ByUser"},"rules":[{"subjects":[{"kind":"Group","group":{"name":"system:unauthenticated"}},{"kind":"Group","group":{"name":"system:authenticated"}}],"resourceRules":[{"verbs":["*"],"apiGroups":["*"],"resources":["*"],"clusterScope":true,"namespaces":["*"]}],"nonResourceRules":[{"verbs":["*"],"nonResourceURLs":["*"]}]}]},"status":{"conditions":[{"type":"Dangling","status":"False","lastTransitionTime":"2020-08-06T01:09:58Z","reason":"Found","message":"This FlowSchema references the PriorityLevelConfiguration object named \"catch-all\" and it exists"}]}}
goroutine 37991583 [running]:
github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP.func1.1(0xc02b0c5080)
        /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/server/filters/timeout.go:108 +0x107
panic(0x3d2a960, 0xc01b669c30)
...
```

Sending the apiserver team for analysis.

Looks like this is around priority & fairness:

E0806 09:33:09.815325       1 runtime.go:76] Observed a panic: No match; rd=RequestDigest{RequestInfo: &request.RequestInfo{IsResourceRequest:true, Path:"/apis/user.openshift.io/v1/users/afuc6@customdomain", Verb:"delete", APIPrefix:"apis", APIGroup:"user.openshift.io", APIVersion:"v1", Namespace:"", Resource:"users", Subresource:"", Name:"afuc6@customdomain", Parts:[]string{"users", "afuc6@customdomain"}}, User: &user.DefaultInfo{Name:"test@customdomain", UID:"", Groups:[]string{"dedicated-admins"}, Extra:map[string][]string{}}}, catchAll={"metadata":{"name":"catch-all","selfLink":"/apis/flowcontrol.apiserver.k8s.io/v1alpha1/flowschemas/catch-all","uid":"6ee022c7-65ef-4ee3-9547-5b804f76e9b5","resourceVersion":"87","generation":1,"creationTimestamp":"2020-08-06T01:09:58Z","managedFields":[{"manager":"api-priority-and-fairness-config-consumer-v1","operation":"Update","apiVersion":"flowcontrol.apiserver.k8s.io/v1alpha1","time":"2020-08-06T01:09:58Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{".":{},"k:{\"type\":\"Dangling\"}":{".":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}}},{"manager":"api-priority-and-fairness-config-producer-v1","operation":"Update","apiVersion":"flowcontrol.apiserver.k8s.io/v1alpha1","time":"2020-08-06T01:09:58Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:distinguisherMethod":{".":{},"f:type":{}},"f:matchingPrecedence":{},"f:priorityLevelConfiguration":{"f:name":{}},"f:rules":{}}}}]},"spec":{"priorityLevelConfiguration":{"name":"catch-all"},"matchingPrecedence":10000,"distinguisherMethod":{"type":"ByUser"},"rules":[{"subjects":[{"kind":"Group","group":{"name":"system:unauthenticated"}},{"kind":"Group","group":{"name":"system:authenticated"}}],"resourceRules":[{"verbs":["*"],"apiGroups":["*"],"resources":["*"],"clusterScope":true,"namespaces":["*"]}],"nonResourceRules":[{"verbs":["*"],"nonResourceURLs":["*"]}]}]},"status":{"conditions":[{"type":"Dangling","status":"False","lastTransitionTime":"2020-08-06T01:09:58Z","reason":"Found","message":"This FlowSchema references the PriorityLevelConfiguration object named \"catch-all\" and it exists"}]}}

Hi haowang,

The impersonated user "test@customdomain" does not have "system:authenticated" in the "Groups".  

  User: &user.DefaultInfo{
    Name:"test@customdomain",
    UID:"",
    Groups:[]string{"dedicated-admins"},
    Extra:map[string][]string{}
  }


This is causing a no match in priority & fairness logic (matches a user to the default set of flow schema) and a panic consequently. I will do more investigation to find a proper solution.

In the meantime, as a workaround, if you add "--as-group=system:authenticated" to your oc command, this should work. You can make the change in your e2e test and let me know if this works.

Hi akashem
The workaround works well.

Opened an upstream PR to resolve the issue - https://github.com/kubernetes/kubernetes/pull/94204

In summary, an impersonated user, given successful authorization check, should be added to "system:authenticated" group.

*** Bug 1873721 has been marked as a duplicate of this bug. ***

Client Version: v4.2.13
Server Version: 4.5.0-rc.7

This is an unsupported version skew.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196