add p&f configuration to protect openshift traffic. Define dedicated flowschema and priority configuration that will protect openshift specific traffic. - subjectaccessreviews (SAR) and tokenreviews from oas or oauth server is very importnant. - openshift controller manager, other `oas` requests, '/metrics' requests from openshift-monitoring is as important as kcm traffic. - control plane operators are important (kas-o, auth operator, etcd operator) - The default `workloads-low` goes below the traffic defined above.
These are all the relevant PRs for this BZ: - https://github.com/openshift/cluster-kube-apiserver-operator/pull/966 - https://github.com/openshift/cluster-etcd-operator/pull/462 - https://github.com/openshift/cluster-authentication-operator/pull/356 - https://github.com/openshift/cluster-openshift-apiserver-operator/pull/398 - https://github.com/openshift/cluster-openshift-controller-manager-operator/pull/181 One way to test this would be to enable `Tracing` level logging for the kube-apiserver. priority and fairness logs which flowschema it used for each incoming request at "--v=7" So, we need to go through each flowschema and take the service account and search with it in the apiserver log to see if the requests have the right priority level selected. For example, take the following flowschema. it says requests from the kube-apiserver-operator should be assigned to "openshift-control-plane-operators" priority level configuration. apiVersion: flowcontrol.apiserver.k8s.io/v1alpha1 kind: FlowSchema metadata: name: openshift-kube-apiserver-operator spec: distinguisherMethod: type: ByUser matchingPrecedence: 2000 priorityLevelConfiguration: name: openshift-control-plane-operators rules: - resourceRules: - apiGroups: - '*' clusterScope: true namespaces: - '*' resources: - '*' verbs: - '*' subjects: - kind: ServiceAccount serviceAccount: name: kube-apiserver-operator namespace: openshift-kube-apiserver-operator We can do the following search > oc -n openshift-kube-apiserver logs kube-apiserver-ip-10-0-142-52.ec2.internal -c kube-apiserver | grep 'dispatching request' | grep 'system:serviceaccount:openshift-kube-apiserver-operator:kube-apiserver-operator' The above search should yield the following line(s) > I1006 13:56:49.528305 1 queueset.go:572] QS(openshift-control-plane-operators) at r=2020-10-06 13:56:49.528295756 v=42732.803929426s: dispatching request &request.RequestInfo{IsResourceRequest:true, Path:"/api/v1/namespaces/openshift-config-managed/secrets", Verb:"list", APIPrefix:"api", APIGroup:"", APIVersion:"v1", Namespace:"openshift-config-managed", Resource:"secrets", Subresource:"", Name:"", Parts:[]string{"secrets"}} &user.DefaultInfo{Name:"system:serviceaccount:openshift-kube-apiserver-operator:kube-apiserver-operator", UID:"929d69a1-fe70-4bb7-a4cf-df2cb386cb5c", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-kube-apiserver-operator", "system:authenticated"}, Extra:map[string][]string(nil)} from queue 58 with virtual start time 42732.803929426s, queue will have 0 waiting & 1 executing The "QS(openshift-control-plane-operators)" matches the desired "priorityLevelConfiguration" defined in the flowschema. Requests can come to any kube-apiserver instance. So we should search logs from all instances for match.
Correction on comment https://bugzilla.redhat.com/show_bug.cgi?id=1885353#c1 The PR for this BZ is: https://github.com/openshift/cluster-kube-apiserver-operator/pull/968
$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.5.0-0.ci.test-2020-10-15-015314-ci-ln-yqv5t5k True False 168m Cluster version is 4.5.0-0.ci.test-2020-10-15-015314-ci-ln-yqv5t5k $ oc get FlowSchema NAME PRIORITYLEVEL MATCHINGPRECEDENCE DISTINGUISHERMETHOD AGE MISSINGPL exempt exempt 1 <none> 3h12m False openshift-apiserver-sar exempt 2 ByUser 3h3m False system-leader-election leader-election 100 ByUser 3h12m False workload-leader-election leader-election 200 ByUser 3h12m False system-nodes system 500 ByUser 3h12m False kube-controller-manager workload-high 800 ByNamespace 3h12m False kube-scheduler workload-high 800 ByNamespace 3h12m False kube-system-service-accounts workload-high 900 ByNamespace 3h12m False openshift-apiserver workload-high 1000 ByUser 3h3m False openshift-controller-manager workload-high 1000 ByUser 3h3m False openshift-oauth-server workload-high 1000 ByUser 3h3m False openshift-apiserver-operator openshift-control-plane-operators 2000 ByUser 3h3m False openshift-authentication-operator openshift-control-plane-operators 2000 ByUser 3h3m False openshift-etcd-operator openshift-control-plane-operators 2000 ByUser 3h3m False openshift-kube-apiserver-operator openshift-control-plane-operators 2000 ByUser 3h3m False openshift-monitoring-metrics workload-high 2000 ByUser 3h3m False service-accounts workload-low 9000 ByUser 3h12m False global-default global-default 9900 ByUser 3h12m False catch-all catch-all 10000 ByUser 3h12m False After changed kubeapiserver/cluster loglevel to Trace, will catch the following message which we want. $ oc get kubeapiserver/cluster -oyaml | grep ' logLevel:' logLevel: Trace $ kas_pods=$(oc get pods -n openshift-kube-apiserver | grep 'kube-apiserver' | awk '{print $1}') $ for pod in $kas_pods; do oc -n openshift-kube-apiserver logs $pod -c kube-apiserver | grep 'dispatching request' | grep 'system:serviceaccount:openshift-authentication:oauth-openshift';done I1015 05:20:13.834740 1 queueset.go:572] QS(workload-high) at r=2020-10-15 05:20:13.834735779 v=120.486811156s: dispatching request &request.RequestInfo{IsResourceRequest:true, Path:"/apis/authentication.k8s.io/v1/tokenreviews", Verb:"create", APIPrefix:"apis", APIGroup:"authentication.k8s.io", APIVersion:"v1", Namespace:"", Resource:"tokenreviews", Subresource:"", Name:"", Parts:[]string{"tokenreviews"}} &user.DefaultInfo{Name:"system:serviceaccount:openshift-authentication:oauth-openshift", UID:"e05cb15b-c3e9-440e-a6fd-74110e81164a", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-authentication", "system:authenticated"}, Extra:map[string][]string(nil)} from queue 97 with virtual start time 120.486811156s, queue will have 0 waiting & 1 executing I1015 05:20:13.837697 1 queueset.go:572] QS(workload-high) at r=2020-10-15 05:20:13.837690596 v=120.488278902s: dispatching request &request.RequestInfo{IsResourceRequest:true, Path:"/apis/authorization.k8s.io/v1/subjectaccessreviews", Verb:"create", APIPrefix:"apis", APIGroup:"authorization.k8s.io", APIVersion:"v1", Namespace:"", Resource:"subjectaccessreviews", Subresource:"", Name:"", Parts:[]string{"subjectaccessreviews"}} &user.DefaultInfo{Name:"system:serviceaccount:openshift-authentication:oauth-openshift", UID:"e05cb15b-c3e9-440e-a6fd-74110e81164a", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-authentication", "system:authenticated"}, Extra:map[string][]string(nil)} from queue 97 with virtual start time 120.488278902s, queue will have 0 waiting & 1 executing I1015 05:20:29.235439 1 queueset.go:572] QS(workload-high) at r=2020-10-15 05:20:29.235433930 v=120.539252022s: dispatching request &request.RequestInfo{IsResourceRequest:true, Path:"/apis/authentication.k8s.io/v1/tokenreviews", Verb:"create", APIPrefix:"apis", APIGroup:"authentication.k8s.io", APIVersion:"v1", Namespace:"", Resource:"tokenreviews", Subresource:"", Name:"", Parts:[]string{"tokenreviews"}} &user.DefaultInfo{Name:"system:serviceaccount:openshift-authentication:oauth-openshift", UID:"e05cb15b-c3e9-440e-a6fd-74110e81164a", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-authentication", "system:authenticated"}, Extra:map[string][]string(nil)} from queue 97 with virtual start time 120.539252022s, queue will have 0 waiting & 1 executing I1015 05:20:29.238075 1 queueset.go:572] QS(workload-high) at r=2020-10-15 05:20:29.238069356 v=120.540424027s: dispatching request &request.RequestInfo{IsResourceRequest:true, Path:"/apis/authorization.k8s.io/v1/subjectaccessreviews", Verb:"create", APIPrefix:"apis", APIGroup:"authorization.k8s.io", APIVersion:"v1", Namespace:"", Resource:"subjectaccessreviews", Subresource:"", Name:"", Parts:[]string{"subjectaccessreviews"}} &user.DefaultInfo{Name:"system:serviceaccount:openshift-authentication:oauth-openshift", UID:"e05cb15b-c3e9-440e-a6fd-74110e81164a", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-authentication", "system:authenticated"}, Extra:map[string][]string(nil)} from queue 97 with virtual start time 120.540424027s, queue will have 0 waiting & 1 executing I1015 05:19:09.386775 1 queueset.go:572] QS(workload-high) at r=2020-10-15 05:19:09.386771429 v=41.212858992s: dispatching request &request.RequestInfo{IsResourceRequest:true, Path:"/apis/authentication.k8s.io/v1/tokenreviews", Verb:"create", APIPrefix:"apis", APIGroup:"authentication.k8s.io", APIVersion:"v1", Namespace:"", Resource:"tokenreviews", Subresource:"", Name:"", Parts:[]string{"tokenreviews"}} &user.DefaultInfo{Name:"system:serviceaccount:openshift-authentication:oauth-openshift", UID:"e05cb15b-c3e9-440e-a6fd-74110e81164a", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-authentication", "system:authenticated"}, Extra:map[string][]string(nil)} from queue 97 with virtual start time 41.212858992s, queue will have 0 waiting & 1 executing I1015 05:19:09.389538 1 queueset.go:572] QS(workload-high) at r=2020-10-15 05:19:09.389530802 v=41.213989872s: dispatching request &request.RequestInfo{IsResourceRequest:true, Path:"/apis/authorization.k8s.io/v1/subjectaccessreviews", Verb:"create", APIPrefix:"apis", APIGroup:"authorization.k8s.io", APIVersion:"v1", Namespace:"", Resource:"subjectaccessreviews", Subresource:"", Name:"", Parts:[]string{"subjectaccessreviews"}} &user.DefaultInfo{Name:"system:serviceaccount:openshift-authentication:oauth-openshift", UID:"e05cb15b-c3e9-440e-a6fd-74110e81164a", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-authentication", "system:authenticated"}, Extra:map[string][]string(nil)} from queue 97 with virtual start time 41.213989872s, queue will have 0 waiting & 1 executing ...
Hi kewang, This looks good, can you do the same for all the service accounts specified in all the flow schemas in the PR? If `Trace` logs at or above '--v=7', then you can also check how P&F are processing requests that are exempt. We have one flowschema "openshift-apiserver-sar" that is exempt - all "subjectaccessreviews" and "tokenreviews" requests from openshift-apiserver are exempt. To check for this you can grep for "startRequest" and within the search results you should see that this flowschema is treated as "immediate". But the log level needs to be at 7 for this. Also, check for panic in the apiserver logs with this message "No match". This indicates that there was no match for a flowschema. WE should never see this.
Hi akashem, I did same for all the service accounts specified in all the flow schemas in the PR, please see the following, is that as expected? $ kas_pods=$(oc get pods -n openshift-kube-apiserver | grep 'kube-apiserver' | awk '{print $1}') $ for pod in $kas_pods; do oc -n openshift-kube-apiserver logs $pod -c kube-apiserver >$pod.log;done Total FlowSchemas need to be checked: $ grep 'dispatching request' *.internal.log | grep 'system:serviceaccount:openshift-authentication:oauth-openshift' | head -1 kube-apiserver-ip-10-0-144-141.us-west-1.compute.internal.log:I1016 15:32:43.698744 1 queueset.go:572] QS(workload-high) at r=2020-10-16 15:32:43.698733040 v=25.713695553s: dispatching request &request.RequestInfo{IsResourceRequest:true, Path:"/apis/authentication.k8s.io/v1/tokenreviews", Verb:"create", APIPrefix:"apis", APIGroup:"authentication.k8s.io", APIVersion:"v1", Namespace:"", Resource:"tokenreviews", Subresource:"", Name:"", Parts:[]string{"tokenreviews"}} &user.DefaultInfo{Name:"system:serviceaccount:openshift-authentication:oauth-openshift", UID:"60d65ed1-c907-466d-96e4-8c0dae079c2d", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-authentication", "system:authenticated"}, Extra:map[string][]string(nil)} from queue 97 with virtual start time 25.713695553s, queue will have 0 waiting & 1 executing $ grep 'dispatching request' *.internal.log | grep 'system:serviceaccount:openshift-controller-manager:openshift-controller-manager-sa' | head -1 kube-apiserver-ip-10-0-144-141.us-west-1.compute.internal.log:I1016 15:32:28.853406 1 queueset.go:572] QS(workload-high) at r=2020-10-16 15:32:28.853393930 v=25.623386077s: dispatching request &request.RequestInfo{IsResourceRequest:true, Path:"/api/v1/namespaces/openshift-controller-manager/configmaps/openshift-master-controllers", Verb:"get", APIPrefix:"api", APIGroup:"", APIVersion:"v1", Namespace:"openshift-controller-manager", Resource:"configmaps", Subresource:"", Name:"openshift-master-controllers", Parts:[]string{"configmaps", "openshift-master-controllers"}} &user.DefaultInfo{Name:"system:serviceaccount:openshift-controller-manager:openshift-controller-manager-sa", UID:"ef86ede2-a316-4503-8048-8893951dd3e4", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-controller-manager", "system:authenticated"}, Extra:map[string][]string(nil)} from queue 29 with virtual start time 25.623386077s, queue will have 0 waiting & 1 executing $ grep 'dispatching request' *.internal.log | grep 'system:serviceaccount:openshift-etcd-operator:etcd-operator' | head -1 kube-apiserver-ip-10-0-167-29.us-west-1.compute.internal.log:I1016 15:30:23.251407 1 queueset.go:572] QS(openshift-control-plane-operators) at r=2020-10-16 15:30:23.251400846 v=5.257184307s: dispatching request &request.RequestInfo{IsResourceRequest:true, Path:"/api/v1/namespaces/openshift-etcd-operator/configmaps/openshift-cluster-etcd-operator-lock", Verb:"get", APIPrefix:"api", APIGroup:"", APIVersion:"v1", Namespace:"openshift-etcd-operator", Resource:"configmaps", Subresource:"", Name:"openshift-cluster-etcd-operator-lock", Parts:[]string{"configmaps", "openshift-cluster-etcd-operator-lock"}} &user.DefaultInfo{Name:"system:serviceaccount:openshift-etcd-operator:etcd-operator", UID:"c5d1548e-4d91-49b4-b526-e2b6d4b48354", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-etcd-operator", "system:authenticated"}, Extra:map[string][]string(nil)} from queue 39 with virtual start time 5.257184307s, queue will have 0 waiting & 1 executing $ grep 'dispatching request' *.internal.log | grep 'system:serviceaccount:openshift-kube-apiserver-operator:kube-apiserver-operator' | head -1 kube-apiserver-ip-10-0-144-141.us-west-1.compute.internal.log:I1016 15:32:27.622469 1 queueset.go:572] QS(openshift-control-plane-operators) at r=2020-10-16 15:32:27.622456200 v=105.833173234s: dispatching request &request.RequestInfo{IsResourceRequest:true, Path:"/api/v1/namespaces/openshift-kube-apiserver/pods", Verb:"list", APIPrefix:"api", APIGroup:"", APIVersion:"v1", Namespace:"openshift-kube-apiserver", Resource:"pods", Subresource:"", Name:"", Parts:[]string{"pods"}} &user.DefaultInfo{Name:"system:serviceaccount:openshift-kube-apiserver-operator:kube-apiserver-operator", UID:"2e2a2649-d4c7-458c-b9f8-796b7674b7fc", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-kube-apiserver-operator", "system:authenticated"}, Extra:map[string][]string(nil)} from queue 58 with virtual start time 105.833173234s, queue will have 0 waiting & 1 executing $ grep 'dispatching request' *.internal.log | grep 'system:serviceaccount:openshift-apiserver-operator:openshift-apiserver-operator' | head -1 kube-apiserver-ip-10-0-144-141.us-west-1.compute.internal.log:I1016 15:32:27.550293 1 queueset.go:572] QS(openshift-control-plane-operators) at r=2020-10-16 15:32:27.550281578 v=105.828712748s: dispatching request &request.RequestInfo{IsResourceRequest:true, Path:"/api/v1/namespaces/openshift-apiserver/pods", Verb:"list", APIPrefix:"api", APIGroup:"", APIVersion:"v1", Namespace:"openshift-apiserver", Resource:"pods", Subresource:"", Name:"", Parts:[]string{"pods"}} &user.DefaultInfo{Name:"system:serviceaccount:openshift-apiserver-operator:openshift-apiserver-operator", UID:"5c201a52-8440-41ac-a342-740750882cc0", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-apiserver-operator", "system:authenticated"}, Extra:map[string][]string(nil)} from queue 56 with virtual start time 105.828712748s, queue will have 0 waiting & 1 executing $ grep 'dispatching request' *.internal.log | grep 'system:serviceaccount:openshift-monitoring:prometheus-k8s' | head -1 kube-apiserver-ip-10-0-144-141.us-west-1.compute.internal.log:I1016 15:32:28.793380 1 queueset.go:572] QS(workload-low) at r=2020-10-16 15:32:28.793369094 v=177.097907112s: dispatching request &request.RequestInfo{IsResourceRequest:true, Path:"/apis/user.openshift.io/v1/users/~", Verb:"get", APIPrefix:"apis", APIGroup:"user.openshift.io", APIVersion:"v1", Namespace:"", Resource:"users", Subresource:"", Name:"~", Parts:[]string{"users", "~"}} &user.DefaultInfo{Name:"system:serviceaccount:openshift-monitoring:prometheus-k8s", UID:"92130880-3bfe-4ae9-b4cd-505232918673", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-monitoring", "system:authenticated"}, Extra:map[string][]string(nil)} from queue 30 with virtual start time 177.097907112s, queue will have 0 waiting & 1 executing $ grep 'dispatching request' *.internal.log | grep 'system:serviceaccount:openshift-authentication-operator:authentication-operator' | head -1 kube-apiserver-ip-10-0-144-141.us-west-1.compute.internal.log:I1016 15:32:28.263738 1 queueset.go:572] QS(openshift-control-plane-operators) at r=2020-10-16 15:32:28.263726189 v=105.858770782s: dispatching request &request.RequestInfo{IsResourceRequest:true, Path:"/api/v1/namespaces/openshift-authentication/configmaps/v4-0-config-system-console-config", Verb:"delete", APIPrefix:"api", APIGroup:"", APIVersion:"v1", Namespace:"openshift-authentication", Resource:"configmaps", Subresource:"", Name:"v4-0-config-system-console-config", Parts:[]string{"configmaps", "v4-0-config-system-console-config"}} &user.DefaultInfo{Name:"system:serviceaccount:openshift-authentication-operator:authentication-operator", UID:"aad703e3-977b-4387-b018-c4cbb4094c74", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-authentication-operator", "system:authenticated"}, Extra:map[string][]string(nil)} from queue 57 with virtual start time 105.858770782s, queue will have 0 waiting & 1 executing one flowschema "openshift-apiserver-sar" that is exempt: grep 'dispatching request' *.internal.log | grep 'system:serviceaccount:openshift-apiserver:openshift-apiserver-sa' grep 'startRequest' *.internal.log panic checking: grep 'panic' *.internal.log grep 'No match' *.internal.log
Hi kewang, > one flowschema "openshift-apiserver-sar" that is exempt: > grep 'startRequest' *.internal.log Can you check if the log level is '--v=7'? If it is lower than 7 then you wouldn't see it in the logs. Otherwise it looks good!
Hi akashem, just like what you said when I changed the logLevel to TraceAll and checked again, $ oc get kubeapiserver/cluster -oyaml | grep ' logLevel:' logLevel: TraceAll the log level is '--v=8'. $ grep 'dispatching request' *.internal.log | grep 'system:serviceaccount:openshift-apiserver:openshift-apiserver-sa' | head -1 kube-apiserver-kewang1953-knqxm-master-0.c.openshift-qe.internal.log:I1019 14:55:53.689292 1 queueset.go:572] QS(workload-low) at r=2020-10-19 14:55:53.689285418 v=119.359099270s: dispatching request &request.RequestInfo{IsResourceRequest:true, Path:"/apis/authorization.k8s.io/v1/subjectaccessreviews", Verb:"create", APIPrefix:"apis", APIGroup:"authorization.k8s.io", APIVersion:"v1", Namespace:"", Resource:"subjectaccessreviews", Subresource:"", Name:"", Parts:[]string{"subjectaccessreviews"}} &user.DefaultInfo{Name:"system:serviceaccount:openshift-apiserver:openshift-apiserver-sa", UID:"c0e4a29c-f0f7-42b5-b2cb-87f64ded4df9", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-apiserver", "system:authenticated"}, Extra:map[string][]string(nil)} from queue 38 with virtual start time 119.359099270s, queue will have 0 waiting & 1 executing $ grep 'startRequest' *.internal.log | head -1 kube-apiserver-kewang1953-knqxm-master-0.c.openshift-qe.internal.log:I1019 14:55:52.710011 1 apf_controller.go:628] startRequest(RequestDigest{RequestInfo: &request.RequestInfo{IsResourceRequest:true, Path:"/api/v1/namespaces/openshift-kube-apiserver/pods", Verb:"list", APIPrefix:"api", APIGroup:"", APIVersion:"v1", Namespace:"openshift-kube-apiserver", Resource:"pods", Subresource:"", Name:"", Parts:[]string{"pods"}}, User: &user.DefaultInfo{Name:"system:serviceaccount:openshift-kube-apiserver-operator:kube-apiserver-operator", UID:"ccb5d42d-e46c-4866-9688-434244598b9c", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-kube-apiserver-operator", "system:authenticated"}, Extra:map[string][]string(nil)}}) Since the results all are as expected, move the bug VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.5.16 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4268
Some corrections for Comment 10, > If `Trace` logs at or above '--v=7', then you can also check how P&F are processing requests that are exempt. > We have one flowschema "openshift-apiserver-sar" that is exempt - all "subjectaccessreviews" and "tokenreviews" requests from openshift-apiserver are exempt. To check for this you can grep for "startRequest" and within the search results you should see that this flowschema is treated as "immediate". But the log level needs to be at 7 for this. Using the following grep command line to get the above described results, ignored Comment 10, $ grep 'startRequest' *master*.log | grep 'system:serviceaccount:openshift-apiserver:openshift-apiserver-sa' | grep immediate ... kube-apiserver-kewang2273-7hpgp-master-1.log:I1022 16:07:22.501799 19 apf_controller.go:663] startRequest(RequestDigest{RequestInfo: &request.RequestInfo{IsResourceRequest:true, Path:"/apis/authorization.k8s.io/v1/subjectaccessreviews", Verb:"create", APIPrefix:"apis", APIGroup:"authorization.k8s.io", APIVersion:"v1", Namespace:"", Resource:"subjectaccessreviews", Subresource:"", Name:"", Parts:[]string{"subjectaccessreviews"}}, User: &user.DefaultInfo{Name:"system:serviceaccount:openshift-apiserver:openshift-apiserver-sa", UID:"18eb7c21-6e8d-4d22-8a20-456efe008c06", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-apiserver", "system:authenticated"}, Extra:map[string][]string(nil)}}) => fsName="openshift-apiserver-sar", distMethod=&v1alpha1.FlowDistinguisherMethod{Type:"ByUser"}, plName="exempt", immediate