p&f: make SAR traffic from oauth and openshift apiserver exempt This is what the flow schema would look like: apiVersion: flowcontrol.apiserver.k8s.io/v1alpha1 kind: FlowSchema metadata: name: openshift-apiserver-sar spec: distinguisherMethod: type: ByUser matchingPrecedence: 2 priorityLevelConfiguration: name: exempt rules: - resourceRules: - apiGroups: - authorization.k8s.io clusterScope: true namespaces: - '*' resources: - subjectaccessreviews verbs: - '*' - apiGroups: - authentication.k8s.io clusterScope: true namespaces: - '*' resources: - tokenreviews verbs: - '*' subjects: - kind: ServiceAccount serviceAccount: name: openshift-apiserver-sa namespace: openshift-apiserver --- apiVersion: flowcontrol.apiserver.k8s.io/v1alpha1 kind: FlowSchema metadata: name: openshift-oauth-apiserver-sar spec: distinguisherMethod: type: ByUser matchingPrecedence: 2 priorityLevelConfiguration: name: exempt rules: - resourceRules: - apiGroups: - authorization.k8s.io clusterScope: true namespaces: - '*' resources: - subjectaccessreviews verbs: - '*' - apiGroups: - authentication.k8s.io clusterScope: true namespaces: - '*' resources: - tokenreviews verbs: - '*' subjects: - kind: ServiceAccount serviceAccount: name: oauth-apiserver-sa namespace: openshift-oauth-apiserver
*** Bug 1888308 has been marked as a duplicate of this bug. ***
$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.6.0-0.nightly-2020-10-28-001707 True False 82m Cluster version is 4.6.0-0.nightly-2020-10-28-001707 $ oc get FlowSchema NAME PRIORITYLEVEL MATCHINGPRECEDENCE DISTINGUISHERMETHOD AGE MISSINGPL exempt exempt 1 <none> 65m False openshift-apiserver-sar exempt 2 ByUser 53m False openshift-oauth-apiserver-sar exempt 2 ByUser 53m False system-leader-election leader-election 100 ByUser 65m False workload-leader-election leader-election 200 ByUser 65m False system-nodes system 500 ByUser 65m False kube-controller-manager workload-high 800 ByNamespace 65m False kube-scheduler workload-high 800 ByNamespace 65m False kube-system-service-accounts workload-high 900 ByNamespace 65m False openshift-apiserver workload-high 1000 ByUser 53m False openshift-controller-manager workload-high 1000 ByUser 53m False openshift-oauth-apiserver workload-high 1000 ByUser 53m False openshift-oauth-server workload-high 1000 ByUser 53m False openshift-apiserver-operator openshift-control-plane-operators 2000 ByUser 53m False openshift-authentication-operator openshift-control-plane-operators 2000 ByUser 53m False openshift-etcd-operator openshift-control-plane-operators 2000 ByUser 53m False openshift-kube-apiserver-operator openshift-control-plane-operators 2000 ByUser 53m False openshift-monitoring-metrics workload-high 2000 ByUser 53m False service-accounts workload-low 9000 ByUser 65m False global-default global-default 9900 ByUser 65m False catch-all catch-all 10000 ByUser 65m False After changed kubeapiserver/cluster loglevel to Trace, will catch the following message which we want. $ oc get kubeapiserver/cluster -oyaml | grep ' logLevel:' logLevel: Trace the log level is '--v=6'. After the kube-apiserver pods rolled out, $ kas_pods=$(oc get pods -n openshift-kube-apiserver | grep 'kube-apiserver' | awk '{print $1}') $ for pod in $kas_pods; do oc -n openshift-kube-apiserver logs $pod -c kube-apiserver >$pod.log;done SAR traffic from flowschema oauth and openshift apiserver are exempt: $ grep 'startRequest' *internal*.log Changed loglevel to 8, SAR traffic from flowschema oauth and openshift apiserver will be seen. $ oc get kubeapiserver/cluster -oyaml | grep ' logLevel:' logLevel: TraceAll the log level is '--v=8'. After the kube-apiserver pods rolled out, Using the following grep command line show us expected. Refer to https://bugzilla.redhat.com/show_bug.cgi?id=1885353#c5, > If `Trace` logs at or above '--v=7', then you can also check how P&F are processing requests that are exempt. > We have one flowschema "openshift-apiserver-sar" that is exempt - all "subjectaccessreviews" or "tokenreviews" requests from openshift-apiserver are exempt. To check for this you can grep for "startRequest" and within the search results you should see that this flowschema is treated as "immediate". But the log level needs to be at 7 for this. $ grep 'startRequest' *internal*.log | grep 'system:serviceaccount:openshift-apiserver:openshift-apiserver-sa' | grep -E 'immediate|exempt' | head -1 kube-apiserver-ip-xx-xx-155-160.us-east-2.compute.internal.log:I1028 09:13:49.785197 17 apf_controller.go:663] startRequest(RequestDigest{RequestInfo: &request.RequestInfo{IsResourceRequest:true, Path:"/apis/authentication.k8s.io/v1/tokenreviews", Verb:"create", APIPrefix:"apis", APIGroup:"authentication.k8s.io", APIVersion:"v1", Namespace:"", Resource:"tokenreviews", Subresource:"", Name:"", Parts:[]string{"tokenreviews"}}, User: &user.DefaultInfo{Name:"system:serviceaccount:openshift-apiserver:openshift-apiserver-sa", UID:"58712a1b-9e33-4924-8871-7bfa7fc01a2a", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-apiserver", "system:authenticated"}, Extra:map[string][]string(nil)}}) => fsName="openshift-apiserver-sar", distMethod=&v1alpha1.FlowDistinguisherMethod{Type:"ByUser"}, plName="exempt", immediate $ grep 'startRequest' *internal*.log | grep 'system:serviceaccount:openshift-oauth-apiserver:oauth-apiserver-sa' | grep -E 'immediate|exempt' | head -1 kube-apiserver-ip-xx-xx-155-160.us-east-2.compute.internal.log:I1028 09:13:50.723315 17 apf_controller.go:663] startRequest(RequestDigest{RequestInfo: &request.RequestInfo{IsResourceRequest:true, Path:"/apis/authorization.k8s.io/v1/subjectaccessreviews", Verb:"create", APIPrefix:"apis", APIGroup:"authorization.k8s.io", APIVersion:"v1", Namespace:"", Resource:"subjectaccessreviews", Subresource:"", Name:"", Parts:[]string{"subjectaccessreviews"}}, User: &user.DefaultInfo{Name:"system:serviceaccount:openshift-oauth-apiserver:oauth-apiserver-sa", UID:"e0366bb0-7fda-4025-b7e6-67e292849436", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-oauth-apiserver", "system:authenticated"}, Extra:map[string][]string(nil)}}) => fsName="openshift-oauth-apiserver-sar", distMethod=&v1alpha1.FlowDistinguisherMethod{Type:"ByUser"}, plName="exempt", immediate We can find the keywords 'startRequest, "exempt", immediate' in logs, so the results are expected, move the bug VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6.3 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4339