+++ This bug was initially created as a clone of Bug #1888309 +++ p&f: make SAR traffic from oauth and openshift apiserver exempt This is what the flow schema would look like: apiVersion: flowcontrol.apiserver.k8s.io/v1alpha1 kind: FlowSchema metadata: name: openshift-apiserver-sar spec: distinguisherMethod: type: ByUser matchingPrecedence: 2 priorityLevelConfiguration: name: exempt rules: - resourceRules: - apiGroups: - authorization.k8s.io clusterScope: true namespaces: - '*' resources: - subjectaccessreviews verbs: - '*' - apiGroups: - authentication.k8s.io clusterScope: true namespaces: - '*' resources: - tokenreviews verbs: - '*' subjects: - kind: ServiceAccount serviceAccount: name: openshift-apiserver-sa namespace: openshift-apiserver --- apiVersion: flowcontrol.apiserver.k8s.io/v1alpha1 kind: FlowSchema metadata: name: openshift-oauth-apiserver-sar spec: distinguisherMethod: type: ByUser matchingPrecedence: 2 priorityLevelConfiguration: name: exempt rules: - resourceRules: - apiGroups: - authorization.k8s.io clusterScope: true namespaces: - '*' resources: - subjectaccessreviews verbs: - '*' - apiGroups: - authentication.k8s.io clusterScope: true namespaces: - '*' resources: - tokenreviews verbs: - '*' subjects: - kind: ServiceAccount serviceAccount: name: oauth-apiserver-sa namespace: openshift-oauth-apiserver
$ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.0-0.nightly-2020-10-21-001511 True False 115m Cluster version is 4.7.0-0.nightly-2020-10-21-001511 $ oc get FlowSchema NAME PRIORITYLEVEL MATCHINGPRECEDENCE DISTINGUISHERMETHOD AGE MISSINGPL exempt exempt 1 <none> 139m False openshift-apiserver-sar exempt 2 ByUser 126m False openshift-oauth-apiserver-sar exempt 2 ByUser 116m False system-leader-election leader-election 100 ByUser 139m False workload-leader-election leader-election 200 ByUser 139m False system-nodes system 500 ByUser 139m False kube-controller-manager workload-high 800 ByNamespace 139m False kube-scheduler workload-high 800 ByNamespace 139m False kube-system-service-accounts workload-high 900 ByNamespace 139m False openshift-apiserver workload-high 1000 ByUser 126m False openshift-controller-manager workload-high 1000 ByUser 138m False openshift-oauth-apiserver workload-high 1000 ByUser 116m False openshift-oauth-server workload-high 1000 ByUser 116m False openshift-apiserver-operator openshift-control-plane-operators 2000 ByUser 126m False openshift-authentication-operator openshift-control-plane-operators 2000 ByUser 116m False openshift-etcd-operator openshift-control-plane-operators 2000 ByUser 131m False openshift-kube-apiserver-operator openshift-control-plane-operators 2000 ByUser 127m False openshift-monitoring-metrics workload-high 2000 ByUser 127m False service-accounts workload-low 9000 ByUser 139m False global-default global-default 9900 ByUser 139m False catch-all catch-all 10000 ByUser 139m False After changed kubeapiserver/cluster loglevel to Trace, will catch the following message which we want. $ oc get kubeapiserver/cluster -oyaml | grep ' logLevel:' logLevel: Trace the log level is '--v=6'. After the kube-apiserver pods rolled out, $ kas_pods=$(oc get pods -n openshift-kube-apiserver | grep 'kube-apiserver' | awk '{print $1}') $ for pod in $kas_pods; do oc -n openshift-kube-apiserver logs $pod -c kube-apiserver >$pod.log;done SAR traffic from flowschema oauth and openshift apiserver are exempt: grep 'startRequest' *master*.log Changed loglevel to 8, SAR traffic from flowschema oauth and openshift apiserver will be seen. $ oc get kubeapiserver/cluster -oyaml | grep ' logLevel:' logLevel: TraceAll the log level is '--v=8'. After the kube-apiserver pods rolled out, $ grep 'startRequest' *master*.log | grep 'system:serviceaccount:openshift-apiserver:openshift-apiserver-sa' | head -1 kube-apiserver-kewang2372-2s7xv-master-0.log:I1023 15:06:41.741834 19 apf_controller.go:655] startRequest(RequestDigest{RequestInfo: &request.RequestInfo{IsResourceRequest:true, Path:"/apis/authentication.k8s.io/v1/tokenreviews", Verb:"create", APIPrefix:"apis", APIGroup:"authentication.k8s.io", APIVersion:"v1", Namespace:"", Resource:"tokenreviews", Subresource:"", Name:"", Parts:[]string{"tokenreviews"}}, User: &user.DefaultInfo{Name:"system:serviceaccount:openshift-apiserver:openshift-apiserver-sa", UID:"6f20ee71-5b5a-4696-a0e3-5c49ff7c9263", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-apiserver", "system:authenticated"}, Extra:map[string][]string(nil)}}) $ grep 'startRequest' *master*.log | grep 'system:serviceaccount:openshift-oauth-apiserver:oauth-apiserver-sa' | head -1 kube-apiserver-kewang2372-2s7xv-master-0.log:I1023 15:06:22.798668 19 apf_controller.go:655] startRequest(RequestDigest{RequestInfo: &request.RequestInfo{IsResourceRequest:true, Path:"/apis/authorization.k8s.io/v1/subjectaccessreviews", Verb:"create", APIPrefix:"apis", APIGroup:"authorization.k8s.io", APIVersion:"v1", Namespace:"", Resource:"subjectaccessreviews", Subresource:"", Name:"", Parts:[]string{"subjectaccessreviews"}}, User: &user.DefaultInfo{Name:"system:serviceaccount:openshift-oauth-apiserver:oauth-apiserver-sa", UID:"b56e69a8-1f2c-450f-a974-1624a2b3f069", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-oauth-apiserver", "system:authenticated"}, Extra:map[string][]string(nil)}})
Correction for above grep command line searching log, using the following grep command line show us expected. > If `Trace` logs at or above '--v=7', then you can also check how P&F are processing requests that are exempt. > We have one flowschema "openshift-apiserver-sar" that is exempt - all "subjectaccessreviews" or "tokenreviews" requests from openshift-apiserver are exempt. To check for this you can grep for "startRequest" and within the search results you should see that this flowschema is treated as "immediate". But the log level needs to be at 7 for this. $ grep 'startRequest' *master*.log | grep 'system:serviceaccount:openshift-apiserver:openshift-apiserver-sa' | grep -E 'immediate|exempt' | head -1 kube-apiserver-kewang2273-7hpgp-master-1.log:I1022 16:07:22.501799 19 apf_controller.go:663] startRequest(RequestDigest{RequestInfo: &request.RequestInfo{IsResourceRequest:true, Path:"/apis/authorization.k8s.io/v1/subjectaccessreviews", Verb:"create", APIPrefix:"apis", APIGroup:"authorization.k8s.io", APIVersion:"v1", Namespace:"", Resource:"subjectaccessreviews", Subresource:"", Name:"", Parts:[]string{"subjectaccessreviews"}}, User: &user.DefaultInfo{Name:"system:serviceaccount:openshift-apiserver:openshift-apiserver-sa", UID:"18eb7c21-6e8d-4d22-8a20-456efe008c06", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-apiserver", "system:authenticated"}, Extra:map[string][]string(nil)}}) => fsName="openshift-apiserver-sar", distMethod=&v1alpha1.FlowDistinguisherMethod{Type:"ByUser"}, plName="exempt", immediate $ grep 'startRequest' *master*.log | grep 'system:serviceaccount:openshift-oauth-apiserver:oauth-apiserver-sa' | grep -E 'immediate|exempt' | head -1 kube-apiserver-kewang2273-7hpgp-master-2.log:I1022 16:09:30.723485 19 apf_controller.go:663] startRequest(RequestDigest{RequestInfo: &request.RequestInfo{IsResourceRequest:true, Path:"/apis/authorization.k8s.io/v1/subjectaccessreviews", Verb:"create", APIPrefix:"apis", APIGroup:"authorization.k8s.io", APIVersion:"v1", Namespace:"", Resource:"subjectaccessreviews", Subresource:"", Name:"", Parts:[]string{"subjectaccessreviews"}}, User: &user.DefaultInfo{Name:"system:serviceaccount:openshift-oauth-apiserver:oauth-apiserver-sa", UID:"a24d5b7b-6deb-4126-a39f-114b73473297", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-oauth-apiserver", "system:authenticated"}, Extra:map[string][]string(nil)}}) => fsName="openshift-oauth-apiserver-sar", distMethod=&v1alpha1.FlowDistinguisherMethod{Type:"ByUser"}, plName="exempt", immediate
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633