Bug 1888309

Summary: p&f: make SAR traffic from oauth and openshift apiserver exempt
Product: OpenShift Container Platform Reporter: Abu Kashem <akashem>
Component: kube-apiserverAssignee: Abu Kashem <akashem>
Status: CLOSED ERRATA QA Contact: Ke Wang <kewang>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 4.5CC: aos-bugs, cpassare, emarquez, mfojtik, naoto30, pamoedom, xxia
Target Milestone: ---   
Target Release: 4.6.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1888311 (view as bug list) Environment:
Last Closed: 2020-11-09 15:50:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1888311    
Bug Blocks:    

Description Abu Kashem 2020-10-14 14:38:21 UTC 
p&f: make SAR traffic from oauth and openshift apiserver exempt

This is what the flow schema would look like:

apiVersion: flowcontrol.apiserver.k8s.io/v1alpha1
kind: FlowSchema
metadata:
  name: openshift-apiserver-sar
spec:
  distinguisherMethod:
    type: ByUser
  matchingPrecedence: 2
  priorityLevelConfiguration:
    name: exempt
  rules:
  - resourceRules:
    - apiGroups:
      - authorization.k8s.io
      clusterScope: true
      namespaces:
      - '*'
      resources:
      - subjectaccessreviews
      verbs:
      - '*'
    - apiGroups:
      - authentication.k8s.io
      clusterScope: true
      namespaces:
      - '*'
      resources:
      - tokenreviews
      verbs:
      - '*'
    subjects:
    - kind: ServiceAccount
      serviceAccount:
        name: openshift-apiserver-sa
        namespace: openshift-apiserver

---

apiVersion: flowcontrol.apiserver.k8s.io/v1alpha1
kind: FlowSchema
metadata:
  name: openshift-oauth-apiserver-sar
spec:
  distinguisherMethod:
    type: ByUser
  matchingPrecedence: 2
  priorityLevelConfiguration:
    name: exempt
  rules:
  - resourceRules:
    - apiGroups:
      - authorization.k8s.io
      clusterScope: true
      namespaces:
      - '*'
      resources:
      - subjectaccessreviews
      verbs:
      - '*'
    - apiGroups:
      - authentication.k8s.io
      clusterScope: true
      namespaces:
      - '*'
      resources:
      - tokenreviews
      verbs:
      - '*'
    subjects:
    - kind: ServiceAccount
      serviceAccount:
        name: oauth-apiserver-sa
        namespace: openshift-oauth-apiserver
Comment 1 Stephen Cuppett 2020-10-14 14:57:54 UTC 
*** Bug 1888308 has been marked as a duplicate of this bug. ***
Comment 4 Ke Wang 2020-10-28 09:28:35 UTC 
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.6.0-0.nightly-2020-10-28-001707   True        False         82m     Cluster version is 4.6.0-0.nightly-2020-10-28-001707

$ oc get FlowSchema
NAME                                PRIORITYLEVEL                       MATCHINGPRECEDENCE   DISTINGUISHERMETHOD   AGE   MISSINGPL
exempt                              exempt                              1                    <none>                65m   False
openshift-apiserver-sar             exempt                              2                    ByUser                53m   False
openshift-oauth-apiserver-sar       exempt                              2                    ByUser                53m   False
system-leader-election              leader-election                     100                  ByUser                65m   False
workload-leader-election            leader-election                     200                  ByUser                65m   False
system-nodes                        system                              500                  ByUser                65m   False
kube-controller-manager             workload-high                       800                  ByNamespace           65m   False
kube-scheduler                      workload-high                       800                  ByNamespace           65m   False
kube-system-service-accounts        workload-high                       900                  ByNamespace           65m   False
openshift-apiserver                 workload-high                       1000                 ByUser                53m   False
openshift-controller-manager        workload-high                       1000                 ByUser                53m   False
openshift-oauth-apiserver           workload-high                       1000                 ByUser                53m   False
openshift-oauth-server              workload-high                       1000                 ByUser                53m   False
openshift-apiserver-operator        openshift-control-plane-operators   2000                 ByUser                53m   False
openshift-authentication-operator   openshift-control-plane-operators   2000                 ByUser                53m   False
openshift-etcd-operator             openshift-control-plane-operators   2000                 ByUser                53m   False
openshift-kube-apiserver-operator   openshift-control-plane-operators   2000                 ByUser                53m   False
openshift-monitoring-metrics        workload-high                       2000                 ByUser                53m   False
service-accounts                    workload-low                        9000                 ByUser                65m   False
global-default                      global-default                      9900                 ByUser                65m   False
catch-all                           catch-all                           10000                ByUser                65m   False

After changed kubeapiserver/cluster loglevel to Trace, will catch the following message which we want.

$ oc get kubeapiserver/cluster -oyaml | grep ' logLevel:'
  logLevel: Trace

the log level is '--v=6'.  

After the kube-apiserver pods rolled out, 
$ kas_pods=$(oc get pods -n openshift-kube-apiserver | grep 'kube-apiserver' | awk '{print $1}')
$ for pod in $kas_pods; do oc -n openshift-kube-apiserver logs $pod -c kube-apiserver >$pod.log;done

SAR traffic from flowschema oauth and openshift apiserver are exempt: 
$ grep 'startRequest' *internal*.log

Changed loglevel to 8, SAR traffic from flowschema oauth and openshift apiserver will be seen.
$ oc get kubeapiserver/cluster -oyaml | grep ' logLevel:'
  logLevel: TraceAll

the log level is '--v=8'.  

After the kube-apiserver pods rolled out,
Using the following grep command line show us expected.

Refer to https://bugzilla.redhat.com/show_bug.cgi?id=1885353#c5, 
> If `Trace` logs at or above '--v=7', then you can also check how P&F are processing requests that are exempt.
> We have one flowschema "openshift-apiserver-sar" that is exempt - all "subjectaccessreviews" or "tokenreviews" requests from openshift-apiserver are exempt. To check for this you can grep for "startRequest" and within the search results you should see that this flowschema is treated as "immediate". But the log level needs to be at 7 for this.

$ grep 'startRequest' *internal*.log | grep 'system:serviceaccount:openshift-apiserver:openshift-apiserver-sa' | grep -E 'immediate|exempt' | head -1
kube-apiserver-ip-xx-xx-155-160.us-east-2.compute.internal.log:I1028 09:13:49.785197      17 apf_controller.go:663] startRequest(RequestDigest{RequestInfo: &request.RequestInfo{IsResourceRequest:true, Path:"/apis/authentication.k8s.io/v1/tokenreviews", Verb:"create", APIPrefix:"apis", APIGroup:"authentication.k8s.io", APIVersion:"v1", Namespace:"", Resource:"tokenreviews", Subresource:"", Name:"", Parts:[]string{"tokenreviews"}}, User: &user.DefaultInfo{Name:"system:serviceaccount:openshift-apiserver:openshift-apiserver-sa", UID:"58712a1b-9e33-4924-8871-7bfa7fc01a2a", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-apiserver", "system:authenticated"}, Extra:map[string][]string(nil)}}) => fsName="openshift-apiserver-sar", distMethod=&v1alpha1.FlowDistinguisherMethod{Type:"ByUser"}, plName="exempt", immediate

$ grep 'startRequest' *internal*.log | grep 'system:serviceaccount:openshift-oauth-apiserver:oauth-apiserver-sa' | grep -E 'immediate|exempt' | head -1
kube-apiserver-ip-xx-xx-155-160.us-east-2.compute.internal.log:I1028 09:13:50.723315      17 apf_controller.go:663] startRequest(RequestDigest{RequestInfo: &request.RequestInfo{IsResourceRequest:true, Path:"/apis/authorization.k8s.io/v1/subjectaccessreviews", Verb:"create", APIPrefix:"apis", APIGroup:"authorization.k8s.io", APIVersion:"v1", Namespace:"", Resource:"subjectaccessreviews", Subresource:"", Name:"", Parts:[]string{"subjectaccessreviews"}}, User: &user.DefaultInfo{Name:"system:serviceaccount:openshift-oauth-apiserver:oauth-apiserver-sa", UID:"e0366bb0-7fda-4025-b7e6-67e292849436", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-oauth-apiserver", "system:authenticated"}, Extra:map[string][]string(nil)}}) => fsName="openshift-oauth-apiserver-sar", distMethod=&v1alpha1.FlowDistinguisherMethod{Type:"ByUser"}, plName="exempt", immediate

We can find the keywords 'startRequest, "exempt", immediate' in logs, so the results are expected, move the bug VERIFIED.
Comment 6 errata-xmlrpc 2020-11-09 15:50:59 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6.3 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4339