Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1888311

Summary: p&f: make SAR traffic from oauth and openshift apiserver exempt
Product: OpenShift Container Platform Reporter: Abu Kashem <akashem>
Component: kube-apiserverAssignee: Abu Kashem <akashem>
Status: CLOSED ERRATA QA Contact: Ke Wang <kewang>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 4.5CC: aos-bugs, kewang, mfojtik, xxia
Target Milestone: ---   
Target Release: 4.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: 1888309 Environment:
Last Closed: 2021-02-24 15:26:11 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1888309    

Description Abu Kashem 2020-10-14 14:40:55 UTC 
+++ This bug was initially created as a clone of Bug #1888309 +++

p&f: make SAR traffic from oauth and openshift apiserver exempt

This is what the flow schema would look like:

apiVersion: flowcontrol.apiserver.k8s.io/v1alpha1
kind: FlowSchema
metadata:
  name: openshift-apiserver-sar
spec:
  distinguisherMethod:
    type: ByUser
  matchingPrecedence: 2
  priorityLevelConfiguration:
    name: exempt
  rules:
  - resourceRules:
    - apiGroups:
      - authorization.k8s.io
      clusterScope: true
      namespaces:
      - '*'
      resources:
      - subjectaccessreviews
      verbs:
      - '*'
    - apiGroups:
      - authentication.k8s.io
      clusterScope: true
      namespaces:
      - '*'
      resources:
      - tokenreviews
      verbs:
      - '*'
    subjects:
    - kind: ServiceAccount
      serviceAccount:
        name: openshift-apiserver-sa
        namespace: openshift-apiserver

---

apiVersion: flowcontrol.apiserver.k8s.io/v1alpha1
kind: FlowSchema
metadata:
  name: openshift-oauth-apiserver-sar
spec:
  distinguisherMethod:
    type: ByUser
  matchingPrecedence: 2
  priorityLevelConfiguration:
    name: exempt
  rules:
  - resourceRules:
    - apiGroups:
      - authorization.k8s.io
      clusterScope: true
      namespaces:
      - '*'
      resources:
      - subjectaccessreviews
      verbs:
      - '*'
    - apiGroups:
      - authentication.k8s.io
      clusterScope: true
      namespaces:
      - '*'
      resources:
      - tokenreviews
      verbs:
      - '*'
    subjects:
    - kind: ServiceAccount
      serviceAccount:
        name: oauth-apiserver-sa
        namespace: openshift-oauth-apiserver
Comment 2 Ke Wang 2020-10-23 15:18:43 UTC 
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.nightly-2020-10-21-001511   True        False         115m    Cluster version is 4.7.0-0.nightly-2020-10-21-001511

$ oc get FlowSchema
NAME                                PRIORITYLEVEL                       MATCHINGPRECEDENCE   DISTINGUISHERMETHOD   AGE    MISSINGPL
exempt                              exempt                              1                    <none>                139m   False
openshift-apiserver-sar             exempt                              2                    ByUser                126m   False
openshift-oauth-apiserver-sar       exempt                              2                    ByUser                116m   False
system-leader-election              leader-election                     100                  ByUser                139m   False
workload-leader-election            leader-election                     200                  ByUser                139m   False
system-nodes                        system                              500                  ByUser                139m   False
kube-controller-manager             workload-high                       800                  ByNamespace           139m   False
kube-scheduler                      workload-high                       800                  ByNamespace           139m   False
kube-system-service-accounts        workload-high                       900                  ByNamespace           139m   False
openshift-apiserver                 workload-high                       1000                 ByUser                126m   False
openshift-controller-manager        workload-high                       1000                 ByUser                138m   False
openshift-oauth-apiserver           workload-high                       1000                 ByUser                116m   False
openshift-oauth-server              workload-high                       1000                 ByUser                116m   False
openshift-apiserver-operator        openshift-control-plane-operators   2000                 ByUser                126m   False
openshift-authentication-operator   openshift-control-plane-operators   2000                 ByUser                116m   False
openshift-etcd-operator             openshift-control-plane-operators   2000                 ByUser                131m   False
openshift-kube-apiserver-operator   openshift-control-plane-operators   2000                 ByUser                127m   False
openshift-monitoring-metrics        workload-high                       2000                 ByUser                127m   False
service-accounts                    workload-low                        9000                 ByUser                139m   False
global-default                      global-default                      9900                 ByUser                139m   False
catch-all                           catch-all                           10000                ByUser                139m   False

After changed kubeapiserver/cluster loglevel to Trace, will catch the following message which we want.

$ oc get kubeapiserver/cluster -oyaml | grep ' logLevel:'
  logLevel: Trace

the log level is '--v=6'.  

After the kube-apiserver pods rolled out, 
$ kas_pods=$(oc get pods -n openshift-kube-apiserver | grep 'kube-apiserver' | awk '{print $1}')
$ for pod in $kas_pods; do oc -n openshift-kube-apiserver logs $pod -c kube-apiserver >$pod.log;done

SAR traffic from flowschema oauth and openshift apiserver are exempt: 
grep 'startRequest' *master*.log

Changed loglevel to 8, SAR traffic from flowschema oauth and openshift apiserver will be seen.
$ oc get kubeapiserver/cluster -oyaml | grep ' logLevel:'
  logLevel: TraceAll

the log level is '--v=8'.  

After the kube-apiserver pods rolled out,
$ grep 'startRequest' *master*.log | grep 'system:serviceaccount:openshift-apiserver:openshift-apiserver-sa' | head -1
kube-apiserver-kewang2372-2s7xv-master-0.log:I1023 15:06:41.741834      19 apf_controller.go:655] startRequest(RequestDigest{RequestInfo: &request.RequestInfo{IsResourceRequest:true, Path:"/apis/authentication.k8s.io/v1/tokenreviews", Verb:"create", APIPrefix:"apis", APIGroup:"authentication.k8s.io", APIVersion:"v1", Namespace:"", Resource:"tokenreviews", Subresource:"", Name:"", Parts:[]string{"tokenreviews"}}, User: &user.DefaultInfo{Name:"system:serviceaccount:openshift-apiserver:openshift-apiserver-sa", UID:"6f20ee71-5b5a-4696-a0e3-5c49ff7c9263", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-apiserver", "system:authenticated"}, Extra:map[string][]string(nil)}})

$ grep 'startRequest' *master*.log | grep 'system:serviceaccount:openshift-oauth-apiserver:oauth-apiserver-sa' | head -1
kube-apiserver-kewang2372-2s7xv-master-0.log:I1023 15:06:22.798668      19 apf_controller.go:655] startRequest(RequestDigest{RequestInfo: &request.RequestInfo{IsResourceRequest:true, Path:"/apis/authorization.k8s.io/v1/subjectaccessreviews", Verb:"create", APIPrefix:"apis", APIGroup:"authorization.k8s.io", APIVersion:"v1", Namespace:"", Resource:"subjectaccessreviews", Subresource:"", Name:"", Parts:[]string{"subjectaccessreviews"}}, User: &user.DefaultInfo{Name:"system:serviceaccount:openshift-oauth-apiserver:oauth-apiserver-sa", UID:"b56e69a8-1f2c-450f-a974-1624a2b3f069", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-oauth-apiserver", "system:authenticated"}, Extra:map[string][]string(nil)}})
Comment 3 Ke Wang 2020-10-27 07:59:16 UTC 
Correction for above grep command line searching log, using the following grep command line show us expected.

> If `Trace` logs at or above '--v=7', then you can also check how P&F are processing requests that are exempt.
> We have one flowschema "openshift-apiserver-sar" that is exempt - all "subjectaccessreviews" or "tokenreviews" requests from openshift-apiserver are exempt. To check for this you can grep for "startRequest" and within the search results you should see that this flowschema is treated as "immediate". But the log level needs to be at 7 for this.

$ grep 'startRequest' *master*.log | grep 'system:serviceaccount:openshift-apiserver:openshift-apiserver-sa' | grep -E 'immediate|exempt' | head -1
kube-apiserver-kewang2273-7hpgp-master-1.log:I1022 16:07:22.501799      19 apf_controller.go:663] startRequest(RequestDigest{RequestInfo: &request.RequestInfo{IsResourceRequest:true, Path:"/apis/authorization.k8s.io/v1/subjectaccessreviews", Verb:"create", APIPrefix:"apis", APIGroup:"authorization.k8s.io", APIVersion:"v1", Namespace:"", Resource:"subjectaccessreviews", Subresource:"", Name:"", Parts:[]string{"subjectaccessreviews"}}, User: &user.DefaultInfo{Name:"system:serviceaccount:openshift-apiserver:openshift-apiserver-sa", UID:"18eb7c21-6e8d-4d22-8a20-456efe008c06", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-apiserver", "system:authenticated"}, Extra:map[string][]string(nil)}}) => fsName="openshift-apiserver-sar", distMethod=&v1alpha1.FlowDistinguisherMethod{Type:"ByUser"}, plName="exempt", immediate

$ grep 'startRequest' *master*.log | grep 'system:serviceaccount:openshift-oauth-apiserver:oauth-apiserver-sa' | grep -E 'immediate|exempt' | head -1
kube-apiserver-kewang2273-7hpgp-master-2.log:I1022 16:09:30.723485      19 apf_controller.go:663] startRequest(RequestDigest{RequestInfo: &request.RequestInfo{IsResourceRequest:true, Path:"/apis/authorization.k8s.io/v1/subjectaccessreviews", Verb:"create", APIPrefix:"apis", APIGroup:"authorization.k8s.io", APIVersion:"v1", Namespace:"", Resource:"subjectaccessreviews", Subresource:"", Name:"", Parts:[]string{"subjectaccessreviews"}}, User: &user.DefaultInfo{Name:"system:serviceaccount:openshift-oauth-apiserver:oauth-apiserver-sa", UID:"a24d5b7b-6deb-4126-a39f-114b73473297", Groups:[]string{"system:serviceaccounts", "system:serviceaccounts:openshift-oauth-apiserver", "system:authenticated"}, Extra:map[string][]string(nil)}}) => fsName="openshift-oauth-apiserver-sar", distMethod=&v1alpha1.FlowDistinguisherMethod{Type:"ByUser"}, plName="exempt", immediate
Comment 6 errata-xmlrpc 2021-02-24 15:26:11 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633