Bug 1991900

Description of problem:
Now with https://bugzilla.redhat.com/show_bug.cgi?id=1986829 verified, still see tokenreview requests from some targets for the prometheus SA. 


Version-Release number of selected component (if applicable):
Latest 4.9 nightly payload


How reproducible:
Always

Steps to Reproduce:
1. Check metric client certificate
 oc get secret -n openshift-monitoring


2.Configure audit profile from default  to WriteRequestBodies in apiserver/cluster and wait to restart kube-apiserver
oc edit apiserver/cluster

spec:
  audit:
    profile: WriteRequestBodies
                                 

Check kube-apiserver restart or not
oc get pod -n openshift-kube-apiserver -l apiserver --show-labels

3. Check and gather audit logs after kube-api server restart and wait for 15mins. Login to all master and gather audit logs.


oc debug node/ci-ln-6t078vk-f76d1-mv5bv-master-2 -T -- chroot /host grep '"requestURI":"/apis/authentication.k8s.io/v1/tokenreviews"' /var/log/kube-apiserver/audit.log > /tmp/all_tokenreviews_requests.log

grep '"status":{"authenticated":true,"user":{"username":"system:serviceaccount:openshift-monitoring:prometheus-k8s"' /tmp/all_tokenreviews_requests.log > /tmp/all_tokenreviews_for_serviceaccount_prometheus-k8s.log

jq '.user.username' /tmp/all_tokenreviews_for_serviceaccount_prometheus-k8s.log > /tmp/all_users_that_make_traffic_to_check_token_of_serviceaccount_prometheus-k8s.log

sort /tmp/all_users_that_make_traffic_to_check_token_of_serviceaccount_prometheus-k8s.log | uniq -c | sort -rh>/tmp/users.txt


Check there is no token validation requests send to  kube-apiserver from below users and there will no output/display.

for i in kube-apiserver openshift-apiserver openshift-controller-manager kube-scheduler kubelet node-exporter kube-controller-manager etcd; do grep "$i" /tmp/users.txt;done;

Actual results:
1 "system:serviceaccount:openshift-controller-manager:openshift-controller-manager-sa"
4 "system:kube-scheduler"

Expected results:
for this command output should be blank

for i in kube-apiserver openshift-apiserver openshift-controller-manager kube-scheduler kubelet node-exporter kube-controller-manager etcd; do grep "$i" /tmp/users.txt;done;