Bug 2063342

** A NOTE ABOUT USING URGENT **

This BZ has been set to urgent severity and priority. When a BZ is marked urgent priority Engineers are asked to stop whatever they are doing, putting everything else on hold.
Please be prepared to have reasonable justification ready to discuss, and ensure your own and engineering management are aware and agree this BZ is urgent. Keep in mind, urgent bugs are very expensive and have maximal management visibility.

NOTE: This bug was automatically assigned to an engineering manager with the severity reset to *unspecified* until the emergency is vetted and confirmed. Please do not manually override the severity.

** INFORMATION REQUIRED **

Please answer these questions before escalation to engineering:

1. Has a link to must-gather output been provided in this BZ? We cannot work without. If must-gather fails to run, attach all relevant logs and provide the error message of must-gather.
2. Give the output of "oc get clusteroperators -o yaml".
3. In case of degraded/unavailable operators, have all their logs and the logs of the operands been analyzed [yes/no]
4. List the top 5 relevant errors from the logs of the operators and operands in (3).
5. Order the list of degraded/unavailable operators according to which is likely the cause of the failure of the other, root-cause at the top.
6. Explain why (5) is likely the right order and list the information used for that assessment.
7. Explain why Engineering is necessary to make progress.

** A NOTE ABOUT USING URGENT **

This BZ has been set to urgent severity and priority. When a BZ is marked urgent priority Engineers are asked to stop whatever they are doing, putting everything else on hold.
Please be prepared to have reasonable justification ready to discuss, and ensure your own and engineering management are aware and agree this BZ is urgent. Keep in mind, urgent bugs are very expensive and have maximal management visibility.

NOTE: This bug was automatically assigned to an engineering manager with the severity reset to *unspecified* until the emergency is vetted and confirmed. Please do not manually override the severity.

** INFORMATION REQUIRED **

Please answer these questions before escalation to engineering:

1. Has a link to must-gather output been provided in this BZ? We cannot work without. If must-gather fails to run, attach all relevant logs and provide the error message of must-gather.
2. Give the output of "oc get clusteroperators -o yaml".
3. In case of degraded/unavailable operators, have all their logs and the logs of the operands been analyzed [yes/no]
4. List the top 5 relevant errors from the logs of the operators and operands in (3).
5. Order the list of degraded/unavailable operators according to which is likely the cause of the failure of the other, root-cause at the top.
6. Explain why (5) is likely the right order and list the information used for that assessment.
7. Explain why Engineering is necessary to make progress.

Sorry as synced in Slack didn't afford time on this due to other work. Today tested in 4.11.0-0.nightly-2022-04-13-072124, now tokenreviews of k8s group are indeed audited as "level":"Metadata", so kube-apiserver/audit*.log don't expose tokens now. But note, it has 2 groups:
$ oc api-resources | grep token
...
tokenreviews                                              authentication.k8s.io/v1                      false        TokenReview
...
tokenreviews                                              oauth.openshift.io/v1                         false        TokenReview
...

The PR only excludes the k8s group authentication.k8s.io/v1. The tokenreviews of group oauth.openshift.io/v1 still expose tokens in oauth-apiserver/audit*.log, like:
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","auditID":"e44f31f1-a134-4c28-926a-1455ce903039","stage":"ResponseComplete","requestURI":"/apis/oauth.openshift.io/v1/tokenreviews?timeout=30s","verb":"create","user":{"username":"system:serviceaccount:openshift-oauth-apiserver:openshift-authenticator","groups":["system:authenticated"]},"sourceIPs":["10.128.0.1"],"userAgent":"Go-http-client/2.0","objectRef":{"resource":"tokenreviews","apiGroup":"oauth.openshift.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":201},"requestObject":{"kind":"TokenReview","apiVersion":"authentication.k8s.io/v1","metadata":{"creationTimestamp":null},"spec":{"token":"sha256~3QXUWU29_yTsssQDUHIwhX4h9sK2QUqThlIJgdn-cXE","audiences":["https://kubernetes.default.svc"]},"status":{"user":{}}},"responseObject":{"kind":"TokenReview","apiVersion":"authentication.k8s.io/v1","metadata":{"creationTimestamp":null},"spec":{"token":"sha256~3QXUWU29_yTsssQDUHIwhX4h9sK2QUqThlIJgdn-cXE","audiences":["https://kubernetes.default.svc"]},"status":{"authenticated":true,"user":{"username":"testuser-30","uid":"069b4328-3e0c-43e9-a8f8-27984c71b013","groups":["system:authenticated:oauth"],"extra":{"scopes.authorization.openshift.io":["user:full"]}},"audiences":["https://kubernetes.default.svc"]}},"requestReceivedTimestamp":"2022-04-13T15:08:02.748219Z","stageTimestamp":"2022-04-13T15:08:02.784684Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"requesting tokenreviews is allowed"}}

As for tokenrequests, I followed https://docs.openshift.com/container-platform/4.10/authentication/bound-service-account-tokens.html to hopefully trigger tokenrequests and then grep tokenrequests in /var/log/, but grep gets nothing at all, even if Metadata level isn't gotten, not sure why? Seems tokenrequests are not audited at all in any level? Should I create a bug for it?

Revisiting my comment 12:
> As for tokenrequests, I followed https://docs.openshift.com/container-platform/4.10/authentication/bound-service-account-tokens.html to hopefully trigger tokenrequests and then grep tokenrequests in /var/log/, but grep gets nothing at all
I thought tokenrequests are in "requestURI" field of audit logs so I did not find, forgot checking https://docs.openshift.com/container-platform/4.10/rest_api/authorization_apis/tokenrequest-authentication-k8s-io-v1.html#tokenrequest-authentication-k8s-io-v1 , which shows requestURI for tokenrequests looks like /api/v1/namespaces/{namespace}/serviceaccounts/{name}/token instead.
So today retested tokenrequests in 4.11.0-0.nightly-2022-06-15-222801 env:

oc edit apiserver cluster
...
spec:
  audit:
    profile: WriteRequestBodies

oc get po -n openshift-kube-apiserver -L revision -l apiserver -w
Wait about 15 mins for kube-apiserver pods to roll out.

Then:
oc new-project test-proj
MASTERS=`oc get no | grep master | grep -o '^[^ ]*'`

NOW=$(date -u "+%s")
echo "$NOW"
echo "$NOW" > now

Create any pod under test-proj (Pod automatically uses bound serviceaccount token, which posts TokenRequests)
oc new-app httpd --name httpd-2 -n test-proj

Check the audit logs in /var/log/kube-apiserver:
DIRECTORY=kube-apiserver
PATTERN='/token'
oc config use-context admin
for i in $MASTERS
do
  oc debug no/$i -- chroot /host bash -c "grep -ihE '$PATTERN' /var/log/$DIRECTORY/audit*.log || true"
done | jq -c 'select (.requestReceivedTimestamp | .[0:19] + "Z" | fromdateiso8601 > '"`cat now`)" > audit_tokenrequest.log

TokenRequests audit logs leak tokens:
vi audit_tokenrequest.log
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","auditID":"f1e2d851-41e5-4f81-a715-02da803533ba","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/test-proj/serviceaccounts/default/token","verb":"create","user":{"username":"system:node:xxia-617c-tgr6k-worker-northcentralus-62hvx","groups":["system:nodes","system:authenticated"]},"sourceIPs":["10.0.128.4"],"userAgent":"kubelet/v1.24.0+cb71478 (linux/amd64) kubernetes/0a57f1f","objectRef":{"resource":"serviceaccounts","namespace":"test-proj","name":"default","apiVersion":"v1","subresource":"token"},"responseStatus":{"metadata":{},"code":201},"requestObject":{"kind":"TokenRequest","apiVersion":"authentication.k8s.io/v1","metadata":{"creationTimestamp":null},"spec":{"audiences":null,"expirationSeconds":3607,"boundObjectRef":{"kind":"Pod","apiVersion":"v1","name":"httpd-2-56dd889b6-2pjp9","uid":"644527ca-aebc-45d9-ab2f-f2b25019c9d7"}},"status":{"token":"","expirationTimestamp":null}},"responseObject":{"kind":"TokenRequest","apiVersion":"authentication.k8s.io/v1","metadata":{"name":"default","namespace":"test-proj","creationTimestamp":"2022-06-17T09:11:11Z"},"spec":{"audiences":["https://kubernetes.default.svc"],"expirationSeconds":3607,"boundObjectRef":{"kind":"Pod","apiVersion":"v1","name":"httpd-2-56dd889b6-2pjp9","uid":"644527ca-aebc-45d9-ab2f-f2b25019c9d7"}},"status":{"token":"eyJhbGciOiJSUzI1NiIsImtpZCI6ImNVUnY1VFJaR0RtQU9NYm9yd1ptNXB2OHZ3TXpJOG51TTFrOS01LTJjV28ifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6MTY4Njk5MzA3MSwiaWF0IjoxNjU1NDU3MDcxLCJpc3MiOiJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMiLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6InRlc3QtcHJvaiIsInBvZCI6eyJuYW1lIjoiaHR0cGQtMi01NmRkODg5YjYtMnBqcDkiLCJ1aWQiOiI2NDQ1MjdjYS1hZWJjLTQ1ZDktYWIyZi1mMmIyNTAxOWM5ZDcifSwic2VydmljZWFjY291bnQiOnsibmFtZSI6ImRlZmF1bHQiLCJ1aWQiOiJiOWM3NDhjYS0zNDZkLTQ3MWItYWNhMC05ZmVmOTE4YzUxODgifSwid2FybmFmdGVyIjoxNjU1NDYwNjc4fSwibmJmIjoxNjU1NDU3MDcxLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6dGVzdC1wcm9qOmRlZmF1bHQifQ.38eFGcw01yNlmgfmNePtwyk68tYDVIX-rnERQPSa1_qEGbQI22ruj-X2XikRXIDwOTABveZsE_X4PQBXucLt_Q3yObwJNRQGnZYBaUBMPRMI_oBuJlTosfMI2Y3YOpDEDQ2nOq9xhxGC4_XI8oUWwaU9IVCLXAwyinncJvVtXgGdxeUF9uBhv9t3-_csRId1hObXbkRoycFx1z4w14ITZ8YOGG6dDN3BBa_uJbj1C1lrAmKlMNudXlicSYThK4sQI9sKZCzYKQYeuenzmVnH60o4aJaiMP8cRkiT98mILKn-07purorrcJ7p6wc2Gu_U6nIKD9e-oTOujoNPy9W_Gp-oPcZ9j1pbn0laTAEkJ1DguntsukImBw1PVYAiLqHzgmWrfdPADlxWQ8hXEOFd2tdgq6t6kukOg2cLlcj0JpQobceIKju606ZHPy0JsSL4kf4kAY55pZNDEhnCQwENjBA77hyUbAd96AITW3Ae5UVSnlpL4-huVDKQ89OABgs6k1T_U8H8vb6BkdJJauHb7x_VQ3OopLvZkakSc8YpS2p59qQspDngz3V-KzGFvHKniE5iRe6Tjm2J5sR7pX6NvP3qBRdiX-WCb8VStzfNT1VBQPaNBywRClnVtphRmrniv8Gp_ivt87Fq8Ihy9Ue2tXifJ0QrQsFkyuMrxqHFYF0","expirationTimestamp":"2022-06-17T10:11:18Z"}},"requestReceivedTimestamp":"2022-06-17T09:11:11.415527Z","stageTimestamp":"2022-06-17T09:11:11.437439Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}

In conclusion, in addition to needing further fix for tokenreviews of group oauth.openshift.io/v1 as said in comment 12, for tokenrequests the fix PR should exclude subresource "token" instead of "tokenrequests" like previous https://github.com/openshift/library-go/pull/1327/files did: `resources: [..., "tokenrequests"] : )

BTW answering Akhil Harchinder:
> I'm wondering how we could identify suspicious/malicious activity related to the leaked tokens. Using system:serviceaccount:openshift-monitoring:prometheus-k8s as an example
Hi, I'm from QE. Checking audit logs for the users of the leaked tokens is the way AFAIK, like:
MASTERS=`oc get no | grep master | grep -o '^[^ ]*'`
PATTERN='"user":{"username":"system:serviceaccount:openshift-monitoring:prometheus-k8s"'
for i in $MASTERS
do
  oc debug no/$i -- chroot /host bash -c "grep -hE '$PATTERN' /var/log/*server/audit*.log || true"
done > audit_user_activities.log

Sort the activities as per timestamp:
jq -cs 'sort_by(.requestReceivedTimestamp)' audit_user_activities.log | jq -c '.[]' > audit_user_activities_sorted.log

Then check audit_user_activities_sorted.log to analyze the activies or do further filtering as your want, e.g. filter "sourceIPs", "userAgent" fields:
...,"sourceIPs":["66.187.232.129"],"userAgent":"oc/4.11.0 (linux/amd64) kubernetes/63bee3e",... <-- this should be some human activities
...,"sourceIPs":["10.0.128.6"],"userAgent":"Prometheus/2.36.1",...                              <-- this should be internal activities

This maybe can do some help?

Dear reporter, we greatly appreciate the bug you have reported here. Unfortunately, due to migration to a new issue-tracking system (https://issues.redhat.com/), we cannot continue triaging bugs reported in Bugzilla. Since this bug has been stale for multiple days, we, therefore, decided to close this bug.
If you think this is a mistake or this bug has a higher priority or severity as set today, please feel free to reopen this bug and tell us why. We are going to move every re-opened bug to https://issues.redhat.com. 

Thank you for your patience and understanding.

Tracked here instead: https://issues.redhat.com/browse/OCPBUGS-2765