Bug 2117679 - kube-controller-manager needs to stop watching all events
Summary: kube-controller-manager needs to stop watching all events
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-controller-manager
Version: 4.12
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.11.z
Assignee: Ben Luddy
QA Contact: RamaKasturi
URL:
Whiteboard:
Depends On: 2117569 2118318
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-08-11 15:15 UTC by OpenShift BugZilla Robot
Modified: 2023-02-07 13:22 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-02-07 13:22:41 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift kubernetes pull 1343 0 None open [release-4.11] Bug 2117679: UPSTREAM: 110888: feat: fix a bug thaat not all event be ignored by gc controller 2023-01-19 12:39:02 UTC
Github openshift kubernetes pull 1379 0 None open [release-4.11] OCPBUGS-1991: UPSTREAM: 110939: don't quota events.k8s.io events by default 2023-01-19 12:39:02 UTC
Red Hat Product Errata RHSA-2023:0565 0 None None None 2023-02-07 13:22:44 UTC

Comment 1 RamaKasturi 2022-10-06 07:47:16 UTC
Have tested this bug via per-merge testing and below are the steps i have followed to test the same.

Steps followed to test the build with fix :
=========================================
1. create a cluster using cluster-bot with the command "launch openshift/kubernetes#1343,openshift/kubernetes#1379 no-spot"
2. Login to the cluster, run command 'oc debug node/<masternode>'; chroot /host
3. Run less /var/log/kube-apiserver/audit.log | jq -c 'select(.verb=="watch" and .objectRef.resource=="events" and .user.username=="system:kube-controller-manager")'
4. Verify that no output gets displayed which is expected.

Results:
===============
sh-4.4# less /var/log/kube-apiserver/audit.log | jq -c 'select(.verb=="watch" and .objectRef.resource=="events" and .user.username=="system:kube-controller-manager")'
sh-4.4# 

Steps followed to test the build with out fix:
====================================================
1. create a 4.11 cluster
2. Login to the cluster, run command 'oc debug node/<masternode>'; chroot /host
3. Run less /var/log/kube-apiserver/audit.log | jq -c 'select(.verb=="watch" and .objectRef.resource=="events" and .user.username=="system:kube-controller-manager")'
4. Verify that there is output that gets displayed which has logs related to events.

Results:
===============
sh-4.4# less /var/log/kube-apiserver/audit.log | jq -c 'select(.verb=="watch" and .objectRef.resource=="events" and .user.username=="system:kube-controller-manager")'
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"f5221016-fe1f-4c54-b559-819223a8ea4f","stage":"ResponseComplete","requestURI":"/apis/events.k8s.io/v1/events?allowWatchBookmarks=true&resourceVersion=38712&timeout=5m27s&timeoutSeconds=327&watch=true","verb":"watch","user":{"username":"system:kube-controller-manager","groups":["system:authenticated"]},"sourceIPs":["10.0.214.219"],"userAgent":"cluster-policy-controller/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"events","apiGroup":"events.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2022-10-06T07:10:41.197122Z","stageTimestamp":"2022-10-06T07:16:08.201028Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"system:kube-controller-manager\" of ClusterRole \"system:kube-controller-manager\" to User \"system:kube-controller-manager\""}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"20af4be9-4a0f-4a49-84ca-db145f9a017a","stage":"ResponseStarted","requestURI":"/apis/events.k8s.io/v1/events?allowWatchBookmarks=true&resourceVersion=41527&timeout=5m17s&timeoutSeconds=317&watch=true","verb":"watch","user":{"username":"system:kube-controller-manager","groups":["system:authenticated"]},"sourceIPs":["10.0.214.219"],"userAgent":"cluster-policy-controller/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"events","apiGroup":"events.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2022-10-06T07:16:08.202558Z","stageTimestamp":"2022-10-06T07:16:08.203681Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"system:kube-controller-manager\" of ClusterRole \"system:kube-controller-manager\" to User \"system:kube-controller-manager\""}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"20af4be9-4a0f-4a49-84ca-db145f9a017a","stage":"ResponseComplete","requestURI":"/apis/events.k8s.io/v1/events?allowWatchBookmarks=true&resourceVersion=41527&timeout=5m17s&timeoutSeconds=317&watch=true","verb":"watch","user":{"username":"system:kube-controller-manager","groups":["system:authenticated"]},"sourceIPs":["10.0.214.219"],"userAgent":"cluster-policy-controller/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"events","apiGroup":"events.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2022-10-06T07:16:08.202558Z","stageTimestamp":"2022-10-06T07:21:25.204262Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"system:kube-controller-manager\" of ClusterRole \"system:kube-controller-manager\" to User \"system:kube-controller-manager\""}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"41deb121-a103-43cf-ad64-54e9315ce7ac","stage":"ResponseStarted","requestURI":"/apis/events.k8s.io/v1/events?allowWatchBookmarks=true&resourceVersion=43433&timeout=6m17s&timeoutSeconds=377&watch=true","verb":"watch","user":{"username":"system:kube-controller-manager","groups":["system:authenticated"]},"sourceIPs":["10.0.214.219"],"userAgent":"cluster-policy-controller/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"events","apiGroup":"events.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2022-10-06T07:21:25.205570Z","stageTimestamp":"2022-10-06T07:21:25.206518Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"system:kube-controller-manager\" of ClusterRole \"system:kube-controller-manager\" to User \"system:kube-controller-manager\""}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"41deb121-a103-43cf-ad64-54e9315ce7ac","stage":"ResponseComplete","requestURI":"/apis/events.k8s.io/v1/events?allowWatchBookmarks=true&resourceVersion=43433&timeout=6m17s&timeoutSeconds=377&watch=true","verb":"watch","user":{"username":"system:kube-controller-manager","groups":["system:authenticated"]},"sourceIPs":["10.0.214.219"],"userAgent":"cluster-policy-controller/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"events","apiGroup":"events.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2022-10-06T07:21:25.205570Z","stageTimestamp":"2022-10-06T07:27:42.207260Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"system:kube-controller-manager\" of ClusterRole \"system:kube-controller-manager\" to User \"system:kube-controller-manager\""}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"c8e3efda-a8da-4c46-8fcb-55382ed9e9cb","stage":"ResponseStarted","requestURI":"/apis/events.k8s.io/v1/events?allowWatchBookmarks=true&resourceVersion=45164&timeout=9m50s&timeoutSeconds=590&watch=true","verb":"watch","user":{"username":"system:kube-controller-manager","groups":["system:authenticated"]},"sourceIPs":["10.0.214.219"],"userAgent":"cluster-policy-controller/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"events","apiGroup":"events.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2022-10-06T07:27:42.208567Z","stageTimestamp":"2022-10-06T07:27:42.209242Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"system:kube-controller-manager\" of ClusterRole \"system:kube-controller-manager\" to User \"system:kube-controller-manager\""}}

Based on the above setting Verified flag to tested to indicate the fix actually works fine.

Comment 2 RamaKasturi 2022-10-06 07:49:37 UTC
@benluddy could you please help add the other PR as well to the bug as that is needed to have complete fix.

Comment 3 RamaKasturi 2022-10-06 07:50:22 UTC
Version where this was tested:
===============================
[knarra@knarra openshift-tests-private]$ oc get clusterversion
NAME      VERSION                                                   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.0-0.ci.test-2022-10-06-065450-ci-ln-bpwy562-latest   True        False         26m     Cluster version is 4.11.0-0.ci.test-2022-10-06-065450-ci-ln-bpwy562-latest

Comment 8 errata-xmlrpc 2023-02-07 13:22:41 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.26 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:0565


Note You need to log in before you can comment on or make changes to this bug.