Bug 2006810
| Summary: | CNAO is bombarding kube API and HCO with status updates | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Container Native Virtualization (CNV) | Reporter: | Petr Horáček <phoracek> | ||||
| Component: | Networking | Assignee: | Ram Lavi <ralavi> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Meni Yakove <myakove> | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 4.8.1 | CC: | cnv-qe-bugs, rbaumgar, ysegev | ||||
| Target Milestone: | --- | ||||||
| Target Release: | 4.9.1 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | v4.9.1-16 cluster-network-addons-operator-container-v4.9.1-5 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2021-12-13 19:59:01 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
There are 2 entities that update the status: - pods-controller (https://github.com/kubevirt/cluster-network-addons-operator/blob/main/pkg/controller/networkaddonsconfig/pod_controller.go#L41) that updates every time an object is updated, - the CR-controller (https://github.com/kubevirt/cluster-network-addons-operator/blob/main/pkg/controller/networkaddonsconfig/networkaddonsconfig_controller.go#L196) that updated every 1 minute. seems like the CR-controller is setting the status multiple times in the same reconcile (https://github.com/kubevirt/cluster-network-addons-operator/blob/main/pkg/controller/networkaddonsconfig/networkaddonsconfig_controller.go#L322, https://github.com/kubevirt/cluster-network-addons-operator/blob/main/pkg/controller/networkaddonsconfig/networkaddonsconfig_controller.go#L316 and https://github.com/kubevirt/cluster-network-addons-operator/blob/e2db6fa042e0f91403b78ee4d17fee54acb1fa1f/pkg/controller/networkaddonsconfig/networkaddonsconfig_controller.go#L511) However, my current efforts to reduce this (PR https://github.com/kubevirt/cluster-network-addons-operator/pull/1032) causes issues with the ci. This need a more in depth investigation. I checked CNAO pod log, and the issue seems to be resolved. The reconciling message + "Successfully updated status conditions" appear once every ~1 minute. Verified on: OCP 4.9.6 CNV 4.9.1 cluster-network-addons-operator v4.9.1-6 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Virtualization 4.9.1 Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:5091 |
Created attachment 1825323 [details] Logs of CNAO Description of problem: HCO reconciles all changes of ClusterNetworkAddons. It was observed that on 4.8.2, these changes happen about 8 times every minute. That is overwhelming both HCO and the Kubernetes API. We should make sure that all these updates are really needed (are not just bumping the timestamp) and we should lower the frequency. Version-Release number of selected component (if applicable): CNV 4.8.2 How reproducible: Unknown Steps to Reproduce: 1. Install CNAO and apply NetworkAddonsConfig Actual results: NetworkAddonsConfig status is updated 8 times a minute. Expected results: The status is updated less often and only when necessary. Additional info: