Bug 2078545 - many KubeVirtComponentExceedsRequestedMemory alerts in Firing state
Summary: many KubeVirtComponentExceedsRequestedMemory alerts in Firing state
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Virtualization
Version: 4.11.0
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ---
: 4.11.4
Assignee: Jed Lejosne
QA Contact: Akriti Gupta
URL:
Whiteboard:
Depends On:
Blocks: 2151170 2166385
TreeView+ depends on / blocked
 
Reported: 2022-04-25 15:00 UTC by Denys Shchedrivyi
Modified: 2023-05-30 15:38 UTC (History)
6 users (show)

Fixed In Version: hco-bundle-registry-container-v4.11.3-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2151170 2166385 (view as bug list)
Environment:
Last Closed: 2023-05-30 15:37:44 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
alerts (123.93 KB, image/png)
2022-04-25 15:00 UTC, Denys Shchedrivyi 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Github kubevirt kubevirt pull 8096 0 None Merged Bump memory requests for virt-(api|controller|operator) 2023-02-13 13:46:36 UTC
Github kubevirt kubevirt pull 8227 0 None Merged [release-0.53] Bump memory requests for virt-(api|controller|operator) 2023-02-13 13:46:37 UTC
Github kubevirt kubevirt pull 8687 0 None Merged [release-0.53] Bump memory requests according to values seen in testing 2023-02-13 13:46:38 UTC
Github kubevirt kubevirt pull 8707 0 None Merged [release-0.49] Bump memory requests according to value seen in testing 2023-02-13 13:46:39 UTC
Github kubevirt kubevirt pull 9125 0 None Merged virt-*: bump memory requests above highest values seen in testing 2023-04-17 14:03:52 UTC
Github kubevirt kubevirt pull 9418 0 None Merged [release-0.53] virt-*: bump memory requests above highest values seen in testing 2023-04-17 14:03:55 UTC
Red Hat Issue Tracker CNV-17819 0 None None None 2022-11-01 13:53:25 UTC
Red Hat Product Errata RHEA-2023:3352 0 None None None 2023-05-30 15:38:08 UTC

Description Denys Shchedrivyi 2022-04-25 15:00:10 UTC 
Created attachment 1874828 [details]
alerts

Description of problem:
 There are many alerts KubeVirtComponentExceedsRequestedMemory in Firing state on a cluster, some of them appeared right after installation.
 May be memory should be adjusted for kubevirt system pods?

Version-Release number of selected component (if applicable):
 4.11


Actual results:
 Alerts are constantly firing, some of them triggered right after cluster installation. Screenshot attached

Expected results:
 kubevirt system pods requested enough memory and alert is not triggered
Comment 1 Kedar Bidarkar 2022-05-10 09:44:33 UTC 
Denys, Is this issue seen recurring with the latest 4.11 builds ?
Comment 2 Denys Shchedrivyi 2022-05-10 21:47:42 UTC 
yes, it is firing for some pods right after installation in v4.11.0-327
Comment 3 sgott 2022-07-13 18:16:17 UTC 
Raising the priority and severity of this because this can actually be a real problem on nodes with memory pressure. KubeVirt resources will be high on the list to be killed in such a case.
Comment 4 Jed Lejosne 2022-08-01 13:17:06 UTC 
The github PR linked with this issue was the one against the main branch.
I just added the (un-merged) backport PR to release-0.53 that will affect CNV 4.11
Comment 7 Antonio Cardace 2022-10-13 15:00:34 UTC 
Moving back to 4.11.1 as all components but virt-handler have been addressed and virt-handler itself is tracked specifically by https://bugzilla.redhat.com/show_bug.cgi?id=2134123.
Comment 8 Denys Shchedrivyi 2022-10-17 17:42:53 UTC 
Moving this BZ to Assigned state since in my opinion all memory requests should be adjusted. After some research I see all of virt components numbers are exceed the limits (or very close to it) on CNV v4.11:

virt-api        - 245Mb
virt-controller - 270Mb
virt-handler    - 248Mb
virt-operator   - 255Mb (There is Bug 2132473 with virt-operator which increases memory usage on every Kubevirt CR update, so it is difficult to predict how much memory it consumes by default)
Comment 16 Antonio Cardace 2023-01-31 09:38:18 UTC 
virt-controller memory request still has to be adjusted, deferring to 4.11.4 as we're past code freeze for 4.11.3.
Comment 25 errata-xmlrpc 2023-05-30 15:37:44 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Virtualization 4.11.4 Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2023:3352
Note You need to log in before you can comment on or make changes to this bug.