Bug 2165618

Summary: Overhead of management layer in virt-launcher is not calculated accurately
Product: Container Native Virtualization (CNV) Reporter: Itamar Holder <iholder>
Component: VirtualizationAssignee: Itamar Holder <iholder>
Status: CLOSED ERRATA QA Contact: Denys Shchedrivyi <dshchedr>
Severity: high Docs Contact:
Priority: urgent    
Version: 4.10.1CC: acardace, bbenshab, fdeutsch, jhopper, mimehta, nashok
Target Milestone: ---   
Target Release: 4.13.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: hco-bundle-registry-container-v4.13.0.rhel9-1849 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2176031 2176032 2176034 (view as bug list) Environment:
Last Closed: 2023-05-18 02:57:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2176031, 2176032, 2176034    
Attachments:
Description Flags
VM manifest with ceph disk none

Description Itamar Holder 2023-01-30 15:08:46 UTC 
Created attachment 1941101 [details]
VM manifest with ceph disk

Description of problem:
Kubevirt computes [1] the estimation of total memory needed for the domain to operate properly. This includes the memory needed for the guest and memory for Qemu and OS overhead.

When running IO intensive workloads, the memory consumption by the management layer (e.g. libvirt, QEMU, and other processes in the compute container) is larger than calculated. The difference between the expected and actual consumption is between 700M and 1.3G approximately.

[1] https://github.com/kubevirt/kubevirt/blob/v0.59.0-alpha.2/pkg/virt-controller/services/renderresources.go#L272

Version-Release number of selected component (if applicable):
4.10.1

How reproducible:
100%

Steps to Reproduce:
1. create a VM with either local or ceph disks (Manifests are attached)
2. create xfs filesystem
3. mount the disks
4. run vdbench test as per example (via "vdbench -f test_file")

Actual results:
Memory used by the virt-launcher pod is as or below what's calculated

Expected results:
The difference between the expected and actual consumption is between 700M and 1.3G approximately.

Additional info:
VM Manifest attached
Comment 4 Itamar Holder 2023-02-27 09:10:47 UTC 
PR is now out there: https://github.com/kubevirt/kubevirt/pull/9322
Comment 5 Denys Shchedrivyi 2023-03-30 16:11:31 UTC 
 Verified on v4.13.0.rhel9-1931, don't see the missing overhead with running vdbench.

 Also verified that overhead memory can be increased with new kubevirt field: additionalGuestMemoryOverheadRatio
Comment 7 errata-xmlrpc 2023-05-18 02:57:02 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Virtualization 4.13.0 Images security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:3205
Comment 8 nijin ashok 2023-09-07 05:59:33 UTC 
Can you please help me understand how to configure additionalGuestMemoryOverheadRatio in OpenShift Virtualization where kubevirt CR is controlled by hco?  I cannot see any similar configuration option in hco.
Comment 9 Denys Shchedrivyi 2023-09-07 14:57:34 UTC 
You can use jsonpatch annotations [1]. For example, from cli you can set it like this:

> $ oc annotate --overwrite -n openshift-cnv hco kubevirt-hyperconverged kubevirt.kubevirt.io/jsonpatch='[{"op": "add", "path": "/spec/configuration/additionalGuestMemoryOverheadRatio", "value": "2"}]'

and it will set neccessary parameter in the KubeVirt CR:

> $ oc get kubevirt kubevirt-kubevirt-hyperconverged -n openshift-cnv -o json | jq .spec.configuration.additionalGuestMemoryOverheadRatio
> "2"


[1] https://github.com/kubevirt/hyperconverged-cluster-operator/blob/main/docs/cluster-configuration.md#jsonpatch-annotations