Created attachment 1941101 [details] VM manifest with ceph disk Description of problem: Kubevirt computes [1] the estimation of total memory needed for the domain to operate properly. This includes the memory needed for the guest and memory for Qemu and OS overhead. When running IO intensive workloads, the memory consumption by the management layer (e.g. libvirt, QEMU, and other processes in the compute container) is larger than calculated. The difference between the expected and actual consumption is between 700M and 1.3G approximately. [1] https://github.com/kubevirt/kubevirt/blob/v0.59.0-alpha.2/pkg/virt-controller/services/renderresources.go#L272 Version-Release number of selected component (if applicable): 4.10.1 How reproducible: 100% Steps to Reproduce: 1. create a VM with either local or ceph disks (Manifests are attached) 2. create xfs filesystem 3. mount the disks 4. run vdbench test as per example (via "vdbench -f test_file") Actual results: Memory used by the virt-launcher pod is as or below what's calculated Expected results: The difference between the expected and actual consumption is between 700M and 1.3G approximately. Additional info: VM Manifest attached
PR is now out there: https://github.com/kubevirt/kubevirt/pull/9322
Verified on v4.13.0.rhel9-1931, don't see the missing overhead with running vdbench. Also verified that overhead memory can be increased with new kubevirt field: additionalGuestMemoryOverheadRatio
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Virtualization 4.13.0 Images security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:3205
Can you please help me understand how to configure additionalGuestMemoryOverheadRatio in OpenShift Virtualization where kubevirt CR is controlled by hco? I cannot see any similar configuration option in hco.
You can use jsonpatch annotations [1]. For example, from cli you can set it like this: > $ oc annotate --overwrite -n openshift-cnv hco kubevirt-hyperconverged kubevirt.kubevirt.io/jsonpatch='[{"op": "add", "path": "/spec/configuration/additionalGuestMemoryOverheadRatio", "value": "2"}]' and it will set neccessary parameter in the KubeVirt CR: > $ oc get kubevirt kubevirt-kubevirt-hyperconverged -n openshift-cnv -o json | jq .spec.configuration.additionalGuestMemoryOverheadRatio > "2" [1] https://github.com/kubevirt/hyperconverged-cluster-operator/blob/main/docs/cluster-configuration.md#jsonpatch-annotations