Bug 2162140

Summary: CNV VirtualMachine pod used a lot more memory then it suppose to
Product: Container Native Virtualization (CNV) Reporter: Sean Haselden <shaselde>
Component: VirtualizationAssignee: Itamar Holder <iholder>
Status: CLOSED DUPLICATE QA Contact: Kedar Bidarkar <kbidarka>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 4.10.1CC: acardace, dgilbert, dshchedr, fdeutsch, gveitmic, ibezukh, jortialc, kchamart, lpivarc, srengan, victortoso, vromanso, ycui
Target Milestone: ---   
Target Release: 4.13.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-03-28 13:45:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sean Haselden 2023-01-18 21:05:52 UTC
Description of problem:

virt-launcher pod consumes more memory than it should be allowed.

- The VM was configured with requests of 8G, and that is the total mem visible when we exec free on the guest os. The actual used mem observed from inside the VM is approx 2Gi. However, when inspecting the pod stats with crictl we can see mem usage by the pod is in dozens of Gi RAM.


Version-Release number of selected component (if applicable):
OpenShift 4.10.16, CNV in version 4.10.1, Kubevirt v0.49.0-155-g32b905704

How reproducible:

- VM has a single network interface attached to a cnv-bridge NAD, used for ingress/egress
- A pvc of 3.2 TB was attached to the VM, which downloaded 2.5 TB of data from an s3 bucket (ceph object storage). It was indeed saved to the machine.
- The workload was supposed to be CPU intensive, not  memory intensive. 
- The VM was configured with requests of 8G, and that is the total mem visible when we exec free on the guest os. The actual used mem observed from inside the VM is approx 2Gi. However, when inspecting the pod stats with crictl we can see mem usage by the pod is in dozens of Gi RAM.

- The attached PVC is an additional disk, not an extension of the root disk, and it is of type block with virtio-scsi driver. The customer used mkfs.xfs on the entire block with no partitions nor LVMs. We see the host cach



Actual results:


Expected results:

VM pod stays within limit of memory assigned to it. 

Additional info:

Comment 10 Antonio Cardace 2023-03-03 16:47:53 UTC
Deferring to 4.13.1 due to capacity.