Bug 1996660 - [4.8] Goroutine count and memory remains high after VMIs are removed
Summary: [4.8] Goroutine count and memory remains high after VMIs are removed
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: SSP
Version: 4.8.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.8.2
Assignee: Kevin Wiesmueller
QA Contact: Sarah Bennert
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-08-23 12:27 UTC by Kevin Wiesmueller
Modified: 2023-01-26 13:04 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-09-21 11:08:02 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github kubevirt kubevirt issues 6056 0 None None None 2021-08-23 12:27:34 UTC
Github kubevirt kubevirt pull 6227 0 None None None 2021-08-23 12:27:34 UTC
Red Hat Issue Tracker CNV-13839 0 None None None 2023-01-26 13:04:58 UTC
Red Hat Product Errata RHSA-2021:3598 0 None None None 2021-09-21 11:08:54 UTC

Internal Links: 2004295 2004299 2004300

Description Kevin Wiesmueller 2021-08-23 12:27:35 UTC
This bug was initially created as a copy of Bug #1996658

I am copying this bug to track the fix for 4.8 as well.


Description of problem:
Originally tracked in https://github.com/kubevirt/kubevirt/issues/6056
When creating VMIs and deleting them, virt-handler does not clean up goroutines properly.

How reproducible:
Create n VMIs, wait until they are all running, delete n VMIs, wait until they are all deleted, sleep 5s, and repeat. Scenarios for 10, 20, 40, 60, 80, 100, 200, 300 VMIs shows an increase in virt-handler mem/cpu usage and go routines, however only cpu usage returns back to the expected level when there are 0 VMIs.

Steps to Reproduce:
1. Create 100 VMIs (not needed to start them), can use Kubevirt Density Test
2. Delete 100 VMIs
3. Observe goroutines metric on virt-handler

Actual results:
Goroutines metric does not go down all the way after deletion.

Expected results:
Goroutines metric should go down all the way after deletion. 

Fixed in https://github.com/kubevirt/kubevirt/pull/6176
Backport for 4.8 is pending until this cherry-pick merges: https://github.com/kubevirt/kubevirt/pull/6227

Comment 3 Sarah Bennert 2021-09-14 23:55:49 UTC
Initial testing[0] was performed using density test, which does start the VMs.
Verified significant reduction in goroutine leakage.


While testing, observed a smaller leak and have opened follow-up bzs.[1,2,3]

[0] https://bugzilla.redhat.com/show_bug.cgi?id=1996660#c0
[1] 2.6 https://bugzilla.redhat.com/show_bug.cgi?id=2004300
[2] 4.8 https://bugzilla.redhat.com/show_bug.cgi?id=2004295
[3] 4.9 https://bugzilla.redhat.com/show_bug.cgi?id=2004299

Comment 8 errata-xmlrpc 2021-09-21 11:08:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Virtualization 4.8.2 Images security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3598


Note You need to log in before you can comment on or make changes to this bug.