Bug 1996658 - Goroutine count and memory remains high after VMIs are removed
Summary: Goroutine count and memory remains high after VMIs are removed
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: SSP
Version: 2.6.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 2.6.9
Assignee: Karel Šimon
QA Contact: Sarah Bennert
URL: https://github.com/kubevirt/kubevirt/...
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-08-23 12:20 UTC by Kevin Wiesmueller
Modified: 2022-02-02 19:45 UTC (History)
3 users (show)

Fixed In Version: kubevirt-2.6.9-215.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-02-02 19:45:05 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github kubevirt kubevirt issues 6056 0 None None None 2021-08-23 12:20:09 UTC
Github kubevirt kubevirt pull 6176 0 None None None 2021-08-23 12:20:09 UTC
Github kubevirt kubevirt pull 6227 0 None None None 2021-08-23 12:20:09 UTC
Red Hat Issue Tracker CNV-13837 0 None None None 2021-10-19 05:47:37 UTC
Red Hat Product Errata RHBA-2022:0414 0 None None None 2022-02-02 19:45:10 UTC

Description Kevin Wiesmueller 2021-08-23 12:20:10 UTC
Description of problem:
Originally tracked in https://github.com/kubevirt/kubevirt/issues/6056
When creating VMIs and deleting them, virt-handler does not clean up goroutines properly.

How reproducible:
Create n VMIs, wait until they are all running, delete n VMIs, wait until they are all deleted, sleep 5s, and repeat. Scenarios for 10, 20, 40, 60, 80, 100, 200, 300 VMIs shows an increase in virt-handler mem/cpu usage and go routines, however only cpu usage returns back to the expected level when there are 0 VMIs.

Steps to Reproduce:
1. Create 100 VMIs (not needed to start them), can use Kubevirt Density Test
2. Delete 100 VMIs
3. Observe goroutines metric on virt-handler

Actual results:
Goroutines metric does not go down all the way after deletion.

Expected results:
Goroutines metric should go down all the way after deletion. 

Fixed in https://github.com/kubevirt/kubevirt/pull/6176
Backport for 2.6 is pending until this cherry-pick merges: https://github.com/kubevirt/kubevirt/pull/6227

Comment 2 Dominik Holler 2021-11-17 12:33:16 UTC
waiting for CI on https://github.com/kubevirt/kubevirt/pull/6291

Comment 3 Sarah Bennert 2022-01-12 14:17:02 UTC
Verified.
After deleting VMs, virt-handler goroutine metric returns to value observed before VM creation. 

v2.6.9-40

Comment 9 errata-xmlrpc 2022-02-02 19:45:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Virtualization 2.6.9 Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:0414


Note You need to log in before you can comment on or make changes to this bug.