Bug 1967526

Summary:	Excessive logging of KMP
Product:	Container Native Virtualization (CNV)	Reporter:	Petr Horáček <phoracek>
Component:	Networking	Assignee:	Ram Lavi <ralavi>
Status:	CLOSED ERRATA	QA Contact:	Meni Yakove <myakove>
Severity:	high	Docs Contact:
Priority:	high
Version:	4.8.0	CC:	awax, cnv-qe-bugs, danken
Target Milestone:	---
Target Release:	4.8.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	cluster-network-addons-operator-container-v4.8.0-19	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-07-27 14:32:39 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Petr Horáček 2021-06-03 10:09:19 UTC

Description of problem:
KubeMacPool is logging cache of all allocated MAC addresses per each operation on a VM into logs. On a cluster with many VMs, this may mean millions of MACs logged every couple of days and exhaustion of available Node resources. Since KMP will be enabled by default, this issue is urgent.


Version-Release number of selected component (if applicable):
CNV 4.8


How reproducible:
Always


Steps to Reproduce:
1. Create multiple VMs
2. Check logs of the KubeMacPool Deploymetn

Actual results:
Each operation on a VM is accompanied by a dump of all allocated MACs. 


Expected results:
Allocated MACs should be logged only in debug mode.

Comment 1 Ram Lavi 2021-06-03 11:45:01 UTC

https://github.com/k8snetworkplumbingwg/kubemacpool/pull/310

Comment 4 awax 2021-07-07 12:51:06 UTC

Version-Release number of selected component:
CNV: v.4.8.0
Server Version: v.4.8.0-fc.7
Kubernetes Version: v1.21.0-rc.0+4b2b6ff
CNAO: v4.8.0-24
KMP: v4.8.0-19


Steps to reproduce:
1. create a namespace.
2. create 4 VMs (vm-1 to vm-4) using the attached files (kmp_log_vm_1.yaml, kmp_log_vm_2.yaml, kmp_log_vm_3.yaml, kmp_log_vm_4.yaml).
3. start all VMs (virtctl start vm-1 and so on)
4. make sure all VM's are running:
$ oc get vmi
5. delete all VMs (oc delete command)
6. look at the logs produced:
$ oc -n openshift-cnv get pods | grep mac
$ oc -n openshift-cnv logs kubemacpool-mac-controller-manager-796bf7557c-4p4pm

Repeat all steps in debug mode.
How to set kubemacpool-mac-controller-manager deployment to debug mode:
1. shut down HCO (hyperconverged-cluster-operator) operator (change replicas of deployment to 0)
$ oc scale deployment.apps/hco-operator -n openshift-cnv --replicas=0                                                                                                              
2. shut down CNAO operator (change replicas of deployment to 0)
Shut-down the cluster network operator:
$ oc scale deployment.apps/cluster-network-addons-operator --replicas=0 -n openshift-cnv
3. change KMP deployment param to -v debug
$ oc edit deployment -n openshift-cnv kubemacpool-mac-controller-manager
(under: spec -> template -> spec -> containers -> to "v=debug")

Comment 9 errata-xmlrpc 2021-07-27 14:32:39 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Virtualization 4.8.0 Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2920