Bug 2039880

Summary: Log level too low for control plane metrics
Product: OpenShift Container Platform Reporter: Martin Kennelly <mkennell>
Component: NetworkingAssignee: Martin Kennelly <mkennell>
Networking sub component: ovn-kubernetes QA Contact: Anurag saxena <anusaxen>
Status: CLOSED ERRATA Docs Contact:
Severity: low    
Priority: medium CC: zzhao
Version: 4.10   
Target Milestone: ---   
Target Release: 4.10.0   
Hardware: All   
OS: All   
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-03-10 16:39:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Martin Kennelly 2022-01-12 16:22:05 UTC
Description of problem:
Increasing log level to decrease log noise.
Pods maybe created and deleted before port binding
events hit the libovsdb cache causing unwanted
log noise. Its fine to discard metrics that do not hit the libovsdb
cache in an order we expect.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Spawn 4.10 cluster
2. Within the logs, you may see log messages starting with "metrics: ", 

Actual results:
Log messages that do not add any value

Expected results:
Log messages hidden in normal operator and only enabled if log level increased.

Additional info:
Upstream PR: https://github.com/ovn-org/ovn-kubernetes/pull/2744/files

Comment 2 Martin Kennelly 2022-01-22 14:54:30 UTC
To verify this bug:
1) Spawn a cluster with ovn-kubernetes CNI plugin
2) View logs of all non-leader OVN-Kubernetes pods container 'ovnkube-master'. You should not see any logs that contain the following strings:
  * "Add Logical Switch Port event expected pod with UID"
  * 'Unexpected last event type'
  * 'Add port binding event expected pod with UID'
  * 'Cache entry expected pod with UID'

Comment 4 zhaozhanqi 2022-01-25 03:25:35 UTC
Verified this bug on 4.10.0-0.nightly-2022-01-24-020644

$ oc logs ovnkube-master-cfkzn -c ovnkube-master -n openshift-ovn-kubernetes | grep Metrics:

Comment 7 errata-xmlrpc 2022-03-10 16:39:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.