Bug 2035190

Summary:	The CNV specific labels are not parsed as expected and are all saeved as "kuberbetes.flat_labels"
Product:	Container Native Virtualization (CNV)	Reporter:	Shirly Radco <sradco>
Component:	Logging	Assignee:	Shirly Radco <sradco>
Status:	CLOSED CURRENTRELEASE	QA Contact:	SATHEESARAN <sasundar>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	4.10.0	CC:	dbasunag, dholler, kmajcher, stirabos
Target Milestone:	---	Keywords:	TestOnly
Target Release:	4.10.7
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-08-08 10:27:11 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Shirly Radco 2021-12-23 08:44:06 UTC

Description of problem:
The CNV specific labels are not parsed as expected and are all saeved as "kuberbetes.flat_labels".

We may need to have the logs in Json format in the logs, based on https://issues.redhat.com/browse/LOG-785

Version-Release number of selected component (if applicable):
4.10

How reproducible:
100%

Steps to Reproduce:
1.Install OCP+CNV+OpenShift Logging
2.Go Kibana -> Discover tab 
3.Filter the logs by the openshift-cnv namespace
4. Check the labels

For exapmple:
name=hyperconverged-cluster-operator,
pod-template-hash=56585bf977,
app_kubernetes_io/component=deployment,
app_kubernetes_io/managed-by=olm,
app_kubernetes_io/part-of=hyperconverged-cluster, app_kubernetes_io/version=4.10.0

Actual results:
All labels are under the "kuberbetes.flat_labels" field.

Expected results:
Each label should be in a separate field so that we can query it separately.

Additional info:

Comment 1 Shirly Radco 2022-02-14 15:18:02 UTC

This is caused due to a Fluentd intentional specification to flatted pod labels.
I asked the logging team to enable specific pod labels that will allow to filter operators logs more easily.
I opened an issue for them to get this fixed.

Comment 4 Krzysztof Majcher 2022-05-10 12:53:16 UTC

It seems that underlying bug is closed - please doublecheck if we an close this one.

Comment 9 Krzysztof Majcher 2022-10-04 12:45:38 UTC

it seems that the underlying bug has been fixed and is available (please check last comment on OBSDA-150) and recommend next steps.

Comment 10 Krzysztof Majcher 2022-10-19 09:36:34 UTC

This can be tested with updated logging operator.

Comment 11 SATHEESARAN 2022-12-13 09:10:07 UTC

Tested with CNV v4.10.7 ( hco-bundle-4.10.7-24 )

Verified that kubernetes.flat_labels doesn't contain all the labels.
Each label is listed as a separate field.

Example:

 kubernetes.flat_labels 	
	app.kubernetes.io/component=compute, app.kubernetes.io/managed-by=virt-operator, app.kubernetes.io/part-of=hyperconverged-cluster, app.kubernetes.io/version=4.10.7, kubevirt.io=virt-api, pod-template-hash=86776f9cc, prometheus.kubevirt.io=true
t kubernetes.host 	
	c01-ss-410a-f9wrk-worker-0-zzmrg
t kubernetes.labels.app.kubernetes.io/component 	
	compute
t kubernetes.labels.app.kubernetes.io/managed-by 	
	virt-operator
t kubernetes.labels.app.kubernetes.io/part-of 	
	hyperconverged-cluster
t kubernetes.labels.app.kubernetes.io/version 	
	4.10.7