Bug 2256459

Summary:	[release-4.13] Provide better logging for ocs-metrics-exporter
Product:	[Red Hat Storage] Red Hat OpenShift Data Foundation	Reporter:	umanga <uchapaga>
Component:	ceph-monitoring	Assignee:	umanga <uchapaga>
Status:	CLOSED ERRATA	QA Contact:	Filip Balák <fbalak>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	4.13	CC:	hnallurv, kramdoss, odf-bz-bot
Target Milestone:	---
Target Release:	ODF 4.13.7
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	4.13.7-1	Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:	2256456	Environment:
Last Closed:	2024-01-29 08:22:11 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2256456
Bug Blocks:	2256458

Description umanga 2024-01-02 12:46:33 UTC

+++ This bug was initially created as a clone of Bug #2256456 +++

Description of problem (please be detailed as possible and provide log
snippests):

ocs-metrics-exporter logs are not good enough to identify some of the internal errors and the log format is not structured.

Version of all relevant components (if applicable):


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
No

Is there any workaround available to the best of your knowledge?
No

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:
Not a regression.

Steps to Reproduce:
1.
2.
3.


Actual results: Logs do not have enough information about internal errors.


Expected results: Logs should have enough information about internal errors.


Additional info:

--- Additional comment from RHEL Program Management on 2024-01-02 18:12:18 IST ---

This bug having no release flag set previously, is now set with release flag 'odf‑4.15.0' to '?', and so is being proposed to be fixed at the ODF 4.15.0 release. Note that the 3 Acks (pm_ack, devel_ack, qa_ack), if any previously set while release flag was missing, have now been reset since the Acks are to be set against a release flag.

--- Additional comment from umanga on 2024-01-02 18:14:12 IST ---

Fixing this will help with debugging some issues which are extremely difficult to catch.
Providing devel_ack+ .

Comment 5 Filip Balák 2024-01-18 08:50:52 UTC

Logs are in json format with a clear timestamp (as requested by reproducer from https://bugzilla.redhat.com/show_bug.cgi?id=2256456#c7).

Example:
{"level":"info","ts":1705567519.9270978,"caller":"cache/pv.go:102","msg":"PV store addition started at 2024-01-18 08:45:19.927091598 +0000 UTC m=+970.298335101 for PV pvc-f0db85f1-8853-4fa2-ac6a-26f845191da0\n"}
{"level":"info","ts":1705567519.995269,"caller":"cache/pv.go:148","msg":"PV store addition completed at 2024-01-18 08:45:19.995244195 +0000 UTC m=+970.366487709\n"}
{"level":"info","ts":1705567519.9953122,"caller":"cache/pv.go:102","msg":"PV store addition started at 2024-01-18 08:45:19.995305666 +0000 UTC m=+970.366549169 for PV pvc-bb02d53c-4e84-4875-9f96-26de4bdc59a3\n"}
{"level":"info","ts":1705567520.0626886,"caller":"cache/pv.go:148","msg":"PV store addition completed at 2024-01-18 08:45:20.062666617 +0000 UTC m=+970.433910129\n"}
{"level":"info","ts":1705567520.0627282,"caller":"cache/pv.go:102","msg":"PV store addition started at 2024-01-18 08:45:20.062722448 +0000 UTC m=+970.433965952 for PV pvc-d7bfc4c7-2981-406e-82f4-c5c61d873f98\n"}
{"level":"info","ts":1705567520.0627403,"caller":"cache/pv.go:106","msg":"Skipping non Ceph CSI RBD volume pvc-d7bfc4c7-2981-406e-82f4-c5c61d873f98\n"}
{"level":"error","ts":1705567526.3450956,"caller":"collectors/ceph-block-pool.go:137","msg":"Invalid image health for pool ocs-storagecluster-cephblockpool. Must be OK, UNKNOWN, WARNING or ERROR\n","stacktrace":"github.com/red-hat-storage/ocs-operator/metrics/internal/collectors.(*CephBlockPoolCollector).collectMirroringImageHealth\n\t/remote-source/app/metrics/internal/collectors/ceph-block-pool.go:137\ngithub.com/red-hat-storage/ocs-operator/metrics/internal/collectors.(*CephBlockPoolCollector).Collect\n\t/remote-source/app/metrics/internal/collectors/ceph-block-pool.go:82\ngithub.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1\n\t/remote-source/app/vendor/github.com/prometheus/client_golang/prometheus/registry.go:456"}
{"level":"error","ts":1705567527.0112205,"caller":"collectors/ceph-block-pool.go:137","msg":"Invalid image health for pool ocs-storagecluster-cephblockpool. Must be OK, UNKNOWN, WARNING or ERROR\n","stacktrace":"github.com/red-hat-storage/ocs-operator/metrics/internal/collectors.(*CephBlockPoolCollector).collectMirroringImageHealth\n\t/remote-source/app/metrics/internal/collectors/ceph-block-pool.go:137\ngithub.com/red-hat-storage/ocs-operator/metrics/internal/collectors.(*CephBlockPoolCollector).Collect\n\t/remote-source/app/metrics/internal/collectors/ceph-block-pool.go:82\ngithub.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1\n\t/remote-source/app/vendor/github.com/prometheus/client_golang/prometheus/registry.go:456"}
{"level":"info","ts":1705567539.710896,"caller":"cache/rbd-mirror.go:296","msg":"RBD mirror store resync started at 2024-01-18 08:45:39.710849441 +0000 UTC m=+990.082093346\n"}
{"level":"info","ts":1705567539.7109604,"caller":"cache/rbd-mirror.go:321","msg":"RBD mirror store resync ended at 2024-01-18 08:45:39.710951387 +0000 UTC m=+990.082194893\n"}
I0118 08:45:45.833000 1 ceph-blocklist.go:105] Blocklist store sync started 2024-01-18 08:45:45.832990285 +0000 UTC m=+996.204233789

No metrics regression was found during testing.

--> VERIFIED

Tested with:
ocp 4.13.0-0.nightly-2024-01-17-220523
ocs 4.13.7-1

Comment 9 errata-xmlrpc 2024-01-29 08:22:11 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenShift Data Foundation 4.13.7 Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2024:0540