Bug 2041570

Summary:	Lost all collected metrics data when both Prometheus pods restarted
Product:	OpenShift Container Platform	Reporter:	Denys Shchedrivyi <dshchedr>
Component:	Monitoring	Assignee:	Brad Ison <brad.ison>
Status:	CLOSED NOTABUG	QA Contact:	Junqi Zhao <juzhao>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	4.10	CC:	amuller, anpicker, aos-bugs, brad.ison, cnv-qe-bugs, erooth, stirabos
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-01-18 10:37:50 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Denys Shchedrivyi 2022-01-17 17:56:24 UTC

Description of problem:
 After draining nodes both pods (prometheus-k8s-0 and prometheus-k8s-1) restarted and all data that was previously collected by metrics removed from a cluster


Version-Release number of selected component (if applicable):
CNV-4.10

How reproducible:
100%

Steps to Reproduce:
1. drain first node where pod prometheus-k8s-0 is running
2. return node back and drain second node where prometheus-k8s-1 is running
3. check metrics on a cluster - old data removed

Actual results:
 Metrics cleared when both pods prometheus-k8s-0 and prometheus-k8s-1 restarted

Expected results:
 Old data should be kept 

Additional info:
 I guess some persistent storage should be using for keeping all metrics:
https://docs.openshift.com/container-platform/4.9/monitoring/configuring-the-monitoring-stack.html#configuring-persistent-storage

Comment 1 Brad Ison 2022-01-18 10:37:50 UTC

If you haven't configured persistent storage, this is expected. Following the procedure in the docs you linked is a prerequisite to having the data persist.