Bug 1889111

Summary:	Prometheus and alertmanager pods won't start on specific node - in Terminated state
Product:	OpenShift Container Platform	Reporter:	Vladislav Walek <vwalek>
Component:	Monitoring	Assignee:	Simon Pasquier <spasquie>
Status:	CLOSED DUPLICATE	QA Contact:	Junqi Zhao <juzhao>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	4.5	CC:	alegrand, anpicker, aos-bugs, erooth, jokerman, kakkoyun, lcosic, mloibl, nnosenzo, pkrupa, spasquie, surbania
Target Milestone:	---
Target Release:	4.7.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-10-21 14:02:27 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Vladislav Walek 2020-10-17 22:11:38 UTC

Description of problem:

- The alertmanager-main-2 and prometheus-k8s-1 pods are not able to start, stalling in Terminated state. 
- The other pods from statefulset are running normally - without any error.
- both problematic pods are started on the same infra node
- other pods are running normally (non monitoring ones)

alertmanager-main-2                            0/5     Terminating   0          1s
prometheus-k8s-1                               0/7     Terminating   0          2s

After checking logs from kubelet and crio - I see that the containers are constantly brought down and up in endless loop without removing the status.

Scaling down and up the sts won't help.

Restart of the kubelet won't solve the issue.

Version-Release number of selected component (if applicable):
OpenShift Container Platform 4.5


How reproducible:
#n/a


Additional info:
will provide data in additional comment

Comment 5 Nicolas Nosenzo 2020-10-19 08:51:01 UTC

The prometheus-operator is erroring with the following message:

"""
2020-10-16T15:33:42.713855516+00:00 stderr F E1016 15:33:42.713828       1 operator.go:996] Sync "openshift-monitoring/k8s" failed: failed to create new ConfigMap 'prometheus-k8s-rulefiles-0': configmaps "prometheus-k8s-rulefiles-0" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>

Then, it repeatedly posts:

"""
2020-10-17T18:22:33.440002053Z E1017 18:22:33.439961       1 operator.go:996] Sync "openshift-monitoring/k8s" failed: failed to create new ConfigMap 'prometheus-k8s-rulefiles-0': configmaps is forbidden: User "system:serviceaccount:openshift-monitoring:prometheus-operator" cannot create resource "configmaps" in API group "" in the namespace "openshift-monitoring"
2020-10-17T18:22:46.314927734Z E1017 18:22:46.314847       1 operator.go:996] Sync "openshift-monitoring/k8s" failed: failed to create ConfigMap 'prometheus-k8s-rulefiles-0': configmaps "prometheus-k8s-rulefiles-0" already exists
"""

Comment 6 Ryan Phillips 2020-10-19 19:32:30 UTC

This looks like a monitoring operator RBAC permissions issue.

Moving over...

Comment 10 Simon Pasquier 2020-10-21 14:02:27 UTC

After discussing off-line with @surbania we concluded that if might be related to bug 1863011 (bug 1887354 for 4.5.z) so closing this one as a duplicate since the resolution is already on-going.

*** This bug has been marked as a duplicate of bug 1887354 ***