Bug 1598642

Summary:	Uninstallation of prometheus fails with error: unable to process template\n processedtemplates.template.openshift.io \"prometheus-node-exporter\" is forbidden
Product:	OpenShift Container Platform	Reporter:	Rutvik <rkshirsa>
Component:	Monitoring	Assignee:	Paul Gier <pgier>
Status:	CLOSED ERRATA	QA Contact:	Junqi Zhao <juzhao>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	3.9.0	CC:	aos-bugs, asathe, jokerman, minden, mmccomas, pgier, rkshirsa, suchaudh
Target Milestone:	---
Target Release:	3.9.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: Installer attempted to deploy node_exporter during a prometheus uninstall. Consequence: Uninstall failed due to node_exporter not able to deploy to prometheus namespace. Fix: Add check to deploy node_exporter if prometheus is installed/present. Result: Prometheus uninstall completes successfully without attempting to install node_exporter into non-existent namespace.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-12-13 19:26:59 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Rutvik 2018-07-06 05:33:55 UTC

Description of problem:

Issue with https://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html/installation_and_configuration/install-config-cluster-metrics#openshift-prometheus-undeploy


Rerunning this uninstallation playbook goes past this issue and it uninstalls the Prometheus. It happens repeatedly.

~~~
TASK [openshift_prometheus : Apply the node exporter template file] **************************************************************************************************************************
Wednesday 27 June 2018  12:41:58 +0000 (0:00:02.410)       0:02:42.225 ******** 
fatal: [master0.xyz.com]: FAILED! => {"changed": true, "cmd": "oc process -f \"/tmp/prometheus-ansible-MtDs2l/node-exporter-template.yaml\" --param IMAGE=\"registry.access.redhat.com/openshift3/prometheus-node-exporter:v3.9.30\" --param MEMORY_REQUESTS=\"30Mi\" --param CPU_REQUESTS=\"100m\" --param MEMORY_LIMITS=\"50Mi\" --param CPU_LIMITS=\"200m\" --config=/tmp/prometheus-ansible-MtDs2l/admin.kubeconfig -n \"openshift-metrics\" | oc apply --config=/tmp/prometheus-ansible-MtDs2l/admin.kubeconfig -f - -n \"openshift-metrics\"", "delta": "0:00:00.907052", "end": "2018-06-27 12:42:00.736659", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2018-06-27 12:41:59.829607", "stderr": "error: unable to process template\n  processedtemplates.template.openshift.io \"prometheus-node-exporter\" is forbidden: unable to create new content in namespace openshift-metrics because it is being terminated.\nerror: no objects passed to apply", "stderr_lines": ["error: unable to process template", "  processedtemplates.template.openshift.io \"prometheus-node-exporter\" is forbidden: unable to create new content in namespace openshift-metrics because it is being terminated.", "error: no objects passed to apply"], "stdout": "", "stdout_lines": []}
~~~



Version-Release number of the following components:

openshift-ansible: 3.9.32-1
prometheus-node-exporter:v3.9.30

Comment 5 Paul Gier 2018-09-25 20:38:33 UTC

This is a bug, looks like it affects at least 3.9 and later.
PR for 3.10: https://github.com/openshift/openshift-ansible/pull/10225
I'll backport to 3.9 once it's merged to 3.10.

Comment 6 Junqi Zhao 2018-10-08 03:47:25 UTC

Tested with the fix, issue is fixed, but the current latest openshift-ansible version is openshift-ansible-3.9.45-1 which does not contain the fix, need new openshift-ansible package to verify, change it to MODIFIED

Comment 8 Paul Gier 2018-11-02 14:36:34 UTC

The fix should be available now as of 3.9.46

Comment 9 Junqi Zhao 2018-11-05 09:01:20 UTC

Tested with openshift-ansible-3.9.50-1, there is not prometheus uninstallation error.

Comment 12 errata-xmlrpc 2018-12-13 19:26:59 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3748