Bug 1598642 - Uninstallation of prometheus fails with error: unable to process template\n processedtemplates.template.openshift.io \"prometheus-node-exporter\" is forbidden
Summary: Uninstallation of prometheus fails with error: unable to process template\n p...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 3.9.z
Assignee: Paul Gier
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-07-06 05:33 UTC by Rutvik
Modified: 2021-12-10 16:34 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Installer attempted to deploy node_exporter during a prometheus uninstall. Consequence: Uninstall failed due to node_exporter not able to deploy to prometheus namespace. Fix: Add check to deploy node_exporter if prometheus is installed/present. Result: Prometheus uninstall completes successfully without attempting to install node_exporter into non-existent namespace.
Clone Of:
Environment:
Last Closed: 2018-12-13 19:26:59 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:3748 0 None None None 2018-12-13 19:27:10 UTC

Description Rutvik 2018-07-06 05:33:55 UTC
Description of problem:

Issue with https://access.redhat.com/documentation/en-us/openshift_container_platform/3.9/html/installation_and_configuration/install-config-cluster-metrics#openshift-prometheus-undeploy


Rerunning this uninstallation playbook goes past this issue and it uninstalls the Prometheus. It happens repeatedly.

~~~
TASK [openshift_prometheus : Apply the node exporter template file] **************************************************************************************************************************
Wednesday 27 June 2018  12:41:58 +0000 (0:00:02.410)       0:02:42.225 ******** 
fatal: [master0.xyz.com]: FAILED! => {"changed": true, "cmd": "oc process -f \"/tmp/prometheus-ansible-MtDs2l/node-exporter-template.yaml\" --param IMAGE=\"registry.access.redhat.com/openshift3/prometheus-node-exporter:v3.9.30\" --param MEMORY_REQUESTS=\"30Mi\" --param CPU_REQUESTS=\"100m\" --param MEMORY_LIMITS=\"50Mi\" --param CPU_LIMITS=\"200m\" --config=/tmp/prometheus-ansible-MtDs2l/admin.kubeconfig -n \"openshift-metrics\" | oc apply --config=/tmp/prometheus-ansible-MtDs2l/admin.kubeconfig -f - -n \"openshift-metrics\"", "delta": "0:00:00.907052", "end": "2018-06-27 12:42:00.736659", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2018-06-27 12:41:59.829607", "stderr": "error: unable to process template\n  processedtemplates.template.openshift.io \"prometheus-node-exporter\" is forbidden: unable to create new content in namespace openshift-metrics because it is being terminated.\nerror: no objects passed to apply", "stderr_lines": ["error: unable to process template", "  processedtemplates.template.openshift.io \"prometheus-node-exporter\" is forbidden: unable to create new content in namespace openshift-metrics because it is being terminated.", "error: no objects passed to apply"], "stdout": "", "stdout_lines": []}
~~~



Version-Release number of the following components:

openshift-ansible: 3.9.32-1
prometheus-node-exporter:v3.9.30

Comment 5 Paul Gier 2018-09-25 20:38:33 UTC
This is a bug, looks like it affects at least 3.9 and later.
PR for 3.10: https://github.com/openshift/openshift-ansible/pull/10225
I'll backport to 3.9 once it's merged to 3.10.

Comment 6 Junqi Zhao 2018-10-08 03:47:25 UTC
Tested with the fix, issue is fixed, but the current latest openshift-ansible version is openshift-ansible-3.9.45-1 which does not contain the fix, need new openshift-ansible package to verify, change it to MODIFIED

Comment 8 Paul Gier 2018-11-02 14:36:34 UTC
The fix should be available now as of 3.9.46

Comment 9 Junqi Zhao 2018-11-05 09:01:20 UTC
Tested with openshift-ansible-3.9.50-1, there is not prometheus uninstallation error.

Comment 12 errata-xmlrpc 2018-12-13 19:26:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3748


Note You need to log in before you can comment on or make changes to this bug.