Created attachment 1715072 [details] prometheus-operator container logs Created attachment 1715072 [details] prometheus-operator container logs Description of problem: # oc -n openshift-monitoring logs $(oc -n openshift-monitoring get po | grep prometheus-operator | awk '{print $1}') -c prometheus-operator | grep "openshift-user-workload-monitoring" level=error ts=2020-09-16T05:24:25.618881767Z caller=operator.go:952 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"openshift-user-workload-monitoring\" does not exist" level=error ts=2020-09-16T07:01:04.049813581Z caller=operator.go:952 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"openshift-user-workload-monitoring\" does not exist" level=error ts=2020-09-16T09:13:41.562393113Z caller=operator.go:952 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"openshift-user-workload-monitoring\" does not exist" level=error ts=2020-09-16T09:21:52.139920279Z caller=operator.go:952 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"openshift-user-workload-monitoring\" does not exist" Version-Release number of selected component (if applicable): 4.6.0-0.nightly-2020-09-15-171211 Prometheus Operator version '0.40.0' How reproducible: always Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: container logs see the attached file
Namespace should be created by CVO was it? I am trying to figure out if this is on monitoring side or on CVO.
(In reply to Pawel Krupa from comment #1) > Namespace should be created by CVO was it? I am trying to figure out if this > is on monitoring side or on CVO. should be handled by prometheus-operator, since we have bug 1806541 which happened in 4.3, and was fixed in previous 4.6 build, but it happen again
@junqi please retest :)
tested with 4.6.0-0.nightly-2020-09-29-170625 # oc -n openshift-monitoring logs $(oc -n openshift-monitoring get po | grep prometheus-operator | awk '{print $1}') -c prometheus-operator | grep "openshift-user-workload-monitoring" | tail -n 5 level=error ts=2020-09-30T08:13:38.48645366Z caller=operator.go:939 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"openshift-user-workload-monitoring\" does not exist" level=error ts=2020-09-30T08:27:56.503909871Z caller=operator.go:939 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"openshift-user-workload-monitoring\" does not exist" level=error ts=2020-09-30T08:28:39.318274231Z caller=operator.go:939 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"openshift-user-workload-monitoring\" does not exist" level=error ts=2020-09-30T08:38:27.355508961Z caller=operator.go:939 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"openshift-user-workload-monitoring\" does not exist" level=error ts=2020-09-30T08:44:53.707142526Z caller=operator.go:939 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"openshift-user-workload-monitoring\" does not exist"
No capacity to work on this currently.
UpcomingSprint: We don't have enough capacity to tackle this one in the next sprint (193).
still reproduced with 4.7.0-0.nightly-2020-12-13-202314 # oc -n openshift-monitoring logs $(oc -n openshift-monitoring get po | grep prometheus-operator | awk '{print $1}') -c prometheus-operator | grep "openshift-user-workload-monitoring" level=error ts=2020-12-14T03:35:14.697304959Z caller=operator.go:931 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"openshift-user-workload-monitoring\" does not exist"
still reproduced with 4.8.0-0.nightly-2021-02-24-063313 # oc -n openshift-monitoring logs $(oc -n openshift-monitoring get po | grep prometheus-operator | awk '{print $1}') -c prometheus-operator | grep "openshift-user-workload-monitoring" level=error ts=2021-02-25T01:55:03.516352066Z caller=operator.go:929 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"openshift-user-workload-monitoring\" does not exist"
Is there any update on this, because it still doesn't work while it has been out of preview since 4.6 and we want to use it. Does this mean we have to create our own instances of Prometheus for application logging? Kind regards, Jacob
@Jacob can you detail what's not working for you exactly? So far, the assumption has been that user workload monitoring is functional despite the log messages.
Hi Somin, Sorry. Looks like I made a mistake quoting the true value in the cluster-monitoring-config :-(. I made the change but didn't see anything happening in the prometheus-user-worload-monitoring namespace and thought it was related to the issue. Now I removed the quotes and it works, regardless of the message. Thnx for your quick response. I see why this has no importance ;-)
# oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.8.0-0.nightly-2021-04-18-101412 True False 3h3m Cluster version is 4.8.0-0.nightly-2021-04-18-101412 fresh cluster, disabled user workload by default # oc -n openshift-monitoring logs $(oc -n openshift-monitoring get po | grep prometheus-operator | awk '{print $1}') -c prometheus-operator | grep "openshift-user-workload-monitoring" no result enabled user workload # oc -n openshift-monitoring get cm cluster-monitoring-config -oyaml apiVersion: v1 data: config.yaml: | enableUserWorkload: true kind: ConfigMap metadata: creationTimestamp: "2021-04-19T02:54:10Z" name: cluster-monitoring-config namespace: openshift-monitoring resourceVersion: "98112" uid: e8513823-e7fa-42de-9e88-2c042671287e # oc -n openshift-monitoring logs $(oc -n openshift-monitoring get po | grep prometheus-operator | awk '{print $1}') -c prometheus-operator | grep "openshift-user-workload-monitoring" level=error ts=2021-04-19T02:54:13.99840747Z caller=operator.go:929 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"openshift-user-workload-monitoring\" does not exist"
Thanks Junqi, very helpful!
*** Bug 1958016 has been marked as a duplicate of this bug. ***
enable or disable UWM, there is not "openshift-user-workload-monitoring\ does not exist" error with # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.8.0-0.nightly-2021-06-07-180258 True False 116m Cluster version is 4.8.0-0.nightly-2021-06-07-180258 # oc -n openshift-monitoring logs $(oc -n openshift-monitoring get po | grep prometheus-operator | awk '{print $1}') -c prometheus-operator | grep "openshift-user-workload-monitoring" no result prometheus-operator version now is 0.48.1 # oc -n openshift-monitoring get po prometheus-operator-5cfbff779b-w88xv --show-labels NAME READY STATUS RESTARTS AGE LABELS prometheus-operator-5cfbff779b-w88xv 2/2 Running 1 149m app.kubernetes.io/component=controller,app.kubernetes.io/name=prometheus-operator,app.kubernetes.io/part-of=openshift-monitoring,app.kubernetes.io/version=0.48.1,pod-template-hash=5cfbff779b
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438