Bug 1806541 - namespace \“openshift-user-workload-monitoring\” does not exist”
Summary: namespace \“openshift-user-workload-monitoring\” does not exist”
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
medium
low
Target Milestone: ---
: 4.6.0
Assignee: Pawel Krupa
QA Contact: Junqi Zhao
URL:
Whiteboard: groom
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-02-24 13:42 UTC by Lili Cosic
Modified: 2020-10-27 15:56 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-27 15:55:19 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift prometheus-operator pull 77 0 None closed Bug 1806541: Update prometheus-operator to 0.40.0 2021-01-26 07:05:36 UTC
Github openshift prometheus-operator pull 80 0 None closed Bug 1806541: Revert of Bug 1853007: Revert of revert: Bump prometheus-operator to v0.40.0 2021-01-26 07:05:36 UTC
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 15:56:00 UTC

Description Lili Cosic 2020-02-24 13:42:49 UTC
Description of problem:

In the logs of user-workload-monitoring on our long lived cluster, there were multiple logs saying the following:

level=error ts=2020-02-19T14:04:28.296931247Z caller=operator.go:894 component=prometheusoperator msg=“get namespace to enqueue Prometheus instances failed: namespace \“openshift-user-workload-monitoring\” does not exist” err=null
level=error ts=2020-02-19T14:04:28.36728741Z caller=operator.go:894 component=prometheusoperator msg=“get namespace to enqueue Prometheus instances failed: namespace \“openshift-user-workload-monitoring\” does not exist” err=null
Version-Release number of selected component (if applicable):


How reproducible:
On our long lived cluster after 4.3.1 upgrade always there.

Comment 2 Junqi Zhao 2020-04-24 04:47:29 UTC
4.5.0-0.nightly-2020-04-23-202137 cluster
# oc -n openshift-user-workload-monitoring logs prometheus-operator-765866997c-6fn65 -c prometheus-operator | grep "does not exist"
level=error ts=2020-04-24T03:27:21.959896725Z caller=operator.go:919 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"openshift-user-workload-monitoring\" does not exist"
level=error ts=2020-04-24T03:27:21.959962346Z caller=operator.go:919 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"openshift-user-workload-monitoring\" does not exist"
level=error ts=2020-04-24T03:27:21.959983855Z caller=operator.go:919 component=prometheusoperator msg="get namespace to enqueue Prometheus instances failed: namespace \"openshift-user-workload-monitoring\" does not exist"
...

Comment 3 Pawel Krupa 2020-05-07 12:17:43 UTC
Status: deffered due to low severity and lack of capacity

Comment 10 Junqi Zhao 2020-07-08 03:20:52 UTC
Tested with 4.6.0-0.nightly-2020-07-07-233934, issue is fixed
# oc -n openshift-user-workload-monitoring logs prometheus-operator-6ccb488d45-7wzgh -c prometheus-operator
ts=2020-07-08T02:41:23.343294809Z caller=main.go:217 msg="Starting Prometheus Operator version '0.40.0'."
ts=2020-07-08T02:41:23.360094744Z caller=main.go:104 msg="Starting insecure server on [::]:8080"
level=info ts=2020-07-08T02:41:23.38488072Z caller=operator.go:461 component=prometheusoperator msg="connection established" cluster-version=v1.18.3+a377312
level=info ts=2020-07-08T02:41:23.385116496Z caller=operator.go:470 component=prometheusoperator msg="CRD API endpoints ready"
level=info ts=2020-07-08T02:41:23.385692962Z caller=operator.go:295 component=thanosoperator msg="connection established" cluster-version=v1.18.3+a377312
level=info ts=2020-07-08T02:41:23.385886291Z caller=operator.go:304 component=thanosoperator msg="CRD API endpoints ready"
level=info ts=2020-07-08T02:41:23.392921891Z caller=operator.go:200 component=alertmanageroperator msg="connection established" cluster-version=v1.18.3+a377312
level=info ts=2020-07-08T02:41:23.400155499Z caller=operator.go:209 component=alertmanageroperator msg="CRD API endpoints ready"
level=info ts=2020-07-08T02:41:23.585536295Z caller=operator.go:407 component=prometheusoperator msg="successfully synced all caches"
level=info ts=2020-07-08T02:41:23.686409677Z caller=operator.go:256 component=thanosoperator msg="successfully synced all caches"
level=info ts=2020-07-08T02:41:23.800366387Z caller=operator.go:171 component=alertmanageroperator msg="successfully synced all caches"
level=info ts=2020-07-08T02:41:34.305856904Z caller=operator.go:1122 component=prometheusoperator msg="sync prometheus" key=openshift-user-workload-monitoring/user-workload
level=info ts=2020-07-08T02:41:34.462307229Z caller=operator.go:1122 component=prometheusoperator msg="sync prometheus" key=openshift-user-workload-monitoring/user-workload
level=info ts=2020-07-08T02:41:34.524649131Z caller=operator.go:1122 component=prometheusoperator msg="sync prometheus" key=openshift-user-workload-monitoring/user-workload
level=info ts=2020-07-08T02:41:34.610527058Z caller=operator.go:1122 component=prometheusoperator msg="sync prometheus" key=openshift-user-workload-monitoring/user-workload
level=info ts=2020-07-08T02:41:34.679739224Z caller=operator.go:1122 component=prometheusoperator msg="sync prometheus" key=openshift-user-workload-monitoring/user-workload
level=info ts=2020-07-08T02:41:34.739046224Z caller=operator.go:1122 component=prometheusoperator msg="sync prometheus" key=openshift-user-workload-monitoring/user-workload
level=info ts=2020-07-08T02:41:35.702815334Z caller=operator.go:586 component=thanosoperator msg="sync thanos-ruler" key=openshift-user-workload-monitoring/user-workload
level=info ts=2020-07-08T02:41:35.763196908Z caller=operator.go:586 component=thanosoperator msg="sync thanos-ruler" key=openshift-user-workload-monitoring/user-workload
level=info ts=2020-07-08T02:41:35.805760139Z caller=operator.go:586 component=thanosoperator msg="sync thanos-ruler" key=openshift-user-workload-monitoring/user-workload
level=info ts=2020-07-08T02:41:35.923138389Z caller=operator.go:586 component=thanosoperator msg="sync thanos-ruler" key=openshift-user-workload-monitoring/user-workload
level=info ts=2020-07-08T02:41:39.005836359Z caller=operator.go:586 component=thanosoperator msg="sync thanos-ruler" key=openshift-user-workload-monitoring/user-workload
level=info ts=2020-07-08T02:41:40.328794036Z caller=operator.go:1122 component=prometheusoperator msg="sync prometheus" key=openshift-user-workload-monitoring/user-workload
level=info ts=2020-07-08T02:41:41.780577398Z caller=operator.go:1122 component=prometheusoperator msg="sync prometheus" key=openshift-user-workload-monitoring/user-workload
level=info ts=2020-07-08T02:41:43.649536464Z caller=operator.go:586 component=thanosoperator msg="sync thanos-ruler" key=openshift-user-workload-monitoring/user-workload

Comment 11 Simon Pasquier 2020-07-09 14:58:16 UTC
Setting back to ASSIGNED since we've reverted the upgrade to v0.40.0 for now.

Comment 12 Pawel Krupa 2020-07-10 08:29:56 UTC
Fix had to be reverted due to unexpected problems.

Comment 14 Junqi Zhao 2020-07-21 02:10:44 UTC
issue is fixed with 4.6.0-0.nightly-2020-07-20-183524
# oc -n openshift-user-workload-monitoring logs prometheus-operator-db5688d96-tp2wh -c prometheus-operator
ts=2020-07-21T01:42:54.961571165Z caller=main.go:217 msg="Starting Prometheus Operator version '0.40.0'."
ts=2020-07-21T01:42:54.975906234Z caller=main.go:104 msg="Starting insecure server on [::]:8080"
level=info ts=2020-07-21T01:42:54.983990034Z caller=operator.go:295 component=thanosoperator msg="connection established" cluster-version=v4.6.0-202007171513.p0-dirty
level=info ts=2020-07-21T01:42:54.98402945Z caller=operator.go:304 component=thanosoperator msg="CRD API endpoints ready"
level=info ts=2020-07-21T01:42:54.98408666Z caller=operator.go:461 component=prometheusoperator msg="connection established" cluster-version=v4.6.0-202007171513.p0-dirty
level=info ts=2020-07-21T01:42:54.984104553Z caller=operator.go:470 component=prometheusoperator msg="CRD API endpoints ready"
level=info ts=2020-07-21T01:42:54.984012764Z caller=operator.go:200 component=alertmanageroperator msg="connection established" cluster-version=v4.6.0-202007171513.p0-dirty
level=info ts=2020-07-21T01:42:55.023762606Z caller=operator.go:209 component=alertmanageroperator msg="CRD API endpoints ready"
level=info ts=2020-07-21T01:42:55.084337088Z caller=operator.go:407 component=prometheusoperator msg="successfully synced all caches"
level=info ts=2020-07-21T01:42:55.123942409Z caller=operator.go:171 component=alertmanageroperator msg="successfully synced all caches"
level=info ts=2020-07-21T01:42:55.18603845Z caller=operator.go:256 component=thanosoperator msg="successfully synced all caches"
level=info ts=2020-07-21T01:43:09.373939535Z caller=operator.go:1122 component=prometheusoperator msg="sync prometheus" key=openshift-user-workload-monitoring/user-workload
level=info ts=2020-07-21T01:43:09.529719636Z caller=operator.go:1122 component=prometheusoperator msg="sync prometheus" key=openshift-user-workload-monitoring/user-workload
level=info ts=2020-07-21T01:43:09.586902305Z caller=operator.go:1122 component=prometheusoperator msg="sync prometheus" key=openshift-user-workload-monitoring/user-workload
level=info ts=2020-07-21T01:43:09.711290073Z caller=operator.go:1122 component=prometheusoperator msg="sync prometheus" key=openshift-user-workload-monitoring/user-workload
level=info ts=2020-07-21T01:43:12.34232516Z caller=operator.go:1122 component=prometheusoperator msg="sync prometheus" key=openshift-user-workload-monitoring/user-workload
level=info ts=2020-07-21T01:43:12.761776246Z caller=operator.go:586 component=thanosoperator msg="sync thanos-ruler" key=openshift-user-workload-monitoring/user-workload
level=info ts=2020-07-21T01:43:12.814861808Z caller=operator.go:586 component=thanosoperator msg="sync thanos-ruler" key=openshift-user-workload-monitoring/user-workload
level=info ts=2020-07-21T01:43:12.843370879Z caller=operator.go:586 component=thanosoperator msg="sync thanos-ruler" key=openshift-user-workload-monitoring/user-workload
level=info ts=2020-07-21T01:43:12.862610251Z caller=operator.go:586 component=thanosoperator msg="sync thanos-ruler" key=openshift-user-workload-monitoring/user-workload
level=info ts=2020-07-21T01:43:12.931709853Z caller=operator.go:586 component=thanosoperator msg="sync thanos-ruler" key=openshift-user-workload-monitoring/user-workload
level=info ts=2020-07-21T01:43:16.219672652Z caller=operator.go:1122 component=prometheusoperator msg="sync prometheus" key=openshift-user-workload-monitoring/user-workload
level=info ts=2020-07-21T01:43:16.223644785Z caller=operator.go:586 component=thanosoperator msg="sync thanos-ruler" key=openshift-user-workload-monitoring/user-workload
level=info ts=2020-07-21T01:43:16.585596726Z caller=operator.go:1122 component=prometheusoperator msg="sync prometheus" key=openshift-user-workload-monitoring/user-workload
level=info ts=2020-07-21T01:43:16.897841642Z caller=operator.go:586 component=thanosoperator msg="sync thanos-ruler" key=openshift-user-workload-monitoring/user-workload

Comment 16 errata-xmlrpc 2020-10-27 15:55:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196


Note You need to log in before you can comment on or make changes to this bug.