Bug 1825324 - Work queue metrics from openshift-controller-manager missing
Summary: Work queue metrics from openshift-controller-manager missing
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: openshift-controller-manager
Version: 4.3.z
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ---
: 4.5.0
Assignee: Adam Kaplan
QA Contact: wewang
URL:
Whiteboard:
Depends On:
Blocks: 1832839
TreeView+ depends on / blocked
 
Reported: 2020-04-17 17:30 UTC by Adam Kaplan
Modified: 2020-07-13 17:28 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: controllers within the openshift-controller-manager did not use named work queues Consequence: useful metrics like workqueue_depth did not appear in Prometheus Fix: refactored controllers to use named work queues Result: workqueue_depth metrics appear in Prometheus
Clone Of:
Environment:
Last Closed: 2020-07-13 17:28:35 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
workqueue_depth metrics (52.62 KB, image/png)
2020-04-17 17:30 UTC, Adam Kaplan
no flags Details
metrics for ocm (48.38 KB, image/png)
2020-04-30 08:23 UTC, wewang
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift openshift-controller-manager pull 95 0 None closed Bug 1825324: Use named queues for all controllers 2020-08-20 09:53:08 UTC
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:28:58 UTC

Description Adam Kaplan 2020-04-17 17:30:49 UTC
Created attachment 1679716 [details]
workqueue_depth metrics

Description of problem:

Work queue metrics for controllers within OCM are missing in Prometheus


Version-Release number of selected component (if applicable): 4.3.z


How reproducible: Always


Steps to Reproduce:
1. Open the prometheus UI from the web console
2. Query the workqueue_depth metric by job - ex `sum(workqueue_depth) by (job)`
3. View the results

Actual results:

No metrics appear from the "openshift-controller-manager" job

Expected results:

Metrics from the "openshift-controller-manager" job should appear, with work queue metrics for each named controller

Comment 1 Oleg Bulatov 2020-04-28 00:23:11 UTC
The queue to check for regression: ImageStreamController. We have it in 3.11, but it disappeared in 4.x

Comment 4 wewang 2020-04-30 08:23:00 UTC
Created attachment 1683219 [details]
metrics for ocm

Comment 5 wewang 2020-04-30 08:23:40 UTC
Verified in version:
4.5.0-0.nightly-2020-04-30-042256

Comment 6 Steve Kuznetsov 2020-05-05 18:33:37 UTC
Bumping to urgent, we need this backported at least to 4.4 before we can migrate again.

Comment 7 errata-xmlrpc 2020-07-13 17:28:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.