Bug 1832839 - Work queue metrics from openshift-controller-manager missing
Summary: Work queue metrics from openshift-controller-manager missing
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: openshift-controller-manager
Version: 4.3.z
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.4.z
Assignee: Gabe Montero
QA Contact: wewang
Depends On: 1825324
TreeView+ depends on / blocked
Reported: 2020-05-07 11:04 UTC by OpenShift BugZilla Robot
Modified: 2020-06-29 15:34 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: controllers within the openshift-controller-manager did not use named work queues Consequence: useful metrics like workqueue_depth did not appear in Prometheus Fix: refactored controllers to use named work queues Result: workqueue_depth metrics appear in Prometheus
Clone Of:
Last Closed: 2020-06-29 15:33:54 UTC
Target Upstream Version:

Attachments (Terms of Use)
metrics for ocm (86.02 KB, image/png)
2020-05-18 03:05 UTC, wewang
no flags Details
metics of ocm (94.71 KB, image/png)
2020-06-22 03:05 UTC, wewang
no flags Details
sum of "image-signature-import" (85.60 KB, image/png)
2020-06-22 03:09 UTC, wewang
no flags Details
sum of "project-finalizer" (85.92 KB, image/png)
2020-06-22 03:09 UTC, wewang
no flags Details
sum(workqueue_adds_total) by (namespace,name) product (185.99 KB, image/png)
2020-06-22 03:20 UTC, wewang
no flags Details

System ID Private Priority Status Summary Last Updated
Github openshift openshift-controller-manager pull 104 0 None closed Bug 1832839: Use named queues for all controllers 2020-10-15 17:10:59 UTC
Github openshift openshift-controller-manager pull 117 0 None closed Bug 1832839: more changes required for named workqueue metrics on pre 1.18 k8s base 2020-10-15 17:10:59 UTC
Red Hat Product Errata RHBA-2020:2713 0 None None None 2020-06-29 15:34:11 UTC

Description OpenShift BugZilla Robot 2020-05-07 11:04:47 UTC
This is a clone of Bug #1825324. This is the description of that bug:
Created attachment 1679716 [details]
workqueue_depth metrics

Description of problem:

Work queue metrics for controllers within OCM are missing in Prometheus

Version-Release number of selected component (if applicable): 4.3.z

How reproducible: Always

Steps to Reproduce:
1. Open the prometheus UI from the web console
2. Query the workqueue_depth metric by job - ex `sum(workqueue_depth) by (job)`
3. View the results

Actual results:

No metrics appear from the "openshift-controller-manager" job

Expected results:

Metrics from the "openshift-controller-manager" job should appear, with work queue metrics for each named controller

Comment 4 wewang 2020-05-15 03:00:04 UTC
Now latest payload is not include the pr, keep waiting

Comment 5 wewang 2020-05-18 03:05:59 UTC
Created attachment 1689502 [details]
metrics for ocm

Comment 6 wewang 2020-05-18 03:10:57 UTC
@Adam, Checked the version: 4.4.0-0.nightly-2020-05-17-221856, which is include the pr104, but still cannot see metrics for ocm, attached the result.

Check the commits:
$ oc adm release info --commits  "registry.svc.ci.openshift.org/ocp/release:4.4.0-0.nightly-2020-05-17-221856" |grep openshift-controller-manager
  openshift-controller-manager                   https://github.com/openshift/openshift-controller-manager                   7d4142c8a468662c7a426cfc413fd29a010e876d

$ git log 7d4142c8a468662c7a426cfc413fd29a010e876d |grep "#104"
    Merge pull request #104 from openshift-cherrypick-robot/cherry-pick-95-to-release-4.4
    0b5d6c6 add firewall troubleshooting section (#1040)
    6745418 Bump OSB Client (#1049)

Comment 14 wewang 2020-06-22 03:05:49 UTC
Created attachment 1698246 [details]
metics of ocm

Comment 15 wewang 2020-06-22 03:09:00 UTC
Created attachment 1698247 [details]
sum of "image-signature-import"

Comment 16 wewang 2020-06-22 03:09:46 UTC
Created attachment 1698248 [details]
sum of "project-finalizer"

Comment 17 wewang 2020-06-22 03:11:00 UTC
Verified in version as follow version, and attached result.

Comment 18 wewang 2020-06-22 03:20:07 UTC
Created attachment 1698249 [details]
sum(workqueue_adds_total) by (namespace,name) product

Comment 20 errata-xmlrpc 2020-06-29 15:33:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.