Bug 1832839

Summary: Work queue metrics from openshift-controller-manager missing
Product: OpenShift Container Platform Reporter: OpenShift BugZilla Robot <openshift-bugzilla-robot>
Component: openshift-controller-managerAssignee: Gabe Montero <gmontero>
Status: CLOSED ERRATA QA Contact: wewang <wewang>
Severity: high Docs Contact:
Priority: urgent    
Version: 4.3.zCC: aaleman, adam.kaplan, aos-bugs, mfojtik, obulatov, skuznets
Target Milestone: ---   
Target Release: 4.4.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: controllers within the openshift-controller-manager did not use named work queues Consequence: useful metrics like workqueue_depth did not appear in Prometheus Fix: refactored controllers to use named work queues Result: workqueue_depth metrics appear in Prometheus
 Story Points: ---
Clone Of: Environment:
Last Closed: 2020-06-29 15:33:54 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1825324    
Bug Blocks:    
Attachments:
Description Flags
metrics for ocm
none
metics of ocm
none
sum of "image-signature-import"
none
sum of "project-finalizer"
none
sum(workqueue_adds_total) by (namespace,name) product none

Description OpenShift BugZilla Robot 2020-05-07 11:04:47 UTC 
This is a clone of Bug #1825324. This is the description of that bug:
Created attachment 1679716 [details]
workqueue_depth metrics

Description of problem:

Work queue metrics for controllers within OCM are missing in Prometheus


Version-Release number of selected component (if applicable): 4.3.z


How reproducible: Always


Steps to Reproduce:
1. Open the prometheus UI from the web console
2. Query the workqueue_depth metric by job - ex `sum(workqueue_depth) by (job)`
3. View the results

Actual results:

No metrics appear from the "openshift-controller-manager" job

Expected results:

Metrics from the "openshift-controller-manager" job should appear, with work queue metrics for each named controller
Comment 4 wewang 2020-05-15 03:00:04 UTC 
Now latest payload is not include the pr, keep waiting
Comment 5 wewang 2020-05-18 03:05:59 UTC 
Created attachment 1689502 [details]
metrics for ocm
Comment 6 wewang 2020-05-18 03:10:57 UTC 
@Adam, Checked the version: 4.4.0-0.nightly-2020-05-17-221856, which is include the pr104, but still cannot see metrics for ocm, attached the result.


Check the commits:
$ oc adm release info --commits  "registry.svc.ci.openshift.org/ocp/release:4.4.0-0.nightly-2020-05-17-221856" |grep openshift-controller-manager
  openshift-controller-manager                   https://github.com/openshift/openshift-controller-manager                   7d4142c8a468662c7a426cfc413fd29a010e876d

$ git log 7d4142c8a468662c7a426cfc413fd29a010e876d |grep "#104"
    Merge pull request #104 from openshift-cherrypick-robot/cherry-pick-95-to-release-4.4
    0b5d6c6 add firewall troubleshooting section (#1040)
    6745418 Bump OSB Client (#1049)
Comment 14 wewang 2020-06-22 03:05:49 UTC 
Created attachment 1698246 [details]
metics of ocm
Comment 15 wewang 2020-06-22 03:09:00 UTC 
Created attachment 1698247 [details]
sum of "image-signature-import"
Comment 16 wewang 2020-06-22 03:09:46 UTC 
Created attachment 1698248 [details]
sum of "project-finalizer"
Comment 17 wewang 2020-06-22 03:11:00 UTC 
Verified in version as follow version, and attached result.
Version:
4.4.0-0.nightly-2020-06-21-210301
Comment 18 wewang 2020-06-22 03:20:07 UTC 
Created attachment 1698249 [details]
sum(workqueue_adds_total) by (namespace,name) product
Comment 20 errata-xmlrpc 2020-06-29 15:33:54 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2713