Bug 1798461 - kube-aggregator: unavailableGauge is wrong
Summary: kube-aggregator: unavailableGauge is wrong
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-apiserver
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.3.z
Assignee: Lukasz Szaszkiewicz
QA Contact: Ke Wang
Depends On: 1798450
TreeView+ depends on / blocked
Reported: 2020-02-05 12:12 UTC by Lukasz Szaszkiewicz
Modified: 2020-05-20 13:48 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1798450
Last Closed: 2020-05-20 13:47:53 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Priority Status Summary Last Updated
Github openshift origin pull 24501 None closed Bug 1798461: makes unavailableGauge metric to always reflect the current state of a service 2020-05-11 14:46:00 UTC
Red Hat Product Errata RHBA-2020:2129 None None None 2020-05-20 13:48:08 UTC

Description Lukasz Szaszkiewicz 2020-02-05 12:12:50 UTC
+++ This bug was initially created as a clone of Bug #1798450 +++

unavailableGauge metric is not always set and might report incorrect values especially in an HA setup where one instance could observe and mark a service as unavailable whereas some other instance might observe it as available. That would prevent the first instance from reflecting that state since it wouldn't observe any changes

Comment 3 Ke Wang 2020-05-12 07:17:41 UTC
Verified with OCP build 4.3.0-0.nightly-2020-05-07-171148,

Verification steps,
1. Make some apiservice fail, e.g. remove openshift-apiserver by:
$ oc patch openshiftapiserver cluster --type=json -p '[{"op": "replace", "path": "/spec/managementState", "value": "Removed"}]'
2. Wait for a while(about half an hour), Recover the apiservice.
$ oc patch openshiftapiserver cluster --type=json -p '[{"op": "replace", "path": "/spec/managementState", "value": "Managed"}]'

3. Open the prometheus UI from OCP web console, enter keyword ‘aggregator_unavailable_apiservice_count’  and Click on'Exuecte', navigate to Console tab, some unavailable apiservices will be displayed with name and count.

Comment 5 errata-xmlrpc 2020-05-20 13:47:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.