Bug 1823261

Summary: [4.4]No prometheus graph for firing alerts except Watchdog
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: Management ConsoleAssignee: David Taylor <dtaylor>
Status: CLOSED ERRATA QA Contact: Junqi Zhao <juzhao>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.4CC: anpicker, aos-bugs, bpeterse, jokerman, jonjacks, yapei
Target Milestone: ---Keywords: Regression
Target Release: 4.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: 1771843 Environment:
Last Closed: 2020-05-04 11:49:15 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
there is diagram for Watchdog alert
none
AlertmanagerReceiversNotConfigured with Prometheus graph
none
alert other than Watchdog is showing prometheus graph (4.4)
none
could show metrics diagram for AlertmanagerReceiversNotConfigured alert
none
No prometheus graph for AlertmanagerReceiversNotConfigured on UPI baremetal cluster
none
API returns value successfully, but not shown on UI
none
chart shown in 4.4-rc.8
none
upi-on-vsphere cluster, alerts other than Watchdog could show chart none

Comment 1 Junqi Zhao 2020-04-13 04:32:51 UTC
UPI on vSphere cluster, payload: 4.4.0-0.nightly-2020-04-09-220855

Comment 2 Junqi Zhao 2020-04-13 04:35:48 UTC
Created attachment 1678367 [details]
there is diagram for Watchdog alert

Comment 3 David Taylor 2020-04-13 18:21:22 UTC
Created attachment 1678499 [details]
AlertmanagerReceiversNotConfigured with Prometheus graph

Comment 4 David Taylor 2020-04-13 18:34:44 UTC
Created attachment 1678501 [details]
alert other than Watchdog is showing prometheus graph (4.4)

Comment 5 David Taylor 2020-04-13 18:46:17 UTC
Hi 
Using 4.4.0-0.ci-2020-04-13-142213 I do see a Prometheus graph for AlertmanagerReceiversNotConfigured (attachment 1678501 [details]).  I also see a Prometheus graph in 4.5.0-0.ci-2020-04-13-003218 (attachment 1678499 [details]).

Not sure if just not working in 4.4.0-0.nightly-2020-04-09-220855, or something to do with 'UPI on vSphere cluster'?

Comment 6 Junqi Zhao 2020-04-14 04:03:51 UTC
Tested with 4.4.0-0.nightly-2020-04-13-113747, it is IPI AWS cluster, there is not such issue now, see the attached picture, maybe the error is just in few payload

Comment 7 Junqi Zhao 2020-04-14 04:04:34 UTC
Created attachment 1678614 [details]
could show metrics diagram for AlertmanagerReceiversNotConfigured alert

Comment 10 Junqi Zhao 2020-04-15 10:41:16 UTC
Created attachment 1679001 [details]
No prometheus graph for AlertmanagerReceiversNotConfigured on UPI baremetal cluster

Comment 11 Junqi Zhao 2020-04-15 10:48:09 UTC
checked, the API returns value successfully, but not shown on UI, see the picture

Comment 12 Junqi Zhao 2020-04-15 10:48:33 UTC
Created attachment 1679003 [details]
API returns value successfully, but not shown on UI

Comment 13 bpeterse 2020-04-15 13:35:31 UTC
This appears to have been fixed in 4.3 and should be working now.

Comment 14 bpeterse 2020-04-15 13:44:06 UTC
Misread the requirement for vsphere.  Back to assigned.

Comment 15 David Taylor 2020-04-15 14:56:35 UTC
Created attachment 1679098 [details]
chart shown in 4.4-rc.8

Comment 16 David Taylor 2020-04-15 14:59:33 UTC
Hi, issued '@cluster-bot launch 4.4 vsphere' which resulted in a 'stable 4.4.0-rc.8' cluster.
I do see a prometheus graph for AlertmanagerReceiversNotConfigured, see attachment 1679098 [details]

Comment 17 Junqi Zhao 2020-04-16 02:10:53 UTC
https://openshift-release.svc.ci.openshift.org/releasestream/4-stable/release/4.4.0-rc.8
Release 4.4.0-rc.8 was created from registry.svc.ci.openshift.org/ocp/release:4.4.0-0.nightly-2020-04-13-113747

see Comment 9, our payload is 4.4.0-0.nightly-2020-04-14-222426, it is newer than 4.4.0-rc.8.

Will try to install a new payload, if it still happen, will assign back

Comment 18 Junqi Zhao 2020-04-16 03:18:11 UTC
upi-on-vsphere 4.4.0-0.nightly-2020-04-15-212458 cluster, no such issue there, maybe the issue is only with a few payload, close it first

Comment 19 Junqi Zhao 2020-04-16 03:20:00 UTC
Created attachment 1679208 [details]
upi-on-vsphere cluster, alerts other than Watchdog could show chart

Comment 22 errata-xmlrpc 2020-05-04 11:49:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581