Bug 1927423 - Happy "Not Found" and no visible error messages on error-list page when /silences 504s
Summary: Happy "Not Found" and no visible error messages on error-list page when /sile...
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.6
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.8.0
Assignee: Andrew Pickering
QA Contact: hongyan li
Depends On:
TreeView+ depends on / blocked
Reported: 2021-02-10 17:11 UTC by W. Trevor King
Modified: 2021-07-27 22:44 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Last Closed: 2021-07-27 22:43:44 UTC
Target Upstream Version:

Attachments (Terms of Use)
silences 504 leads to "Not Found" (91.49 KB, image/png)
2021-02-10 17:11 UTC, W. Trevor King
no flags Details
alert tab (36.80 KB, image/png)
2021-03-28 12:37 UTC, hongyan li
no flags Details
silence tab (31.35 KB, image/png)
2021-03-28 12:38 UTC, hongyan li
no flags Details
alert rule tab (111.00 KB, image/png)
2021-03-28 12:39 UTC, hongyan li
no flags Details

System ID Private Priority Status Summary Last Updated
Github openshift console pull 8461 0 None open Bug 1927423: Monitoring: Add warning to list pages when silences cannot be loaded 2021-03-25 03:30:48 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 22:44:07 UTC

Description W. Trevor King 2021-02-10 17:11:02 UTC
Created attachment 1756280 [details]
silences 504 leads to "Not Found"

Seen in 4.6.  Maybe 4.6.16?  I'll check.  But due to compute-node issues, /api/alertmanager/api/v2/silences 504ed.  Apparently because of that, the alert-listing logic was unable to list alerts, and it showed "Not Found" (screenshot attached).  Ideally it would show an error message about the silence 504, to let folks know that the issue was "I don't know which unsilenced alerts match your filter", which is much more serious than "no unsilenced alerts match your filter, all is well".

And it might also be worth listing all filter-matching alerts with some note about "maybe these are silenced, but we can't tell because the silences API is 504ing".  Or maybe that risks being too distracting?

Comment 2 hongyan li 2021-03-28 12:37:07 UTC
verified with payload 4.8.0-0.nightly-2021-03-25-191436

oc -n openshift-monitoring scale sts alertmanager-main --replicas 0
access api /api/alertmanager/api/v2/silences get 504 gateway timeout
check monitoring-alert page
Get alert for alert tab and alert rule tab, get 504 error for silence tab, for detail see screenshot

Comment 3 hongyan li 2021-03-28 12:37:47 UTC
Created attachment 1767096 [details]
alert tab

Comment 4 hongyan li 2021-03-28 12:38:22 UTC
Created attachment 1767097 [details]
silence tab

Comment 5 hongyan li 2021-03-28 12:39:10 UTC
Created attachment 1767098 [details]
alert rule tab

Comment 8 errata-xmlrpc 2021-07-27 22:43:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.