If a route were to break the HAProxy config files, and thus break HAProxy reloads, the HAProxyReloadFail alert will only fire for ~5 minutes. Instead, the HAProxyReloadFail alert (& base metric) should be reworked. The template_router_reload_fails metric should be dropped in exchange for a metric that tracks the status of the most recent reload. ie template_router_reload_success which is pinned to 1 on successful reloads, and 0 on failed reloads. The current template_router_reload_fails reports an increasing value of failed reloads, which is difficult to alert on properly. A flag/boolean metric is trivial to alert on for the actual duration of the problem. This affects 4.7, 4.6, and 4.5, so backports will be required.
I'd mentioned a positive name, but a negative name like template_router_reload_failure might be more convenient if you wanted a label with a reason slug, or some such. You could always add a reason to a positive label too, but template_router_reload_success{failure_reason="whatever"} feels more awkward than template_router_reload_failure{reason="whatever"}.
(In reply to W. Trevor King from comment #1) > I'd mentioned a positive name, but a negative name like > template_router_reload_failure might be more convenient if you wanted a > label with a reason slug, or some such. You could always add a reason to a > positive label too, but > template_router_reload_success{failure_reason="whatever"} feels more awkward > than template_router_reload_failure{reason="whatever"}. Noted, I will make sure to use a negative name instead. :)
Not a regression, so it's hard to imagine holding 4.7.0 on a fix for this.
Tested in "4.7.0-0.nightly-2020-11-25-114114" payload. It is noted that the new metric and the associated Prometheus rules are added as intended: ---- $ curl -sS -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IjhjRDJ6ZnJsdnhKczlVQ2R6TndrOW1RS29BdS1LSEhzbGtQUEVPZFNsVUkifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJvcGVuc2hpZnQtbW9uaXRvcmluZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJwcm9tZXRoZXVzLWs4cy10b2tlbi1yZmhtdiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJwcm9tZXRoZXVzLWs4cyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjFiYmIyOGQwLThkMDgtNGI2MC1hNDk4LTBjYzc3NzE4OWY1YSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpvcGVuc2hpZnQtbW9uaXRvcmluZzpwcm9tZXRoZXVzLWs4cyJ9.b2nqVlbW4xHGuh3EYuXTsCV9fcjc6G5Yq9TABxqAYaUuRSmp79lH5dfhC9k9TRKDITlVxsXDvMuH_CN392RlwXIMyytEidnNP_zTH-rqpl12NrDTxGfurf2WtfZefPGDM1tSadcAm_jGmebDLzmjWGPGm5mWIYIdRiBaILku0HDrhDLgfhG-BpsZ5WXTikJhdskmCs38Ru9oQcuyXIJEXSOnGqKJZYQFqdzPsA8zh-aCotI49R42qo903tXvoh5kvBU4kx0gAgwdaXYBPOrKe8kWWDvE6gL_NHcb3pLIoI05vXvMW0BwBnMX8h9X7KCXymsU7aH-IiBVnNaRQ8BxYQ" -k https://10.129.2.22:1936/metrics | grep -i template_router_reload_failure # HELP template_router_reload_failure Metric to track the status of the most recent HAProxy reload # TYPE template_router_reload_failure gauge template_router_reload_failure 0 ----
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633