Bug 1861455
| Summary: | Defunct router doesn't trigger alerts | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | aaleman | |
| Component: | Networking | Assignee: | Stephen Greene <sgreene> | |
| Networking sub component: | router | QA Contact: | Arvind iyengar <aiyengar> | |
| Status: | CLOSED ERRATA | Docs Contact: | ||
| Severity: | urgent | |||
| Priority: | urgent | CC: | amcdermo, aos-bugs, hongli, jeder, rrackow, sgreene, skuznets, wking | |
| Version: | 4.4 | Keywords: | ServiceDeliveryImpact | |
| Target Milestone: | --- | |||
| Target Release: | 4.6.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | Bug Fix | ||
| Doc Text: |
Cause:
Updating the cluster when routes that break HAProxy exist in the cluster.
Consequence:
HAProxy does not initialize properly and does not respect cluster routes on upgrade. Upgrade succeeds into a cluster with not functioning routes.
Fix:
Correct HAProxy initial sync logic so that upgrades with broken routes will fail. Add router template reload failure metric and alert.
Result:
Upgrading with broken routes will not be successful, and HAProxy failed reloads at any time within a cluster will be reported via alerts.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1871175 (view as bug list) | Environment: | ||
| Last Closed: | 2020-10-27 16:21:16 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1866454, 1871175 | |||
| Attachments: | ||||
|
Description
aaleman
2020-07-28 16:36:57 UTC
Created attachment 1702783 [details]
router logs
Adding router logs with loglevel set to 10 to easier follow up what's happening
In this case, the operand was entirely nonfunctional. The fact that this did not cause an alert to be fired, the operator to be considered degraded, or the upgrade to fail is a serious bug. Target a fix for 4.6 and will backport. Added UpcomingSprint. I am working on a fix for this bug. There are 2 interesting parts to this issue. The HAProxy "up" metric is actually working. First, we identified an issue with how the router syncs on startup. Currently, the router performs an initial reload before syncing the state of all route resources. This is a defect. This initial "routeless" reload starts HAProxy, just in a defunct state with respect to cluster ingress. Removing this premature reload should cause new router pods to never become ready in the event that improper routes exist, since HAProxy will not start without having all the route resources available. The reason that the HAProxy "up" metric was reporting "1" during the mentioned upgrade to 4.4 was because the previous "routeless" HAProxy process was still running after the first "route in" reload failed. The router intentionally does not kill the running "old" HAProxy process if a reload fails. Second, there is no metric that tracks HAProxy router reload failures. A router reload failure that happens after the router pod successfully starts will not crash the router pod, and this is intentional. By adding a router reload failure count metric, we can alert cluster admins when the router is not respecting newly created/modified routes. This new metric, in addition to the existing HAProxy "up" metric, will serve as great metrics for HAProxy up/down alerts. I will have the fix PR for this BZ up shortly after I perform some more tests. I am providing this comment to scope this BZ and to help QA during the verification process. The PR was merged and made into "4.6.0-0.nightly-2020-08-11-235456" release. With this payload, it is noted that "template_router_reload_fail" metric is now functional and display the reload failures for the routers whereas, the "haproxy_up" metric triggers warning in the alert manager during router down events. Created attachment 1711166 [details]
Prometheus graph data from patched cluster version
Created attachment 1711167 [details]
Prometheus graph data from unpatched cluster version
Created attachment 1711168 [details]
Alermanager data from patched cluster version
Created attachment 1711301 [details]
alertmanager dashboard example - 2
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196 |