Bug 1925659

Especially transient errors.

This is happening in about 1% of upgrades

https://search.ci.openshift.org/?search=clusteroperator%2Finsights+is+Degraded&maxAge=48h&context=1&type=bug%2Bjunit&name=&maxMatches=5&maxBytes=20971520&groupBy=job

It looks like the symptom is fairly similar, but the correct behavior is not "hide this one failure from causing degraded" it is "don't degrade at all if not all data can be gathered".

OK. I didn't know this. Yes we parse some logs (sdn, sdn-controller and some more) for interesting messages and these are very likely not ready yet. I think the fix should be pretty easy. I am about to open a PR.

Let me ask once again here please. The original behaviour (4.6 and lower) was following (if I am not mistaken). If there was any error (and most of the time there were probably none, because there was few gatherers) during the gathering then the state was set as degraded, but the new gathering was triggered immediately (which probably means that the degraded state lasted a very short time). In fact, we only changed the second part - the new gathering is not triggered immediately but there's some time before the next run. So we don't really need the degraded state caused by some gatherer error at all right?

The part we changed is that we retry 5 times before setting the degraded status using ExponentialBackoff. We implemented this because in the original implementation an infinite loop was created if a gather was failing consistently.

Here's another failure:

        {
            s: "query failed: ALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards\",alertstate=\"firing\",severity!=\"info\"} >= 1: promQL query: ALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards\",alertstate=\"firing\",severity!=\"info\"} >= 1 had reported incorrect results:\n[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"ClusterOperatorDegraded\",\"alertstate\":\"firing\",\"condition\":\"Degraded\",\"endpoint\":\"metrics\",\"instance\":\"10.0.239.234:9099\",\"job\":\"cluster-version-operator\",\"name\":\"insights\",\"namespace\":\"openshift-cluster-version\",\"pod\":\"cluster-version-operator-64d68cd48d-r5h7f\",\"reason\":\"PeriodicGatherFailed\",\"service\":\"cluster-version-operator\",\"severity\":\"critical\"},\"value\":[1612980990.956,\"1\"]},{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"ClusterOperatorDown\",\"alertstate\":\"firing\",\"endpoint\":\"metrics\",\"instance\":\"10.0.239.234:9099\",\"job\":\"cluster-version-operator\",\"name\":\"insights\",\"namespace\":\"openshift-cluster-version\",\"pod\":\"cluster-version-operator-64d68cd48d-r5h7f\",\"service\":\"cluster-version-operator\",\"severity\":\"critical\",\"version\":\"4.7.0-0.ci.test-2021-02-10-172859-ci-ln-xb3dpyb\"},\"value\":[1612980990.956,\"1\"]}]",

Tomas' last question is what I consider fundamental.

Insights-operator may not go degraded due to gather failures, but it should signal those via the Disabled (after enough consistent failures, maybe 3-4).

Put in another way, insights operator may not operationally bring down a cluster (which going degraded on a purely cosmetic failure causes).  Telemetry failing doesn't cause monitoring to report degraded.

Thanks Clayton. We updated the PR (4.7 is waiting for group lead approval) so that the IO is disabled when the gathering failures threshold is exceeded. It looks like it would be good to backport it to 4.6 as well.

(In reply to Tomas Remes from comment #9)
> It looks like it would be good to backport it to 4.6 as well.

"Issue exists in 4.6" sounds like "not a 4.6 -> 4.7 regression", so I'm going to set blocker- on this to avoid delaying the 4.7 GA.

this would be challenging to verify directly,
steps to verify:
1) change a gatherer of IO to always return error (modify IO code)
2) build & replace IO on cluster
3) check that IO is not degraded

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.7.1 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:0678