https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/25861/pull-ci-openshift-origin-master-e2e-gcp-upgrade/1357748792526376960 ClusterOperators did not settle: clusteroperator/insights is Degraded for 17m19.717543653s because "Source clusterconfig could not be retrieved: Get \"https://10.0.0.5:10250/containerLogs/openshift-sdn/sdn-controller-j59d2/sdn-controller?limitBytes=65536&sinceSeconds=86400\": dial tcp 10.0.0.5:10250: connect: connection refused, Get \"https://10.0.0.5:10250/containerLogs/openshift-sdn/sdn-z5nx8/sdn?limitBytes=65536&sinceSeconds=86400\": dial tcp 10.0.0.5:10250: connect: connection refused" The insights operator is not allowed to go degraded and block a cluster upgrade because of failures in gathering data from other components. The failure that caused this will be independently debugged, insights is not allowed to go degraded because of a failure to gather data from one specific subsystem.
Especially transient errors.
This is happening in about 1% of upgrades https://search.ci.openshift.org/?search=clusteroperator%2Finsights+is+Degraded&maxAge=48h&context=1&type=bug%2Bjunit&name=&maxMatches=5&maxBytes=20971520&groupBy=job It looks like the symptom is fairly similar, but the correct behavior is not "hide this one failure from causing degraded" it is "don't degrade at all if not all data can be gathered".
OK. I didn't know this. Yes we parse some logs (sdn, sdn-controller and some more) for interesting messages and these are very likely not ready yet. I think the fix should be pretty easy. I am about to open a PR.
Let me ask once again here please. The original behaviour (4.6 and lower) was following (if I am not mistaken). If there was any error (and most of the time there were probably none, because there was few gatherers) during the gathering then the state was set as degraded, but the new gathering was triggered immediately (which probably means that the degraded state lasted a very short time). In fact, we only changed the second part - the new gathering is not triggered immediately but there's some time before the next run. So we don't really need the degraded state caused by some gatherer error at all right?
The part we changed is that we retry 5 times before setting the degraded status using ExponentialBackoff. We implemented this because in the original implementation an infinite loop was created if a gather was failing consistently.
Here's another failure: { s: "query failed: ALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards\",alertstate=\"firing\",severity!=\"info\"} >= 1: promQL query: ALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards\",alertstate=\"firing\",severity!=\"info\"} >= 1 had reported incorrect results:\n[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"ClusterOperatorDegraded\",\"alertstate\":\"firing\",\"condition\":\"Degraded\",\"endpoint\":\"metrics\",\"instance\":\"10.0.239.234:9099\",\"job\":\"cluster-version-operator\",\"name\":\"insights\",\"namespace\":\"openshift-cluster-version\",\"pod\":\"cluster-version-operator-64d68cd48d-r5h7f\",\"reason\":\"PeriodicGatherFailed\",\"service\":\"cluster-version-operator\",\"severity\":\"critical\"},\"value\":[1612980990.956,\"1\"]},{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"ClusterOperatorDown\",\"alertstate\":\"firing\",\"endpoint\":\"metrics\",\"instance\":\"10.0.239.234:9099\",\"job\":\"cluster-version-operator\",\"name\":\"insights\",\"namespace\":\"openshift-cluster-version\",\"pod\":\"cluster-version-operator-64d68cd48d-r5h7f\",\"service\":\"cluster-version-operator\",\"severity\":\"critical\",\"version\":\"4.7.0-0.ci.test-2021-02-10-172859-ci-ln-xb3dpyb\"},\"value\":[1612980990.956,\"1\"]}]", Tomas' last question is what I consider fundamental. Insights-operator may not go degraded due to gather failures, but it should signal those via the Disabled (after enough consistent failures, maybe 3-4).
Put in another way, insights operator may not operationally bring down a cluster (which going degraded on a purely cosmetic failure causes). Telemetry failing doesn't cause monitoring to report degraded.
Thanks Clayton. We updated the PR (4.7 is waiting for group lead approval) so that the IO is disabled when the gathering failures threshold is exceeded. It looks like it would be good to backport it to 4.6 as well.
(In reply to Tomas Remes from comment #9) > It looks like it would be good to backport it to 4.6 as well. "Issue exists in 4.6" sounds like "not a 4.6 -> 4.7 regression", so I'm going to set blocker- on this to avoid delaying the 4.7 GA.
this would be challenging to verify directly, steps to verify: 1) change a gatherer of IO to always return error (modify IO code) 2) build & replace IO on cluster 3) check that IO is not degraded
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.7.1 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:0678