Bug 1986983

Summary: Revise Alert Severity in OCP 4.8
Product: OpenShift Container Platform Reporter: Haoyu Sun <hasun>
Component: MonitoringAssignee: Haoyu Sun <hasun>
Status: CLOSED DUPLICATE QA Contact: Junqi Zhao <juzhao>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.9CC: alegrand, amuller, anpicker, aos-bugs, arajkuma, dgrisonn, erooth, jeder, kakkoyun, pkrupa, rrackow, rsandu, spasquie
Target Milestone: ---Keywords: ServiceDeliveryImpact
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-08-10 07:13:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Haoyu Sun 2021-07-28 16:11:13 UTC 
This is a backport of modifications that we are going to make to fix bug 1986981.

Description of problem:

After reviewing critical alerts in OCP, we find out the 21 alerts that need adjustments:
- Recommend changing Critical to Warning:  13
  - KubePersistentVolumeErrors
  - PrometheusBadConfig
  - PrometheusRemoteStorageFailures
  - PrometheusRuleFailures
  - AlertmanagerMembersInconsistent
  - AlertmanagerClusterFailedToSendAlerts
  - AlertmanagerConfigInconsistent
  - AlertmanagerClusterDown
  - KubeStateMetricsListErrors
  - KubeStateMetricsWatchErrors
  - ThanosRuleSenderIsFailingAlerts
  - ThanosRuleHighRuleEvaluationFailures
  - ThanosNoRuleEvaluations

- Recommend removing alert:  2
  - PrometheusErrorSendingAlertsToAnyAlertmanager
  - AlertmanagerClusterCrashlooping

- Recommend changing Critical to Info:  1
  - PrometheusRemoteWriteBehind
  
- Threshold Tweaks:  5
  - KubePersistentVolumeFillingUp
  - KubeletDown
  - NodeFilesystemFilesFillingUp
  - NodeFilesystemSpaceFillingUp
  - PrometheusRemoteStorageFailures

Please refer to this table for details(proposed modification are in column F "Comments") : https://docs.google.com/spreadsheets/d/10rL3loHz6a8lBfKsU2W9TVZSrSqndrnVmkzDeA3Z2kI/edit?usp=sharing
This table can be also found in the attachment.


Version-Release number of selected component (if applicable): 4.8


How reproducible:
N/A

Steps to Reproduce:
N/A

Actual results:
N/A

Expected results:
N/A

Additional info:
Comment 1 Damien Grisonnet 2021-08-10 07:13:43 UTC 

*** This bug has been marked as a duplicate of bug 1986981 ***