Bug 1820229

Summary: CMO reverts to the default configuration when the 'cluster-monitoring-config' config map is invalid
Product: OpenShift Container Platform Reporter: Simon Pasquier <spasquie>
Component: MonitoringAssignee: Simon Pasquier <spasquie>
Status: CLOSED ERRATA QA Contact: Junqi Zhao <juzhao>
Severity: low Docs Contact:
Priority: unspecified    
Version: 4.4CC: alegrand, anpicker, erooth, juzhao, kakkoyun, lcosic, mloibl, pkrupa, surbania
Target Milestone: ---   
Target Release: 4.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: when the "cluster-monitoring-config" config map is invalid (e.g. it can't be decoded), the cluster monitoring operator uses the default configuration. Consequence: any change made to customize the cluster monitoring configuration (such as PVC, node selectors, toleration) would be lost and the monitoring stack would be reconciled to its default state. Fix: when the cluster monitoring operator can't decode "cluster-monitoring-config" config map, it doesn't try to reconcile the monitoring stack. An alert fires when this happens. Result: the monitoring stack isn't modified when an invalid configuration is given.
 Story Points: ---
Clone Of: 1807430 Environment:
Last Closed: 2020-05-04 11:48:02 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1807430    
Bug Blocks:    

Comment 4 Junqi Zhao 2020-04-10 07:33:21 UTC 
Test with 4.4.0-0.nightly-2020-04-09-194724, set invalid value in cluster-monitoring-config confimap, there is error in cluster-monitoring-operator pod and trigger ClusterMonitoringOperatorReconciliationErrors/ClusterOperatorDegraded/ClusterOperatorDown alerts, no resource is created under openshift-user-workload-monitoring

kind: ConfigMap
apiVersion: v1
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    techPreviewUserWorkload:
      enabled: friday

# oc get co/monitoring -oyaml
...
  - lastTransitionTime: "2020-04-10T07:23:49Z"
    message: 'Failed to rollout the stack. Error: the Cluster Monitoring ConfigMap
      could not be parsed: error unmarshaling JSON: while decoding JSON: json: cannot
      unmarshal string into Go struct field UserWorkloadConfig.techPreviewUserWorkload.enabled
      of type bool'
    reason: InvalidConfiguration
    status: "True"
    type: Degraded
...


# oc -n openshift-monitoring logs cluster-monitoring-operator-5df74c575d-xjz5b | tail
E0410 07:23:55.408097       1 operator.go:274] sync "openshift-monitoring/cluster-monitoring-config" failed: the Cluster Monitoring ConfigMap could not be parsed: error unmarshaling JSON: while decoding JSON: json: cannot unmarshal string into Go struct field UserWorkloadConfig.techPreviewUserWorkload.enabled of type bool
I0410 07:24:00.528324       1 operator.go:299] Updating ClusterOperator status to failed. Err: the Cluster Monitoring ConfigMap could not be parsed: error unmarshaling JSON: while decoding JSON: json: cannot unmarshal string into Go struct field UserWorkloadConfig.techPreviewUserWorkload.enabled of type bool
...
Comment 6 errata-xmlrpc 2020-05-04 11:48:02 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581