Bug 1820229 - CMO reverts to the default configuration when the 'cluster-monitoring-config' config map is invalid
Summary: CMO reverts to the default configuration when the 'cluster-monitoring-config'...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.4
Hardware: Unspecified
OS: Unspecified
unspecified
low
Target Milestone: ---
: 4.4.0
Assignee: Simon Pasquier
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On: 1807430
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-04-02 14:44 UTC by Simon Pasquier
Modified: 2020-05-04 11:48 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: when the "cluster-monitoring-config" config map is invalid (e.g. it can't be decoded), the cluster monitoring operator uses the default configuration. Consequence: any change made to customize the cluster monitoring configuration (such as PVC, node selectors, toleration) would be lost and the monitoring stack would be reconciled to its default state. Fix: when the cluster monitoring operator can't decode "cluster-monitoring-config" config map, it doesn't try to reconcile the monitoring stack. An alert fires when this happens. Result: the monitoring stack isn't modified when an invalid configuration is given.
Clone Of: 1807430
Environment:
Last Closed: 2020-05-04 11:48:02 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-monitoring-operator pull 737 0 None closed [release-4.4] Bug 1820229: don't sync on invalid configuration 2020-11-11 09:57:46 UTC
Red Hat Product Errata RHBA-2020:0581 0 None None None 2020-05-04 11:48:21 UTC

Comment 4 Junqi Zhao 2020-04-10 07:33:21 UTC 
Test with 4.4.0-0.nightly-2020-04-09-194724, set invalid value in cluster-monitoring-config confimap, there is error in cluster-monitoring-operator pod and trigger ClusterMonitoringOperatorReconciliationErrors/ClusterOperatorDegraded/ClusterOperatorDown alerts, no resource is created under openshift-user-workload-monitoring

kind: ConfigMap
apiVersion: v1
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    techPreviewUserWorkload:
      enabled: friday

# oc get co/monitoring -oyaml
...
  - lastTransitionTime: "2020-04-10T07:23:49Z"
    message: 'Failed to rollout the stack. Error: the Cluster Monitoring ConfigMap
      could not be parsed: error unmarshaling JSON: while decoding JSON: json: cannot
      unmarshal string into Go struct field UserWorkloadConfig.techPreviewUserWorkload.enabled
      of type bool'
    reason: InvalidConfiguration
    status: "True"
    type: Degraded
...


# oc -n openshift-monitoring logs cluster-monitoring-operator-5df74c575d-xjz5b | tail
E0410 07:23:55.408097       1 operator.go:274] sync "openshift-monitoring/cluster-monitoring-config" failed: the Cluster Monitoring ConfigMap could not be parsed: error unmarshaling JSON: while decoding JSON: json: cannot unmarshal string into Go struct field UserWorkloadConfig.techPreviewUserWorkload.enabled of type bool
I0410 07:24:00.528324       1 operator.go:299] Updating ClusterOperator status to failed. Err: the Cluster Monitoring ConfigMap could not be parsed: error unmarshaling JSON: while decoding JSON: json: cannot unmarshal string into Go struct field UserWorkloadConfig.techPreviewUserWorkload.enabled of type bool
...
Comment 6 errata-xmlrpc 2020-05-04 11:48:02 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581
Note You need to log in before you can comment on or make changes to this bug.