Created attachment 1603252 [details]
prometheus-k8s pod logs

Description of problem:
# oc -n openshift-monitoring logs prometheus-k8s-0 -c prometheus | grep many
level=warn ts=2019-08-13T06:53:04.487Z caller=manager.go:513 component="rule manager" group=alertmanager.rules msg="Evaluating rule failed" rule="alert: AlertmanagerConfigInconsistent\nexpr: count_values by(service) (\"config_hash\", alertmanager_config_hash{job=\"alertmanager-main\",namespace=\"openshift-monitoring\"})\n  / on(service) group_left() label_replace(prometheus_operator_spec_replicas{controller=\"alertmanager\",job=\"prometheus-operator\",namespace=\"openshift-monitoring\"},\n  \"service\", \"alertmanager-$1\", \"name\", \"(.*)\") != 1\nfor: 5m\nlabels:\n  severity: critical\nannotations:\n  message: The configuration of the instances of the Alertmanager cluster `{{$labels.service}}`\n    are out of sync.\n" err="found duplicate series for the match group {service=\"alertmanager-main\"} on the right hand-side of the operation: [{__name__=\"prometheus_operator_spec_replicas\", controller=\"alertmanager\", endpoint=\"http\", instance=\"10.131.0.17:8080\", job=\"prometheus-operator\", name=\"main\", namespace=\"openshift-monitoring\", pod=\"prometheus-operator-7665c99b6f-njsgm\", service=\"alertmanager-main\"}, {__name__=\"prometheus_operator_spec_replicas\", controller=\"alertmanager\", endpoint=\"http\", instance=\"10.129.2.21:8080\", job=\"prometheus-operator\", name=\"main\", namespace=\"openshift-monitoring\", pod=\"prometheus-operator-57d45fcf98-8jslj\", service=\"alertmanager-main\"}];many-to-many matching not allowed: matching labels must be unique on one side"
*********************************************

alert: AlertmanagerConfigInconsistent
expr: count_values
  by(service) ("config_hash", alertmanager_config_hash{job="alertmanager-main",namespace="openshift-monitoring"})
  / on(service) group_left() label_replace(prometheus_operator_spec_replicas{controller="alertmanager",job="prometheus-operator",namespace="openshift-monitoring"},
  "service", "alertmanager-$1", "name", "(.*)") !=
  1 
for: 5m 
labels: severity: critical 
annotations: message: The configuration of the instances of the Alertmanager cluster `{{$labels.service}}` are out of sync.

Note: did not meet this error before, it is a 4.2 regression, but it seems it does not affect the function

Version-Release number of selected component (if applicable):
4.2.0-0.nightly-2019-08-12-153437

How reproducible:
Always

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info: