Description of problem:
cluster_quantile:apiserver_request_duration_seconds:histogram_quantile is shipped both by CMO via the openshift-monitoring/kubernetes-monitoring-rules PrometheusRule resource and cluster-kube-apiserver-operator via the openshift-kube-apiserver/kube-apiserver-slos PrometheusRule resource.

Version-Release number of selected component (if applicable):
4.9

How reproducible:
Always

Steps to Reproduce:
Run the following command:

oc get prometheusrule -A -o json | jq '.items | map({name: .metadata.name, ns: .metadata.namespace, groups: .spec.groups})  | map( select( .groups | any(.rules | any(.record == "cluster_quantile:apiserver_request_duration_seconds:histogram_quantile")))) | map(.ns + "/" + .name)'  

Actual results:

[
  "openshift-kube-apiserver/kube-apiserver-slos",
  "openshift-monitoring/kubernetes-monitoring-rules"
]

Expected results:


[
  "openshift-kube-apiserver/kube-apiserver-slos"
]

Additional info:

AFAICT the issue is only present in 4.9 because the rules have been reorganized/split by https://github.com/kubernetes-monitoring/kubernetes-mixin/pull/632 which hasn't been pulled in 4.8.