Created attachment 1679579 [details]
No prometheus graph for AlertmanagerReceiversNotConfigured if configured prometheus externalLabels

Description of problem:
Prometheus externalLabels setting would block prometheus chart for alert on "Monitoring -> Alerting", the exception is Watchdog alert, since its expression is just "vector(1)"
configure externalLabels for prometheus via cluster-monitoring-config configmap
*********************
# oc -n openshift-monitoring get cm cluster-monitoring-config -oyaml
apiVersion: v1
data:
  config.yaml: |
    prometheusK8s:
      externalLabels:
        region: us-east-2
        environment: AWS
kind: ConfigMap
metadata:
  creationTimestamp: "2020-04-17T05:25:31Z"
  name: cluster-monitoring-config
  namespace: openshift-monitoring
  resourceVersion: "164707"
  selfLink: /api/v1/namespaces/openshift-monitoring/configmaps/cluster-monitoring-config
  uid: 82d78eb3-88a0-44c6-b090-79b61d45b221
*********************
the alerts are with environment/region label
# token=`oc sa get-token prometheus-k8s -n openshift-monitoring`
# oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://alertmanager-main.openshift-monitoring.svc:9094/api/v1/alerts' | jq '.data[].labels | {alertname,environment,region}'
{
  "alertname": "CustomResourceDetected",
  "environment": "AWS",
  "region": "us-east-2"
}
{
  "alertname": "ServiceCatalogAPIServerEnabled",
  "environment": "AWS",
  "region": "us-east-2"
}
{
  "alertname": "ServiceCatalogControllerManagerEnabled",
  "environment": "AWS",
  "region": "us-east-2"
}
{
  "alertname": "Watchdog",
  "environment": "AWS",
  "region": "us-east-2"
}
{
  "alertname": "AlertmanagerReceiversNotConfigured",
  "environment": "AWS",
  "region": "us-east-2"
}
{
  "alertname": "FailingOperator",
  "environment": "AWS",
  "region": "us-east-2"
}
{
  "alertname": "FailingOperator",
  "environment": "AWS",
  "region": "us-east-2"
}

take AlertmanagerReceiversNotConfigured as an example, the prometheus chart can not be shown on the alert datil UI of "Monitoring -> Alerting".
remove the externalLabels settings and wait for the environment/region are null, check the AlertmanagerReceiversNotConfigured chart, there is prometheus chart then

# oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://alertmanager-main.openshift-monitoring.svc:9094/api/v1/alerts' | jq '.data[].labels | {alertname,environment,region}'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  6600    0  6600    0     0   1171      0 --:--:--  0:00:05 --:--:--  1600
{
  "alertname": "AlertmanagerReceiversNotConfigured",
  "environment": null,
  "region": null
}
{
  "alertname": "FailingOperator",
  "environment": null,
  "region": null
}
{
  "alertname": "ServiceCatalogControllerManagerEnabled",
  "environment": null,
  "region": null
}
{
  "alertname": "CustomResourceDetected",
  "environment": null,
  "region": null
}
{
  "alertname": "ServiceCatalogAPIServerEnabled",
  "environment": null,
  "region": null
}
{
  "alertname": "FailingOperator",
  "environment": null,
  "region": null
}
{
  "alertname": "Watchdog",
  "environment": null,
  "region": null
}


Version-Release number of selected component (if applicable):
4.4.0-0.nightly-2020-04-16-205909

How reproducible:
Always

Steps to Reproduce:
1. See the description
2.
3.

Actual results:


Expected results:


Additional info: