Created attachment 1877998 [details]
Observe dashboard capture file which is missing instance label in the alert messages.

Description of problem:

When firing FluentdNodeDown alert, the alert message is missing "$labels.instance" as follows.
As a result, it does not honor "$labels.instance" label in the alert message. Check the image capture file together please.

Displayed messages: 
~~~
Prometheus could not scrape fluentd  for more than 10m.
~~~

Alert rule definition:
~~~
- name: logging_fluentd.alerts
  rules:
  - alert: FluentdNodeDown
    annotations:
      message: Prometheus could not scrape fluentd {{ $labels.instance }} for more    <--- HERE
        than 10m.
      summary: Fluentd cannot be scraped
    expr: absent(up{job="collector",namespace="openshift-logging"} == 1)
    for: 10m
    labels:
      namespace: openshift-logging
      service: fluentd
      severity: critical
~~~

Is it expected result ? How to display the "$labels.instance" label in the alert message ?


Version-Release number of selected component (if applicable):

$ oc version
Client Version: 4.9.23
Server Version: 4.9.23
Kubernetes Version: v1.22.3+b93fd35

How reproducible:

You can always reproduce this when firing the "FluentdNodeDown".

Steps to Reproduce:
1.
2.
3.

Actual results:

Missing the "$labels.instance" value in the alert messages, even though the label is configured in the alert rules.

Expected results:

The "$labels.instance" value should be displayed as the alert rule is defined.


Additional info: