Bug 2020489

Description of problem:
My test environment is 4.6.13, but 4.x also has this issue.

It found that the metrics of 'coredns_dns_*' such as 'coredns_dns_request_count_total' don't included the custom zone metrics data, due to CoreDNS prometheus plugin is not defined in custom zone block.

The documentation about 'prometheus enables Prometheus metrics'.
https://coredns.io/plugins/metrics/
It mentioned that 'This plugin can only be used once per Server Block'.

Currently, the coredns configuration is defined the prometheus plugin in '.' zone, but others zone is not defined.

OpenShift release version:
4.x

Cluster Platform:
all

How reproducible:
Please see the details.

Steps to Reproduce (in detail):
1. Configure the DNS forwarding for the custom zones.
https://docs.openshift.com/container-platform/4.9/networking/dns-operator.html#nw-dns-forward_dns-operator

# oc edit dns.operator/default
apiVersion: operator.openshift.io/v1
kind: DNS
metadata:
  name: default
spec:
  servers:
  - name: foo-server 
    zones: 
      - foo.com
    forwardPlugin:
      upstreams: 
        - 1.1.1.1
        - 2.2.2.2:5353
  - name: bar-server
    zones:
      - bar.com
      - example.com
    forwardPlugin:
      upstreams:
        - 3.3.3.3
        - 4.4.4.4:5454

2. Go to the OAuth pods and do the dns query for the custom forwwarding zone.
# oc -n openshift-authentication rsh oauth-openshift-xxxxxxxxxx
$ curl example.com
$ curl foo.com

3. Open the Prometheus WebConsole and do the PromQL
sum by (zone) (coredns_dns_request_count_total)

Element	        Value
{zone="."}	804685

Actual results:
It is only '.' zone metrics.

PromQL-> sum by (zone) (coredns_dns_request_count_total)
Element	        Value
{zone="."}	804685

# oc -n openshift-dns edit cm dns-default
  Corefile: |
    # foo-server
    foo.com:5353 {
        forward . 1.1.1.1 2.2.2.2:5353
    }
    # bar-server
    bar.com:5353 example.com:5353 {
        forward . 3.3.3.3 4.4.4.4:5454
    }
    .:5353 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            upstream
            fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf {
            policy sequential
        }
        cache 30
        reload
    }

Expected results:
All zone metrics should be included.

PromQL-> sum by (zone) (coredns_dns_request_count_total)
Element	Value
{zone="."}	804685
{zone="example.com."}	4644
{zone="foo.com."}	10

# oc -n openshift-dns edit cm dns-default
  Corefile: |
    # foo-server
    foo.com:5353 {
        forward . 1.1.1.1 2.2.2.2:5353
        prometheus :9153    <- prometheus plugin should be defined for this zone
    }
    # bar-server
    bar.com:5353 example.com:5353 {
        forward . 3.3.3.3 4.4.4.4:5454
        prometheus :9153    <- prometheus plugin should be defined for this zone
    }
    .:5353 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            upstream
            fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf {
            policy sequential
        }
        cache 30
        reload
    }

Impact of the problem:
The customer cannot get the custom zone metrics for the data analysis and the troubleshooting.

Additional info:
This issue can be verified via coredns operator unmanaged and modifying the coredns configuration.

To set up to unmanaged status for the dns operator, the spec of the clusterversion should be changed as below.
# oc edit clusterversion

Append the overrides args for the coredns operator.
spec:
  overrides:
  - group: apps/v1
    kind: Deployment
    name: dns-operator
    namespace: openshift-dns-operator
    unmanaged: true

Change the replicas to 0 for the deployment of the dns-operator.
# oc -n openshift-dns-operator scale --replicas=0 deployment/dns-operator

Check the deployment of the dns-operator is 0.
# oc -n  openshift-dns-operator  get deployment
NAME           READY   UP-TO-DATE   AVAILABLE   AGE
dns-operator   0/0     0            0           15d

Modify the default CoreDNS configuration.
# oc -n openshift-dns edit cm dns-default
  Corefile: |
    # foo-server
    foo.com:5353 {
        forward . 1.1.1.1 2.2.2.2:5353
        prometheus :9153    <- Add this line, prometheus plugin will collect the metrics for this zone
    }
    # bar-server
    bar.com:5353 example.com:5353 {
        forward . 3.3.3.3 4.4.4.4:5454
        prometheus :9153    <- Add this line, prometheus plugin will collect the metrics for this zone
    }
    .:5353 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            upstream
            fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf {
            policy sequential
        }
        cache 30
        reload
    }

** Please do not disregard the report template; filling the template out as much as possible will allow us to help you. Please consider attaching a must-gather archive (via `oc adm must-gather`). Please review must-gather contents for sensitive information before attaching any must-gathers to a bugzilla report.  You may also mark the bug private if you wish.