Description of problem: telemetry-config configmap is the metrics allowlist for telemeter client, we could see the result from aws cluster for "cluster:ingress_controller_aws_nlb_active:sum". # token=`oc sa get-token prometheus-k8s -n openshift-monitoring` # oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://thanos-querier.openshift-monitoring.svc:9091/api/v1/query?query=cluster:ingress_controller_aws_nlb_active:sum' | jq { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "__name__": "cluster:ingress_controller_aws_nlb_active:sum", "prometheus": "openshift-monitoring/k8s" }, "value": [ 1645712191.009, "0" ] } ] } } but no result from telemeter server: https://infogw-proxy.api.openshift.com/ reason is the metrics is not added to telemeter server allowlist, see from https://github.com/rhobs/configuration/blob/main/configuration/telemeter/metrics.json # oc -n openshift-monitoring get cm telemetry-config -o jsonpath="{.data.metrics\.yaml}" matches: ... # # owners: (@openshift/network-edge) # # cluster:ingress_controller_aws_nlb_active:sum informs how many NLBs are active in AWS. # Zero would indicate ELB (legacy). This metric is only emitted on AWS. - '{__name__="cluster:ingress_controller_aws_nlb_active:sum"}' # ... OpenShift release version: 4.10.0-0.nightly-2022-02-22-093600 Cluster Platform: AWS How reproducible: always Steps to Reproduce (in detail): 1. AWS cluster, search "cluster:ingress_controller_aws_nlb_active:sum" from local cluster 2. check in telemeter server 3. Actual results: no result from telemeter server: https://infogw-proxy.api.openshift.com/ Expected results: should see the result Impact of the problem: Additional info: Docs on the process on why and how to send these metrics: https://docs.google.com/document/d/1a6n5iBGM2QaIQRg9Lw4-Npj6QY9--Hpx3XYut-BrUSY/edit#heading=h.i05nm4wmvn7x
Thomas, I believe this is related to <https://issues.redhat.com/browse/NE-704>, which you implemented with <https://github.com/openshift/cluster-ingress-operator/pull/675>. Can you take a look at this BZ and assess priority and blocker status?
setting blocker- PR for rhobs is here: https://github.com/rhobs/configuration/pull/157
PR merged, asked for manual rollout in https://coreos.slack.com/archives/C010A637DGB/p1646043172619779
was just deployed confirmed by Jessica
@juzhao Arvind iyenga @
ok, I can see metrics coming in from tenants on thanos now since the deployment happened: https://telemeter-lts.datahub.redhat.com/graph?g0.range_input=1w&g0.max_source_resolution=auto&g0.expr=cluster%3Aingress_controller_aws_nlb_active%3Asum&g0.tab=0