Bug 2058219

Summary: "cluster:ingress_controller_aws_nlb_active:sum" is not found from telemeter server
Product: OpenShift Container Platform Reporter: Junqi Zhao <juzhao>
Component: NetworkingAssignee: Thomas Jungblut <tjungblu>
Networking sub component: router QA Contact: Arvind iyengar <aiyengar>
Status: CLOSED CURRENTRELEASE Docs Contact:
Severity: low    
Priority: high CC: aos-bugs, hongli, mmasters, vlaad
Version: 4.10   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
URL: https://github.com/rhobs/configuration/pull/157
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2058225 (view as bug list) Environment:
Last Closed: 2022-08-26 15:24:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Junqi Zhao 2022-02-24 14:21:25 UTC
Description of problem:
telemetry-config configmap is the metrics allowlist for telemeter client, we could see the result from aws cluster for "cluster:ingress_controller_aws_nlb_active:sum".
# token=`oc sa get-token prometheus-k8s -n openshift-monitoring`
# oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://thanos-querier.openshift-monitoring.svc:9091/api/v1/query?query=cluster:ingress_controller_aws_nlb_active:sum' | jq
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "cluster:ingress_controller_aws_nlb_active:sum",
          "prometheus": "openshift-monitoring/k8s"
        },
        "value": [
          1645712191.009,
          "0"
        ]
      }
    ]
  }
}

but no result from telemeter server: https://infogw-proxy.api.openshift.com/
reason is the metrics is not added to telemeter server allowlist, see from
https://github.com/rhobs/configuration/blob/main/configuration/telemeter/metrics.json

#  oc -n openshift-monitoring get cm telemetry-config -o jsonpath="{.data.metrics\.yaml}"
matches:
...
#
# owners: (@openshift/network-edge)
#
# cluster:ingress_controller_aws_nlb_active:sum informs how many NLBs are active in AWS.
# Zero would indicate ELB (legacy). This metric is only emitted on AWS.
- '{__name__="cluster:ingress_controller_aws_nlb_active:sum"}'
#
...

OpenShift release version:
4.10.0-0.nightly-2022-02-22-093600

Cluster Platform:
AWS

How reproducible:
always

Steps to Reproduce (in detail):
1. AWS cluster, search "cluster:ingress_controller_aws_nlb_active:sum" from local cluster
2. check in telemeter server 
3.


Actual results:
no result from telemeter server: https://infogw-proxy.api.openshift.com/

Expected results:
should see the result

Impact of the problem:


Additional info:
Docs on the process on why and how to send these metrics:
https://docs.google.com/document/d/1a6n5iBGM2QaIQRg9Lw4-Npj6QY9--Hpx3XYut-BrUSY/edit#heading=h.i05nm4wmvn7x

Comment 1 Miciah Dashiel Butler Masters 2022-02-24 17:35:07 UTC
Thomas, I believe this is related to <https://issues.redhat.com/browse/NE-704>, which you implemented with <https://github.com/openshift/cluster-ingress-operator/pull/675>.  Can you take a look at this BZ and assess priority and blocker status?

Comment 2 Thomas Jungblut 2022-02-25 12:19:13 UTC
setting blocker- PR for rhobs is here: https://github.com/rhobs/configuration/pull/157

Comment 3 Thomas Jungblut 2022-02-28 10:13:24 UTC
PR merged, asked for manual rollout in https://coreos.slack.com/archives/C010A637DGB/p1646043172619779

Comment 4 Thomas Jungblut 2022-02-28 14:46:54 UTC
was just deployed confirmed by Jessica

Comment 5 Thomas Jungblut 2022-02-28 14:48:52 UTC
@juzhao

Arvind iyenga


@

Comment 7 Thomas Jungblut 2022-03-02 07:35:27 UTC
ok, I can see metrics coming in from tenants on thanos now since the deployment happened:

https://telemeter-lts.datahub.redhat.com/graph?g0.range_input=1w&g0.max_source_resolution=auto&g0.expr=cluster%3Aingress_controller_aws_nlb_active%3Asum&g0.tab=0