Bug 2058219 - "cluster:ingress_controller_aws_nlb_active:sum" is not found from telemeter server
Summary: "cluster:ingress_controller_aws_nlb_active:sum" is not found from telemeter s...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.10
Hardware: Unspecified
OS: Unspecified
high
low
Target Milestone: ---
: ---
Assignee: Thomas Jungblut
QA Contact: Arvind iyengar
URL: https://github.com/rhobs/configuratio...
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-02-24 14:21 UTC by Junqi Zhao
Modified: 2022-08-26 15:24 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2058225 (view as bug list)
Environment:
Last Closed: 2022-08-26 15:24:40 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Junqi Zhao 2022-02-24 14:21:25 UTC
Description of problem:
telemetry-config configmap is the metrics allowlist for telemeter client, we could see the result from aws cluster for "cluster:ingress_controller_aws_nlb_active:sum".
# token=`oc sa get-token prometheus-k8s -n openshift-monitoring`
# oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://thanos-querier.openshift-monitoring.svc:9091/api/v1/query?query=cluster:ingress_controller_aws_nlb_active:sum' | jq
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "cluster:ingress_controller_aws_nlb_active:sum",
          "prometheus": "openshift-monitoring/k8s"
        },
        "value": [
          1645712191.009,
          "0"
        ]
      }
    ]
  }
}

but no result from telemeter server: https://infogw-proxy.api.openshift.com/
reason is the metrics is not added to telemeter server allowlist, see from
https://github.com/rhobs/configuration/blob/main/configuration/telemeter/metrics.json

#  oc -n openshift-monitoring get cm telemetry-config -o jsonpath="{.data.metrics\.yaml}"
matches:
...
#
# owners: (@openshift/network-edge)
#
# cluster:ingress_controller_aws_nlb_active:sum informs how many NLBs are active in AWS.
# Zero would indicate ELB (legacy). This metric is only emitted on AWS.
- '{__name__="cluster:ingress_controller_aws_nlb_active:sum"}'
#
...

OpenShift release version:
4.10.0-0.nightly-2022-02-22-093600

Cluster Platform:
AWS

How reproducible:
always

Steps to Reproduce (in detail):
1. AWS cluster, search "cluster:ingress_controller_aws_nlb_active:sum" from local cluster
2. check in telemeter server 
3.


Actual results:
no result from telemeter server: https://infogw-proxy.api.openshift.com/

Expected results:
should see the result

Impact of the problem:


Additional info:
Docs on the process on why and how to send these metrics:
https://docs.google.com/document/d/1a6n5iBGM2QaIQRg9Lw4-Npj6QY9--Hpx3XYut-BrUSY/edit#heading=h.i05nm4wmvn7x

Comment 1 Miciah Dashiel Butler Masters 2022-02-24 17:35:07 UTC
Thomas, I believe this is related to <https://issues.redhat.com/browse/NE-704>, which you implemented with <https://github.com/openshift/cluster-ingress-operator/pull/675>.  Can you take a look at this BZ and assess priority and blocker status?

Comment 2 Thomas Jungblut 2022-02-25 12:19:13 UTC
setting blocker- PR for rhobs is here: https://github.com/rhobs/configuration/pull/157

Comment 3 Thomas Jungblut 2022-02-28 10:13:24 UTC
PR merged, asked for manual rollout in https://coreos.slack.com/archives/C010A637DGB/p1646043172619779

Comment 4 Thomas Jungblut 2022-02-28 14:46:54 UTC
was just deployed confirmed by Jessica

Comment 5 Thomas Jungblut 2022-02-28 14:48:52 UTC
@juzhao

Arvind iyenga


@

Comment 7 Thomas Jungblut 2022-03-02 07:35:27 UTC
ok, I can see metrics coming in from tenants on thanos now since the deployment happened:

https://telemeter-lts.datahub.redhat.com/graph?g0.range_input=1w&g0.max_source_resolution=auto&g0.expr=cluster%3Aingress_controller_aws_nlb_active%3Asum&g0.tab=0


Note You need to log in before you can comment on or make changes to this bug.