Bug 1971589 - [4.8.0] Telemetry-client won't report metrics in case the cluster was installed using the assisted operator
Summary: [4.8.0] Telemetry-client won't report metrics in case the cluster was install...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: assisted-installer
Version: 4.8
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ---
: 4.8.0
Assignee: Yoni Bettan
QA Contact: bjacot
URL:
Whiteboard: AI-Team-Core
Depends On: 1971312
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-06-14 11:29 UTC by Ronnie Lazar
Modified: 2021-07-27 23:13 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1971312
Environment:
Last Closed: 2021-07-27 23:12:53 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift assisted-service pull 2003 0 None open Bug 1971589: Change default telemeter server of installed clusters to prod. (#1988) 2021-06-15 10:06:00 UTC
Red Hat Bugzilla 1971312 1 urgent CLOSED [master] Telemetry-client won't report metrics in case the cluster was installed using the assisted operator 2022-08-28 08:47:34 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 23:13:09 UTC

Description Ronnie Lazar 2021-06-14 11:29:13 UTC
+++ This bug was initially created as a clone of Bug #1971312 +++

Description of problem:

The telemetry-client isn't forwarding metrics 
Version-Release number of selected component (if applicable):


How reproducible:
100%

Steps to Reproduce:
1. Install a cluster using the assisted-service-operator
2.
3.

Actual results:

level=info caller=main.go:259 ts=2021-06-12T02:01:12.403052205Z msg="starting telemeter-client" from=https://prometheus-k8s.openshift-monitoring.svc:9091 to=https://dummy.com listen=localhost:8080
level=error caller=forwarder.go:268 ts=2021-06-13T11:03:07.595840144Z component=forwarder/worker msg="unable to forward results" err="unable to authorize to server: unable to parse the authentication response: invalid character '<' looking for beginning of value"
level=error caller=forwarder.go:268 ts=2021-06-13T11:04:07.998038114Z component=forwarder/worker msg="unable to forward results" err="unable to authorize to server: unable to parse the authentication response: invalid character '<' looking for beginning of value"
level=error caller=forwarder.go:268 ts=2021-06-13T11:05:08.389192787Z component=forwarder/worker msg="unable to forward results" err="unable to authorize to server: unable to parse the authentication response: invalid character '<' looking for beginning of value"

Expected results:

level=info caller=main.go:259 ts=2021-06-13T11:33:06.758863733Z msg="starting telemeter-client" from=https://prometheus-k8s.openshift-monitoring.svc:9091 to=https://infogw.api.openshift.com/ listen=localhost:8080

Additional info:

The root cause of this issue seems to be this PR https://github.com/openshift/assisted-service/pull/1773
The aim of the PR was to redirect the cluster metrics to the correct Telemeter server.
In case the cluster is installed using the assisted-service operator the service URL isn't "https://api.openshift.com" nor "https://api.stage.openshift.com" resulting a telemeter URL of:  "https://dummy.com"

--- Additional comment from ybettan on 20210613T15:26:49

What would be the correct URLs for the operator on different envs ?

--- Additional comment from ercohen on 20210613T16:22:34

I think the desired behavior when installing with the operator is to set the Telemeter URL to: https://infogw.api.openshift.com 
The operator/service should allow setting another URL but by default, it should get to prod.

Comment 2 Yoni Bettan 2021-06-21 13:25:34 UTC
Nir, in order to test it, in addition to checking the logs, please make sure that the metrics arrived to telemeter server and have the correct labels.

You can do it as follow:
1. go to https://telemeter-lts-dashboards.datahub.redhat.com/d/rFFxyJ5Mz/assisted-installer?panelId=16&fullscreen&orgId=1&from=now-7d&to=now and to https://telemeter-lts-dashboards.datahub.redhat.com/d/rFFxyJ5Mz/assisted-installer?panelId=41&fullscreen&orgId=1&from=now-7d&to=now
2. make sure both panels contain some cluster with the label "infrastructure-operator"

Comment 6 errata-xmlrpc 2021-07-27 23:12:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438


Note You need to log in before you can comment on or make changes to this bug.