Bug 1794616 - Prometheus federation write is failing on a large scale cluster
Summary: Prometheus federation write is failing on a large scale cluster
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.3.0
Hardware: Unspecified
OS: Linux
unspecified
medium
Target Milestone: ---
: 4.4.0
Assignee: Sergiusz Urbaniak
QA Contact: Junqi Zhao
URL:
Whiteboard: aos-scalability-43
Depends On:
Blocks: 1804574 1805116
TreeView+ depends on / blocked
 
Reported: 2020-01-24 02:50 UTC by Naga Ravi Chaitanya Elluri
Modified: 2020-05-04 11:27 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-04 11:26:35 UTC
Target Upstream Version:


Attachments (Terms of Use)
Total time series in an 800 nodes cluster: 8301 (55.19 KB, image/png)
2020-01-27 14:41 UTC, Sergiusz Urbaniak
no flags Details


Links
System ID Priority Status Summary Last Updated
Github coreos kube-prometheus pull 397 None closed jsonnet: add general rules for up/down targets 2020-07-22 14:53:54 UTC
Github openshift cluster-monitoring-operator pull 650 None closed Bug 1794616: replace up with count:up[0,1], remove node_uname_info 2020-07-22 14:53:54 UTC
Github openshift cluster-monitoring-operator pull 655 None closed Bug 1794616: pkg/manifests: set modified telemeter-client command line arguments 2020-07-22 14:53:54 UTC
Red Hat Product Errata RHBA-2020:0581 None None None 2020-05-04 11:27:08 UTC

Description Naga Ravi Chaitanya Elluri 2020-01-24 02:50:59 UTC
Description of problem:
we observed bunch of errors related to "caller=federate.go:184 component=web msg="federation failed" err="write tcp 127.0.0.1:9090->127.0.0.1:36894: write: broken pipe"" ( full log attached ) on a large scale 4.3 cluster ( 800 nodes ) built using 4.3.0-rc.3 payload. Looking at the logs, increasing the scrape_timeout and scrape_interval might help avoid this at large node counts. We are not exactly sure why the federation is enabled by default.

Version-Release number of selected component (if applicable):
OpenShift build/payload version: 4.3.0-rc.3

How reproducible:
We haven't seen this error before as we didn't really scale the cluster to large number of nodes.

Steps to Reproduce:
1. Build a cluster using 4.3.0-rc.3 payload.
2. Scale the cluster to larger node counts.
3. Look at one of the prom replica logs.

Actual results:
caller=federate.go:184 component=web msg="federation failed" err="write tcp 127.0.0.1:9090->127.0.0.1:36894: write: broken pipe"

Expected results:
Metrics scraped within the scrape_timeout interval.

Additional info:
Full logs are at http://dell-r510-01.perf.lab.eng.rdu2.redhat.com/4.3/logs/prometheus_scrape/

Comment 1 Sergiusz Urbaniak 2020-01-27 14:41:46 UTC
Created attachment 1655695 [details]
Total time series in an 800 nodes cluster: 8301

Comment 15 Junqi Zhao 2020-02-19 09:10:18 UTC
Tested with 4.4.0-0.nightly-2020-02-17-211020,
the "up" metric is replaced by "count:up0" and "count:up1", "node_uname_info" is removed

Comment 17 errata-xmlrpc 2020-05-04 11:26:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.