Back port https://github.com/thanos-io/thanos/pull/3146 which cuts latency by half for streaming read requests. This will significantly improve request latency of Thanos querier (including evaluations from Thanos ruler) when a query returns lots of series (e.g. several hundreds). The fix should be available in Thanos v0.16.0 upstream.
https://github.com/thanos-io/thanos/pull/2783 might also be useful.
Created attachment 1718402 [details] CPU utilization dashboard I've tested locally with Thanos v0.16.0-rc.0 and the performances are much better. For the same workload, the overall CPU usage of the prometheus pod goes from 700Mi (right side of the dashboard, 4.6 downstream version) to 450Mi (left side, upstream v0.16.0-rc.0).
tested with 4.7.0-0.nightly-2020-11-05-010603, Thanos sidecar performance is improved and thanos version is 0.16.0 # token=`oc sa get-token prometheus-k8s -n openshift-monitoring` # oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -g -H "Authorization: Bearer $token" 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=thanos_build_info' | jq { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "__name__": "thanos_build_info", "branch": "rhaos-4.7-rhel-8", "container": "oauth-proxy", "endpoint": "web", "goversion": "go1.15.2", "instance": "10.128.2.9:9091", "job": "thanos-querier", "namespace": "openshift-monitoring", "pod": "thanos-querier-66679569c6-gxf7p", "revision": "c8ee6fabf9fd5f3e44701f981204a1361965e59d", "service": "thanos-querier", "version": "0.16.0" }, "value": [ 1604559215.114, "1" ] }, ... } }
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633