Bug 2066457 - Prometheus CI failure: 503 Service Unavailable
Summary: Prometheus CI failure: 503 Service Unavailable
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.11
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.11.0
Assignee: Simon Pasquier
QA Contact: Junqi Zhao
Depends On:
Blocks: 2077516
TreeView+ depends on / blocked
Reported: 2022-03-21 20:01 UTC by mkenigsb
Modified: 2022-08-10 10:55 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Last Closed: 2022-08-10 10:55:17 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift origin pull 26984 0 None open Bug 2066457: don't fail when a query returns warnings 2022-04-07 08:33:25 UTC
Red Hat Product Errata RHSA-2022:5069 0 None None None 2022-08-10 10:55:55 UTC

Description mkenigsb 2022-03-21 20:01:08 UTC
Description of problem:
e2e-agnostic-upgrade failed with:

{  fail [github.com/openshift/origin/test/e2e/upgrade/alert/alert.go:212]: unable to check firing alerts during upgrade
Unexpected error:
    <errors.aggregate | len:1, cap:1>: [
            s: "receive series from Addr: LabelSets:  Mint: -62167219200000 Maxt: 9223372036854775807: rpc error: code = Unknown desc = query Prometheus: request failed with code 503 Service Unavailable; msg Service Unavailable",
    receive series from Addr: LabelSets:  Mint: -62167219200000 Maxt: 9223372036854775807: rpc error: code = Unknown desc = query Prometheus: request failed with code 503 Service Unavailable; msg Service Unavailable

Prow job:

How reproducible:
Looked on search.ci.openshift.org and it looks like there have been quite a few failures with the same error

Comment 1 Nikolaos Leandros Moraitis 2022-03-23 12:06:00 UTC
@mkenigsb Please re-assign the bug to the responsible team. Thank you.

Comment 2 Colin Walters 2022-04-04 14:59:27 UTC
This is causing a quite high flake rate across certain job types.  Look at this query:

Specifically e.g.
periodic-ci-openshift-release-master-ci-4.11-upgrade-from-stable-4.10-e2e-azure-ovn-upgrade (all) - 7 runs, 100% failed, 43% of failures match = 43% impact

I can't speak to where this should fall in the team priority list but I think it's higher than "low".

Comment 3 Simon Pasquier 2022-04-05 08:15:00 UTC
Increasing priority on this one since it may be a regression in the Thanos sidecar reporting ready while it isn't.

Comment 13 errata-xmlrpc 2022-08-10 10:55:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.