Bug 1948771 - ~50% of GCP upgrade jobs in 4.8 failing with "AggregatedAPIDown" alert on packages.coreos.com
Summary: ~50% of GCP upgrade jobs in 4.8 failing with "AggregatedAPIDown" alert on pac...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.8
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.8.0
Assignee: Kevin Rizza
QA Contact: Jian Zhang
URL:
Whiteboard:
: 1949115 (view as bug list)
Depends On:
Blocks: 1949115
TreeView+ depends on / blocked
 
Reported: 2021-04-12 21:14 UTC by Clayton Coleman
Modified: 2021-07-27 23:00 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-07-27 22:59:54 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift operator-framework-olm pull 51 0 None closed Bug 1948771: Revert "set packageserver replicas to 1 for single node" 2021-04-14 04:49:33 UTC
Github operator-framework operator-lifecycle-manager pull 2095 0 None closed Revert "set packageserver replicas to 1 for single node" 2021-04-14 04:49:35 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 23:00:11 UTC

Description Clayton Coleman 2021-04-12 21:14:24 UTC
As much as 50% (possibly a bit lower) of jobs that upgrade GCP in PRs are failing because of the following alert firing during the run:

disruption_tests: [sig-arch] Check if alerts are firing during or after upgrade success expand_less 	1h11m50s
Apr 12 16:28:39.842: Unexpected alerts fired or pending during the upgrade:

alert AggregatedAPIDown fired for 136 seconds with labels: {name="v1.packages.operators.coreos.com", namespace="default", severity="warning"}

https://search.ci.openshift.org/?search=AggregatedAPIDown.*packages&maxAge=120h&context=1&type=bug%2Bjunit&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

It looks like this started happening in the last 48h (so sometime after Friday 4/9 and before Sunday 4/11).

Marking urgent because most repos using GCP upgrade jobs are failing.  Appears to be impacting AWS and vSphere jobs as well.

Comment 2 tflannag 2021-04-13 13:18:41 UTC
*** Bug 1949115 has been marked as a duplicate of this bug. ***

Comment 4 Jian Zhang 2021-04-14 09:51:36 UTC
I didn't find the AggregatedAPIDown alert of packageserver for the latest 12 hours. Verify it.

Comment 7 errata-xmlrpc 2021-07-27 22:59:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438


Note You need to log in before you can comment on or make changes to this bug.