The router needs to retry updating a route's status when receiving a forbidden error (generally, the router should retry on all API errors).
+++ This bug was initially created as a clone of Bug #1780398 +++
Description of problem:
Upgrading a cluster with 2K projects from 4.2.9 to 4.3.0-0.nightly-2019-12-05-073829 failed with monitoring hung for an hour.
--- Additional comment from Miciah Dashiel Butler Masters on 2019-12-06 18:58:26 UTC ---
The route in question was created at time 2019-12-05T20:09:20Z. From 2019-12-05T20:09:41Z to 2019-12-05T20:12:15Z, the API was returning "forbidden: not yet ready to handle request" errors (not only to the router: I see the same error in logs for the auth operator, samples operator, CVO, and kube-controller-manager). The router may have admitted the route, but the router failed to update the route's status due to the API outage. The router is designed to be able to function with limited privileges. It is not clear to me whether the router should retry on forbidden errors, or whether the API is incorrect in returning a forbidden error for a request that should be retried.
verification blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1778904
Verified on registry.svc.ci.openshift.org/ocp/release:4.3.0-0.nightly-2019-12-19-050538
Multiple upgrades successful without hitting this issue.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.