Bug 1925740
Summary: | GCP jobs exhaust zone listing query quota sometimes due to too many initializations of cloud provider in tests | ||
---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Clayton Coleman <ccoleman> |
Component: | Test Infrastructure | Assignee: | Clayton Coleman <ccoleman> |
Status: | CLOSED ERRATA | QA Contact: | Jian Zhang <jiazha> |
Severity: | high | Docs Contact: | |
Priority: | medium | ||
Version: | 4.7 | CC: | jhou, jiazha, wking |
Target Milestone: | --- | ||
Target Release: | 4.7.z | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | No Doc Update | |
Doc Text: | Story Points: | --- | |
Clone Of: | 1920221 | Environment: | |
Last Closed: | 2021-03-10 11:24:01 UTC | Type: | --- |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | 1920221, 1926258 | ||
Bug Blocks: | 1926262 |
Description
Clayton Coleman
2021-02-06 03:40:54 UTC
*** Bug 1926258 has been marked as a duplicate of this bug. *** The link https://search.ci.openshift.org/?search=Quota+exceeded+for+quota+group&maxAge=168h&context=1&type=junit&name=4%5C.7&maxMatches=5&maxBytes=20971520&groupBy=job returns many rateLimitExceeded errors, I think this still needs to be looked into, moving to assigned We've had a bunch of rateLimitExceeded for other types [1]. Better query for the zone-listing this bug is trying to address is [2]. That gives three hits in release-openshift-origin-installer-e2e-gcp-ovn-upgrade-4.6-stable-to-4.7-ci, and I'm not sure if that uses the unpatched 4.6 suite or the patched 4.7 suite. But still, three 4.7 failures on this mode is a lot better than the 4.6 [3]. Moving back to ON_QA in case this is a convincing case for verification ;). [1]: https://github.com/openshift/release/pull/16256 [2]: https://search.ci.openshift.org/?search=listing+zones%3A+googleapi.*ListGroup.*rateLimitExceeded&maxAge=168h&context=1&type=junit&name=4%5C.7&maxMatches=5&maxBytes=20971520&groupBy=job [3]: https://search.ci.openshift.org/?search=listing+zones%3A+googleapi.*ListGroup.*rateLimitExceeded&maxAge=168h&context=1&type=junit&name=4%5C.6&maxMatches=5&maxBytes=20971520&groupBy=job When querying "Quota exceeded for quota group", it may return those tests that failures with "Error reading InstanceGroup Members: googleapi: Error 403: Quota exceeded for quota group 'ListGroup' and limit 'List requests per 100 seconds' of service 'compute.googleapis.com' for consumer 'project_number:1053217076791'., rateLimitExceeded". This failure is didferent with the origin issue: "response listing zones" So, I queried the 4.7 jobs with "listing zones: googleapi.*ListGroup.*rateLimitExceeded". Only some tests hit this failure in there test templates: release-openshift-origin-installer-e2e-gcp-ovn-upgrade-4.6-stable-to-4.7-ci release-openshift-origin-installer-e2e-gcp-upgrade-4.7 release-openshift-ocp-installer-e2e-gcp-ovn-4.7 Details: https://search.ci.openshift.org/?search=listing+zones%3A+googleapi.*ListGroup.*rateLimitExceeded&maxAge=168h&context=1&type=junit&name=4%5C.7&maxMatches=5&maxBytes=20971520&groupBy=job It's much better than 4.6 that without the fixed PR: https://search.ci.openshift.org/?search=listing+zones%3A+googleapi.*ListGroup.*rateLimitExceeded&maxAge=168h&context=1&type=junit&name=4%5C.6&maxMatches=5&maxBytes=20971520&groupBy=job LGTM, verify it. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.7.1 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:0678 This is just an issue for folks who run hundreds of GCP e2e jobs in the same account each day, so probably not customer facing. Setting as no-docs. |