In CI, we're hitting errors like this: ``` level=error msg=Error: Error when reading or editing Target Pool "ci-op-x5j99sbj-82914-2f74l-api": googleapi: Error 403: Quota exceeded for quota group 'ReadGroup' and limit 'Read requests per 100 seconds' of service 'compute.googleapis.com' for consumer 'project_number:711936183532'., rateLimitExceeded ``` This was fixed in terraform-provider-google v3.62.0, however v3.62.0 uses v2 of the terraform sdk. The installer should be pointed to the openshift fork that contains 3.27.0 + the retry patches.
This may be difficult to reproduce. TRT has a wealth of data from which they can easily assess whether or not this fix has improved things. They should have enough data tomorrow to reach a conclusion on both effectiveness and whether or not these changes broke something, as such I welcome them to mark the bug VERIFIED once they have that data if QE hasn't been able to verify it independently.
We got 108 runs in the 24 hours since my 4.10 installer PR merged. GCP is doing a little better compared to the 24 hours before that: After PR: https://sippy.ci.openshift.org/sippy-ng/jobs/4.10/analysis?filters=%7B%22items%22%3A%5B%7B%22columnField%22%3A%22name%22%2C%22not%22%3Afalse%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%224.10-e2e-gcp%22%7D%2C%7B%22columnField%22%3A%22name%22%2C%22not%22%3Atrue%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%224.9%22%7D%2C%7B%22columnField%22%3A%22timestamp%22%2C%22operatorValue%22%3A%22%3E%22%2C%22value%22%3A%221638423780000%22%7D%5D%2C%22linkOperator%22%3A%22and%22%7D Before PR: https://sippy.ci.openshift.org/sippy-ng/jobs/4.10/analysis?filters=%7B%22items%22%3A%5B%7B%22columnField%22%3A%22name%22%2C%22not%22%3Afalse%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%224.10-e2e-gcp%22%7D%2C%7B%22columnField%22%3A%22name%22%2C%22not%22%3Atrue%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%224.9%22%7D%2C%7B%22columnField%22%3A%22timestamp%22%2C%22operatorValue%22%3A%22%3C%22%2C%22value%22%3A%221638423780000%22%7D%2C%7B%22columnField%22%3A%22timestamp%22%2C%22operatorValue%22%3A%22%3E%3D%22%2C%22value%22%3A%221638337380000%22%7D%5D%2C%22linkOperator%22%3A%22and%22%7D search.ci also confirms we didn't get any terraform-generated read quota messages from GCP in the last 24 hours: https://search.ci.openshift.org/chart?search=Error+when+reading.*403.*Quota+exceeded.*Read.*&maxAge=48h&context=1&type=bug%2Bjunit&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job Marking this verified based on this data and comment #2.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056