Bug 1977226
| Summary: | [DOC] Azure storage account is disconnected when too many migplans are created | ||
|---|---|---|---|
| Product: | Migration Toolkit for Containers | Reporter: | Sergio <sregidor> |
| Component: | Documentation | Assignee: | Avital Pinnick <apinnick> |
| Status: | CLOSED NEXTRELEASE | QA Contact: | Xin jiang <xjiang> |
| Severity: | medium | Docs Contact: | Avital Pinnick <apinnick> |
| Priority: | medium | ||
| Version: | 1.5.0 | CC: | ernelson, whu, xjiang |
| Target Milestone: | --- | ||
| Target Release: | 1.5.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-06-30 11:51:44 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Will add to "Known issues" of release notes. We have reviewed the docs PR. It looks good to me. Thank you very much! we move the BZ to VERIFIED. Changes merged. |
Description of problem: When we create too many migplans, MTC reaches the requests limit for the Azure storage account and it starts rejecting requests. Version-Release number of selected component (if applicable): MTC 1.5.0 Even if it was detected in 1.5.0, probably it happens in all releases. How reproducible: Always Steps to Reproduce: 1. Create 500 empty namespaces $ for i in {1..500}; do oc new-project ocp-42100-$i; done 2. Create a migplan for evey namespace # Fill in the right values in the yaml info $ for i in {1..500}; do cat << EOF | oc create -f - apiVersion: migration.openshift.io/v1alpha1 kind: MigPlan metadata: labels: controller-tools.k8s.io: "1.0" name: migplan-ocp-42100-$i namespace: openshift-migration spec: indirectImageMigration: true srcMigClusterRef: name: source_cluster namespace: openshift-migration destMigClusterRef: name: host namespace: openshift-migration migStorageRef: name: automatic namespace: openshift-migration namespaces: - "ocp-42100-$i" EOF done Actual results: When 400 migplans or so become ready, the migration storage will begin to be disconnected, and we will find the following error in the migstorage resource $ oc get migstorage -o yaml ... status: conditions: - category: Critical lastTransitionTime: "2021-06-28T16:28:31Z" message: 'The `backupStorageConfig` settings [storage.AccountsClient#ListKeys: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="TooManyRequests" Message="The request is being throttled as the limit has been reached for operation type - Read_ObservationWindow_00:05:00. For more information, see - https://aka.ms/srpthrottlinglimits"] not provided in secret openshift-migration/automatic-creds not valid.' reason: TestFailed status: "True" type: BackupStorageProviderTestFailed observedDigest: 0bd042adcc384904f0a1296b96ba737f75b9a09a35edc76e99222765e4d044ac Expected results: The number of requests to the Azure storage account should not depend on the number of idle migration plans. Additional info: When we close the migplans the storage account becomes ready again. We don't need to delete them, only to close them.