Bug 1977226 - [DOC] Azure storage account is disconnected when too many migplans are created
Summary: [DOC] Azure storage account is disconnected when too many migplans are created
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: Migration Toolkit for Containers
Classification: Red Hat
Component: Documentation
Version: 1.5.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 1.5.0
Assignee: Avital Pinnick
QA Contact: Xin jiang
Avital Pinnick
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-06-29 08:56 UTC by Sergio
Modified: 2021-06-30 11:51 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-06-30 11:51:44 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Sergio 2021-06-29 08:56:08 UTC
Description of problem:
When we create too many migplans, MTC reaches the requests limit for the Azure storage account and it starts rejecting requests.

Version-Release number of selected component (if applicable):
MTC 1.5.0

Even if it was detected in 1.5.0, probably it happens in all releases.

How reproducible:
Always

Steps to Reproduce:
1. Create 500 empty namespaces

$ for i in {1..500}; do oc new-project ocp-42100-$i; done

2. Create a migplan for evey namespace

# Fill in the right values in the yaml info

$ for i in {1..500}; do
    cat << EOF | oc create -f -
apiVersion: migration.openshift.io/v1alpha1
kind: MigPlan
metadata:
  labels:
    controller-tools.k8s.io: "1.0"
  name: migplan-ocp-42100-$i
  namespace: openshift-migration
spec:
  indirectImageMigration: true
  srcMigClusterRef:
    name: source_cluster
    namespace: openshift-migration
  destMigClusterRef:
    name:  host
    namespace: openshift-migration
  migStorageRef:
    name: automatic
    namespace: openshift-migration
  namespaces:
  - "ocp-42100-$i"
EOF
done



Actual results:
When 400 migplans or so become ready, the migration storage will begin to be disconnected, and we will find the following error in the migstorage resource

$ oc get migstorage -o yaml
...
  status:
    conditions:
    - category: Critical
      lastTransitionTime: "2021-06-28T16:28:31Z"
      message: 'The `backupStorageConfig` settings [storage.AccountsClient#ListKeys: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="TooManyRequests" Message="The request is being throttled as the limit has been reached for operation type - Read_ObservationWindow_00:05:00. For more information, see - https://aka.ms/srpthrottlinglimits"] not provided in secret openshift-migration/automatic-creds not valid.'
      reason: TestFailed
      status: "True"
      type: BackupStorageProviderTestFailed
    observedDigest: 0bd042adcc384904f0a1296b96ba737f75b9a09a35edc76e99222765e4d044ac


Expected results:
The number of requests to the Azure storage account should not depend on the number of idle migration plans.

Additional info:

When we close the migplans the storage account becomes ready again. We don't need to delete them, only to close them.

Comment 1 Avital Pinnick 2021-06-30 08:04:17 UTC
Will add to "Known issues" of release notes.

Comment 3 Sergio 2021-06-30 10:38:32 UTC
We have reviewed the docs PR. It looks good to me.

Thank you very much! we move the BZ to VERIFIED.

Comment 4 Avital Pinnick 2021-06-30 11:51:44 UTC
Changes merged.


Note You need to log in before you can comment on or make changes to this bug.