Created attachment 1612391 [details] workload creation script Description of problem: Migration of 50 (or above) projects fails with restic timeout. Version-Release number of selected component (if applicable): # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.2.0-0.nightly-2019-09-04-090826 True False 47h Cluster version is 4.2.0-0.nightly-2019-09-04-090826 # oc describe pod/controller-manager-7485dfd885-6d74r | grep Image Image: quay.io/ocpmigrate/mig-controller:stable Image ID: quay.io/ocpmigrate/mig-controller@sha256:053523b27ab2ae23dc046ca6565271c62370cf29ff294b46cf138fa9c8859c52 # oc describe pod/controller-manager-7485dfd885-6d74r | grep Image Image: quay.io/ocpmigrate/mig-controller:stable Image ID: quay.io/ocpmigrate/mig-controller@sha256:053523b27ab2ae23dc046ca6565271c62370cf29ff294b46cf138fa9c8859c52 # oc describe pod/registry-migplan-sample-8r667-1-557hx | grep Image Image: registry:2 Image ID: docker.io/library/registry@sha256:b1165286043f2745f45ea637873d61939bff6d9a59f76539d6228abf79f87774 # oc describe pod/velero-7474bd69fd-x8f8f | grep Image Image: quay.io/ocpmigrate/migration-plugin:stable Image ID: quay.io/ocpmigrate/migration-plugin@sha256:4b2ddc8094db229c3edc621d3fc77f7ec77da9644b29eeca23569b02e6fbdc1c Image: quay.io/ocpmigrate/velero:stable Image ID: quay.io/ocpmigrate/velero@sha256:957725dec5f0fb6a46dee78bd49de9ec4ab66903eabb4561b62ad8f4ad9e6f05 # oc describe pod/migration-operator-bfdbbd946-tv7nh | grep Image Image: quay.io/ocpmigrate/mig-operator:stable Image ID: quay.io/ocpmigrate/mig-operator@sha256:89ee280659a198a106243b1a0494cb8d98b6b2dbc3f072118b902b989d3ea491 Image: quay.io/ocpmigrate/mig-operator:stable Image ID: quay.io/ocpmigrate/mig-operator@sha256:89ee280659a198a106243b1a0494cb8d98b6b2dbc3f072118b902b989d3ea491 How reproducible: always Steps to Reproduce: 1. run the attached script using cluster-loader program on OCP 3.11 cluster. This will create 50 projects and mongodb/nodejs apps under it. curl to the app route to test the workload 2. Create migration operator and velero on the OCP 3.11 cluster 3. Create migration controller and operator on OCP 4.2 cluster. 4. Configure the migration plan with the namespaces and pv information that needs to be migrated. 5. Start migration Actual results: Migration fails during stage backup. # oc describe migmigration Name: migmigration-sample Namespace: openshift-migration Labels: controller-tools.k8s.io=1.0 Annotations: kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"migration.openshift.io/v1alpha1","kind":"MigMigration","metadata":{"annotations":{},"labels":{"controller-tools.k8s.io":"1.... touch: a5709eab-66b5-4156-b11c-8143e1f83dda API Version: migration.openshift.io/v1alpha1 Kind: MigMigration Metadata: Creation Timestamp: 2019-09-06T00:29:42Z Generation: 21 Owner References: API Version: migration.openshift.io/v1alpha1 Kind: MigPlan Name: migplan-sample UID: 6b6c7b46-d03d-11e9-9644-0210457668b8 Resource Version: 816838 Self Link: /apis/migration.openshift.io/v1alpha1/namespaces/openshift-migration/migmigrations/migmigration-sample UID: 6b699ae7-d03d-11e9-9644-0210457668b8 Spec: Mig Plan Ref: Name: migplan-sample Namespace: openshift-migration Quiesce Pods: true Stage: false Status: Conditions: Category: Advisory Durable: true Last Transition Time: 2019-09-06T01:09:04Z Message: The migration has failed. See: Errors. Reason: StageRestoreFailed Status: True Type: Failed Errors: Restore: openshift-migration/migmigration-sample-fwhmm partially failed. Phase: Completed Start Timestamp: 2019-09-06T00:30:08Z Events: <none> Log messages under velero on the destination. # oc logs velero-7474bd69fd-x8f8f | grep restic | grep error time="2019-09-06T01:07:54Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-0/mongodb-1-rrwlk-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:07:54Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-1/mongodb-1-hxfll-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:07:54Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-10/mongodb-1-pvxvz-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:07:55Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-11/mongodb-1-qssc9-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:07:55Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-12/mongodb-1-8j86b-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:07:55Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-13/mongodb-1-z8fl9-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:07:55Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-14/mongodb-1-rb5mp-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:07:55Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-15/mongodb-1-slvzf-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:07:56Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-16/mongodb-1-nnr27-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:07:56Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-17/mongodb-1-jqnjd-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:07:56Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-18/mongodb-1-bgjjw-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:07:56Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-19/mongodb-1-mpq6v-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:07:56Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-2/mongodb-1-ns9db-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:07:57Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-20/mongodb-1-q566f-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:07:57Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-21/mongodb-1-lggjd-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:07:57Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-22/mongodb-1-bg948-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:07:57Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-23/mongodb-1-ktpwb-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:07:57Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-24/mongodb-1-c88qf-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:07:58Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-25/mongodb-1-wv6cg-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:07:58Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-26/mongodb-1-pmwch-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:07:58Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-27/mongodb-1-sttkb-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:07:58Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-28/mongodb-1-vbz92-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:07:58Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-29/mongodb-1-b49sq-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:07:59Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-3/mongodb-1-vm7b9-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:07:59Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-30/mongodb-1-gbfdp-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:07:59Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-31/mongodb-1-ltsr6-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:07:59Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-32/mongodb-1-tdzd2-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:07:59Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-33/mongodb-1-hs9f8-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:08:00Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-34/mongodb-1-ks8z8-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:08:00Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-35/mongodb-1-jdc24-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:08:00Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-36/mongodb-1-krdkw-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:08:00Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-37/mongodb-1-qxvwh-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:08:00Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-38/mongodb-1-mmrb9-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:08:01Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-39/mongodb-1-5vhph-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:08:01Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-4/mongodb-1-zr4r2-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:08:01Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-40/mongodb-1-7wzwb-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:08:01Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-41/mongodb-1-8t7ws-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:08:01Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-42/mongodb-1-txfpm-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:08:02Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-43/mongodb-1-s8452-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:08:02Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-44/mongodb-1-2dxmh-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:08:02Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-45/mongodb-1-wdd5s-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:08:02Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-46/mongodb-1-5l7x5-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:08:02Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-47/mongodb-1-4fg5j-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:08:03Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-48/mongodb-1-bgf4v-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:08:03Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-49/mongodb-1-8lgv7-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:08:03Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-5/mongodb-1-mbzcm-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:08:03Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-6/mongodb-1-drmj8-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:08:03Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-7/mongodb-1-jwvhx-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:08:04Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-8/mongodb-1-jm2qd-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:08:04Z" level=error msg="Using default resource values, couldn't parse resource requirements: couldn't parse CPU request \"\": quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'." cmd=/velero logSource="pkg/restore/restic_restore_action.go:112" pluginName=velero pod=migtest-9/mongodb-1-szvtv-stage restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:09:03Z" level=error msg="unable to successfully complete restic restores of pod's volumes" error="timed out waiting for restic repository to become ready" logSource="pkg/restore/restore.go:1126" restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:09:03Z" level=error msg="unable to successfully complete restic restores of pod's volumes" error="timed out waiting for restic repository to become ready" logSource="pkg/restore/restore.go:1126" restore=openshift-migration/migmigration-sample-fwhmm time="2019-09-06T01:09:03Z" level=error msg="unable to successfully complete restic restores of pod's volumes" error="timed out waiting for restic repository to become ready" logSource="pkg/restore/restore.go:1126" restore=openshift-migration/migmigration-sample-fwhmm Expected results: Migration should succeed. Additional info:
The highest number of projects that migrated successfully was 25.
The error in the logs looks like it's coming from here: https://github.com/fusor/velero/blob/fusor-dev/pkg/restic/repository_ensurer.go#L180-L183 It looks like there is a 1 minute timeout for creation of the repository for each namespace. There are some debug statements around here that might give a better sense of what's actually going on. I can work on a PR to the operator allow getting debug output from velero.
This should make it possible to get debug output. https://github.com/fusor/mig-operator/pull/114
You should be able to enabled debug with the downstream images now. Add `velero_debug: true` in the MigrationController CR (on at least the backup side) and please reattempt a many namespace backup and attach the full velero container log.
Created attachment 1624737 [details] controller pod log Using the latest downstream images. I attempted a migration of 60 namespaces. When I attempted Stage before migration, it hung at "StagePodsCreated" stage for more than 18 hours. Attaching all the relevant logs.
Created attachment 1624749 [details] velero pod log
Hi, Jason I heard that developers were working on this bug and try to fix in OCP4.2. What about the current status? In another word, we will verify Doc bug https://bugzilla.redhat.com/show_bug.cgi?id=1757484 if dev have no plan to fix in current release. Thanks.
Hi Zhang, I need more logs with debugging to see if we can figure out what's going on. The last attachment does not look like it's showing the same problem. The one Roshni attached was hung up at, "StagePodsCreated", which is probably a different issue. It doesn't look like the velero/restic backup was even attempted. I'm suspicious that starting 60 stage pods might have caused a problem with resources (memory, cpu, storage, etc.), but it's impossible to say without more details. It is probably worthy of it's own bug for further investigation, though if it was a small cluster hitting resource limitations I'm not sure what we can do to guard against that at the moment. Still, I would say it should be investigated to make sure it isn't something we have control over.
Hi, Jason Thanks for your response. Base on the current status, I think it's OKay to fix or do more research in next release. But, this notes should be recorded in official doc. So, I will verify the doc bug https://bugzilla.redhat.com/show_bug.cgi?id=1757484
Created attachment 1626752 [details] velero debug log Hi Jason, Attaching the velero log after enabling debug log for the pod.
Moving to ON_QA so QE may verify and close.
I am marking this bug verified because I was not able reproduce the issue.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:0440