Description of problem: When migration from MTC 1.5.1 to MTC 1.6.0 with indirect mode, the migration failed for PVC could not be found in StageBackup phase Version-Release number of selected component (if applicable): SOURCE CLUSTER: AWS OCP 3.11 (MTC 1.5.1) TARGET CLUSTER: AWS OCP 4.9 (MTC 1.6.0) REPLICATION REPOSITORY: AWS S3 Storage How reproducible: Always Steps to Reproduce: 1 Create a common nginx application in ocp3.11 source cluster $ ansible-playbook deploy-app.yml -e use_role=ocp-nginxpv -e namespace= ocp-22222-nginx 2 Create indirect migplan against ocp-22222-nginx in web console as normal 3. Trigger the migration Actual results: The migration failed for PVC could not be found in StageBackup phase Expected results: The migration should be successful Additional information: $ oc describe migmigration --namespace openshift-migration migration-ab233 Name: migration-ab233 Namespace: openshift-migration Labels: migration.openshift.io/migplan-name=nginx-gp2-gp2 Annotations: openshift.io/touch: bdfd3aaa-04bb-11ec-ad1f-0a580a81020f API Version: migration.openshift.io/v1alpha1 Kind: MigMigration Metadata: ………... Spec: Mig Plan Ref: Name: nginx-gp2-gp2 Namespace: openshift-migration Quiesce Pods: true Stage: false Status: Conditions: Category: Advisory Durable: true Last Transition Time: 2021-08-24T09:14:53Z Message: The migration has failed. See: Errors. Reason: AnnotateResources Status: True Type: Failed Errors: persistentvolumeclaims "nginx-logs-012345678901234567890123456789012345678901234567890123456789:nginx-logs-012345678901234567890123456789012345678901234567890123456789" not found Itinerary: Failed Observed Digest: 11ce3d95629b7d6e8f021c7de395d0c0247d505f4f56292744acd20a55b0301c Phase: Completed Pipeline: Completed: 2021-08-24T09:14:27Z Message: Completed Name: Prepare Started: 2021-08-24T09:14:22Z Completed: 2021-08-24T09:14:36Z Message: Completed Name: Backup Progress: Backup openshift-migration/migration-ab233-initial-hx6v8: 39 out of estimated total of 39 objects backed up (7s) Started: 2021-08-24T09:14:27Z Completed: 2021-08-24T09:14:53Z Failed: true Message: Failed Name: StageBackup Started: 2021-08-24T09:14:36Z Message: Skipped Name: StageRestore Skipped: true Message: Skipped Name: Restore Skipped: true Completed: 2021-08-24T09:14:53Z Message: Completed Name: Cleanup Started: 2021-08-24T09:14:53Z Completed: 2021-08-24T09:14:53Z Message: Completed Name: CleanupHelpers Started: 2021-08-24T09:14:53Z Start Timestamp: 2021-08-24T09:14:22Z Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Running 63s migmigration_controller Step: 2/48 Normal Running 63s migmigration_controller Step: 3/48 Normal Running 60s (x3 over 63s) migmigration_controller Step: 4/48 Normal Running 60s migmigration_controller Step: 5/48 Normal Ready 59s (x12 over 63s) migmigration_controller The migration is ready. Normal Running 59s (x2 over 60s) migmigration_controller Step: 6/48 Normal Running 59s migmigration_controller Step: 7/48 Normal Running 59s migmigration_controller Step: 8/48 Normal Running 59s migmigration_controller Step: 9/48 Normal Running 59s migmigration_controller Step: 10/48 Normal Running 59s migmigration_controller (combined from similar events): Step: 12/48 $ oc get migplan nginx-gp2-gp2 -o yaml apiVersion: migration.openshift.io/v1alpha1 kind: MigPlan metadata: ……….. name: nginx-gp2-gp2 namespace: openshift-migration resourceVersion: "197535" uid: bfcbe5f5-09e3-4501-8962-33ce1aed0213 spec: destMigClusterRef: name: host namespace: openshift-migration indirectImageMigration: true indirectVolumeMigration: true migStorageRef: name: automatic namespace: openshift-migration namespaces: - ocp-22222-nginx persistentVolumes: - capacity: 1Gi name: pvc-0727928a-04bb-11ec-9dee-0e7ac224013d proposedCapacity: 1Gi pvc: accessModes: - ReadWriteOnce hasReference: true name: nginx-logs-012345678901234567890123456789012345678901234567890123456789:nginx-logs-012345678901234567890123456789012345678901234567890123456789 namespace: ocp-22222-nginx selection: action: copy copyMethod: filesystem storageClass: gp2 storageClass: gp2 supported: actions: - skip - copy - move copyMethods: - filesystem - snapshot - capacity: 1Gi name: pvc-0747f0fd-04bb-11ec-9dee-0e7ac224013d proposedCapacity: 1Gi pvc: accessModes: - ReadWriteOnce hasReference: true name: nginx-html-012345678901234567890123456789012345678901234567890123456789:nginx-html-012345678901234567890123456789012345678901234567890123456789 namespace: ocp-22222-nginx selection: action: copy copyMethod: filesystem storageClass: gp2 storageClass: gp2 supported: actions: - skip - copy - move copyMethods: - filesystem - snapshot srcMigClusterRef: name: source-cluster namespace: openshift-migration status: conditions: - category: Required lastTransitionTime: "2021-08-24T09:11:40Z" message: The `persistentVolumes` list has been updated with discovered PVs. reason: Done status: "True" type: PvsDiscovered - category: Required lastTransitionTime: "2021-08-24T09:11:40Z" message: The storage resources have been created. reason: Done status: "True" type: StorageEnsured - category: Required lastTransitionTime: "2021-08-24T09:11:40Z" message: The migration plan is ready. status: "True" type: Ready - category: Warn lastTransitionTime: "2021-08-24T09:12:31Z" message: 'Failed to compute PV resizing data for the following volumes. PV resizing will be disabled for these volumes and the migration may fail if the volumes are full or their requested and actual capacities differ in the source cluster. Please ensure that the volumes are attached to one or more running Pods for PV resizing to work correctly: [pvc-0727928a-04bb-11ec-9dee-0e7ac224013d,pvc-0747f0fd-04bb-11ec-9dee-0e7ac224013d]' reason: NotDone status: "True" type: PvUsageAnalysisFailed destStorageClasses: - accessModes: - ReadWriteOnce default: true name: gp2 provisioner: kubernetes.io/aws-ebs - accessModes: - ReadWriteOnce name: gp2-csi provisioner: ebs.csi.aws.com - accessModes: - ReadWriteOnce name: ocs-storagecluster-ceph-rbd provisioner: openshift-storage.rbd.csi.ceph.com - accessModes: - ReadWriteOnce - ReadOnlyMany - ReadWriteMany name: ocs-storagecluster-cephfs provisioner: openshift-storage.cephfs.csi.ceph.com - accessModes: - ReadWriteOnce name: openshift-storage.noobaa.io provisioner: openshift-storage.noobaa.io/obc …….. [In source cluster] $ oc get pvc -n ocp-22222-nginx NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE nginx-html-012345678901234567890123456789012345678901234567890123456789 Bound pvc-0747f0fd-04bb-11ec-9dee-0e7ac224013d 1Gi RWO gp2 30s nginx-logs-012345678901234567890123456789012345678901234567890123456789 Bound pvc-0727928a-04bb-11ec-9dee-0e7ac224013d 1Gi RWO gp2 30s $ oc get all -n ocp-22222-nginx NAME READY STATUS RESTARTS AGE pod/nginx-deployment-56477766-46dbv 1/1 Running 0 27s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/my-nginx ClusterIP 172.30.94.94 <none> 8081/TCP 27s NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE deployment.apps/nginx-deployment 1 1 1 1 27s NAME DESIRED CURRENT READY AGE replicaset.apps/nginx-deployment-56477766 1 1 1 27s NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD route.route.openshift.io/my-nginx my-nginx-ocp-22222-nginx.apps.0824-kya.qe.rhcloud.com my-nginx 8081 None
This was fixed in a PR that was submitted after the image build that was used here, and before this BZ was created, so it did not kick in the usual BZ automation. I'm manually moving this to MODIFIED to reflect that it has been merged into the v1.6.0 release branch, but has not been built yet. https://github.com/konveyor/mig-controller/pull/1175
Verfied using: SOURCE CLUSTER: AWS OCP 3.11 (MTC 1.5.1) TARGET CLUSTER: AWS OCP 4.8 (MTC 1.6.0) image: registry.redhat.io/rhmtc/openshift-migration-controller-rhel8@sha256:61b1303933af81fed2390fd2f24f098ecf3a96693a1caab7988c4575c03b2132 Moved to VERIFIED status.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Migration Toolkit for Containers (MTC) 1.6.0 security & bugfix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3694