Bug 1997051 - Indirect migration from MTC 1.5.1 to 1.6.0 fails during "StageBackup" phase
Summary: Indirect migration from MTC 1.5.1 to 1.6.0 fails during "StageBackup" phase
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Migration Toolkit for Containers
Classification: Red Hat
Component: General
Version: 1.6.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 1.6.0
Assignee: Pranav Gaikwad
QA Contact: Xin jiang
Avital Pinnick
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-08-24 10:21 UTC by whu
Modified: 2021-09-29 14:35 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-09-29 14:35:20 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2021:3694 0 None None None 2021-09-29 14:35:31 UTC

Description whu 2021-08-24 10:21:28 UTC
Description of problem:
When migration from  MTC 1.5.1 to MTC 1.6.0 with indirect mode, the migration failed for PVC could not be found in StageBackup phase

Version-Release number of selected component (if applicable):
SOURCE CLUSTER: AWS OCP 3.11 (MTC 1.5.1)
TARGET CLUSTER: AWS OCP 4.9 (MTC 1.6.0)
REPLICATION REPOSITORY: AWS S3 Storage

How reproducible:
Always

Steps to Reproduce:
1 Create a common nginx application in ocp3.11 source cluster
$ ansible-playbook deploy-app.yml -e use_role=ocp-nginxpv -e namespace= ocp-22222-nginx

2 Create indirect migplan against ocp-22222-nginx in web console as normal

3. Trigger the migration

Actual results:
The migration failed for PVC could not be found in  StageBackup phase

Expected results:
The  migration  should be successful

Additional information:
$ oc describe migmigration --namespace openshift-migration migration-ab233
Name:         migration-ab233
Namespace:    openshift-migration
Labels:       migration.openshift.io/migplan-name=nginx-gp2-gp2
Annotations:  openshift.io/touch: bdfd3aaa-04bb-11ec-ad1f-0a580a81020f
API Version:  migration.openshift.io/v1alpha1
Kind:         MigMigration
Metadata:
      ………...
Spec:
  Mig Plan Ref:
    Name:        nginx-gp2-gp2
    Namespace:   openshift-migration
  Quiesce Pods:  true
  Stage:         false
Status:
  Conditions:
    Category:              Advisory
    Durable:               true
    Last Transition Time:  2021-08-24T09:14:53Z
    Message:               The migration has failed.  See: Errors.
    Reason:                AnnotateResources
    Status:                True
    Type:                  Failed
  Errors:
    persistentvolumeclaims "nginx-logs-012345678901234567890123456789012345678901234567890123456789:nginx-logs-012345678901234567890123456789012345678901234567890123456789" not found
  Itinerary:        Failed
  Observed Digest:  11ce3d95629b7d6e8f021c7de395d0c0247d505f4f56292744acd20a55b0301c
  Phase:            Completed
  Pipeline:
    Completed:  2021-08-24T09:14:27Z
    Message:    Completed
    Name:       Prepare
    Started:    2021-08-24T09:14:22Z
    Completed:  2021-08-24T09:14:36Z
    Message:    Completed
    Name:       Backup
    Progress:
      Backup openshift-migration/migration-ab233-initial-hx6v8: 39 out of estimated total of 39 objects backed up (7s)
    Started:        2021-08-24T09:14:27Z
    Completed:      2021-08-24T09:14:53Z
    Failed:         true
    Message:        Failed
    Name:           StageBackup
    Started:        2021-08-24T09:14:36Z
    Message:        Skipped
    Name:           StageRestore
    Skipped:        true
    Message:        Skipped
    Name:           Restore
    Skipped:        true
    Completed:      2021-08-24T09:14:53Z
    Message:        Completed
    Name:           Cleanup
    Started:        2021-08-24T09:14:53Z
    Completed:      2021-08-24T09:14:53Z
    Message:        Completed
    Name:           CleanupHelpers
    Started:        2021-08-24T09:14:53Z
  Start Timestamp:  2021-08-24T09:14:22Z
Events:
  Type    Reason   Age                 From                     Message
  ----    ------   ----                ----                     -------
  Normal  Running  63s                 migmigration_controller  Step: 2/48
  Normal  Running  63s                 migmigration_controller  Step: 3/48
  Normal  Running  60s (x3 over 63s)   migmigration_controller  Step: 4/48
  Normal  Running  60s                 migmigration_controller  Step: 5/48
  Normal  Ready    59s (x12 over 63s)  migmigration_controller  The migration is ready.
  Normal  Running  59s (x2 over 60s)   migmigration_controller  Step: 6/48
  Normal  Running  59s                 migmigration_controller  Step: 7/48
  Normal  Running  59s                 migmigration_controller  Step: 8/48
  Normal  Running  59s                 migmigration_controller  Step: 9/48
  Normal  Running  59s                 migmigration_controller  Step: 10/48
  Normal  Running  59s                 migmigration_controller  (combined from similar events): Step: 12/48


$ oc get migplan nginx-gp2-gp2 -o yaml
apiVersion: migration.openshift.io/v1alpha1
kind: MigPlan
metadata:
 ………..
  name: nginx-gp2-gp2
  namespace: openshift-migration
  resourceVersion: "197535"
  uid: bfcbe5f5-09e3-4501-8962-33ce1aed0213
spec:
  destMigClusterRef:
    name: host
    namespace: openshift-migration
  indirectImageMigration: true
  indirectVolumeMigration: true
  migStorageRef:
    name: automatic
    namespace: openshift-migration
  namespaces:
  - ocp-22222-nginx
  persistentVolumes:
  - capacity: 1Gi
    name: pvc-0727928a-04bb-11ec-9dee-0e7ac224013d
    proposedCapacity: 1Gi
    pvc:
      accessModes:
      - ReadWriteOnce
      hasReference: true
      name: nginx-logs-012345678901234567890123456789012345678901234567890123456789:nginx-logs-012345678901234567890123456789012345678901234567890123456789
      namespace: ocp-22222-nginx
    selection:
      action: copy
      copyMethod: filesystem
      storageClass: gp2
    storageClass: gp2
    supported:
      actions:
      - skip
      - copy
      - move
      copyMethods:
      - filesystem
      - snapshot
  - capacity: 1Gi
    name: pvc-0747f0fd-04bb-11ec-9dee-0e7ac224013d
    proposedCapacity: 1Gi
    pvc:
      accessModes:
      - ReadWriteOnce
      hasReference: true
      name: nginx-html-012345678901234567890123456789012345678901234567890123456789:nginx-html-012345678901234567890123456789012345678901234567890123456789
      namespace: ocp-22222-nginx
    selection:
      action: copy
      copyMethod: filesystem
      storageClass: gp2
    storageClass: gp2
    supported:
      actions:
      - skip
      - copy
      - move
      copyMethods:
      - filesystem
      - snapshot
  srcMigClusterRef:
    name: source-cluster
    namespace: openshift-migration
status:
  conditions:
  - category: Required
    lastTransitionTime: "2021-08-24T09:11:40Z"
    message: The `persistentVolumes` list has been updated with discovered PVs.
    reason: Done
    status: "True"
    type: PvsDiscovered
  - category: Required
    lastTransitionTime: "2021-08-24T09:11:40Z"
    message: The storage resources have been created.
    reason: Done
    status: "True"
    type: StorageEnsured
  - category: Required
    lastTransitionTime: "2021-08-24T09:11:40Z"
    message: The migration plan is ready.
    status: "True"
    type: Ready
  - category: Warn
    lastTransitionTime: "2021-08-24T09:12:31Z"
    message: 'Failed to compute PV resizing data for the following volumes. PV resizing will be disabled for these volumes and the migration may fail if the volumes are full or their requested and actual capacities differ in the source cluster. Please ensure that the volumes are attached to one or more running Pods for PV resizing to work correctly: [pvc-0727928a-04bb-11ec-9dee-0e7ac224013d,pvc-0747f0fd-04bb-11ec-9dee-0e7ac224013d]'
    reason: NotDone
    status: "True"
    type: PvUsageAnalysisFailed
  destStorageClasses:
  - accessModes:
    - ReadWriteOnce
    default: true
    name: gp2
    provisioner: kubernetes.io/aws-ebs
  - accessModes:
    - ReadWriteOnce
    name: gp2-csi
    provisioner: ebs.csi.aws.com
  - accessModes:
    - ReadWriteOnce
    name: ocs-storagecluster-ceph-rbd
    provisioner: openshift-storage.rbd.csi.ceph.com
  - accessModes:
    - ReadWriteOnce
    - ReadOnlyMany
    - ReadWriteMany
    name: ocs-storagecluster-cephfs
    provisioner: openshift-storage.cephfs.csi.ceph.com
  - accessModes:
    - ReadWriteOnce
    name: openshift-storage.noobaa.io
    provisioner: openshift-storage.noobaa.io/obc
  ……..


[In source cluster]
$ oc get pvc -n ocp-22222-nginx
NAME                                                                      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
nginx-html-012345678901234567890123456789012345678901234567890123456789   Bound     pvc-0747f0fd-04bb-11ec-9dee-0e7ac224013d   1Gi        RWO            gp2            30s
nginx-logs-012345678901234567890123456789012345678901234567890123456789   Bound     pvc-0727928a-04bb-11ec-9dee-0e7ac224013d   1Gi        RWO            gp2            30s

$ oc get all -n ocp-22222-nginx
NAME                                  READY     STATUS    RESTARTS   AGE
pod/nginx-deployment-56477766-46dbv   1/1       Running   0          27s

NAME               TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/my-nginx   ClusterIP   172.30.94.94   <none>        8081/TCP   27s

NAME                               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-deployment   1         1         1            1           27s

NAME                                        DESIRED   CURRENT   READY     AGE
replicaset.apps/nginx-deployment-56477766   1         1         1         27s

NAME                                HOST/PORT                                               PATH      SERVICES   PORT      TERMINATION   WILDCARD
route.route.openshift.io/my-nginx   my-nginx-ocp-22222-nginx.apps.0824-kya.qe.rhcloud.com             my-nginx   8081                    None

Comment 1 Erik Nelson 2021-08-25 14:32:30 UTC
This was fixed in a PR that was submitted after the image build that was used here, and before this BZ was created, so it did not kick in the usual BZ automation. I'm manually moving this to MODIFIED to reflect that it has been merged into the v1.6.0 release branch, but has not been built yet.

https://github.com/konveyor/mig-controller/pull/1175

Comment 5 Prasad Joshi 2021-09-20 09:28:15 UTC
Verfied using:
SOURCE CLUSTER: AWS OCP 3.11 (MTC 1.5.1)
TARGET CLUSTER: AWS OCP 4.8 (MTC 1.6.0)

image: registry.redhat.io/rhmtc/openshift-migration-controller-rhel8@sha256:61b1303933af81fed2390fd2f24f098ecf3a96693a1caab7988c4575c03b2132

Moved to VERIFIED status.

Comment 7 errata-xmlrpc 2021-09-29 14:35:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Migration Toolkit for Containers (MTC) 1.6.0 security & bugfix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3694


Note You need to log in before you can comment on or make changes to this bug.