Bug 1920455 - Migration status is "Succeeded" but the application did not migrate to GCP target cluster
Summary: Migration status is "Succeeded" but the application did not migrate to GCP ta...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Migration Toolkit for Containers
Classification: Red Hat
Component: General
Version: 1.4.0
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ---
: 1.5.0
Assignee: Dylan Murray
QA Contact: Xin jiang
Avital Pinnick
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-01-26 11:04 UTC by whu
Modified: 2023-09-15 00:59 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-07-28 04:08:00 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2021:2929 0 None None None 2021-07-28 04:08:07 UTC

Description whu 2021-01-26 11:04:54 UTC 
Description of problem:
Migrate basic nginx application from GCP OCP 3.11 to OCP 4.3 cluster by direct migration mode, the migration process completed successfully, but check nginx application in GCP OCP 4.3target cluster, there was nothing in the nginx namespace. 

Version-Release number of selected component (if applicable):
MTC 1.4.0
registry.stage.redhat.io/rhmtc/openshift-migration-log-reader-rhel8@sha256:6dbd4c4aa27dcaede49f68159b9923840732d67bfb4f14e4107e8ff28f56defa
source cluster: GCP OCP 3.11
target cluster: GCP OCP 4.3

How reproducible:
Always

Steps to Reproduce:
1. deploy nginx application in source cluster
oc process -p LOGS_ACCESSMODE=ReadWriteOnce  -p LOGS_STORAGECLASS=standard  -p HTML_ACCESSMODE=ReadWriteOnce  -p HTML_STORAGECLASS=standard -p namespace=ocp-00000-nginx -f https://gitlab.cee.redhat.com/app-mig/cam-helper/raw/master/ocp-24706/nginx_with_pv_template.yml  | oc create -f -

2. Create migration plan, choose direct migration mode, trigger migration 

3. After migration completed successfully, check the nginx application in target cluster

Actual results:
After migration completed successfully, there is nothing in nginx namespace

Expected results:
After migration completed successfully, the nginx should work in target cluster

Additional info:
After migration
[In source cluster]
$ oc get all -n ocp-00000-nginx
NAME               TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/my-nginx   ClusterIP   172.30.101.0   <none>        8081/TCP   1h

NAME                               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-deployment   0         0         0            0           1h

NAME                                          DESIRED   CURRENT   READY     AGE
replicaset.apps/nginx-deployment-557dd97bf8   0         0         0         1h

NAME                                HOST/PORT                                               PATH      SERVICES   PORT      TERMINATION   WILDCARD
route.route.openshift.io/my-nginx   my-nginx-ocp-00000-nginx.apps.0126-dt9.qe.rhcloud.com             my-nginx   8081                    None

[In target cluster]
$ oc get all -n ocp-00000-nginx
No resources found in ocp-00000-nginx namespace.

$ oc get pvc  -n ocp-00000-nginx
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
nginx-html   Bound    pvc-b616e669-6913-406b-b1e6-c723bf891f18   1Gi        RWO            standard       61m
nginx-logs   Bound    pvc-c3a6dc50-3ad2-4470-a1b6-b04c31a8f146   1Gi        RWO            standard       61m

$ oc get migmigration 09c3ebd0-5fbb-11eb-9617-11ad526c33f0 -o yaml
apiVersion: migration.openshift.io/v1alpha1
kind: MigMigration
metadata:
  .....
  labels:
    migration.openshift.io/migplan-name: ocp-00000-nginx-dvm
  name: 09c3ebd0-5fbb-11eb-9617-11ad526c33f0
  namespace: openshift-migration
  ..... 
status:
  conditions:
  - category: Advisory
    durable: true
    lastTransitionTime: "2021-01-26T09:45:45Z"
    message: The migration has completed successfully.
    reason: Completed
    status: "True"
    type: Succeeded
  itinerary: Final
  observedDigest: f3b4d0397ccd665980cc5a69468fc65ab170daf69d633495cc4a2c443da79f87
  phase: Completed
  pipeline:
  - completed: "2021-01-26T09:44:16Z"
    message: Completed
    name: Prepare
    started: "2021-01-26T09:44:09Z"
  - completed: "2021-01-26T09:44:28Z"
    message: Completed
    name: Backup
    progress:
    - 'Backup openshift-migration/09c3ebd0-5fbb-11eb-9617-11ad526c33f0-2h6x8: 37 out
      of estimated total of 37 objects backed up (6s)'
    started: "2021-01-26T09:44:16Z"
  - completed: "2021-01-26T09:44:33Z"
    message: Completed
    name: StageBackup
    started: "2021-01-26T09:44:28Z"
  - message: Skipped
    name: StageRestore
    skipped: true
  - completed: "2021-01-26T09:44:33Z"
    message: Completed
    name: DirectImage
    progress:
    - 0 total ImageStreams; 0 running; 0 successful; 0 failed
    started: "2021-01-26T09:44:33Z"
  - completed: "2021-01-26T09:45:34Z"
    message: Completed
    name: DirectVolume
    progress:
    - 'Rsync Client Pod ocp-00000-nginx/directvolumemigration-rsync-transfer-nginx-html:  100%
      completed'
    - 'Rsync Client Pod ocp-00000-nginx/directvolumemigration-rsync-transfer-nginx-logs:  100%
      completed'
    started: "2021-01-26T09:44:33Z"
  - completed: "2021-01-26T09:45:44Z"
    message: Completed
    name: Restore
    progress:
    - 'Restore openshift-migration/09c3ebd0-5fbb-11eb-9617-11ad526c33f0-lq64b: Completed'
    started: "2021-01-26T09:45:34Z"
  - completed: "2021-01-26T09:45:45Z"
    message: Completed
    name: Cleanup
    started: "2021-01-26T09:45:44Z"
  startTimestamp: "2021-01-26T09:44:09Z"

$ oc get dvm 09c3ebd0-5fbb-11eb-9617-11ad526c33f0-tmgdj  -o yaml
apiVersion: migration.openshift.io/v1alpha1
kind: DirectVolumeMigration
metadata:
  annotations:
    openshift.io/touch: 3c81add3-5fbb-11eb-aa22-0a580a82020f
  creationTimestamp: "2021-01-26T09:44:32Z"
  generateName: 09c3ebd0-5fbb-11eb-9617-11ad526c33f0-
  generation: 24
  labels:
    app.kubernetes.io/part-of: openshift-migration
    migmigration: 08f4566d-6001-487f-88fd-d15e1b75cb66
    migration-direct-volume: 08f4566d-6001-487f-88fd-d15e1b75cb66
  name: 09c3ebd0-5fbb-11eb-9617-11ad526c33f0-tmgdj
  namespace: openshift-migration
  ownerReferences:
  - apiVersion: migration.openshift.io/v1alpha1
    controller: true
    kind: MigMigration
    name: 09c3ebd0-5fbb-11eb-9617-11ad526c33f0
    uid: 08f4566d-6001-487f-88fd-d15e1b75cb66
  resourceVersion: "126080"
  selfLink: /apis/migration.openshift.io/v1alpha1/namespaces/openshift-migration/directvolumemigrations/09c3ebd0-5fbb-11eb-9617-11ad526c33f0-tmgdj
  uid: eb6d60c4-5326-4684-9e92-884ce8a04cbe
spec:
  createDestinationNamespaces: true
  destMigClusterRef:
    name: host
    namespace: openshift-migration
  persistentVolumeClaims:
  - name: nginx-logs
    namespace: ocp-00000-nginx
    targetAccessModes:
    - ReadWriteOnce
    targetStorageClass: standard
    verify: false
  - name: nginx-html
    namespace: ocp-00000-nginx
    targetAccessModes:
    - ReadWriteOnce
    targetStorageClass: standard
    verify: false
  srcMigClusterRef:
    name: source-cluster
    namespace: openshift-migration
status:
  conditions:
  - category: Advisory
    durable: true
    lastTransitionTime: "2021-01-26T09:45:34Z"
    message: The migration has succeeded
    reason: Completed
    status: "True"
    type: Succeeded
  itinerary: VolumeMigration
  observedDigest: f424539f304884932d94ebf358671a4b85bcc1c54421fed89bb413932d810bee
  phase: Completed
  phaseDescription: Complete
  startTimestamp: "2021-01-26T09:44:32Z"
  successfulPods:
  - lastObservedProgressPercent: 100%
    name: directvolumemigration-rsync-transfer-nginx-logs
    namespace: ocp-00000-nginx
  - lastObservedProgressPercent: 100%
    name: directvolumemigration-rsync-transfer-nginx-html
    namespace: ocp-00000-nginx

$ oc get dvmp -o yaml
apiVersion: v1
items:
- apiVersion: migration.openshift.io/v1alpha1
  kind: DirectVolumeMigrationProgress
  metadata:
    annotations:
      openshift.io/touch: 40fd57e3-5fc3-11eb-aa22-0a580a82020f
    creationTimestamp: "2021-01-26T09:44:40Z"
    generation: 5
    labels:
      app.kubernetes.io/part-of: openshift-migration
      directvolumemigration: eb6d60c4-5326-4684-9e92-884ce8a04cbe
    name: 3aecb48b078cdbc7101d708c59df570e
    namespace: openshift-migration
    ownerReferences:
    - apiVersion: migration.openshift.io/v1alpha1
      controller: true
      kind: DirectVolumeMigration
      name: 09c3ebd0-5fbb-11eb-9617-11ad526c33f0-tmgdj
      uid: eb6d60c4-5326-4684-9e92-884ce8a04cbe
    resourceVersion: "159880"
    selfLink: /apis/migration.openshift.io/v1alpha1/namespaces/openshift-migration/directvolumemigrationprogresses/3aecb48b078cdbc7101d708c59df570e
    uid: 5f5664de-cd3a-4f0c-8e9e-62687a6fd69e
  spec:
    clusterRef:
      name: source-cluster
      namespace: openshift-migration
    podRef:
      name: directvolumemigration-rsync-transfer-nginx-logs
      namespace: ocp-00000-nginx
  status:
    conditions:
    - category: Critical
      lastTransitionTime: "2021-01-26T09:45:22Z"
      message: 'The spec.podRef ocp-00000-nginx/directvolumemigration-rsync-transfer-nginx-logs
        must reference a valid `Pod` '
      reason: NotFound
      status: "True"
      type: InvalidPod
    lastObservedProgressPercent: 100%
    observedDigest: 43837eaa4b1844a2bab2c4dbda1ff8c2945f6bcc8612102d29ef02378eacce81
- apiVersion: migration.openshift.io/v1alpha1
  kind: DirectVolumeMigrationProgress
  metadata:
    annotations:
      openshift.io/touch: 40dc1f88-5fc3-11eb-aa22-0a580a82020f
    creationTimestamp: "2021-01-26T09:44:40Z"
    generation: 6
    labels:
      app.kubernetes.io/part-of: openshift-migration
      directvolumemigration: eb6d60c4-5326-4684-9e92-884ce8a04cbe
    name: 48ef53b5cae10e336f53f6c69ff5d2d5
    namespace: openshift-migration
    ownerReferences:
    - apiVersion: migration.openshift.io/v1alpha1
      controller: true
      kind: DirectVolumeMigration
      name: 09c3ebd0-5fbb-11eb-9617-11ad526c33f0-tmgdj
      uid: eb6d60c4-5326-4684-9e92-884ce8a04cbe
    resourceVersion: "159877"
    selfLink: /apis/migration.openshift.io/v1alpha1/namespaces/openshift-migration/directvolumemigrationprogresses/48ef53b5cae10e336f53f6c69ff5d2d5
    uid: 26d54453-b1b6-44f9-8020-8d9ecca53f78
  spec:
    clusterRef:
      name: source-cluster
      namespace: openshift-migration
    podRef:
      name: directvolumemigration-rsync-transfer-nginx-html
      namespace: ocp-00000-nginx
  status:
    conditions:
    - category: Critical
      lastTransitionTime: "2021-01-26T09:45:21Z"
      message: 'The spec.podRef ocp-00000-nginx/directvolumemigration-rsync-transfer-nginx-html
        must reference a valid `Pod` '
      reason: NotFound
      status: "True"
      type: InvalidPod
    lastObservedProgressPercent: 100%
    logMessage: |-
      2021/01/26 09:45:19 [1] sent 191 bytes  received 1.58K bytes
      sent 191 bytes  received 1.58K bytes  3.54K bytes/sec
      2021/01/26 09:45:19 [1] total size is 21  speedup is 0.01
      total size is 21  speedup is 0.01
    observedDigest: 0db0cb338cce102a89c911fb7deac8eca30fc239d6cba194522622944f1c6ba6
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""


Filter some errors in the log
[migration-log-reader in source cluster]
velero-5bd7fdd9d6-bpxmd velero time="2021-01-26T09:51:03Z" level=error msg="Error in syncHandler, re-adding item to queue" controller=backup-deletion error="error downloading backup: error copying Backup to temp file: rpc error: code = Unknown desc = storage: object doesn't exist" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/restore_controller.go:558" error.function=github.com/vmware-tanzu/velero/pkg/controller.downloadToTempFile key=openshift-migration/max-pvs-limit-mig-1608576299-dgtpj-8vjdt logSource="pkg/controller/generic_controller.go:140"
velero-5bd7fdd9d6-bpxmd velero time="2021-01-26T09:51:03Z" level=error msg="Error in syncHandler, re-adding item to queue" controller=backup-deletion error="error downloading backup: error copying Backup to temp file: rpc error: code = Unknown desc = storage: object doesn't exist" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/restore_controller.go:558" error.function=github.com/vmware-tanzu/velero/pkg/controller.downloadToTempFile key=openshift-migration/ocp-24871-datagrid-mig-1608582419-ljn9c-q2dw5 logSource="pkg/controller/generic_controller.go:140"
velero-5bd7fdd9d6-bpxmd velero time="2021-01-26T09:51:04Z" level=error msg="Error in syncHandler, re-adding item to queue" controller=backup-deletion error="error downloading backup: error copying Backup to temp file: rpc error: code = Unknown desc = storage: object doesn't exist" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/restore_controller.go:558" error.function=github.com/vmware-tanzu/velero/pkg/controller.downloadToTempFile key=openshift-migration/ocp-24871-datagrid-mig-1608582419-n88gc-wbknr logSource="pkg/controller/generic_controller.go:140"
velero-5bd7fdd9d6-bpxmd velero time="2021-01-26T09:51:04Z" level=error msg="Error in syncHandler, re-adding item to queue" controller=backup-deletion error="error downloading backup: error copying Backup to temp file: rpc error: code = Unknown desc = storage: object doesn't exist" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/restore_controller.go:558" error.function=github.com/vmware-tanzu/velero/pkg/controller.downloadToTempFile key=openshift-migration/ocp-29073-internalis-mig-1608560485-ctftx-686mj logSource="pkg/controller/generic_controller.go:140"
velero-5bd7fdd9d6-bpxmd velero time="2021-01-26T09:51:05Z" level=error msg="Error in syncHandler, re-adding item to queue" controller=backup-deletion error="error downloading backup: error copying Backup to temp file: rpc error: code = Unknown desc = storage: object doesn't exist" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/restore_controller.go:558" error.function=github.com/vmware-tanzu/velero/pkg/controller.downloadToTempFile key=openshift-migration/ocp-32189-cronjob-with-pvc-mig-1608572764-dkrz5-x7klg logSource="pkg/controller/generic_controller.go:140"


[migration-log-reader in target cluster]
velero-85575cc6b5-6v2f5 velero time="2021-01-26T09:45:37Z" level=warning msg="unable to restore additional item" additionalResource=persistentvolumeclaims additionalResourceName=nginx-logs additionalResourceNamespace=ocp-00000-nginx error="stat /tmp/750959154/resources/persistentvolumeclaims/namespaces/ocp-00000-nginx/nginx-logs.json: no such file or directory" logSource="pkg/restore/restore.go:1034" restore=openshift-migration/09c3ebd0-5fbb-11eb-9617-11ad526c33f0-lq64b
velero-85575cc6b5-6v2f5 velero time="2021-01-26T09:45:37Z" level=warning msg="unable to restore additional item" additionalResource=persistentvolumeclaims additionalResourceName=nginx-html additionalResourceNamespace=ocp-00000-nginx error="stat /tmp/750959154/resources/persistentvolumeclaims/namespaces/ocp-00000-nginx/nginx-html.json: no such file or directory" logSource="pkg/restore/restore.go:1034" restore=openshift-migration/09c3ebd0-5fbb-11eb-9617-11ad526c33f0-lq64b
velero-85575cc6b5-6v2f5 velero time="2021-01-26T09:48:02Z" level=error msg="Error in syncHandler, re-adding item to queue" controller=backup-deletion error="error downloading backup: error copying Backup to temp file: rpc error: code = Unknown desc = storage: object doesn't exist" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/restore_controller.go:558" error.function=github.com/vmware-tanzu/velero/pkg/controller.downloadToTempFile key=openshift-migration/ocp-32189-cronjob-with-pvc-mig-1608572764-dkrz5-tvf4l logSource="pkg/controller/generic_controller.go:140"
velero-85575cc6b5-6v2f5 velero time="2021-01-26T09:48:02Z" level=error msg="Error in syncHandler, re-adding item to queue" controller=backup-deletion error="error downloading backup: error copying Backup to temp file: rpc error: code = Unknown desc = storage: object doesn't exist" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/restore_controller.go:558" error.function=github.com/vmware-tanzu/velero/pkg/controller.downloadToTempFile key=openshift-migration/max-pvs-limit-mig-1608579231-mhlmt-s896r logSource="pkg/controller/generic_controller.go:140"
velero-85575cc6b5-6v2f5 velero time="2021-01-26T09:48:03Z" level=error msg="Error in syncHandler, re-adding item to queue" controller=backup-deletion error="error downloading backup: error copying Backup to temp file: rpc error: code = Unknown desc = storage: object doesn't exist" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/restore_controller.go:558" error.function=github.com/vmware-tanzu/velero/pkg/controller.downloadToTempFile key=openshift-migration/ocp-25022-cronjob-quiesced-mig-1608578651-m5g78-h5p2k logSource="pkg/controller/generic_controller.go:140"
velero-85575cc6b5-6v2f5 velero time="2021-01-26T09:48:03Z" level=error msg="Error in syncHandler, re-adding item to queue" controller=backup-deletion error="error downloading backup: error copying Backup to temp file: rpc error: code = Unknown desc = storage: object doesn't exist" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/restore_controller.go:558" error.function=github.com/vmware-tanzu/velero/pkg/controller.downloadToTempFile key=openshift-migration/ocp-34824-decrease-data-mig-1608583584-stage-phase-vsgxx-mhhtz logSource="pkg/controller/generic_controller.go:140"
velero-85575cc6b5-6v2f5 velero time="2021-01-26T09:48:03Z" level=error msg="Error in syncHandler, re-adding item to queue" controller=backup-deletion error="error downloading backup: error copying Backup to temp file: rpc error: code = Unknown desc = storage: object doesn't exist" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/restore_controller.go:558" error.function=github.com/vmware-tanzu/velero/pkg/controller.downloadToTempFile key=openshift-migration/ocp-24871-datagrid-mig-1608582419-n88gc-llvqr logSource="pkg/controller/generic_controller.go:140"
velero-85575cc6b5-6v2f5 velero time="2021-01-26T09:48:03Z" level=error msg="Error in syncHandler, re-adding item to queue" controller=backup-deletion error="error downloading backup: error copying Backup to temp file: rpc error: code = Unknown desc = storage: object doesn't exist" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/restore_controller.go:558" error.function=github.com/vmware-tanzu/velero/pkg/controller.downloadToTempFile key=openshift-migration/ocp-29073-internalis-mig-1608560485-ctftx-x4nrz logSource="pkg/controller/generic_controller.go:140"
velero-85575cc6b5-6v2f5 velero time="2021-01-26T09:48:04Z" level=error msg="Error in syncHandler, re-adding item to queue" controller=backup-deletion error="error downloading backup: error copying Backup to temp file: rpc error: code = Unknown desc = storage: object doesn't exist" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/restore_controller.go:558" error.function=github.com/vmware-tanzu/velero/pkg/controller.downloadToTempFile key=openshift-migration/max-pvs-limit-mig-1608576299-dgtpj-gl7rh logSource="pkg/controller/generic_controller.go:140"


But also get "The migration has completed successfully" log 
migration-controller-d76bd6468-v4ww9 mtc {"level":"info","ts":1611654345.820521,"logger":"migration|grxsd","msg":"CR","migration":"openshift-migration/09c3ebd0-5fbb-11eb-9617-11ad526c33f0","conditions":{"conditions":[{"type":"Succeeded","status":"True","reason":"Completed","category":"Advisory","message":"The migration has completed successfully.","lastTransitionTime":"2021-01-26T09:45:45Z","durable":true}]}}

$ oc get migrationcontroller  migration-controller -o yaml
apiVersion: migration.openshift.io/v1alpha1
kind: MigrationController
metadata:
  creationTimestamp: "2021-01-26T06:08:01Z"
  generation: 2
  name: migration-controller
  namespace: openshift-migration
  resourceVersion: "125097"
  selfLink: /apis/migration.openshift.io/v1alpha1/namespaces/openshift-migration/migrationcontrollers/migration-controller
  uid: 92e807d6-861a-4542-8c0d-7da8cced52cf
spec:
  azure_resource_group: ""
  cluster_name: host
  mig_namespace_limit: "10"
  mig_pod_limit: "100"
  mig_pv_limit: "100"
  migration_controller: true
  migration_log_reader: true
  migration_ui: true
  migration_velero: true
  olm_managed: true
  restic_timeout: 1h
  version: 1.0 (OLM)
status:
  phase: Reconciled
Comment 1 whu 2021-01-27 03:48:35 UTC 
In indirect migration mode, this issue exists too.

Version-Release number of selected component (if applicable):
MTC 1.4.0
registry.stage.redhat.io/rhmtc/openshift-migration-rhel7-operator@sha256:79f524931e7188bfbfddf1e3d23f491b627d691ef7849a42432c7aec2d5f8a54
source cluster: GCP OCP 3.11
target cluster: GCP OCP 4.3



$ oc get migplan mig-plan-00000-ocp-00000-nginx  -o yaml
apiVersion: migration.openshift.io/v1alpha1
kind: MigPlan
metadata:
  ......
  name: mig-plan-00000-ocp-00000-nginx
  namespace: openshift-migration
  ......
spec:
  destMigClusterRef:
    name: host
    namespace: openshift-migration
  indirectImageMigration: true
  indirectVolumeMigration: true
  migStorageRef:
    name: automatic
    namespace: openshift-migration
  namespaces:
  - ocp-00000-nginx
  persistentVolumes:
  - capacity: 1Gi
    name: pvc-ef257b85-604c-11eb-880f-0ededc9b8185
    pvc:
      accessModes:
      - ReadWriteOnce
      hasReference: true
      name: nginx-logs
      namespace: ocp-00000-nginx
    selection:
      action: copy
      copyMethod: filesystem
      storageClass: gp2
    storageClass: gp2
    supported:
      actions:
      - copy
      - move
      copyMethods:
      - filesystem
      - snapshot
  - capacity: 1Gi
    name: pvc-ef2ba5e8-604c-11eb-880f-0ededc9b8185
    pvc:
      accessModes:
      - ReadWriteOnce
      hasReference: true
      name: nginx-html
      namespace: ocp-00000-nginx
    selection:
      action: copy
      copyMethod: filesystem
      storageClass: gp2
    storageClass: gp2
    supported:
      actions:
      - copy
      - move
      copyMethods:
      - filesystem
      - snapshot
  srcMigClusterRef:
    name: source-cluster
    namespace: openshift-migration
status:
  conditions:
  ......
  - category: Required
    lastTransitionTime: "2021-01-27T03:23:54Z"
    message: The migration plan is ready.
    status: "True"
    type: Ready


$ oc get migmigration mig-migration-00000-ocp-00000-nginx -o yaml
apiVersion: migration.openshift.io/v1alpha1
kind: MigMigration
metadata:
 .....
  name: mig-migration-00000-ocp-00000-nginx
  namespace: openshift-migration
  ownerReferences:
  - apiVersion: migration.openshift.io/v1alpha1
    kind: MigPlan
    name: mig-plan-00000-ocp-00000-nginx
 .....
spec:
  migPlanRef:
    name: mig-plan-00000-ocp-00000-nginx
    namespace: openshift-migration
  quiescePods: true
  stage: false
status:
  conditions:
  - category: Advisory
    durable: true
    lastTransitionTime: "2021-01-27T03:30:58Z"
    message: The migration has completed successfully.
    reason: Completed
    status: "True"
    type: Succeeded
  itinerary: Final
  observedDigest: 9ecf816fba1c0c4560be4973ce222c7b625305241d652b30dfe3f718c64fc5e4
  phase: Completed
  pipeline:
  - completed: "2021-01-27T03:28:27Z"
    message: Completed
    name: Prepare
    started: "2021-01-27T03:27:52Z"
  - completed: "2021-01-27T03:28:44Z"
    message: Completed
    name: Backup
    progress:
    - 'Backup openshift-migration/mig-migration-00000-ocp-00000-nginx-t5zjc: 39 out
      of estimated total of 39 objects backed up (6s)'
    started: "2021-01-27T03:28:27Z"
  - completed: "2021-01-27T03:29:52Z"
    message: Completed
    name: StageBackup
    progress:
    - 'Backup openshift-migration/mig-migration-00000-ocp-00000-nginx-n746j: 7 out
      of estimated total of 7 objects backed up (11s)'
    - 'PodVolumeBackup openshift-migration/mig-migration-00000-ocp-00000-nginx-n746j-99t77:
      429 bytes out of 429 bytes backed up (4s)'
    - 'PodVolumeBackup openshift-migration/mig-migration-00000-ocp-00000-nginx-n746j-m6xtg:
      21 bytes out of 21 bytes backed up (3s)'
    started: "2021-01-27T03:28:44Z"
  - completed: "2021-01-27T03:30:36Z"
    message: Completed
    name: StageRestore
    progress:
    - 'Restore openshift-migration/mig-migration-00000-ocp-00000-nginx-84fwn: Completed'
    - stage-nginx-deployment-557dd97bf8-pmj4l-mfjhj is missing. Migration might fail
    started: "2021-01-27T03:29:52Z"
  - message: Skipped
    name: DirectImage
    skipped: true
  - message: Skipped
    name: DirectVolume
    skipped: true
  - completed: "2021-01-27T03:30:55Z"
    message: Completed
    name: Restore
    progress:
    - 'Restore openshift-migration/mig-migration-00000-ocp-00000-nginx-n4w7t: Completed'
    started: "2021-01-27T03:30:36Z"
  - completed: "2021-01-27T03:30:58Z"
    message: Completed
    name: Cleanup
    started: "2021-01-27T03:30:55Z"
  startTimestamp: "2021-01-27T03:27:52Z"


After migration in target cluster

$ oc get all -n ocp-00000-nginx
No resources found in ocp-00000-nginx namespace.

$ oc get pvc  -n ocp-00000-nginx
No resources found in ocp-00000-nginx namespace.


Filter error from log in target cluster

$ grep -i error ocp-00000-nginx-idvm-target.log
velero-85575cc6b5-bvk4m velero time="2021-01-27T03:30:47Z" level=warning msg="unable to restore additional item" additionalResource=persistentvolumeclaims additionalResourceName=nginx-logs additionalResourceNamespace=ocp-00000-nginx error="stat /tmp/764925421/resources/persistentvolumeclaims/namespaces/ocp-00000-nginx/nginx-logs.json: no such file or directory" logSource="pkg/restore/restore.go:1034" restore=openshift-migration/mig-migration-00000-ocp-00000-nginx-n4w7t
velero-85575cc6b5-bvk4m velero time="2021-01-27T03:30:47Z" level=warning msg="unable to restore additional item" additionalResource=persistentvolumeclaims additionalResourceName=nginx-html additionalResourceNamespace=ocp-00000-nginx error="stat /tmp/764925421/resources/persistentvolumeclaims/namespaces/ocp-00000-nginx/nginx-html.json: no such file or directory" logSource="pkg/restore/restore.go:1034" restore=openshift-migration/mig-migration-00000-ocp-00000-nginx-n4w7t


Also get "The migration has completed successfully." message from log

migration-controller-84799ff68c-mcl27 mtc {"level":"info","ts":1611718258.3558404,"logger":"migration|m4n64","msg":"CR","migration":"openshift-migration/mig-migration-00000-ocp-00000-nginx","conditions":{"conditions":[{"type":"Succeeded","status":"True","reason":"Completed","category":"Advisory","message":"The migration has completed successfully.","lastTransitionTime":"2021-01-27T03:30:58Z","durable":true}]}}
Comment 2 Xin jiang 2021-01-29 06:54:54 UTC 
I can reproduce the GCP issue in a fresh MTC clusters GCP 3.11->4.3.
The current behavior I observed:
Deploy an application with PVC
2.  Create a migplan with DIM+DVM
3, migration the application
result:
migration process completed successfully
only imagestream was migrated to target cluster, others were NOT migrated to target cluster



errors:
1.
time="2021-01-28T10:24:59Z" level=error msg="Error getting a backup store" backupstoragelocation=automatic-495v4 controller=backupstoragelocation error="rpc error: code = Unknown desc = google: error getting credentials using GOOGLE_APPLICATION_CREDENTIALS environment variable: unexpected end of JSON input" error.file="/go/src/github.com/konveyor/openshift-migration-velero-plugin-for-gcp/velero-plugin-for-gcp/object_store.go:87" error.function="main.(*ObjectStore).Init" logSource="pkg/controller/backupstoragelocation_controller.go:87"

2. 
time="2021-01-28T10:29:59Z" level=warning msg="unable to restore additional item" additionalResource=persistentvolumeclaims additionalResourceName=mysql additionalResourceNamespace=ocp-cakephpaaaa error="stat /tmp/411562558/resources/persistentvolumeclaims/namespaces/ocp-cakephpaaaa/mysql.json: no such file or directory" logSource="pkg/restore/restore.go:1034" restore=openshift-migration/776b9090-6153-11eb-a0b1-458bf8b2b8de-dm6vh
time="2021-01-28T10:29:59Z" level=info msg="Executing item action for pods" logSource="pkg/restore/restore.go:1002" restore=openshift-migration/776b9090-6153-11eb-a0b1-458bf8b2b8de-dm6vh
Comment 3 Xin jiang 2021-01-29 11:55:02 UTC 
We tried indirect migration on GCP 3.11->GCP 4.7, it worked fine.
Comment 4 Dylan Murray 2021-01-29 21:56:14 UTC 
So I'm a little confused based on what I've read here and in slack. It sounds like sometimes indirect migration is successful, and sometimes direct migration is unsuccessful. The error itself is GCP specific, that the BSL data references a secret that does not have a valid google credentials string. See more information here: https://github.com/vmware-tanzu/velero/issues/1530.

Please if you can reproduce this I would be curious about what data exists in a few resources:

`oc get backupstoragelocation -n openshift-migration`
`oc get migstorage -n openshift-migration`
and `oc get secret -n openshift-migration cloud-credentials -o yaml`

Please run these on both the source and destination clusters at the moment of failure that would be very helpful. Thank you.
Comment 5 Erik Nelson 2021-02-01 03:08:36 UTC 
Pusing to 1.4.1 based on Dylan's investigation Friday -- this is an upstream bug that appears to be some kind of race condition with how the replication repo data gets updated and how it's ultimately propagated to the related secret.
Comment 6 Xin jiang 2021-02-01 09:36:26 UTC 
Dylan, you are right. this error is GCP specific. The data.cloud is empty string.

$ oc get backupstoragelocation -n openshift-migration -o yaml
apiVersion: v1
items:
- apiVersion: velero.io/v1
  kind: BackupStorageLocation
  metadata:
    creationTimestamp: "2021-02-01T06:07:48Z"
    generateName: automatic-
    generation: 1
    labels:
      app.kubernetes.io/part-of: openshift-migration
      migplan: 8e31b300-20e5-4c02-a276-c94fb5b74b23
      migrationcontroller: edd9fa5c-ec77-4a05-88d4-cf3afdcfb02e
    name: automatic-bsr5m
    namespace: openshift-migration
    resourceVersion: "80131"
    selfLink: /apis/velero.io/v1/namespaces/openshift-migration/backupstoragelocations/automatic-bsr5m
    uid: 5aa7d1e8-5729-428e-82e3-3dc91e210dbb
  spec:
    objectStorage:
      bucket: camreplication
      prefix: velero
    provider: gcp
  status:
    lastSyncedTime: "2021-02-01T06:35:37Z"
    lastValidationTime: "2021-02-01T06:35:38Z"
    phase: Available
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""






$ oc get migstorage -n openshift-migration -o yaml
apiVersion: v1
items:
- apiVersion: migration.openshift.io/v1alpha1
  kind: MigStorage
  metadata:
    annotations:
      openshift.io/touch: dff6d7a9-6447-11eb-bfd3-0a580a810207
    creationTimestamp: "2021-02-01T04:42:21Z"
    generation: 2
    labels:
      controller-tools.k8s.io: "1.0"
    name: automatic
    namespace: openshift-migration
    resourceVersion: "47506"
    selfLink: /apis/migration.openshift.io/v1alpha1/namespaces/openshift-migration/migstorages/automatic
    uid: 32f9f044-f650-4aab-9042-704ff9f930bf
  spec:
    backupStorageConfig:
      credsSecretRef:
        name: automatic-creds
        namespace: openshift-migration
      gcpBucket: camreplication
    backupStorageProvider: gcp
    volumeSnapshotConfig:
      credsSecretRef:
        name: automatic-creds
        namespace: openshift-migration
    volumeSnapshotProvider: gcp
  status:
    conditions:
    - category: Required
      lastTransitionTime: "2021-02-01T04:42:22Z"
      message: The storage is ready.
      status: "True"
      type: Ready
    observedDigest: 6f0b70974a47a9343ea8f67e3949b1f5f6eef1be762fc90d3b41f54db1785766
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""



$ oc get secret -n openshift-migration cloud-credentials -o yaml
apiVersion: v1
data:
  cloud: ""
kind: Secret
metadata:
  creationTimestamp: "2021-02-01T04:40:02Z"
  name: cloud-credentials
  namespace: openshift-migration
  ownerReferences:
  - apiVersion: migration.openshift.io/v1alpha1
    kind: MigrationController
    name: migration-controller
    uid: edd9fa5c-ec77-4a05-88d4-cf3afdcfb02e
  resourceVersion: "46440"
  selfLink: /api/v1/namespaces/openshift-migration/secrets/cloud-credentials
  uid: 56e894a3-235b-4d81-be33-15639f8aa481
type: Opaque
Comment 7 Erik Nelson 2021-06-22 15:13:43 UTC 
We believe this issue to have been fixed upstream, and as of our Velero rebase for 1.5.0 this should have been picked up: https://github.com/vmware-tanzu/velero/issues/1530

Moving to MODIFIED for Rayford to place onto the errata and alert QE this is ready for verification with upcoming build.
Comment 11 Sergio 2021-06-29 13:55:51 UTC 
Verified using MTC 1.5.0

   openshift-migration-rhel7-operator@sha256:00e77706ca22bcb557d13c16822180fc877e6ea1639a72fda8eb9f5488b039a2
    - name: MIG_CONTROLLER_REPO
      value: openshift-migration-controller-rhel8@sha256
    - name: MIG_CONTROLLER_TAG
      value: 7f657df15e9514df4ef42da3431f558a19b8d3233a2ef1222cd8e27793c93816

Verified with: GCP 4.3 -> GCP 4.7 with GCP replication repository.

We could migrate applications in GCP using direct and indirect migrations.

Moved to VERIFIED.
Comment 17 errata-xmlrpc 2021-07-28 04:08:00 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Migration Toolkit for Containers (MTC) image release advisory 1.5.0), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:2929
Comment 18 Red Hat Bugzilla 2023-09-15 00:59:12 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days
Note You need to log in before you can comment on or make changes to this bug.