Bug 2004566

Description of problem:
When IDIM + DVM, the State migration failed at StageBackup and reported error "migration registry service not found"

Version-Release number of selected component (if applicable):
MTC 1.6.0
source: OCP4.7 on GCP + 1.6.0
target: OCP 4.9 on GCP + 1.6.0

How reproducible:
always

Steps to Reproduce:
1. Deployment an application with PVCs
2. Create a migplan with IDIM + DVM
3. Trigger State migration

Actual results:
the State migration failed at StageBackup and reported error "migration registry service not found"

Expected results:
I should completed without error

Additional info:
1. migplan
$ oc get migplan test1 -o yaml
.....
spec:
  destMigClusterRef:
    name: target-cluster
    namespace: openshift-migration
  indirectImageMigration: true
  migStorageRef:
    name: automatic
    namespace: openshift-migration
  namespaces:
  - test1
  persistentVolumes:
  - capacity: 1Gi
    name: pvc-64aa2457-f2b4-4e66-b589-be9cfe371694
    proposedCapacity: 1Gi
    pvc:
      accessModes:
      - ReadWriteOnce
      hasReference: true
      name: postgresql:postgresql
      namespace: test1
    selection:
      action: copy
      copyMethod: filesystem
      storageClass: standard
    storageClass: standard
    supported:
      actions:
      - skip
      - copy
      copyMethods:
      - filesystem
      - snapshot
  srcMigClusterRef:
    name: host
    namespace: openshift-migration


2. migmigration
$ oc get migmigration state-migration-980db -o yaml
apiVersion: migration.openshift.io/v1alpha1
kind: MigMigration
metadata:
  annotations:
    migration.openshift.io/state-transfer: "true"
    openshift.io/touch: a46baf83-162d-11ec-9be4-0a580a80020a
  creationTimestamp: "2021-09-15T14:02:43Z"
  generation: 18
  labels:
    migration.openshift.io/migplan-name: test1
    migration.openshift.io/migration-uid: 9d412982-46ee-4c53-a788-dc6e0b792bfa
.....
spec:
  migPlanRef:
    name: test1
    namespace: openshift-migration
  stage: true
status:
  conditions:
  - category: Advisory
    durable: true
    lastTransitionTime: "2021-09-15T14:03:02Z"
    message: 'The migration has failed.  See: Errors.'
    reason: EnsureStageBackup
    status: "True"
    type: Failed
  errors:
  - migration registry service not found
  itinerary: Failed
  observedDigest: 53b455b5286200bbaafc398d33aaf13c7c0cc6e2b0d8bc607f26c5a7589fe8ea
  phase: Completed
  pipeline:
  - completed: "2021-09-15T14:03:01Z"
    message: Completed
    name: Prepare
    started: "2021-09-15T14:02:43Z"
  - completed: "2021-09-15T14:03:02Z"
    failed: true
    message: Failed
    name: StageBackup
    started: "2021-09-15T14:03:01Z"
  - message: Skipped
    name: StageRestore
    skipped: true
  - message: Skipped
    name: DirectVolume
    skipped: true
  - completed: "2021-09-15T14:03:02Z"
    message: Completed
    name: Cleanup
    started: "2021-09-15T14:03:02Z"
  - completed: "2021-09-15T14:03:02Z"
    message: Completed
    name: CleanupHelpers
    started: "2021-09-15T14:03:02Z"
  startTimestamp: "2021-09-15T14:02:43Z"


3. Contoller logs

{"level":"info","ts":1631714582.7220912,"logger":"migration","msg":"Phase completed","migMigration":"state-migration-980db","phase":"EnsureCloudSecretPropagated","phaseElapsed":1.722090208}
{"level":"error","ts":1631714582.7366307,"logger":"migration","msg":"","migMigration":"state-migration-980db","error":"Operation cannot be fulfilled on migmigrations.migration.openshift.io \"state-migration-980db\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/remote-source/app/vendor/github.com/go-logr/zapr/zapr.go:132\ngithub.com/konveyor/controller/pkg/logging.(*Logger).Error\n\t/remote-source/app/vendor/github.com/konveyor/controller/pkg/logging/logger.go:109\ngithub.com/konveyor/controller/pkg/logging.(*Logger).Trace\n\t/remote-source/app/vendor/github.com/konveyor/controller/pkg/logging/logger.go:115\ngithub.com/konveyor/mig-controller/pkg/controller/migmigration.(*ReconcileMigMigration).Reconcile\n\t/remote-source/app/pkg/controller/migmigration/migmigration_controller.go:278\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:263\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:198\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1\n\t/remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext\n\t/remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185\nk8s.io/apimachinery/pkg/util/wait.UntilWithContext\n\t/remote-source/app/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:99"}
{"level":"info","ts":1631714582.7428343,"logger":"migration","msg":"[RUN] (Step 26/37) Creating a stage backup.","migMigration":"state-migration-980db","phase":"EnsureStageBackup"}