Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1845090

Summary: Migrating more than one namespaces with more than one PVCs results in a migration error
Product: OpenShift Container Platform Reporter: Sergio <sregidor>
Component: Migration ToolingAssignee: Scott Seago <sseago>
Status: CLOSED ERRATA QA Contact: Xin jiang <xjiang>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.5CC: chezhang, mberube, rjohnson, rpattath, whu, xjiang
Target Milestone: ---   
Target Release: 4.4.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1845096 (view as bug list) Environment:
Last Closed: 2020-06-17 00:04:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1845096    
Bug Blocks:    
Attachments:
Description Flags
restic and velero logs
none
velero backup logs
none
controller logs none

Description Sergio 2020-06-08 12:59:43 UTC 
Created attachment 1696083 [details]
restic and velero logs

Description of problem:
When we try to migrate more than one PVC to check the PVCs limit the migration fails because it cannot find the volumes.

Version-Release number of selected component (if applicable):
CAM 1.2.2 stage
SOURCE: OCP 3.11
TARGET: OCP 4.4

How reproducible:
Always

Steps to Reproduce:
1. In the cluster that contains the controller, configure a maximum of 5 PVs per migration plan in MigrationController resouce

$ oc patch -n openshift-migration MigrationController migration-controller --type=json -p='[{"op":"add", "path": "/spec/mig_pv_limit", "value": "5"}]'


2. Create 3 namespaces with a nginx with 2 PVs in every namespace. It makes 6 PVs created, 2 in each namespace.

$ oc process -p NAMESPACE=max-pvs-1 -f  https://gitlab.cee.redhat.com/app-mig/cam-helper/raw/master/ocp-26160/nginx_with_pv_defaultsc_template.yml | oc create -f -

$ oc process -p NAMESPACE=max-pvs-2 -f https://gitlab.cee.redhat.com/app-mig/cam-helper/raw/master/ocp-26160/nginx_with_pv_defaultsc_template.yml  | oc create -f -

$ oc process -p NAMESPACE=max-pvs-3 -f https://gitlab.cee.redhat.com/app-mig/cam-helper/raw/master/ocp-26160/nginx_with_pv_defaultsc_template.yml  | oc create -f -


3. Feed that data once all pods are running

$ oc -n max-pvs-1 rsh $(oc get pods -n max-pvs-1 -o jsonpath='{.items[0].metadata.name}') sh -c 'echo "<h1>HELLO WORLD</h1>" > /usr/share/nginx/html/index.html'

$ oc -n max-pvs-2 rsh $(oc get pods -n max-pvs-2 -o jsonpath='{.items[0].metadata.name}') sh -c 'echo "<h1>HELLO WORLD</h1>" > /usr/share/nginx/html/index.html'

$ oc -n max-pvs-2 rsh $(oc get pods -n max-pvs-2 -o jsonpath='{.items[0].metadata.name}') sh -c 'echo "<h1>HELLO WORLD</h1>" > /usr/share/nginx/html/index.html'


4.Create a Migration Plan and select all those namespaces in order to be migrated. The three namespaces in the same migration plan. It will try to migrate a total of 6 persistent volumes.


5. Execute the migration plan


Actual results:
The migration fails in StageBackupCreated state.

We can find the following log in the VeleroBackup logs
gSource="pkg/backup/resource_backupper.go:283" name=nginx-html namespace=max-pvs-3 resource=persistentvolumeclaims
time="2020-06-08T09:37:13Z" level=error msg="Error backing up item" backup=openshift-migration/max-pvs-mig-1591604420-hdt9c error="error getting vol
ume info: rpc error: code = Unknown desc = InvalidVolume.NotFound: The volume 'vol-0dc329baa28ba7b8e' does not exist.\n\tstatus code: 400, request i
d: 67599552-c075-4cd3-98cc-d6f58d510a81" group=v1 logSource="pkg/backup/resource_backupper.go:287" name=nginx-html namespace=max-pvs-3 resource=pers
istentvolumeclaims


Expected results:
The migration should finish without errors and all namespaces and pvs should be migrated properly

Additional info:
All logs attached.
Comment 1 Sergio 2020-06-08 13:00:30 UTC 
Created attachment 1696084 [details]
velero backup logs
Comment 2 Sergio 2020-06-08 13:00:49 UTC 
Created attachment 1696085 [details]
controller logs
Comment 3 Sergio 2020-06-08 14:39:37 UTC 
I add the migration plan that triggered this error

apiVersion: migration.openshift.io/v1alpha1
kind: MigPlan
metadata:
  annotations:
    openshift.io/touch: ad9599bd-a995-11ea-9e56-0a580a81020f
  creationTimestamp: "2020-06-08T10:20:31Z"
  generation: 6
  labels:
    controller-tools.k8s.io: "1.0"
  name: max-pvs-migplan-1591611025
  namespace: openshift-migration
  resourceVersion: "144343"
  selfLink: /apis/migration.openshift.io/v1alpha1/namespaces/openshift-migration/migplans/max-pvs-migplan-1591611025
  uid: ce90b6cd-f2bb-4cd8-88e4-84ff6863089f
spec:
  destMigClusterRef:
    name: host
    namespace: openshift-migration
  migStorageRef:
    name: automatic
    namespace: openshift-migration
  namespaces:
  - max-pvs-1
  - max-pvs-2
  - max-pvs-3
  persistentVolumes:
  - capacity: 1Gi
    name: pvc-556360b5-a971-11ea-ba19-0e2ac7ac682f
    pvc:
      accessModes:
      - ReadWriteOnce
      hasReference: true
      name: nginx-logs
      namespace: max-pvs-1
    selection:
      action: copy
      copyMethod: filesystem
      storageClass: gp2
    storageClass: gp2
    supported:
      actions:
      - copy
      - move
      copyMethods:
      - filesystem
      - snapshot
  - capacity: 1Gi
    name: pvc-55791c8b-a971-11ea-ba19-0e2ac7ac682f
    pvc:
      accessModes:
      - ReadWriteOnce
      hasReference: true
      name: nginx-html
      namespace: max-pvs-1
    selection:
      action: copy
      copyMethod: filesystem
      storageClass: gp2
    storageClass: gp2
    supported:
      actions:
      - copy
      - move
      copyMethods:
      - filesystem
      - snapshot
  - capacity: 1Gi
    name: pvc-7122b185-a971-11ea-ba19-0e2ac7ac682f
    pvc:
      accessModes:
      - ReadWriteOnce
      hasReference: true
      name: nginx-logs
      namespace: max-pvs-2
    selection:
      action: copy
      copyMethod: filesystem
      storageClass: gp2
    storageClass: gp2
    supported:
      actions:
      - copy
      - move
      copyMethods:
      - filesystem
      - snapshot
  - capacity: 1Gi
    name: pvc-712c23a5-a971-11ea-ba19-0e2ac7ac682f
    pvc:
      accessModes:
      - ReadWriteOnce
      hasReference: true
      name: nginx-html
      namespace: max-pvs-2
    selection:
      action: copy
      copyMethod: filesystem
      storageClass: gp2
    storageClass: gp2
    supported:
      actions:
      - copy
      - move
      copyMethods:
      - filesystem
      - snapshot
  - capacity: 1Gi
    name: pvc-8daba631-a971-11ea-ba19-0e2ac7ac682f
    pvc:
      accessModes:
      - ReadWriteOnce
      hasReference: true
      name: nginx-logs
      namespace: max-pvs-3
    selection:
      action: copy
      copyMethod: filesystem
      storageClass: gp2
    storageClass: gp2
    supported:
      actions:
      - copy
      - move
      copyMethods:
      - filesystem
      - snapshot
  - capacity: 1Gi
    name: pvc-8e1fb3b5-a971-11ea-ba19-0e2ac7ac682f
    pvc:
      accessModes:
      - ReadWriteOnce
      hasReference: true
      name: nginx-html
      namespace: max-pvs-3
    selection:
      action: copy
      copyMethod: filesystem
      storageClass: gp2
    storageClass: gp2
    supported:
      actions:
      - copy
      - move
      copyMethods:
      - filesystem
      - snapshot
  srcMigClusterRef:
    name: source-cluster
    namespace: openshift-migration
status:
  conditions:
  - category: Required
    lastTransitionTime: "2020-06-08T10:20:39Z"
    message: The `persistentVolumes` list has been updated with discovered PVs.
    reason: Done
    status: "True"
    type: PvsDiscovered
  - category: Required
    lastTransitionTime: "2020-06-08T10:20:41Z"
    message: The storage resources have been created.
    status: "True"
    type: StorageEnsured
  - category: Required
    lastTransitionTime: "2020-06-08T10:20:42Z"
    message: The migration registry resources have been created.
    status: "True"
    type: RegistriesEnsured
  - category: Required
    lastTransitionTime: "2020-06-08T10:20:42Z"
    message: The migration plan is ready.
    status: "True"
    type: Ready
  observedDigest: 27c14e6dd61ee66f23e0d505810beecc647f805995757a38ca633f9c4aba2b91
Comment 4 Scott Seago 2020-06-09 22:30:13 UTC 
bugfix in PR: https://github.com/konveyor/mig-controller/pull/564
Comment 8 Sergio 2020-06-11 14:05:50 UTC 
Verified using CAM 1.2.2 stage

All volumes and namespaces were migrated properly.

openshift-migration-rhel7-operator@sha256:ab124c3917a2ea22e03618f287c629e727bbcdf7ec76db5e7d0f8654064b7a52

    - name: MIG_CONTROLLER_REPO
      value: openshift-migration-controller-rhel8@sha256
    - name: MIG_CONTROLLER_TAG
      value: ca9ab7ecf0d939afa1aae2540bb3daf5d7ce651ad58b94c6987484d12af1d211
    - name: MIG_UI_REPO
      value: openshift-migration-ui-rhel8@sha256
    - name: MIG_UI_TAG
      value: 6abfaea8ac04e3b5bbf9648a3479b420b4baec35201033471020c9cae1fe1e11
    - name: MIGRATION_REGISTRY_REPO
      value: openshift-migration-registry-rhel8@sha256
    - name: MIGRATION_REGISTRY_TAG
      value: ea6301a15277d448c8756881c7e2e712893ca8041c913476640f52da9e76cad9
    - name: VELERO_REPO
      value: openshift-migration-velero-rhel8@sha256
    - name: VELERO_TAG
      value: 1a33e327dd610f0eebaaeae5b3c9b4170ab5db572b01a170be35b9ce946c0281
    - name: VELERO_PLUGIN_REPO
      value: openshift-migration-plugin-rhel8@sha256
    - name: VELERO_PLUGIN_TAG
      value: 7eba00127497c4ca6452f9be0c167c2276bed462b648edf51d8bbe7265392879
    - name: VELERO_AWS_PLUGIN_REPO
      value: openshift-migration-velero-plugin-for-aws-rhel8@sha256
    - name: VELERO_AWS_PLUGIN_TAG
      value: 22c58f575ce2f54bf995fced82f89ba173329d9b88409cf371122f9ae8cabda1
Comment 10 errata-xmlrpc 2020-06-17 00:04:13 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:2571