Description of problem: When we create a migration plan to execute an intra-cluster migration from OCS4 (rbd) storage class to OCS4(rbd) storage class the migration plan is reporting this warning "Failed gathering extended PV usage information for PVs [test-data-app1], please see MigAnalytic openshift-migration/local-tvq7x for details" Version-Release number of selected component (if applicable): TARGET CLUSTER: AWS OCP 4.9 (MTC 1.6.0) OCS4 (controller) SOURCE CLUSTER: AWS OCP 3.11 (MTC 1.5.1) NFS RREPLICATION REPOSITORY: AWS S3 How reproducible: Always Steps to Reproduce: 1. Create an applicatin with a pvc in the OCP 4.9 cluster. Use rbd OCS4 storage class the pvc is like this: $ oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE test-data-app1 Bound pvc-413d08ca-382d-4a44-97d4-0d3ddc95def9 1Gi RWO ocs-storagecluster-ceph-rbd 43m 2. Create a migration plan to execute an inter-cluster migration. From OCP4.9 to the same OCP4.9 cluster. Use rbd OCS4 as the destination storage class too. Actual results: The migration plan is reporting a miganalytic warning - category: Warn lastTransitionTime: "2021-09-08T14:43:12Z" message: Failed gathering extended PV usage information for PVs [test-data-app1], please see MigAnalytic openshift-migration/local-tvq7x for details reason: FailedRunningDf status: "True" type: ExtendedPVAnalysisFailed Expected results: Miganalytic should have no problem gathering the PV usage information Additional info: This is the miganalytic executed command {"level":"info","ts":1631112159.3300958,"logger":"analytics","msg":"Executing df command inside source cluster Restic Pod to measure actual usage for extended PV analysis","migAnalytic":"local-tvq7x","pod":"openshift-migration/restic-8tns9","command":["/bin/bash","-c","df -BMB /host_pods/cb95d243-5187-43fb-8882-0d102b333727/volumes/*/pvc-413d08ca-382d-4a44-97d4-0d3ddc95def9"]} This is the mounted volume in the pod: $ df -h Filesystem Size Used Available Use% Mounted on /dev/rbd0 975.9M 602.5M 357.4M 63% /data/test This is the full migration plan: apiVersion: migration.openshift.io/v1alpha1 kind: MigPlan metadata: annotations: openshift.io/touch: 17bea80f-10b3-11ec-afae-0a580a800226 creationTimestamp: "2021-09-08T14:42:36Z" generation: 5 name: local namespace: openshift-migration resourceVersion: "232299" uid: c7a2ea14-af25-4e24-aac3-4e88160a8a4e spec: destMigClusterRef: name: host namespace: openshift-migration indirectImageMigration: true indirectVolumeMigration: true migStorageRef: name: automatic namespace: openshift-migration namespaces: - ocp-44561-state-intralocal persistentVolumes: - capacity: 1Gi name: pvc-413d08ca-382d-4a44-97d4-0d3ddc95def9 proposedCapacity: "0" pvc: accessModes: - ReadWriteOnce hasReference: true name: test-data-app1:test-data-app1 namespace: ocp-44561-state-intralocal selection: action: copy copyMethod: filesystem storageClass: ocs-storagecluster-ceph-rbd storageClass: ocs-storagecluster-ceph-rbd supported: actions: - skip - copy copyMethods: - filesystem - snapshot srcMigClusterRef: name: host namespace: openshift-migration status: conditions: - category: Warn lastTransitionTime: "2021-09-08T14:42:47Z" message: Source namespaces [ocp-44561-state-intralocal] are mapped to namespaces which result in conflicts. Stage/Final migrations will be blocked on this migration plan. Only state migrations are possible provided there are no conflicts in the PVC names. reason: ConflictingNamespaces status: "True" type: IntraClusterMigration - category: Required lastTransitionTime: "2021-09-08T14:42:47Z" message: The `persistentVolumes` list has been updated with discovered PVs. reason: Done status: "True" type: PvsDiscovered - category: Required lastTransitionTime: "2021-09-08T14:42:47Z" message: The storage resources have been created. reason: Done status: "True" type: StorageEnsured - category: Warn lastTransitionTime: "2021-09-08T14:43:12Z" message: Failed gathering extended PV usage information for PVs [test-data-app1], please see MigAnalytic openshift-migration/local-tvq7x for details reason: FailedRunningDf status: "True" type: ExtendedPVAnalysisFailed - category: Required lastTransitionTime: "2021-09-08T14:42:47Z" message: The migration plan is ready. status: "True" type: Ready - category: Warn lastTransitionTime: "2021-09-08T14:42:47Z" message: Source PVCs [ocp-44561-state-intralocal/test-data-app1] are mapped to destination PVCs which result in conflicts. Please map each source PVC to a distinct destination PVC before running a State Migration. reason: NotDistinct status: "True" type: PvNameConflict destStorageClasses: - accessModes: - ReadWriteOnce name: gp2 provisioner: kubernetes.io/aws-ebs - accessModes: - ReadWriteOnce name: gp2-csi provisioner: ebs.csi.aws.com - accessModes: - ReadWriteOnce default: true name: ocs-storagecluster-ceph-rbd provisioner: openshift-storage.rbd.csi.ceph.com - accessModes: - ReadWriteOnce - ReadOnlyMany - ReadWriteMany name: ocs-storagecluster-cephfs provisioner: openshift-storage.cephfs.csi.ceph.com - accessModes: - ReadWriteOnce name: openshift-storage.noobaa.io provisioner: openshift-storage.noobaa.io/obc excludedResources: - imagetags - templateinstances - clusterserviceversions - packagemanifests - subscriptions - servicebrokers - servicebindings - serviceclasses - serviceinstances - serviceplans - operatorgroups - events - events.events.k8s.io observedDigest: 27ca5d44ecab1de03a8f0223595b4cf3044df1bc019434b743f8b335d00557c0 srcStorageClasses: - accessModes: - ReadWriteOnce name: gp2 provisioner: kubernetes.io/aws-ebs - accessModes: - ReadWriteOnce name: gp2-csi provisioner: ebs.csi.aws.com - accessModes: - ReadWriteOnce default: true name: ocs-storagecluster-ceph-rbd provisioner: openshift-storage.rbd.csi.ceph.com - accessModes: - ReadWriteOnce - ReadOnlyMany - ReadWriteMany name: ocs-storagecluster-cephfs provisioner: openshift-storage.cephfs.csi.ceph.com - accessModes: - ReadWriteOnce name: openshift-storage.noobaa.io provisioner: openshift-storage.noobaa.io/obc
I see this problem using "ibmc-vpc-block-10iops-tier" in ROKS, so it does not seem related to the storage class.
I could confirm that this issue was caused in OCS due to the location of the volume mounts in the Restic pod. OCS mounts a volume in a subdirectory of the mount on the node. I dont know exactly when, but it looks like the mount path was changed in some of newer releases of OCS/OCP (in general, it could be a property of newer CSI volumes as the same issue is reported for ROKS as well). I have updated the df command to find the mounts correctly no matter where the directory is mounted. This fix is actually part of sparse files fix, and I believe it should solve this problem. I am updating the BZ with links.
Verified with MTC 1.7.0 metadata_nvr: openshift-migration-operator-metadata-container-v1.7.0-27 Created a intra cluster migplan with src storage class as an OCS4 and target storageclass as an OCS4, Migplan is not showing ExtendedPVAnalysisFailed warning. Moving this to verified status.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Migration Toolkit for Containers (MTC) 1.7.0 release advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:1043