Bug 2002360 - MigAnalytic is reporting a warning in intra-cluster migrations from ocs4 to ocs4 storage classes
Summary: MigAnalytic is reporting a warning in intra-cluster migrations from ocs4 to o...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Migration Toolkit for Containers
Classification: Red Hat
Component: General
Version: 1.6.0
Hardware: Unspecified
OS: Unspecified
medium
low
Target Milestone: ---
: 1.7.0
Assignee: Pranav Gaikwad
QA Contact: Xin jiang
Avital Pinnick
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-09-08 15:49 UTC by Sergio
Modified: 2022-03-24 06:32 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-24 06:32:26 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github konveyor mig-controller pull 1241 0 None open Identify sparse files and automatically set `--sparse` option if they are found 2021-12-16 17:40:06 UTC
Red Hat Product Errata RHBA-2022:1043 0 None None None 2022-03-24 06:32:38 UTC

Description Sergio 2021-09-08 15:49:16 UTC
Description of problem:
When we create a migration plan to execute an intra-cluster migration from OCS4 (rbd) storage class to OCS4(rbd) storage class the migration plan is reporting this warning "Failed gathering extended PV usage information for PVs [test-data-app1], please see MigAnalytic openshift-migration/local-tvq7x for details"

Version-Release number of selected component (if applicable):
TARGET CLUSTER: AWS OCP 4.9 (MTC 1.6.0) OCS4  (controller)
SOURCE CLUSTER: AWS OCP 3.11 (MTC 1.5.1) NFS
RREPLICATION REPOSITORY: AWS S3

How reproducible:
Always

Steps to Reproduce:
1. Create an applicatin with a pvc in the OCP 4.9 cluster. Use rbd OCS4 storage class

the pvc is like this:
$ oc get pvc
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
test-data-app1   Bound    pvc-413d08ca-382d-4a44-97d4-0d3ddc95def9   1Gi        RWO            ocs-storagecluster-ceph-rbd   43m


2. Create a migration plan to execute an inter-cluster migration. From OCP4.9 to the same OCP4.9 cluster. Use rbd OCS4 as the destination storage class too.


Actual results:
The migration plan is reporting a miganalytic warning
  - category: Warn
    lastTransitionTime: "2021-09-08T14:43:12Z"
    message: Failed gathering extended PV usage information for PVs [test-data-app1],
      please see MigAnalytic openshift-migration/local-tvq7x for details
    reason: FailedRunningDf
    status: "True"
    type: ExtendedPVAnalysisFailed

Expected results:
Miganalytic should have no problem gathering the PV usage information

Additional info:

This is the miganalytic executed command
{"level":"info","ts":1631112159.3300958,"logger":"analytics","msg":"Executing df command inside source cluster Restic Pod to measure actual usage for extended PV analysis","migAnalytic":"local-tvq7x","pod":"openshift-migration/restic-8tns9","command":["/bin/bash","-c","df -BMB /host_pods/cb95d243-5187-43fb-8882-0d102b333727/volumes/*/pvc-413d08ca-382d-4a44-97d4-0d3ddc95def9"]}


This is the mounted volume in the pod:

$ df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/rbd0               975.9M    602.5M    357.4M  63% /data/test 


This is the full migration plan:
apiVersion: migration.openshift.io/v1alpha1
kind: MigPlan
metadata:
  annotations:
    openshift.io/touch: 17bea80f-10b3-11ec-afae-0a580a800226
  creationTimestamp: "2021-09-08T14:42:36Z"
  generation: 5

  name: local
  namespace: openshift-migration
  resourceVersion: "232299"
  uid: c7a2ea14-af25-4e24-aac3-4e88160a8a4e
spec:
  destMigClusterRef:
    name: host
    namespace: openshift-migration
  indirectImageMigration: true
  indirectVolumeMigration: true
  migStorageRef:
    name: automatic
    namespace: openshift-migration
  namespaces:
  - ocp-44561-state-intralocal
  persistentVolumes:
  - capacity: 1Gi
    name: pvc-413d08ca-382d-4a44-97d4-0d3ddc95def9
    proposedCapacity: "0"
    pvc:
      accessModes:
      - ReadWriteOnce
      hasReference: true
      name: test-data-app1:test-data-app1
      namespace: ocp-44561-state-intralocal
    selection:
      action: copy
      copyMethod: filesystem
      storageClass: ocs-storagecluster-ceph-rbd
    storageClass: ocs-storagecluster-ceph-rbd
    supported:
      actions:
      - skip
      - copy
      copyMethods:
      - filesystem
      - snapshot
  srcMigClusterRef:
    name: host
    namespace: openshift-migration
status:
  conditions:
  - category: Warn
    lastTransitionTime: "2021-09-08T14:42:47Z"
    message: Source namespaces [ocp-44561-state-intralocal] are mapped to namespaces
      which result in conflicts. Stage/Final migrations will be blocked on this migration
      plan. Only state migrations are possible provided there are no conflicts in
      the PVC names.
    reason: ConflictingNamespaces
    status: "True"
    type: IntraClusterMigration
  - category: Required
    lastTransitionTime: "2021-09-08T14:42:47Z"
    message: The `persistentVolumes` list has been updated with discovered PVs.
    reason: Done
    status: "True"
    type: PvsDiscovered
  - category: Required
    lastTransitionTime: "2021-09-08T14:42:47Z"
    message: The storage resources have been created.
    reason: Done
    status: "True"
    type: StorageEnsured
  - category: Warn
    lastTransitionTime: "2021-09-08T14:43:12Z"
    message: Failed gathering extended PV usage information for PVs [test-data-app1],
      please see MigAnalytic openshift-migration/local-tvq7x for details
    reason: FailedRunningDf
    status: "True"
    type: ExtendedPVAnalysisFailed
  - category: Required
    lastTransitionTime: "2021-09-08T14:42:47Z"
    message: The migration plan is ready.
    status: "True"
    type: Ready
  - category: Warn
    lastTransitionTime: "2021-09-08T14:42:47Z"
    message: Source PVCs [ocp-44561-state-intralocal/test-data-app1] are mapped to
      destination PVCs which result in conflicts. Please map each source PVC to a
      distinct destination PVC before running a State Migration.
    reason: NotDistinct
    status: "True"
    type: PvNameConflict
  destStorageClasses:
  - accessModes:
    - ReadWriteOnce
    name: gp2
    provisioner: kubernetes.io/aws-ebs
  - accessModes:
    - ReadWriteOnce
    name: gp2-csi
    provisioner: ebs.csi.aws.com
  - accessModes:
    - ReadWriteOnce
    default: true
    name: ocs-storagecluster-ceph-rbd
    provisioner: openshift-storage.rbd.csi.ceph.com
  - accessModes:
    - ReadWriteOnce
    - ReadOnlyMany
    - ReadWriteMany
    name: ocs-storagecluster-cephfs
    provisioner: openshift-storage.cephfs.csi.ceph.com
  - accessModes:
    - ReadWriteOnce
    name: openshift-storage.noobaa.io
    provisioner: openshift-storage.noobaa.io/obc
  excludedResources:
  - imagetags
  - templateinstances
  - clusterserviceversions
  - packagemanifests
  - subscriptions
  - servicebrokers
  - servicebindings
  - serviceclasses
  - serviceinstances
  - serviceplans
  - operatorgroups
  - events
  - events.events.k8s.io
  observedDigest: 27ca5d44ecab1de03a8f0223595b4cf3044df1bc019434b743f8b335d00557c0
  srcStorageClasses:
  - accessModes:
    - ReadWriteOnce
    name: gp2
    provisioner: kubernetes.io/aws-ebs
  - accessModes:
    - ReadWriteOnce
    name: gp2-csi
    provisioner: ebs.csi.aws.com
  - accessModes:
    - ReadWriteOnce
    default: true
    name: ocs-storagecluster-ceph-rbd
    provisioner: openshift-storage.rbd.csi.ceph.com
  - accessModes:
    - ReadWriteOnce
    - ReadOnlyMany
    - ReadWriteMany
    name: ocs-storagecluster-cephfs
    provisioner: openshift-storage.cephfs.csi.ceph.com
  - accessModes:
    - ReadWriteOnce
    name: openshift-storage.noobaa.io
    provisioner: openshift-storage.noobaa.io/obc

Comment 1 Sergio 2021-09-16 08:38:09 UTC
I see this problem using "ibmc-vpc-block-10iops-tier" in ROKS, so it does not seem related to the storage class.

Comment 2 Pranav Gaikwad 2021-12-16 17:39:25 UTC
I could confirm that this issue was caused in OCS due to the location of the volume mounts in the Restic pod. OCS mounts a volume in a subdirectory of the mount on the node. I dont know exactly when, but it looks like the mount path was changed in some of newer releases of OCS/OCP (in general, it could be a property of newer CSI volumes as the same issue is reported for ROKS as well). I have updated the df command to find the mounts correctly no matter where the directory is mounted. This fix is actually part of sparse files fix, and I believe it should solve this problem. I am updating the BZ with links.

Comment 8 Prasad Joshi 2022-02-03 16:20:38 UTC
Verified with MTC 1.7.0

metadata_nvr: openshift-migration-operator-metadata-container-v1.7.0-27


Created a intra cluster migplan with src storage class as an OCS4 and target storageclass as an OCS4, Migplan is not showing ExtendedPVAnalysisFailed warning.


Moving this to verified status.

Comment 10 errata-xmlrpc 2022-03-24 06:32:26 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Migration Toolkit for Containers (MTC) 1.7.0 release advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:1043


Note You need to log in before you can comment on or make changes to this bug.