Bug 1973148 - The calculation of PV resize with gp2 did not work as expected
Summary: The calculation of PV resize with gp2 did not work as expected
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Migration Toolkit for Containers
Classification: Red Hat
Component: General
Version: 1.5.0
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ---
: ---
Assignee: John Matthews
QA Contact: Xin jiang
Avital Pinnick
URL:
Whiteboard:
Depends On:
Blocks: 1975694
TreeView+ depends on / blocked
 
Reported: 2021-06-17 10:38 UTC by whu
Modified: 2021-06-24 08:38 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1975694 (view as bug list)
Environment:
Last Closed: 2021-06-18 19:19:21 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description whu 2021-06-17 10:38:35 UTC
Description of problem:
The calculation of PV resize against gp2 did not work as expected.

Version-Release number of selected component (if applicable):
MTC 1.5.0
image: registry.redhat.io/rhmtc/openshift-migration-rhel7-operator@sha256:046840aee4bf73c44ab6fe53f4f5df49a4ca808af8e4496f7e957ef8681415d7
Source cluster: AWS 3.11 gp2
Target cluster: AWS 4.6 gp2

How reproducible:
Always

Steps to Reproduce:

1.  deploy test application in source cluster
$ oc create namespace ocp-41883-resizeconfig
namespace/ocp-41883-resizeconfig created

$ cat  PersistentVolumeClaim yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  namespace: ocp-41883-resizeconfig
  name: test-data
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

$ oc create -f PersistentVolumeClaim.yml
persistentvolumeclaim/test-data created

$ cat pvresize-test-pod.yml
apiVersion:  apps/v1
kind: Deployment
metadata:
  labels:
    app: pvresize-test
  name: pvresize-test
  namespace: ocp-41883-resizeconfig
spec:
  replicas:  1
  selector:
    matchLabels:
      app: pvresize-test
  template:
    metadata:
      labels:
        app: pvresize-test
    spec:
      containers:
      - name: podtest
        imagePullPolicy: Always
        image: quay.io/openshifttest/alpine
        command: [ "/bin/sh", "-c", "--" ]
        args:
        - while true; do sleep 30; done;
        restartPolicy: Never
        volumeMounts:
        - name: testvolume
          mountPath: /data/test
      volumes:
      - name: testvolume
        persistentVolumeClaim:
          claimName: test-data

$ oc create -f pvresize-test-pod.yml
deployment.apps/pvresize-test created

$ oc -n ocp-41883-resizeconfig exec $(oc get pods -l app=pvresize-test -o jsonpath='{.items[0].metadata.name}' -n ocp-41883-resizeconfig) -- dd if=/dev/zero of=/data/test/file_1 bs=20M count=10
10+0 records in
10+0 records out

$ oc -n ocp-41883-resizeconfig exec $(oc get pods -l app=pvresize-test -o jsonpath='{.items[0].metadata.name}' -n ocp-41883-resizeconfig) -- dd if=/dev/zero of=/data/test/file_2 bs=20M count=10
10+0 records in
10+0 records out

$ oc -n ocp-41883-resizeconfig exec $(oc get pods -l app=pvresize-test -o jsonpath='{.items[0].metadata.name}' -n ocp-41883-resizeconfig) -- dd if=/dev/zero of=/data/test/file_3 bs=20M count=10
10+0 records in
10+0 records out

$ oc -n ocp-41883-resizeconfig exec $(oc get pods -l app=pvresize-test -o jsonpath='{.items[0].metadata.name}' -n ocp-41883-resizeconfig) -- df -BMB /data/test
Filesystem           977K-blocks    Used Available Use% Mounted on
/dev/xvdci                1023       632       375  63% /data/test

2 Enable the DVM PV resizing in the migrationcontroller configuration
$ oc patch -n openshift-migration  MigrationController migration-controller --type=json -p='[{"op":"add", "path": "/spec/enable_dvm_pv_resizing", "value": "true"}]'
migrationcontroller.migration.openshift.io/migration-controller patched

$ oc get migrationcontroller migration-controller -o yaml -n  openshift-migration
        f:enable_dvm_pv_resizing: {}
  enable_dvm_pv_resizing: "true"

3 after controller pod restart, create a migration plan for this namespace using DVM

4 In controller cluster. configure a new threshold for the resize, 38% threshold
$ oc -n openshift-migration patch migrationcontroller migration-controller  -p '{"spec":{"pv_resizing_threshold": 38}  }' --type='merge'
migrationcontroller.migration.openshift.io/migration-controller patched

5 after controller pod restart, refresh the migplan
$ oc -n  openshift-migration  patch  migplan mig-plan-41883-ocp-41883-resizeconfig -p '{"spec": {"refresh":true} }'  --type='merge'
migplan.migration.openshift.io/mig-plan-41883-ocp-41883-resizeconfig patched

6. Check the migplan status
$ oc get migplan mig-plan-41883-ocp-41883-resizeconfig -o yaml
apiVersion: migration.openshift.io/v1alpha1
kind: MigPlan
….
status:
  conditions:
  - category: Required
    lastTransitionTime: "2021-06-17T08:01:40Z"
    message: The `persistentVolumes` list has been updated with discovered PVs.
    reason: Done
    status: "True"
    type: PvsDiscovered
  - category: Required
    lastTransitionTime: "2021-06-17T08:01:40Z"
    message: The storage resources have been created.
    reason: Done
    status: "True"
    type: StorageEnsured
  - category: Required
    lastTransitionTime: "2021-06-17T08:01:40Z"
    message: The migration plan is ready.
    status: "True"
    type: Ready


7 In controller cluster. configure a new threshold for the resize, 41% threshold
$ oc -n openshift-migration patch migrationcontroller migration-controller  -p '{"spec":{"pv_resizing_threshold": 41}  }' --type='merge'
migrationcontroller.migration.openshift.io/migration-controller patched

8 after controller pod restart, refresh the migplan
$ oc -n  openshift-migration  patch  migplan mig-plan-41883-ocp-41883-resizeconfig -p '{"spec": {"refresh":true} }'  --type='merge'
migplan.migration.openshift.io/mig-plan-41883-ocp-41883-resizeconfig patched

9. Check the migplan status
$ oc get migplan mig-plan-41883-ocp-41883-resizeconfig -o yaml
apiVersion: migration.openshift.io/v1alpha1
kind: MigPlan
….
status:
  conditions:
  - category: Required
    lastTransitionTime: "2021-06-17T08:01:40Z"
    message: The `persistentVolumes` list has been updated with discovered PVs.
    reason: Done
    status: "True"
    type: PvsDiscovered
  - category: Required
    lastTransitionTime: "2021-06-17T08:01:40Z"
    message: The storage resources have been created.
    reason: Done
    status: "True"
    type: StorageEnsured
  - category: Required
    lastTransitionTime: "2021-06-17T08:01:40Z"
    message: The migration plan is ready.
    status: "True"
    type: Ready

10 In controller cluster. configure a new threshold for the resize, 42% threshold
$ oc -n openshift-migration patch migrationcontroller migration-controller  -p '{"spec":{"pv_resizing_threshold": 42}  }' --type='merge'
migrationcontroller.migration.openshift.io/migration-controller patched

11 after controller pod restart, refresh the migplan
$ oc -n  openshift-migration  patch  migplan mig-plan-41883-ocp-41883-resizeconfig -p '{"spec": {"refresh":true} }'  --type='merge'
migplan.migration.openshift.io/mig-plan-41883-ocp-41883-resizeconfig patched

12. Check the migplan status
$ oc get migplan mig-plan-41883-ocp-41883-resizeconfig -o yaml
apiVersion: migration.openshift.io/v1alpha1
kind: MigPlan
….
status:
  conditions:
   ……..
  - category: Warn
    durable: true
    lastTransitionTime: "2021-06-17T08:57:01Z"
    message: 'Capacity of the following volumes will be automatically adjusted to avoid disk capacity issues in the target cluster:  [pvc-b800eb7b-cf3b-11eb-a3f7-0eae3e0555f3]'
    reason: Done
    status: "False"
    type: PvCapacityAdjustmentRequired


Actual results:
The storage usage in the source cluster is 63%, after setting the pv_resizing_threshold to 38%. There is no warning message in migplan. after setting the pv_resizing_threshold to 41%. There is no warning message in migplan. after setting the pv_resizing_threshold to 42%. There is a warning message in migplan. 

Expected results:
The storage usage in the source cluster is 63%,  after setting the value pv_resizing_threshold bigger than 37%, there should be a warning message “Capacity of the following volumes will be automatically adjusted to avoid disk capacity issues in the target cluster” in migplan.

Additional info:
In gp2, if the storage usage in source cluster is 98% and don’t set pv_resizing_threshold (use the default threshold 3%), won’t get warning in migplan either. 
The calculation of PV resize works as expected with glusterfs.
Refer to another BZ https://bugzilla.redhat.com/show_bug.cgi?id=1963927

Comment 1 Erik Nelson 2021-06-18 19:19:21 UTC
Discussed this one with Pranav, this is a quirk of GP2 and the way that usage and size are calculated in slightly different ways.

From Pranav:
"the problem is that when a gp2 volume shows that its been provisioned 2Gi, the actual useable volume size is of 98% of 2Gi which is already 2% less than the size. our calculations go off since we consider 2 Gi as the base size"

For me, the spirit of exposing a configurable threshold here is so that users can alter that % difference from something like 98% to 70%. I don't think that it will be of significance to users that 38% specifically doesn't trigger the warning, while 39% does. It's somewhat rough.

Due to the difficulty on our side of addressing this based on limitations of GP2 and the relatively low severity, we're going to close as a WONTFIX and provide some documentation that describes what's happening here in our intelligent resize doc: https://github.com/konveyor/mig-operator/blob/master/docs/usage/IntelligentPVResizing.md

Comment 2 Avital Pinnick 2021-06-24 08:38:17 UTC
(In reply to Erik Nelson from comment #1)
> 
> Due to the difficulty on our side of addressing this based on limitations of
> GP2 and the relatively low severity, we're going to close as a WONTFIX and
> provide some documentation that describes what's happening here in our
> intelligent resize doc:
> https://github.com/konveyor/mig-operator/blob/master/docs/usage/
> IntelligentPVResizing.md

Cloning for 1.5.0 documentation


Note You need to log in before you can comment on or make changes to this bug.