Bug 1973148
| Summary: | The calculation of PV resize with gp2 did not work as expected | |||
|---|---|---|---|---|
| Product: | Migration Toolkit for Containers | Reporter: | whu | |
| Component: | General | Assignee: | John Matthews <jmatthew> | |
| Status: | CLOSED WONTFIX | QA Contact: | Xin jiang <xjiang> | |
| Severity: | medium | Docs Contact: | Avital Pinnick <apinnick> | |
| Priority: | high | |||
| Version: | 1.5.0 | CC: | ernelson, sregidor, whu, xjiang | |
| Target Milestone: | --- | |||
| Target Release: | --- | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1975694 (view as bug list) | Environment: | ||
| Last Closed: | 2021-06-18 19:19:21 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1975694 | |||
Discussed this one with Pranav, this is a quirk of GP2 and the way that usage and size are calculated in slightly different ways. From Pranav: "the problem is that when a gp2 volume shows that its been provisioned 2Gi, the actual useable volume size is of 98% of 2Gi which is already 2% less than the size. our calculations go off since we consider 2 Gi as the base size" For me, the spirit of exposing a configurable threshold here is so that users can alter that % difference from something like 98% to 70%. I don't think that it will be of significance to users that 38% specifically doesn't trigger the warning, while 39% does. It's somewhat rough. Due to the difficulty on our side of addressing this based on limitations of GP2 and the relatively low severity, we're going to close as a WONTFIX and provide some documentation that describes what's happening here in our intelligent resize doc: https://github.com/konveyor/mig-operator/blob/master/docs/usage/IntelligentPVResizing.md (In reply to Erik Nelson from comment #1) > > Due to the difficulty on our side of addressing this based on limitations of > GP2 and the relatively low severity, we're going to close as a WONTFIX and > provide some documentation that describes what's happening here in our > intelligent resize doc: > https://github.com/konveyor/mig-operator/blob/master/docs/usage/ > IntelligentPVResizing.md Cloning for 1.5.0 documentation |
Description of problem: The calculation of PV resize against gp2 did not work as expected. Version-Release number of selected component (if applicable): MTC 1.5.0 image: registry.redhat.io/rhmtc/openshift-migration-rhel7-operator@sha256:046840aee4bf73c44ab6fe53f4f5df49a4ca808af8e4496f7e957ef8681415d7 Source cluster: AWS 3.11 gp2 Target cluster: AWS 4.6 gp2 How reproducible: Always Steps to Reproduce: 1. deploy test application in source cluster $ oc create namespace ocp-41883-resizeconfig namespace/ocp-41883-resizeconfig created $ cat PersistentVolumeClaim yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: namespace: ocp-41883-resizeconfig name: test-data spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi $ oc create -f PersistentVolumeClaim.yml persistentvolumeclaim/test-data created $ cat pvresize-test-pod.yml apiVersion: apps/v1 kind: Deployment metadata: labels: app: pvresize-test name: pvresize-test namespace: ocp-41883-resizeconfig spec: replicas: 1 selector: matchLabels: app: pvresize-test template: metadata: labels: app: pvresize-test spec: containers: - name: podtest imagePullPolicy: Always image: quay.io/openshifttest/alpine command: [ "/bin/sh", "-c", "--" ] args: - while true; do sleep 30; done; restartPolicy: Never volumeMounts: - name: testvolume mountPath: /data/test volumes: - name: testvolume persistentVolumeClaim: claimName: test-data $ oc create -f pvresize-test-pod.yml deployment.apps/pvresize-test created $ oc -n ocp-41883-resizeconfig exec $(oc get pods -l app=pvresize-test -o jsonpath='{.items[0].metadata.name}' -n ocp-41883-resizeconfig) -- dd if=/dev/zero of=/data/test/file_1 bs=20M count=10 10+0 records in 10+0 records out $ oc -n ocp-41883-resizeconfig exec $(oc get pods -l app=pvresize-test -o jsonpath='{.items[0].metadata.name}' -n ocp-41883-resizeconfig) -- dd if=/dev/zero of=/data/test/file_2 bs=20M count=10 10+0 records in 10+0 records out $ oc -n ocp-41883-resizeconfig exec $(oc get pods -l app=pvresize-test -o jsonpath='{.items[0].metadata.name}' -n ocp-41883-resizeconfig) -- dd if=/dev/zero of=/data/test/file_3 bs=20M count=10 10+0 records in 10+0 records out $ oc -n ocp-41883-resizeconfig exec $(oc get pods -l app=pvresize-test -o jsonpath='{.items[0].metadata.name}' -n ocp-41883-resizeconfig) -- df -BMB /data/test Filesystem 977K-blocks Used Available Use% Mounted on /dev/xvdci 1023 632 375 63% /data/test 2 Enable the DVM PV resizing in the migrationcontroller configuration $ oc patch -n openshift-migration MigrationController migration-controller --type=json -p='[{"op":"add", "path": "/spec/enable_dvm_pv_resizing", "value": "true"}]' migrationcontroller.migration.openshift.io/migration-controller patched $ oc get migrationcontroller migration-controller -o yaml -n openshift-migration f:enable_dvm_pv_resizing: {} enable_dvm_pv_resizing: "true" 3 after controller pod restart, create a migration plan for this namespace using DVM 4 In controller cluster. configure a new threshold for the resize, 38% threshold $ oc -n openshift-migration patch migrationcontroller migration-controller -p '{"spec":{"pv_resizing_threshold": 38} }' --type='merge' migrationcontroller.migration.openshift.io/migration-controller patched 5 after controller pod restart, refresh the migplan $ oc -n openshift-migration patch migplan mig-plan-41883-ocp-41883-resizeconfig -p '{"spec": {"refresh":true} }' --type='merge' migplan.migration.openshift.io/mig-plan-41883-ocp-41883-resizeconfig patched 6. Check the migplan status $ oc get migplan mig-plan-41883-ocp-41883-resizeconfig -o yaml apiVersion: migration.openshift.io/v1alpha1 kind: MigPlan …. status: conditions: - category: Required lastTransitionTime: "2021-06-17T08:01:40Z" message: The `persistentVolumes` list has been updated with discovered PVs. reason: Done status: "True" type: PvsDiscovered - category: Required lastTransitionTime: "2021-06-17T08:01:40Z" message: The storage resources have been created. reason: Done status: "True" type: StorageEnsured - category: Required lastTransitionTime: "2021-06-17T08:01:40Z" message: The migration plan is ready. status: "True" type: Ready 7 In controller cluster. configure a new threshold for the resize, 41% threshold $ oc -n openshift-migration patch migrationcontroller migration-controller -p '{"spec":{"pv_resizing_threshold": 41} }' --type='merge' migrationcontroller.migration.openshift.io/migration-controller patched 8 after controller pod restart, refresh the migplan $ oc -n openshift-migration patch migplan mig-plan-41883-ocp-41883-resizeconfig -p '{"spec": {"refresh":true} }' --type='merge' migplan.migration.openshift.io/mig-plan-41883-ocp-41883-resizeconfig patched 9. Check the migplan status $ oc get migplan mig-plan-41883-ocp-41883-resizeconfig -o yaml apiVersion: migration.openshift.io/v1alpha1 kind: MigPlan …. status: conditions: - category: Required lastTransitionTime: "2021-06-17T08:01:40Z" message: The `persistentVolumes` list has been updated with discovered PVs. reason: Done status: "True" type: PvsDiscovered - category: Required lastTransitionTime: "2021-06-17T08:01:40Z" message: The storage resources have been created. reason: Done status: "True" type: StorageEnsured - category: Required lastTransitionTime: "2021-06-17T08:01:40Z" message: The migration plan is ready. status: "True" type: Ready 10 In controller cluster. configure a new threshold for the resize, 42% threshold $ oc -n openshift-migration patch migrationcontroller migration-controller -p '{"spec":{"pv_resizing_threshold": 42} }' --type='merge' migrationcontroller.migration.openshift.io/migration-controller patched 11 after controller pod restart, refresh the migplan $ oc -n openshift-migration patch migplan mig-plan-41883-ocp-41883-resizeconfig -p '{"spec": {"refresh":true} }' --type='merge' migplan.migration.openshift.io/mig-plan-41883-ocp-41883-resizeconfig patched 12. Check the migplan status $ oc get migplan mig-plan-41883-ocp-41883-resizeconfig -o yaml apiVersion: migration.openshift.io/v1alpha1 kind: MigPlan …. status: conditions: …….. - category: Warn durable: true lastTransitionTime: "2021-06-17T08:57:01Z" message: 'Capacity of the following volumes will be automatically adjusted to avoid disk capacity issues in the target cluster: [pvc-b800eb7b-cf3b-11eb-a3f7-0eae3e0555f3]' reason: Done status: "False" type: PvCapacityAdjustmentRequired Actual results: The storage usage in the source cluster is 63%, after setting the pv_resizing_threshold to 38%. There is no warning message in migplan. after setting the pv_resizing_threshold to 41%. There is no warning message in migplan. after setting the pv_resizing_threshold to 42%. There is a warning message in migplan. Expected results: The storage usage in the source cluster is 63%, after setting the value pv_resizing_threshold bigger than 37%, there should be a warning message “Capacity of the following volumes will be automatically adjusted to avoid disk capacity issues in the target cluster” in migplan. Additional info: In gp2, if the storage usage in source cluster is 98% and don’t set pv_resizing_threshold (use the default threshold 3%), won’t get warning in migplan either. The calculation of PV resize works as expected with glusterfs. Refer to another BZ https://bugzilla.redhat.com/show_bug.cgi?id=1963927