[Cinder-csi-driver] Filesystem did not expand for on-line volume resize storage/Openstack CSI Driver H Description of problem: Filesystem did not expand for on-line volume resizing although the pvc is expanded Version-Release number of selected component (if applicable): 4.7.0-0.nightly-2021-01-21-215614 How reproducible: Always Steps to Reproduce: 1. Create cluster on OSP with Cinder-csi-driver installed 2. Create pod(mypod) and pvc(mypvc) with sc standard-csi $ oc get pod NAME READY STATUS RESTARTS AGE mypod 1/1 Running 0 21m $ oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE mypvc Bound pvc-0e27b8cb-ed0e-4291-8b74-a7faf29e7c6d 2Gi RWO standard-csi 21m 3. Resize pvc from 2Gi to 4Gi: $ oc patch pvc mypvc -p '{"spec": {"resources": {"requests": {"storage": "4Gi"}}}}' persistentvolumeclaim/mypvc patched 4. Check the pvc status is expanded and MountVolume.NodeExpandVolume succeeded for volume from the log $ oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE mypvc Bound pvc-0e27b8cb-ed0e-4291-8b74-a7faf29e7c6d 4Gi RWO standard-csi 22m $ oc describe pvc Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal WaitForPodScheduled 10m persistentvolume-controller waiting for pod mypod to be scheduled Normal Provisioning 10m cinder.csi.openstack.org_wduan-0122a-ccdlk-master-0_3aaa58ac-9c3a-4390-af9d-bc38c21ca0ae External provisioner is provisioning volume for claim "wduan-resize/mypvc" Normal ExternalProvisioning 10m persistentvolume-controller waiting for a volume to be created, either by external provisioner "cinder.csi.openstack.org" or manually created by system administrator Normal ProvisioningSucceeded 10m cinder.csi.openstack.org_wduan-0122a-ccdlk-master-0_3aaa58ac-9c3a-4390-af9d-bc38c21ca0ae Successfully provisioned volume pvc-0e27b8cb-ed0e-4291-8b74-a7faf29e7c6d Warning ExternalExpanding <invalid> volume_expand Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC. Normal Resizing <invalid> external-resizer cinder.csi.openstack.org External resizer is resizing volume pvc-0e27b8cb-ed0e-4291-8b74-a7faf29e7c6d Normal FileSystemResizeRequired <invalid> external-resizer cinder.csi.openstack.org Require file system resize of volume on node Normal FileSystemResizeSuccessful <invalid> kubelet, wduan-0122a-ccdlk-worker-0-z8z56 MountVolume.NodeExpandVolume succeeded for volume "pvc-0e27b8cb-ed0e-4291-8b74-a7faf29e7c6d" 5. Check in pod the volume is 4Gi but the filesysyem is still 2Gi: $ oc rsh mypod # lsblk | grep vdb vdb 252:16 0 4G 0 disk /mnt/local # df -h | grep vdb /dev/vdb 2.0G 6.0M 1.9G 1% /mnt/local Actual results: Filesystem did not be expanded. Expected results: Filesystem should be expanded. Master Log: Node Log (of failed PODs): PV Dump: PVC Dump: StorageClass Dump (if StorageClass used by PV/PVC): Additional info:
Is this a regression from 4.6? Was volume resizing available in 4.6? We might not be able to fix within 4.7; targeting 4.8 and definitely planning on backporting.
It is not a regression as we just have the cinder-csi-driver. Maybe you want to ask about cinder in-tree, it doesn't support on-line expandtion, so agree to change targeting 4.8 and backporting. Thanks.
It is an upstream issue, the fix is on review
https://github.com/kubernetes/cloud-provider-openstack/pull/1434
*** Bug 1906391 has been marked as a duplicate of this bug. ***
Verified pass on 4.8.0-0.nightly-2021-03-30-160509. $ oc get pvc -w NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE mypvc Bound pvc-c2cbf54e-07df-4e14-870e-ec1de20175b5 2Gi RWO standard-csi 22m mypvc Bound pvc-c2cbf54e-07df-4e14-870e-ec1de20175b5 3Gi RWO standard-csi 23m # df -h Filesystem Size Used Avail Use% Mounted on ... /dev/vdb 2.9G 1.9G 1008M 66% /mnt/local ...
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438