Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 1535314 - Failed to expand PVC of cinder volume
Failed to expand PVC of cinder volume
Status: CLOSED ERRATA
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage (Show other bugs)
3.9.0
Unspecified Unspecified
medium Severity medium
: ---
: 3.9.0
Assigned To: Tomas Smetana
Qin Ping
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2018-01-17 01:16 EST by Qin Ping
Modified: 2018-03-28 10:20 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The OpenStack cloud provider did not fill the size attribute of the volumes for the V3 volume types. Consequence: The dynamic resizing of the OpenStack volumes did not work. Fix: The code was fixed and the internal structures are now being initialized correctly. Result: The dynamic volume resizing works correctly for the OpenStack volumes.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2018-03-28 10:20:40 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:0489 None None None 2018-03-28 10:20 EDT

  None (edit)
Description Qin Ping 2018-01-17 01:16:34 EST
Description of problem:
Failed to expand PVC of cinder volume

Version-Release number of selected component (if applicable):
oc v3.9.0-0.16.0
openshift v3.9.0-0.16.0
kubernetes v1.9.0-beta1


How reproducible:
Always

Steps to Reproduce:
1. Enable feature-gates ExpandPersistentVolumes and admission-control PersistentVolumeClaimResize
2. Create a default storageclass, and set allowVolumeExpansion=true
3. Create a PVC using the default storageclass
4. Edit PVC, and change spec.resources.requests.storage from 5Gi to 6Gi
5. Check PVC and PV


Actual results:
PV expand from 5Gi to 6Gi
# oc get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                                   STORAGECLASS   REASON    AGE
pvc-10de6a9c-f6a2-11e7-abbc-fa163e1b6c1d   6Gi        RWO            Delete           Bound     piqin/claim-1                           standard                 6m
pvc-8d319cdb-f673-11e7-aa75-fa163e1b6c1d   1Gi        RWO            Delete           Bound     openshift-ansible-service-broker/etcd   standard                 5h

PVC report VolumeResizeFailed event and exist in "Resizing" condition
# oc describe pvc
Name:          claim-1
Namespace:     piqin
StorageClass:  standard
Status:        Bound
Volume:        pvc-10de6a9c-f6a2-11e7-abbc-fa163e1b6c1d
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed=yes
               pv.kubernetes.io/bound-by-controller=yes
               volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/cinder
Finalizers:    []
Capacity:      5Gi
Access Modes:  RWO
Events:
  Type     Reason                 Age   From                         Message
  ----     ------                 ----  ----                         -------
  Normal   ProvisioningSucceeded  1m    persistentvolume-controller  Successfully provisioned volume pvc-10de6a9c-f6a2-11e7-abbc-fa163e1b6c1d using kubernetes.io/cinder
  Warning  VolumeResizeFailed     18s   volume_expand                Invalid request due to incorrect syntax or missing required parameters.

# oc get pvc -o yaml
apiVersion: v1
items:
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      pv.kubernetes.io/bind-completed: "yes"
      pv.kubernetes.io/bound-by-controller: "yes"
      volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/cinder
    creationTimestamp: 2018-01-11T07:35:58Z
    name: claim-1
    namespace: piqin
    resourceVersion: "35013"
    selfLink: /api/v1/namespaces/piqin/persistentvolumeclaims/claim-1
    uid: 10de6a9c-f6a2-11e7-abbc-fa163e1b6c1d
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 6Gi
    storageClassName: standard
    volumeName: pvc-10de6a9c-f6a2-11e7-abbc-fa163e1b6c1d
  status:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 5Gi
    conditions:
    - lastProbeTime: null
      lastTransitionTime: 2018-01-11T07:42:39Z
      status: "True"
      type: Resizing
    phase: Bound
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""


Expected results:
PVC expand successfully.

Additional info:
# oc get sc -o yaml
apiVersion: v1
items:
- allowVolumeExpansion: true
  apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
    annotations:
      storageclass.beta.kubernetes.io/is-default-class: "true"
    creationTimestamp: 2018-01-11T02:00:17Z
    name: standard
    namespace: ""
    resourceVersion: "24919"
    selfLink: /apis/storage.k8s.io/v1/storageclasses/standard
    uid: 2bb0bd92-f673-11e7-aa75-fa163e1b6c1d
  parameters:
    fstype: xfs
  provisioner: kubernetes.io/cinder
  reclaimPolicy: Delete
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
Comment 1 Tomas Smetana 2018-01-22 07:49:11 EST
I have the problem reproduced with the latest kubernetes master (v1.10.0-alpha.1.1156+23226c24d4fd11) and OpenShift v3.9.0-0.22.0. The error comes from gophercloud failed REST call (essentially 404) and the confusing part is that sometimes the PVC stays in "Resizing" state without it (or I was not patient enough).

This needs to be fixed upstream first, so I'm focusing on kubernetes now.
Comment 3 Tomas Smetana 2018-01-23 07:01:28 EST
The problem is in the openstack cloud provider: it does not fill in the "Size" attribute for the V3 API type volumes. It's a one-line patch.
Comment 4 Tomas Smetana 2018-01-23 07:35:02 EST
https://github.com/kubernetes/kubernetes/pull/58685
Comment 5 Tomas Smetana 2018-01-23 10:59:53 EST
https://github.com/openshift/origin/pull/18237
Comment 7 Qin Ping 2018-02-22 02:24:45 EST
Verified in openshift v3.9.0-0.47.0
Comment 10 errata-xmlrpc 2018-03-28 10:20:40 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0489

Note You need to log in before you can comment on or make changes to this bug.