Bug 1535314 - Failed to expand PVC of cinder volume
Summary: Failed to expand PVC of cinder volume
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 3.9.0
Assignee: Tomas Smetana
QA Contact: Qin Ping
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-01-17 06:16 UTC by Qin Ping
Modified: 2018-03-28 14:20 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The OpenStack cloud provider did not fill the size attribute of the volumes for the V3 volume types. Consequence: The dynamic resizing of the OpenStack volumes did not work. Fix: The code was fixed and the internal structures are now being initialized correctly. Result: The dynamic volume resizing works correctly for the OpenStack volumes.
Clone Of:
Environment:
Last Closed: 2018-03-28 14:20:40 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:0489 0 None None None 2018-03-28 14:20:59 UTC

Description Qin Ping 2018-01-17 06:16:34 UTC
Description of problem:
Failed to expand PVC of cinder volume

Version-Release number of selected component (if applicable):
oc v3.9.0-0.16.0
openshift v3.9.0-0.16.0
kubernetes v1.9.0-beta1


How reproducible:
Always

Steps to Reproduce:
1. Enable feature-gates ExpandPersistentVolumes and admission-control PersistentVolumeClaimResize
2. Create a default storageclass, and set allowVolumeExpansion=true
3. Create a PVC using the default storageclass
4. Edit PVC, and change spec.resources.requests.storage from 5Gi to 6Gi
5. Check PVC and PV


Actual results:
PV expand from 5Gi to 6Gi
# oc get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                                   STORAGECLASS   REASON    AGE
pvc-10de6a9c-f6a2-11e7-abbc-fa163e1b6c1d   6Gi        RWO            Delete           Bound     piqin/claim-1                           standard                 6m
pvc-8d319cdb-f673-11e7-aa75-fa163e1b6c1d   1Gi        RWO            Delete           Bound     openshift-ansible-service-broker/etcd   standard                 5h

PVC report VolumeResizeFailed event and exist in "Resizing" condition
# oc describe pvc
Name:          claim-1
Namespace:     piqin
StorageClass:  standard
Status:        Bound
Volume:        pvc-10de6a9c-f6a2-11e7-abbc-fa163e1b6c1d
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed=yes
               pv.kubernetes.io/bound-by-controller=yes
               volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/cinder
Finalizers:    []
Capacity:      5Gi
Access Modes:  RWO
Events:
  Type     Reason                 Age   From                         Message
  ----     ------                 ----  ----                         -------
  Normal   ProvisioningSucceeded  1m    persistentvolume-controller  Successfully provisioned volume pvc-10de6a9c-f6a2-11e7-abbc-fa163e1b6c1d using kubernetes.io/cinder
  Warning  VolumeResizeFailed     18s   volume_expand                Invalid request due to incorrect syntax or missing required parameters.

# oc get pvc -o yaml
apiVersion: v1
items:
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      pv.kubernetes.io/bind-completed: "yes"
      pv.kubernetes.io/bound-by-controller: "yes"
      volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/cinder
    creationTimestamp: 2018-01-11T07:35:58Z
    name: claim-1
    namespace: piqin
    resourceVersion: "35013"
    selfLink: /api/v1/namespaces/piqin/persistentvolumeclaims/claim-1
    uid: 10de6a9c-f6a2-11e7-abbc-fa163e1b6c1d
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 6Gi
    storageClassName: standard
    volumeName: pvc-10de6a9c-f6a2-11e7-abbc-fa163e1b6c1d
  status:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 5Gi
    conditions:
    - lastProbeTime: null
      lastTransitionTime: 2018-01-11T07:42:39Z
      status: "True"
      type: Resizing
    phase: Bound
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""


Expected results:
PVC expand successfully.

Additional info:
# oc get sc -o yaml
apiVersion: v1
items:
- allowVolumeExpansion: true
  apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
    annotations:
      storageclass.beta.kubernetes.io/is-default-class: "true"
    creationTimestamp: 2018-01-11T02:00:17Z
    name: standard
    namespace: ""
    resourceVersion: "24919"
    selfLink: /apis/storage.k8s.io/v1/storageclasses/standard
    uid: 2bb0bd92-f673-11e7-aa75-fa163e1b6c1d
  parameters:
    fstype: xfs
  provisioner: kubernetes.io/cinder
  reclaimPolicy: Delete
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Comment 1 Tomas Smetana 2018-01-22 12:49:11 UTC
I have the problem reproduced with the latest kubernetes master (v1.10.0-alpha.1.1156+23226c24d4fd11) and OpenShift v3.9.0-0.22.0. The error comes from gophercloud failed REST call (essentially 404) and the confusing part is that sometimes the PVC stays in "Resizing" state without it (or I was not patient enough).

This needs to be fixed upstream first, so I'm focusing on kubernetes now.

Comment 3 Tomas Smetana 2018-01-23 12:01:28 UTC
The problem is in the openstack cloud provider: it does not fill in the "Size" attribute for the V3 API type volumes. It's a one-line patch.

Comment 4 Tomas Smetana 2018-01-23 12:35:02 UTC
https://github.com/kubernetes/kubernetes/pull/58685

Comment 5 Tomas Smetana 2018-01-23 15:59:53 UTC
https://github.com/openshift/origin/pull/18237

Comment 7 Qin Ping 2018-02-22 07:24:45 UTC
Verified in openshift v3.9.0-0.47.0

Comment 10 errata-xmlrpc 2018-03-28 14:20:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0489


Note You need to log in before you can comment on or make changes to this bug.