Bug 1660946 - expanded pvcs for azure storage don't reflect new size
Summary: expanded pvcs for azure storage don't reflect new size
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 3.10.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Hemant Kumar
QA Contact: Liang Xia
URL:
Whiteboard:
Depends On:
Blocks: 1724792
TreeView+ depends on / blocked
 
Reported: 2018-12-19 16:41 UTC by Mark LaBonte
Modified: 2019-06-28 16:04 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-01-04 17:39:20 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Mark LaBonte 2018-12-19 16:41:40 UTC
Description of problem:

After expanding a pvc which was  dynamically provisioned using the azure-disk provisioner, the new size isn't reflected for status.capacity.storage or from within the associated pod.

Version-Release number of selected component (if applicable):

[cloud-user@ocp-master-1 ~]$ oc version
oc v3.10.83
kubernetes v1.10.0+b81c8f8
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://mark-auto-test.eastus.cloudapp.azure.com:8443
openshift v3.10.83
kubernetes v1.10.0+b81c8f8

How reproducible:

100%

Steps to Reproduce:
1. configure pvc expansion according to documentation
2. oc new-app jenkins-persistent
3. oc edit pvc jenkins (change spec.resources.requests.storage from 1Gi to 2Gi)
4. scale jenkins pod to 0, then back to 1
5 check status.capacity.storage and check filesystem size for jenkins pod


PV Dump:

[cloud-user@ocp-master-1 ~]$ oc get pv pvc-8830c458-02f4-11e9-92bd-000d3a1fa049 -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/bound-by-controller: "yes"
    pv.kubernetes.io/provisioned-by: kubernetes.io/azure-disk
    volumehelper.VolumeDynamicallyCreatedByKey: azure-disk-dynamic-provisioner
  creationTimestamp: 2018-12-18T18:41:34Z
  finalizers:
  - kubernetes.io/pv-protection
  name: pvc-8830c458-02f4-11e9-92bd-000d3a1fa049
  resourceVersion: "1222377"
  selfLink: /api/v1/persistentvolumes/pvc-8830c458-02f4-11e9-92bd-000d3a1fa049
  uid: 8b8b0d67-02f4-11e9-9953-000d3a1fa13e
spec:
  accessModes:
  - ReadWriteOnce
  azureDisk:
    cachingMode: None
    diskName: kubernetes-dynamic-pvc-8830c458-02f4-11e9-92bd-000d3a1fa049
    diskURI: /subscriptions/dcc74c29-4db6-4c49-9a0f-ac0ee03fa17e/resourceGroups/mark-auto-test/providers/Microsoft.Compute/disks/kubernetes-dynamic-pvc-8830c458-02f4-11e9-92bd-000d3a1fa049
    fsType: ""
    kind: Managed
    readOnly: false
  capacity:
    storage: 1Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: jenkins
    namespace: test2
    resourceVersion: "1222339"
    uid: 8830c458-02f4-11e9-92bd-000d3a1fa049
  persistentVolumeReclaimPolicy: Delete
  storageClassName: azure-storage
status:
  phase: Bound
[cloud-user@ocp-master-1 ~]$


PVC Dump:

[cloud-user@ocp-master-1 ~]$ oc get pvc jenkins -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    openshift.io/generated-by: OpenShiftNewApp
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/azure-disk
  creationTimestamp: 2018-12-18T18:41:29Z
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: jenkins-persistent
    template: jenkins-persistent-template
  name: jenkins
  namespace: test2
  resourceVersion: "1235561"
  selfLink: /api/v1/namespaces/test2/persistentvolumeclaims/jenkins
  uid: 8830c458-02f4-11e9-92bd-000d3a1fa049
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: azure-storage
  volumeName: pvc-8830c458-02f4-11e9-92bd-000d3a1fa049
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  conditions:
  - lastProbeTime: null
    lastTransitionTime: 2018-12-18T19:57:50Z
    status: "True"
    type: Resizing
  phase: Bound
[cloud-user@ocp-master-1 ~]$ 


StorageClass Dump (if StorageClass used by PV/PVC):

[cloud-user@ocp-master-1 ~]$ oc get storageclass azure-storage -o yaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.beta.kubernetes.io/is-default-class: "true"
  creationTimestamp: 2018-12-17T20:02:48Z
  name: azure-storage
  resourceVersion: "987739"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/azure-storage
  uid: ba16ce03-0236-11e9-92bd-000d3a1fa049
parameters:
  kind: managed
  storageaccounttype: Standard_LRS
provisioner: kubernetes.io/azure-disk
reclaimPolicy: Delete
volumeBindingMode: Immediate
[cloud-user@ocp-master-1 ~]$ 


Additional info:

[cloud-user@ocp-master-1 ~]$ sudo cat /etc/origin/master/master-config.yaml
admissionConfig:
  pluginConfig:
    BuildDefaults:
      configuration:
        apiVersion: v1
        env: []
        kind: BuildDefaultsConfig
        resources:
          limits: {}
          requests: {}
    BuildOverrides:
      configuration:
        apiVersion: v1
        kind: BuildOverridesConfig
    openshift.io/ImagePolicy:
      configuration:
        apiVersion: v1
        executionRules:
        - matchImageAnnotations:
          - key: images.openshift.io/deny-execution
            value: 'true'
          name: execution-denied
          onResources:
          - resource: pods
          - resource: builds
          reject: true
          skipOnResolutionFailure: true
        kind: ImagePolicyConfig
aggregatorConfig:
  proxyClientInfo:
    certFile: aggregator-front-proxy.crt
    keyFile: aggregator-front-proxy.key
apiLevels:
- v1
apiVersion: v1
authConfig:
  requestHeader:
    clientCA: front-proxy-ca.crt
    clientCommonNames:
    - aggregator-front-proxy
    extraHeaderPrefixes:
    - X-Remote-Extra-
    groupHeaders:
    - X-Remote-Group
    usernameHeaders:
    - X-Remote-User
controllerConfig:
  election:
    lockName: openshift-master-controllers
  serviceServingCert:
    signer:
      certFile: service-signer.crt
      keyFile: service-signer.key
controllers: '*'
corsAllowedOrigins:
- (?i)//127\.0\.0\.1(:|\z)
- (?i)//localhost(:|\z)
- (?i)//192\.168\.0\.4(:|\z)
- (?i)//mark\-auto\-test\.eastus\.cloudapp\.azure\.com(:|\z)
- (?i)//kubernetes\.default(:|\z)
- (?i)//kubernetes\.default\.svc\.cluster\.local(:|\z)
- (?i)//kubernetes(:|\z)
- (?i)//openshift\.default(:|\z)
- (?i)//openshift\.default\.svc(:|\z)
- (?i)//kubernetes\.default\.svc(:|\z)
- (?i)//172\.30\.0\.1(:|\z)
- (?i)//openshift\.default\.svc\.cluster\.local(:|\z)
- (?i)//ocp\-master\-1(:|\z)
- (?i)//openshift(:|\z)
dnsConfig:
  bindAddress: 0.0.0.0:8053
  bindNetwork: tcp4
etcdClientInfo:
  ca: master.etcd-ca.crt
  certFile: master.etcd-client.crt
  keyFile: master.etcd-client.key
  urls:
  - https://ocp-master-1:2379
  - https://ocp-master-2:2379
  - https://ocp-master-3:2379
etcdStorageConfig:
  kubernetesStoragePrefix: kubernetes.io
  kubernetesStorageVersion: v1
  openShiftStoragePrefix: openshift.io
  openShiftStorageVersion: v1
imageConfig:
  format: registry.access.redhat.com/openshift3/ose-${component}:${version}
  latest: false
imagePolicyConfig:
  internalRegistryHostname: docker-registry.default.svc:5000
kind: MasterConfig
kubeletClientInfo:
  ca: ca-bundle.crt
  certFile: master.kubelet-client.crt
  keyFile: master.kubelet-client.key
  port: 10250
kubernetesMasterConfig:
  apiServerArguments:
    cloud-config:
    - /etc/origin/cloudprovider/azure.conf
    cloud-provider:
    - azure
    storage-backend:
    - etcd3
    storage-media-type:
    - application/vnd.kubernetes.protobuf
    feature-gates:
    - ExpandPersistentVolumes=true
  controllerArguments:
    cloud-config:
    - /etc/origin/cloudprovider/azure.conf
    cloud-provider:
    - azure
    cluster-signing-cert-file:
    - /etc/origin/master/ca.crt
    cluster-signing-key-file:
    - /etc/origin/master/ca.key
  masterCount: 3
  masterIP: 192.168.0.4
  podEvictionTimeout: null
  proxyClientInfo:
    certFile: master.proxy-client.crt
    keyFile: master.proxy-client.key
  schedulerArguments: null
  schedulerConfigFile: /etc/origin/master/scheduler.json
  servicesNodePortRange: ''
  servicesSubnet: 172.30.0.0/16
  staticNodeNames: []
masterClients:
  externalKubernetesClientConnectionOverrides:
    acceptContentTypes: application/vnd.kubernetes.protobuf,application/json
    burst: 400
    contentType: application/vnd.kubernetes.protobuf
    qps: 200
  externalKubernetesKubeConfig: ''
  openshiftLoopbackClientConnectionOverrides:
    acceptContentTypes: application/vnd.kubernetes.protobuf,application/json
    burst: 600
    contentType: application/vnd.kubernetes.protobuf
    qps: 300
  openshiftLoopbackKubeConfig: openshift-master.kubeconfig
masterPublicURL: https://mark-auto-test.eastus.cloudapp.azure.com:8443
networkConfig:
  clusterNetworks:
  - cidr: 10.128.0.0/14
    hostSubnetLength: 9
  externalIPNetworkCIDRs:
  - 0.0.0.0/0
  networkPluginName: redhat/openshift-ovs-multitenant
  serviceNetworkCIDR: 172.30.0.0/16
oauthConfig:
  assetPublicURL: https://mark-auto-test.eastus.cloudapp.azure.com:8443/console/
  grantConfig:
    method: auto
  identityProviders:
  - challenge: true
    login: true
    mappingMethod: claim
    name: allow_all
    provider:
      apiVersion: v1
      kind: AllowAllPasswordIdentityProvider
  masterCA: ca-bundle.crt
  masterPublicURL: https://mark-auto-test.eastus.cloudapp.azure.com:8443
  masterURL: https://mark-auto-test.eastus.cloudapp.azure.com:8443
  sessionConfig:
    sessionMaxAgeSeconds: 3600
    sessionName: ssn
    sessionSecretsFile: /etc/origin/master/session-secrets.yaml
  tokenConfig:
    accessTokenMaxAgeSeconds: 86400
    authorizeTokenMaxAgeSeconds: 500
pauseControllers: false
policyConfig:
  bootstrapPolicyFile: /etc/origin/master/policy.json
  openshiftInfrastructureNamespace: openshift-infra
  openshiftSharedResourcesNamespace: openshift
projectConfig:
  defaultNodeSelector: node-role.kubernetes.io/compute=true
  projectRequestMessage: ''
  projectRequestTemplate: ''
  securityAllocator:
    mcsAllocatorRange: s0:/2
    mcsLabelsPerProject: 5
    uidAllocatorRange: 1000000000-1999999999/10000
routingConfig:
  subdomain: 40.80.144.55.nip.io
serviceAccountConfig:
  limitSecretReferences: false
  managedNames:
  - default
  - builder
  - deployer
  masterCA: ca-bundle.crt
  privateKeyFile: serviceaccounts.private.key
  publicKeyFiles:
  - serviceaccounts.public.key
servingInfo:
  bindAddress: 0.0.0.0:8443
  bindNetwork: tcp4
  certFile: master.server.crt
  clientCA: ca.crt
  keyFile: master.server.key
  maxRequestsInFlight: 500
  requestTimeoutSeconds: 3600
volumeConfig:
  dynamicProvisioningEnabled: true

Comment 1 Hemant Kumar 2019-01-04 17:39:20 UTC
3.10 does not support Azure volume resizing. This was added in 3.11. If you do "oc describe pvc", PVC should have events that indicate that.


Note You need to log in before you can comment on or make changes to this bug.