Description of problem: Pods are unable to mount persistent volumes, since PVs are created with nodeAffinity rules and zone labels. Customer has clouds.conf configured with "ignore-volume-az = true" but this doesn't seem to be respected by the controllers when creating the PVs. Version-Release number of selected component (if applicable): Openshift 4.2.12 How reproducible: Everytime PV is created either dynamic or static. Steps to Reproduce: Customer used a Kubernetes 1.14 deployment with and without PersistentVolumeLabel on the admission controller and saw that without PersistentVolumeLabel the PVs are not created with zone labels and nodeAffinity rules, but seems to be hard coded on OCP. Also we noticed that this admission controller along with the flags --cloud-provider= have been depecrated on Kubernetes 1.14. Actual results: Persistent volumes are created with zone labels . Expected results: Customer would like to have the same behaviour as it was on OCP v3.11. Master Log: Node Log (of failed PODs): events on the pods when trying to attach the volumes: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 28s default-scheduler pv "pvc-758068e4-09ce-11ea-bca5-fa163ec4f8d9" node affinity doesn't match node "sbx42-69jrk-worker-sn5dj": No matching NodeSelectorTerms Warning FailedScheduling 28s default-scheduler 0/6 nodes are available: 3 node(s) had taints that the pod didn't tolerate, 3 node(s) had volume node affinity conflict. PV Dump: apiVersion: v1 kind: PersistentVolume metadata: annotations: kubernetes.io/createdby: cinder-dynamic-provisioner pv.kubernetes.io/bound-by-controller: "yes" pv.kubernetes.io/provisioned-by: kubernetes.io/cinder creationTimestamp: 2019-11-18T06:41:48Z finalizers: - kubernetes.io/pv-protection labels: failure-domain.beta.kubernetes.io/region: Groningen2 failure-domain.beta.kubernetes.io/zone: nova name: pvc-758068e4-09ce-11ea-bca5-fa163ec4f8d9 resourceVersion: "1116946" selfLink: /api/v1/persistentvolumes/pvc-758068e4-09ce-11ea-bca5-fa163ec4f8d9 uid: 7eff40e8-09ce-11ea-ace3-fa163ef78756 spec: accessModes: - ReadWriteOnce capacity: storage: 1Gi cinder: volumeID: d652d12a-a43c-42fc-936d-fb8b1b04d294 claimRef: apiVersion: v1 kind: PersistentVolumeClaim name: postgresql namespace: postgres resourceVersion: "1116934" uid: 758068e4-09ce-11ea-bca5-fa163ec4f8d9 nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: failure-domain.beta.kubernetes.io/zone operator: In values: - nova - key: failure-domain.beta.kubernetes.io/region operator: In values: - Groningen2 persistentVolumeReclaimPolicy: Delete storageClassName: standard volumeMode: Filesystem status: phase: Bound PVC Dump: apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{"openshift.io/generated-by":"OpenShiftNewApp"},"creationTimestamp":null,"labels":{"app":"postgresql-persistent","template":"postgresql-persistent-template"},"name":"postgresql","namespace":"postgres"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}}},"status":{"phase":"Pending"}} openshift.io/generated-by: OpenShiftNewApp pv.kubernetes.io/bind-completed: "yes" pv.kubernetes.io/bound-by-controller: "yes" volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/cinder volume.kubernetes.io/selected-node: sbx42-69jrk-worker-sn5dj creationTimestamp: 2019-11-18T06:41:32Z finalizers: - kubernetes.io/pvc-protection labels: app: postgresql-persistent template: postgresql-persistent-template name: postgresql namespace: postgres resourceVersion: "1116949" selfLink: /api/v1/namespaces/postgres/persistentvolumeclaims/postgresql uid: 758068e4-09ce-11ea-bca5-fa163ec4f8d9 spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: standard volumeMode: Filesystem volumeName: pvc-758068e4-09ce-11ea-bca5-fa163ec4f8d9 status: accessModes: - ReadWriteOnce capacity: storage: 1Gi phase: Bound StorageClass Dump (if StorageClass used by PV/PVC): apiVersion: v1 items: - allowVolumeExpansion: true apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: annotations: storageclass.kubernetes.io/is-default-class: "true" creationTimestamp: 2019-12-20T15:39:31Z name: standard namespace: "" ownerReferences: - apiVersion: v1 kind: clusteroperator name: storage uid: 5dda2672-233d-11ea-a418-fa163e2e4435 resourceVersion: "10892" selfLink: /apis/storage.k8s.io/v1/storageclasses/standard uid: ea764510-233e-11ea-a1f4-fa163e5cb724 provisioner: kubernetes.io/cinder reclaimPolicy: Delete volumeBindingMode: WaitForFirstConsumer kind: List metadata: resourceVersion: "" selfLink: "" Additional info:
Can we backport this change in 4.4?
Rebase to 1.18 has finally landed in RHCOS. Tested on today's nightly, 4.5.0-0.nightly-2020-04-16-034119. $ oc get node NAME STATUS ROLES AGE VERSION ip-10-0-134-66.us-east-2.compute.internal Ready master 4h48m v1.18.0-rc.1
Hi @Jaspreet, not sure if I misleading you as I paste two PV info for comparison, and the following one is verified with "ignore-volume-az = yes". There is no label and volume node affinity. [wduan@MINT 01_general]$ oc get pv pvc-f8368565-dc1c-4344-9e11-329382c24e37 -o yaml apiVersion: v1 kind: PersistentVolume metadata: annotations: kubernetes.io/createdby: cinder-dynamic-provisioner pv.kubernetes.io/bound-by-controller: "yes" pv.kubernetes.io/provisioned-by: kubernetes.io/cinder creationTimestamp: "2020-04-20T13:14:42Z" finalizers: - kubernetes.io/pv-protection managedFields: - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:kubernetes.io/createdby: {} f:pv.kubernetes.io/bound-by-controller: {} f:pv.kubernetes.io/provisioned-by: {} f:spec: f:accessModes: {} f:capacity: .: {} f:storage: {} f:cinder: .: {} f:volumeID: {} f:claimRef: .: {} f:apiVersion: {} f:kind: {} f:name: {} f:namespace: {} f:resourceVersion: {} f:uid: {} f:persistentVolumeReclaimPolicy: {} f:storageClassName: {} f:volumeMode: {} f:status: f:phase: {} manager: kube-controller-manager operation: Update time: "2020-04-20T13:14:42Z" name: pvc-f8368565-dc1c-4344-9e11-329382c24e37 resourceVersion: "48905" selfLink: /api/v1/persistentvolumes/pvc-f8368565-dc1c-4344-9e11-329382c24e37 uid: 68d4e1ad-0846-43c9-a05d-bc884d4f5d17 spec: accessModes: - ReadWriteOnce capacity: storage: 2Gi cinder: volumeID: ed511641-d28d-4848-8937-9a61b4c3e34f claimRef: apiVersion: v1 kind: PersistentVolumeClaim name: mypvc02 namespace: wduan resourceVersion: "48896" uid: f8368565-dc1c-4344-9e11-329382c24e37 persistentVolumeReclaimPolicy: Delete storageClassName: standard volumeMode: Filesystem status: phase: Bound
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409