Bug 1786446 - [BUG] Controllers don't seem to respect 'ignore-volume-az = true'
Summary: [BUG] Controllers don't seem to respect 'ignore-volume-az = true'
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 4.2.z
Hardware: All
OS: Linux
high
high
Target Milestone: ---
: 4.5.0
Assignee: Hemant Kumar
QA Contact: Wei Duan
URL:
Whiteboard:
Depends On:
Blocks: 1822667
TreeView+ depends on / blocked
 
Reported: 2019-12-25 09:04 UTC by Andre Costa
Modified: 2023-10-06 18:57 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1822667 1822680 (view as bug list)
Environment:
Last Closed: 2020-07-13 17:13:00 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:13:16 UTC

Description Andre Costa 2019-12-25 09:04:04 UTC
Description of problem:
Pods are unable to mount persistent volumes, since PVs are created with nodeAffinity rules and zone labels. Customer has clouds.conf configured with "ignore-volume-az = true" but this doesn't seem to be respected by the controllers when creating the PVs.

Version-Release number of selected component (if applicable):
Openshift 4.2.12

How reproducible:
Everytime PV is created either dynamic or static.

Steps to Reproduce:
Customer used a Kubernetes 1.14 deployment with and without PersistentVolumeLabel on the admission controller and saw that without PersistentVolumeLabel the PVs are not created with zone labels and nodeAffinity rules, but seems to be hard coded on OCP.
Also we noticed that this admission controller along with the flags --cloud-provider= have been depecrated on Kubernetes 1.14.

Actual results:
Persistent volumes are created with zone labels .

Expected results:
Customer would like to have the same behaviour as it was on OCP v3.11.

Master Log:

Node Log (of failed PODs):
events on the pods when trying to attach the volumes:
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  28s   default-scheduler  pv "pvc-758068e4-09ce-11ea-bca5-fa163ec4f8d9" node affinity doesn't match node "sbx42-69jrk-worker-sn5dj": No matching NodeSelectorTerms
  Warning  FailedScheduling  28s   default-scheduler  0/6 nodes are available: 3 node(s) had taints that the pod didn't tolerate, 3 node(s) had volume node affinity conflict.

PV Dump:
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    kubernetes.io/createdby: cinder-dynamic-provisioner
    pv.kubernetes.io/bound-by-controller: "yes"
    pv.kubernetes.io/provisioned-by: kubernetes.io/cinder
  creationTimestamp: 2019-11-18T06:41:48Z
  finalizers:
  - kubernetes.io/pv-protection
  labels:
    failure-domain.beta.kubernetes.io/region: Groningen2
    failure-domain.beta.kubernetes.io/zone: nova
  name: pvc-758068e4-09ce-11ea-bca5-fa163ec4f8d9
  resourceVersion: "1116946"
  selfLink: /api/v1/persistentvolumes/pvc-758068e4-09ce-11ea-bca5-fa163ec4f8d9
  uid: 7eff40e8-09ce-11ea-ace3-fa163ef78756
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  cinder:
    volumeID: d652d12a-a43c-42fc-936d-fb8b1b04d294
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: postgresql
    namespace: postgres
    resourceVersion: "1116934"
    uid: 758068e4-09ce-11ea-bca5-fa163ec4f8d9
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: failure-domain.beta.kubernetes.io/zone
          operator: In
          values:
          - nova
        - key: failure-domain.beta.kubernetes.io/region
          operator: In
          values:
          - Groningen2
  persistentVolumeReclaimPolicy: Delete
  storageClassName: standard
  volumeMode: Filesystem
status:
  phase: Bound


PVC Dump:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{"openshift.io/generated-by":"OpenShiftNewApp"},"creationTimestamp":null,"labels":{"app":"postgresql-persistent","template":"postgresql-persistent-template"},"name":"postgresql","namespace":"postgres"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}}},"status":{"phase":"Pending"}}
    openshift.io/generated-by: OpenShiftNewApp
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/cinder
    volume.kubernetes.io/selected-node: sbx42-69jrk-worker-sn5dj
  creationTimestamp: 2019-11-18T06:41:32Z
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: postgresql-persistent
    template: postgresql-persistent-template
  name: postgresql
  namespace: postgres
  resourceVersion: "1116949"
  selfLink: /api/v1/namespaces/postgres/persistentvolumeclaims/postgresql
  uid: 758068e4-09ce-11ea-bca5-fa163ec4f8d9
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: standard
  volumeMode: Filesystem
  volumeName: pvc-758068e4-09ce-11ea-bca5-fa163ec4f8d9
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  phase: Bound


StorageClass Dump (if StorageClass used by PV/PVC):

apiVersion: v1
items:
- allowVolumeExpansion: true
  apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
    annotations:
      storageclass.kubernetes.io/is-default-class: "true"
    creationTimestamp: 2019-12-20T15:39:31Z
    name: standard
    namespace: ""
    ownerReferences:
    - apiVersion: v1
      kind: clusteroperator
      name: storage
      uid: 5dda2672-233d-11ea-a418-fa163e2e4435
    resourceVersion: "10892"
    selfLink: /apis/storage.k8s.io/v1/storageclasses/standard
    uid: ea764510-233e-11ea-a1f4-fa163e5cb724
  provisioner: kubernetes.io/cinder
  reclaimPolicy: Delete
  volumeBindingMode: WaitForFirstConsumer
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Additional info:

Comment 6 Mike Fedosin 2020-03-17 21:48:49 UTC
Can we backport this change in 4.4?

Comment 8 Jan Safranek 2020-04-16 12:18:43 UTC
Rebase to 1.18 has finally landed in RHCOS. Tested on today's nightly, 4.5.0-0.nightly-2020-04-16-034119.

$ oc get node
NAME                                         STATUS   ROLES    AGE     VERSION
ip-10-0-134-66.us-east-2.compute.internal    Ready    master   4h48m   v1.18.0-rc.1

Comment 15 Wei Duan 2020-04-22 02:21:04 UTC
Hi @Jaspreet, not sure if I misleading you as I paste two PV info for comparison, and the following one is verified with "ignore-volume-az = yes".
There is no label and volume node affinity. 

[wduan@MINT 01_general]$ oc get pv pvc-f8368565-dc1c-4344-9e11-329382c24e37 -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    kubernetes.io/createdby: cinder-dynamic-provisioner
    pv.kubernetes.io/bound-by-controller: "yes"
    pv.kubernetes.io/provisioned-by: kubernetes.io/cinder
  creationTimestamp: "2020-04-20T13:14:42Z"
  finalizers:
  - kubernetes.io/pv-protection
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kubernetes.io/createdby: {}
          f:pv.kubernetes.io/bound-by-controller: {}
          f:pv.kubernetes.io/provisioned-by: {}
      f:spec:
        f:accessModes: {}
        f:capacity:
          .: {}
          f:storage: {}
        f:cinder:
          .: {}
          f:volumeID: {}
        f:claimRef:
          .: {}
          f:apiVersion: {}
          f:kind: {}
          f:name: {}
          f:namespace: {}
          f:resourceVersion: {}
          f:uid: {}
        f:persistentVolumeReclaimPolicy: {}
        f:storageClassName: {}
        f:volumeMode: {}
      f:status:
        f:phase: {}
    manager: kube-controller-manager
    operation: Update
    time: "2020-04-20T13:14:42Z"
  name: pvc-f8368565-dc1c-4344-9e11-329382c24e37
  resourceVersion: "48905"
  selfLink: /api/v1/persistentvolumes/pvc-f8368565-dc1c-4344-9e11-329382c24e37
  uid: 68d4e1ad-0846-43c9-a05d-bc884d4f5d17
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 2Gi
  cinder:
    volumeID: ed511641-d28d-4848-8937-9a61b4c3e34f
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: mypvc02
    namespace: wduan
    resourceVersion: "48896"
    uid: f8368565-dc1c-4344-9e11-329382c24e37
  persistentVolumeReclaimPolicy: Delete
  storageClassName: standard
  volumeMode: Filesystem
status:
  phase: Bound

Comment 22 errata-xmlrpc 2020-07-13 17:13:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.