Hide Forgot
Description of problem: After installation of OCP 4.11 with eCCM and application credentials, PVC creation fails to provision a StorageClass: ``` failed to provision volume with StorageClass "csi-manila-default": rpc error: code = InvalidArgument desc = invalid OpenStack secrets: parameter 'os-authURL' requires exactly one of [os-password os-trustID os-applicationCredentialSecret os-clientCertPath] parameters ``` Version-Release number of selected component (if applicable): OCP 4.11.0-0.nightly-2022-05-08-193101 on OSP RHOS-16.1-RHEL-8-20220329.n.1. How reproducible: Always Steps to Reproduce: 1. Install OCP 4.11 with eCCM and application credentials on top of OSP: ``` $ source shiftstackrc && openstack application credential list +----------------------------------+--------------------+----------------------------------+-----------------------+------------+ | ID | Name | Project ID | Description | Expires At | +----------------------------------+--------------------+----------------------------------+-----------------------+------------+ | f3993887ec294b4fb4b956461f6ac6fe | AppCredsshiftstack | 62a8ea4499a54281bed4e839aedf5971 | App Creds - All roles | None | +----------------------------------+--------------------+----------------------------------+-----------------------+------------+ $ grep -A 10 shiftstack clouds.yaml #BEGIN shiftstack PARAMETERS shiftstack: auth: application_credential_id: f3993887ec294b4fb4b956461f6ac6fe application_credential_secret: <omited_as_this_is_secret> auth_url: https://10.46.43.75:13000 auth_type: v3applicationcredential cacert: /etc/pki/ca-trust/source/anchors/undercloud-cacert.pem identity_api_version: '3' region_name: regionOne #END shiftstack PARAMETERS $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.11.0-0.nightly-2022-05-08-193101 True False 6h20m Cluster version is 4.11.0-0.nightly-2022-05-08-193101 ``` 2. Make sure manila is present on the OSP installation: ``` $ source ~/shiftstackrc && openstack catalog show manila -c name +-------+--------+ | Field | Value | +-------+--------+ | name | manila | +-------+--------+ ``` 3. Apply manila manifests for a namespace: ``` $ cat manila.manifests.yaml --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: "pvc-manila" namespace: "manila-test" spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: csi-manila-default --- apiVersion: apps/v1 kind: Deployment metadata: name: demo namespace: "manila-test" spec: replicas: 2 selector: matchLabels: app: demo template: metadata: labels: app: demo spec: containers: - name: demo image: quay.io/kuryr/demo ports: - containerPort: 80 protocol: TCP volumeMounts: - mountPath: /var/lib/www/data name: mydata volumes: - name: mydata persistentVolumeClaim: claimName: pvc-manila readOnly: false $ oc apply -f manila.manifests.yaml ``` 4. The pvc-manila creation fails to use application credentials ``` $ oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE pvc-manila Pending csi-manila-default 97m $ oc get pvc pvc-manila -o yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"pvc-manila","namespace":"manila-test"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}},"storageClassName":"csi-manila-default"}} volume.beta.kubernetes.io/storage-provisioner: manila.csi.openstack.org volume.kubernetes.io/storage-provisioner: manila.csi.openstack.org creationTimestamp: "2022-05-09T06:36:55Z" finalizers: - kubernetes.io/pvc-protection name: pvc-manila namespace: manila-test resourceVersion: "58900" uid: 35090e06-2f4e-466d-9125-1c986c5bc44d spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: csi-manila-default volumeMode: Filesystem status: phase: Pending $ oc describe pvc pvc-manila Name: pvc-manila Namespace: manila-test StorageClass: csi-manila-default Status: Pending Volume: Labels: <none> Annotations: volume.beta.kubernetes.io/storage-provisioner: manila.csi.openstack.org volume.kubernetes.io/storage-provisioner: manila.csi.openstack.org Finalizers: [kubernetes.io/pvc-protection] Capacity: Access Modes: VolumeMode: Filesystem Used By: demo-59c59d6d55-554sw demo-59c59d6d55-rfptf Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning ProvisioningFailed 76m (x14 over 97m) manila.csi.openstack.org_openstack-manila-csi-controllerplugin-6bb98796c8-zrcqj_22d16e05-2a02-48c3-ad09-ec80263ba656 failed to provision volume with StorageClass "csi-manila-default": rpc error: code = InvalidArgument desc = invalid OpenStack secrets: parameter 'os-authURL' requires exactly one of [os-password os-trustID os-applicationCredentialSecret os-clientCertPath] parameters Normal ExternalProvisioning 2m28s (x393 over 97m) persistentvolume-controller waiting for a volume to be created, either by external provisioner "manila.csi.openstack.org" or manually created by system administrator Normal Provisioning 100s (x35 over 97m) manila.csi.openstack.org_openstack-manila-csi-controllerplugin-6bb98796c8-zrcqj_22d16e05-2a02-48c3-ad09-ec80263ba656 External provisioner is provisioning volume for claim "manila-test/pvc-manila" ``` Actual results: Unable to use application credentials for pvc-manila creation on OpenStack. Expected results: Successful in creating a pvc-manila.
CSI issues reproduced in the Prow CI with Cinder: https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_release/23126/rehearse-23126-periodic-ci-shiftstack-shiftstack-ci-main-periodic-4.11-e2e-openstack-appcreds/1522586239746707456 Failing tests: [sig-arch][bz-etcd][Late] Alerts alert/etcdMemberCommunicationSlow should not be at or above info [Suite:openshift/conformance/parallel] [sig-cluster-lifecycle] TestAdminAck should succeed [Suite:openshift/conformance/parallel] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (block volmode)] volumes should store data [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup applied to the volume contents [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup skips ownership changes to the volume contents [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] subPath should be able to unmount after the subpath directory is deleted [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] subPath should support existing directories when readOnly specified in the volumeSource [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] subPath should support existing directory [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] subPath should support existing single file [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] subPath should support file as subpath [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] subPath should support non-existent path [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] subPath should support readOnly directory specified in the volumeMount [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] subPath should support readOnly file specified in the volumeMount [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] volumes should allow exec of files on the volume [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] volumes should store data [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (delayed binding)] topology should provision a volume and schedule a pod with AllowedTopologies [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (immediate binding)] topology should provision a volume and schedule a pod with AllowedTopologies [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral should create read/write inline ephemeral volume [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral should support multiple inline ephemeral volumes [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral should support two pods which have the same volume definition [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should create read-only inline ephemeral volume [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should create read/write inline ephemeral volume [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should support two pods which have the same volume definition [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should create read-only inline ephemeral volume [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should create read/write inline ephemeral volume [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should support multiple inline ephemeral volumes [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should support two pods which have the same volume definition [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Inline-volume (default fs)] volumes should allow exec of files on the volume [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Inline-volume (default fs)] volumes should store data [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Pre-provisioned PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Pre-provisioned PV (block volmode)] volumes should store data [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Pre-provisioned PV (default fs)] volumes should allow exec of files on the volume [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Pre-provisioned PV (default fs)] volumes should store data [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Pre-provisioned PV (filesystem volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s] Setting medium priority as the external cloud provider, which is a dependency, is not set for GA in 4.11.
Flagged blocker- as application credentials depends on the external cloud provider, which is not expected to go GA in 4.11 any more.
Verified with OCP 4.11.0-0.nightly-2022-06-06-025509 on top of RHOS-16.1-RHEL-8-20220329.n.1 with OpenshiftSDN. Verification steps: 1. Install OCP 4.11 with eCCM enabled on featureGate and application credentials on top of OSP: ``` $ source shiftstackrc && openstack application credential list +----------------------------------+--------------------+----------------------------------+-----------------------+------------+ | ID | Name | Project ID | Description | Expires At | +----------------------------------+--------------------+----------------------------------+-----------------------+------------+ | bac8ef3636c1403ea4f9089c0d5c4183 | AppCredsshiftstack | c8c72dacabe54ccf80816bed8cae137b | App Creds - All roles | None | +----------------------------------+--------------------+----------------------------------+-----------------------+------------+ $ grep -A 10 shiftstack clouds.yaml #BEGIN shiftstack PARAMETERS shiftstack: auth: application_credential_id: bac8ef3636c1403ea4f9089c0d5c4183 application_credential_secret: <omited_as_this_is_secret> auth_url: https://10.46.44.140:13000 auth_type: v3applicationcredential identity_api_version: '3' region_name: regionOne #END shiftstack PARAMETERS $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.11.0-0.nightly-2022-06-06-025509 True False 43m Cluster version is 4.11.0-0.nightly-2022-06-06-025509 $ oc get featureGate/cluster -o yaml apiVersion: config.openshift.io/v1 kind: FeatureGate metadata: annotations: include.release.openshift.io/self-managed-high-availability: "true" include.release.openshift.io/single-node-developer: "true" release.openshift.io/create-only: "true" creationTimestamp: "2022-06-07T06:37:37Z" generation: 1 name: cluster resourceVersion: "1214" uid: 77a85e07-cc08-4707-8626-74abaa88217b spec: customNoUpgrade: enabled: - ExternalCloudProvider featureSet: CustomNoUpgrade ``` 2. Make sure manila is present on the OSP installation: ``` $ source ~/shiftstackrc && openstack catalog show manila -c name +-------+--------+ | Field | Value | +-------+--------+ | name | manila | +-------+--------+ ``` 3. Apply manila manifests for a namespace: ``` $ cat manila.manifests.yaml --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: "pvc-manila" namespace: "manila-test" spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: csi-manila-default --- apiVersion: apps/v1 kind: Deployment metadata: name: demo namespace: "manila-test" spec: replicas: 2 selector: matchLabels: app: demo template: metadata: labels: app: demo spec: containers: - name: demo image: quay.io/kuryr/demo ports: - containerPort: 80 protocol: TCP volumeMounts: - mountPath: /var/lib/www/data name: mydata volumes: - name: mydata persistentVolumeClaim: claimName: pvc-manila readOnly: false $ oc apply -f manila.manifests.yaml ``` 4. The pvc-manila creation success to use application credentials: ``` $ oc get pvc -n manila-test NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE pvc-manila Bound pvc-bb7e0034-8c2d-4788-bc7c-370dedcc3cee 1Gi RWO csi-manila-default 56m $ oc describe pvc -n manila-test pvc-manila Name: pvc-manila Namespace: manila-test StorageClass: csi-manila-default Status: Bound Volume: pvc-bb7e0034-8c2d-4788-bc7c-370dedcc3cee Labels: <none> Annotations: pv.kubernetes.io/bind-completed: yes pv.kubernetes.io/bound-by-controller: yes volume.beta.kubernetes.io/storage-provisioner: manila.csi.openstack.org volume.kubernetes.io/storage-provisioner: manila.csi.openstack.org Finalizers: [kubernetes.io/pvc-protection] Capacity: 1Gi Access Modes: RWO VolumeMode: Filesystem Used By: demo-54b84f78df-927qq demo-54b84f78df-zvk6v Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ExternalProvisioning 57m persistentvolume-controller waiting for a volume to be created, either by external provisioner "manila.csi.openstack.org" or manually created by system administrator Normal Provisioning 57m manila.csi.openstack.org_openstack-manila-csi-controllerplugin-75dcfd4b6d-qdhnp_e4a753e7-55d6-4276-bc8e-cff7f6a6dbc5 External provisioner is provisioning volume for claim "manila-test/pvc-manila" Normal ProvisioningSucceeded 57m manila.csi.openstack.org_openstack-manila-csi-controllerplugin-75dcfd4b6d-qdhnp_e4a753e7-55d6-4276-bc8e-cff7f6a6dbc5 Successfully provisioned volume pvc-bb7e0034-8c2d-4788-bc7c-370dedcc3cee ```
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5069