Bug 2083153
| Summary: | Unable to use application credentials for Manila PVC creation on OpenStack | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Itay Matza <imatza> |
| Component: | Cloud Compute | Assignee: | Pierre Prinetti <pprinett> |
| Cloud Compute sub component: | OpenStack Provider | QA Contact: | Itay Matza <imatza> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | high | ||
| Priority: | medium | CC: | m.andre, mfedosin, pprinett |
| Version: | 4.11 | Keywords: | Triaged |
| Target Milestone: | --- | ||
| Target Release: | 4.11.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
Cause: The Manila operator is only passing username/password to the Manila driver, because application credentials are not supported by the in-tree cloud provider.
Consequence: When running a techpreview cluster, which supports application credentials, Manila won't operate.
Fix: With this fix, application credentials are copied from the clouds.yaml to Manila's storage class secret.
Result: In clusters running the external cloud provider, Manila is able to authenticate against OpenStack and properly operate with application credentials.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-08-10 11:10:43 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
CSI issues reproduced in the Prow CI with Cinder: https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_release/23126/rehearse-23126-periodic-ci-shiftstack-shiftstack-ci-main-periodic-4.11-e2e-openstack-appcreds/1522586239746707456 Failing tests: [sig-arch][bz-etcd][Late] Alerts alert/etcdMemberCommunicationSlow should not be at or above info [Suite:openshift/conformance/parallel] [sig-cluster-lifecycle] TestAdminAck should succeed [Suite:openshift/conformance/parallel] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (block volmode)] volumes should store data [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup applied to the volume contents [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup skips ownership changes to the volume contents [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] subPath should be able to unmount after the subpath directory is deleted [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] subPath should support existing directories when readOnly specified in the volumeSource [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] subPath should support existing directory [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] subPath should support existing single file [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] subPath should support file as subpath [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] subPath should support non-existent path [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] subPath should support readOnly directory specified in the volumeMount [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] subPath should support readOnly file specified in the volumeMount [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] volumes should allow exec of files on the volume [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] volumes should store data [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (delayed binding)] topology should provision a volume and schedule a pod with AllowedTopologies [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (immediate binding)] topology should provision a volume and schedule a pod with AllowedTopologies [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral should create read/write inline ephemeral volume [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral should support multiple inline ephemeral volumes [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral should support two pods which have the same volume definition [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should create read-only inline ephemeral volume [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should create read/write inline ephemeral volume [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should support two pods which have the same volume definition [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should create read-only inline ephemeral volume [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should create read/write inline ephemeral volume [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should support multiple inline ephemeral volumes [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should support two pods which have the same volume definition [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Inline-volume (default fs)] volumes should allow exec of files on the volume [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Inline-volume (default fs)] volumes should store data [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Pre-provisioned PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Pre-provisioned PV (block volmode)] volumes should store data [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Pre-provisioned PV (default fs)] volumes should allow exec of files on the volume [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Pre-provisioned PV (default fs)] volumes should store data [Suite:openshift/conformance/parallel] [Suite:k8s] [sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Pre-provisioned PV (filesystem volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s] Setting medium priority as the external cloud provider, which is a dependency, is not set for GA in 4.11. Flagged blocker- as application credentials depends on the external cloud provider, which is not expected to go GA in 4.11 any more. Verified with OCP 4.11.0-0.nightly-2022-06-06-025509 on top of RHOS-16.1-RHEL-8-20220329.n.1 with OpenshiftSDN.
Verification steps:
1. Install OCP 4.11 with eCCM enabled on featureGate and application credentials on top of OSP:
```
$ source shiftstackrc && openstack application credential list
+----------------------------------+--------------------+----------------------------------+-----------------------+------------+
| ID | Name | Project ID | Description | Expires At |
+----------------------------------+--------------------+----------------------------------+-----------------------+------------+
| bac8ef3636c1403ea4f9089c0d5c4183 | AppCredsshiftstack | c8c72dacabe54ccf80816bed8cae137b | App Creds - All roles | None |
+----------------------------------+--------------------+----------------------------------+-----------------------+------------+
$ grep -A 10 shiftstack clouds.yaml
#BEGIN shiftstack PARAMETERS
shiftstack:
auth:
application_credential_id: bac8ef3636c1403ea4f9089c0d5c4183
application_credential_secret: <omited_as_this_is_secret>
auth_url: https://10.46.44.140:13000
auth_type: v3applicationcredential
identity_api_version: '3'
region_name: regionOne
#END shiftstack PARAMETERS
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.11.0-0.nightly-2022-06-06-025509 True False 43m Cluster version is 4.11.0-0.nightly-2022-06-06-025509
$ oc get featureGate/cluster -o yaml
apiVersion: config.openshift.io/v1
kind: FeatureGate
metadata:
annotations:
include.release.openshift.io/self-managed-high-availability: "true"
include.release.openshift.io/single-node-developer: "true"
release.openshift.io/create-only: "true"
creationTimestamp: "2022-06-07T06:37:37Z"
generation: 1
name: cluster
resourceVersion: "1214"
uid: 77a85e07-cc08-4707-8626-74abaa88217b
spec:
customNoUpgrade:
enabled:
- ExternalCloudProvider
featureSet: CustomNoUpgrade
```
2. Make sure manila is present on the OSP installation:
```
$ source ~/shiftstackrc && openstack catalog show manila -c name
+-------+--------+
| Field | Value |
+-------+--------+
| name | manila | +-------+--------+
```
3. Apply manila manifests for a namespace:
```
$ cat manila.manifests.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: "pvc-manila"
namespace: "manila-test"
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: csi-manila-default
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: demo
namespace: "manila-test"
spec:
replicas: 2
selector:
matchLabels:
app: demo
template:
metadata:
labels:
app: demo
spec:
containers:
- name: demo
image: quay.io/kuryr/demo
ports:
- containerPort: 80
protocol: TCP
volumeMounts:
- mountPath: /var/lib/www/data
name: mydata
volumes:
- name: mydata
persistentVolumeClaim:
claimName: pvc-manila
readOnly: false
$ oc apply -f manila.manifests.yaml
```
4. The pvc-manila creation success to use application credentials:
```
$ oc get pvc -n manila-test
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pvc-manila Bound pvc-bb7e0034-8c2d-4788-bc7c-370dedcc3cee 1Gi RWO csi-manila-default 56m
$ oc describe pvc -n manila-test pvc-manila
Name: pvc-manila
Namespace: manila-test
StorageClass: csi-manila-default
Status: Bound
Volume: pvc-bb7e0034-8c2d-4788-bc7c-370dedcc3cee
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: manila.csi.openstack.org
volume.kubernetes.io/storage-provisioner: manila.csi.openstack.org
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 1Gi
Access Modes: RWO
VolumeMode: Filesystem
Used By: demo-54b84f78df-927qq
demo-54b84f78df-zvk6v
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ExternalProvisioning 57m persistentvolume-controller waiting for a volume to be created, either by external provisioner "manila.csi.openstack.org" or manually created by system administrator
Normal Provisioning 57m manila.csi.openstack.org_openstack-manila-csi-controllerplugin-75dcfd4b6d-qdhnp_e4a753e7-55d6-4276-bc8e-cff7f6a6dbc5 External provisioner is provisioning volume for claim "manila-test/pvc-manila"
Normal ProvisioningSucceeded 57m manila.csi.openstack.org_openstack-manila-csi-controllerplugin-75dcfd4b6d-qdhnp_e4a753e7-55d6-4276-bc8e-cff7f6a6dbc5 Successfully provisioned volume pvc-bb7e0034-8c2d-4788-bc7c-370dedcc3cee
```
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5069 |
Description of problem: After installation of OCP 4.11 with eCCM and application credentials, PVC creation fails to provision a StorageClass: ``` failed to provision volume with StorageClass "csi-manila-default": rpc error: code = InvalidArgument desc = invalid OpenStack secrets: parameter 'os-authURL' requires exactly one of [os-password os-trustID os-applicationCredentialSecret os-clientCertPath] parameters ``` Version-Release number of selected component (if applicable): OCP 4.11.0-0.nightly-2022-05-08-193101 on OSP RHOS-16.1-RHEL-8-20220329.n.1. How reproducible: Always Steps to Reproduce: 1. Install OCP 4.11 with eCCM and application credentials on top of OSP: ``` $ source shiftstackrc && openstack application credential list +----------------------------------+--------------------+----------------------------------+-----------------------+------------+ | ID | Name | Project ID | Description | Expires At | +----------------------------------+--------------------+----------------------------------+-----------------------+------------+ | f3993887ec294b4fb4b956461f6ac6fe | AppCredsshiftstack | 62a8ea4499a54281bed4e839aedf5971 | App Creds - All roles | None | +----------------------------------+--------------------+----------------------------------+-----------------------+------------+ $ grep -A 10 shiftstack clouds.yaml #BEGIN shiftstack PARAMETERS shiftstack: auth: application_credential_id: f3993887ec294b4fb4b956461f6ac6fe application_credential_secret: <omited_as_this_is_secret> auth_url: https://10.46.43.75:13000 auth_type: v3applicationcredential cacert: /etc/pki/ca-trust/source/anchors/undercloud-cacert.pem identity_api_version: '3' region_name: regionOne #END shiftstack PARAMETERS $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.11.0-0.nightly-2022-05-08-193101 True False 6h20m Cluster version is 4.11.0-0.nightly-2022-05-08-193101 ``` 2. Make sure manila is present on the OSP installation: ``` $ source ~/shiftstackrc && openstack catalog show manila -c name +-------+--------+ | Field | Value | +-------+--------+ | name | manila | +-------+--------+ ``` 3. Apply manila manifests for a namespace: ``` $ cat manila.manifests.yaml --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: "pvc-manila" namespace: "manila-test" spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: csi-manila-default --- apiVersion: apps/v1 kind: Deployment metadata: name: demo namespace: "manila-test" spec: replicas: 2 selector: matchLabels: app: demo template: metadata: labels: app: demo spec: containers: - name: demo image: quay.io/kuryr/demo ports: - containerPort: 80 protocol: TCP volumeMounts: - mountPath: /var/lib/www/data name: mydata volumes: - name: mydata persistentVolumeClaim: claimName: pvc-manila readOnly: false $ oc apply -f manila.manifests.yaml ``` 4. The pvc-manila creation fails to use application credentials ``` $ oc get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE pvc-manila Pending csi-manila-default 97m $ oc get pvc pvc-manila -o yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"pvc-manila","namespace":"manila-test"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}},"storageClassName":"csi-manila-default"}} volume.beta.kubernetes.io/storage-provisioner: manila.csi.openstack.org volume.kubernetes.io/storage-provisioner: manila.csi.openstack.org creationTimestamp: "2022-05-09T06:36:55Z" finalizers: - kubernetes.io/pvc-protection name: pvc-manila namespace: manila-test resourceVersion: "58900" uid: 35090e06-2f4e-466d-9125-1c986c5bc44d spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi storageClassName: csi-manila-default volumeMode: Filesystem status: phase: Pending $ oc describe pvc pvc-manila Name: pvc-manila Namespace: manila-test StorageClass: csi-manila-default Status: Pending Volume: Labels: <none> Annotations: volume.beta.kubernetes.io/storage-provisioner: manila.csi.openstack.org volume.kubernetes.io/storage-provisioner: manila.csi.openstack.org Finalizers: [kubernetes.io/pvc-protection] Capacity: Access Modes: VolumeMode: Filesystem Used By: demo-59c59d6d55-554sw demo-59c59d6d55-rfptf Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning ProvisioningFailed 76m (x14 over 97m) manila.csi.openstack.org_openstack-manila-csi-controllerplugin-6bb98796c8-zrcqj_22d16e05-2a02-48c3-ad09-ec80263ba656 failed to provision volume with StorageClass "csi-manila-default": rpc error: code = InvalidArgument desc = invalid OpenStack secrets: parameter 'os-authURL' requires exactly one of [os-password os-trustID os-applicationCredentialSecret os-clientCertPath] parameters Normal ExternalProvisioning 2m28s (x393 over 97m) persistentvolume-controller waiting for a volume to be created, either by external provisioner "manila.csi.openstack.org" or manually created by system administrator Normal Provisioning 100s (x35 over 97m) manila.csi.openstack.org_openstack-manila-csi-controllerplugin-6bb98796c8-zrcqj_22d16e05-2a02-48c3-ad09-ec80263ba656 External provisioner is provisioning volume for claim "manila-test/pvc-manila" ``` Actual results: Unable to use application credentials for pvc-manila creation on OpenStack. Expected results: Successful in creating a pvc-manila.