Bug 2083153 - Unable to use application credentials for Manila PVC creation on OpenStack
Summary: Unable to use application credentials for Manila PVC creation on OpenStack
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cloud Compute
Version: 4.11
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ---
: 4.11.0
Assignee: Pierre Prinetti
QA Contact: Itay Matza
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-05-09 13:06 UTC by Itay Matza
Modified: 2022-08-10 11:11 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The Manila operator is only passing username/password to the Manila driver, because application credentials are not supported by the in-tree cloud provider. Consequence: When running a techpreview cluster, which supports application credentials, Manila won't operate. Fix: With this fix, application credentials are copied from the clouds.yaml to Manila's storage class secret. Result: In clusters running the external cloud provider, Manila is able to authenticate against OpenStack and properly operate with application credentials.
Clone Of:
Environment:
Last Closed: 2022-08-10 11:10:43 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift csi-driver-manila-operator pull 148 0 None open Bug 2083153: Fetch application credentials 2022-05-31 14:30:39 UTC
Red Hat Product Errata RHSA-2022:5069 0 None None None 2022-08-10 11:10:59 UTC

Description Itay Matza 2022-05-09 13:06:58 UTC
Description of problem:
After installation of OCP 4.11 with eCCM and application credentials, PVC creation fails to provision a StorageClass:
```
failed to provision volume with StorageClass "csi-manila-default": rpc error: code = InvalidArgument desc = invalid OpenStack secrets: parameter 'os-authURL' requires exactly one of [os-password os-trustID os-applicationCredentialSecret os-clientCertPath] parameters
```


Version-Release number of selected component (if applicable):
OCP 4.11.0-0.nightly-2022-05-08-193101 on OSP RHOS-16.1-RHEL-8-20220329.n.1.


How reproducible:
Always


Steps to Reproduce:
1. Install OCP 4.11 with eCCM and application credentials on top of OSP:
```
$ source shiftstackrc && openstack application credential list
+----------------------------------+--------------------+----------------------------------+-----------------------+------------+
| ID                               | Name               | Project ID                       | Description           | Expires At |
+----------------------------------+--------------------+----------------------------------+-----------------------+------------+
| f3993887ec294b4fb4b956461f6ac6fe | AppCredsshiftstack | 62a8ea4499a54281bed4e839aedf5971 | App Creds - All roles | None       |
+----------------------------------+--------------------+----------------------------------+-----------------------+------------+

$ grep -A 10 shiftstack clouds.yaml 
#BEGIN shiftstack PARAMETERS
 shiftstack:
    auth:
        application_credential_id: f3993887ec294b4fb4b956461f6ac6fe
        application_credential_secret: <omited_as_this_is_secret>
        auth_url: https://10.46.43.75:13000
    auth_type: v3applicationcredential
    cacert: /etc/pki/ca-trust/source/anchors/undercloud-cacert.pem
    identity_api_version: '3'
    region_name: regionOne
#END shiftstack PARAMETERS

$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.0-0.nightly-2022-05-08-193101   True        False         6h20m   Cluster version is 4.11.0-0.nightly-2022-05-08-193101
```

2. Make sure manila is present on the OSP installation:
```
$ source ~/shiftstackrc && openstack catalog show manila -c name
+-------+--------+
| Field | Value  |
+-------+--------+
| name  | manila |
+-------+--------+
```

3. Apply manila manifests for a namespace:
```
$ cat manila.manifests.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: "pvc-manila"
  namespace: "manila-test"
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-manila-default
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo
  namespace: "manila-test"
spec:
  replicas: 2
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      containers:
      - name: demo
        image: quay.io/kuryr/demo
        ports:
        - containerPort: 80
          protocol: TCP
        volumeMounts:
          - mountPath: /var/lib/www/data
            name: mydata
      volumes:
        - name: mydata
          persistentVolumeClaim:
            claimName: pvc-manila
            readOnly: false


$ oc apply -f manila.manifests.yaml

```

4. The pvc-manila creation fails to use application credentials
```
$ oc get pvc
NAME         STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS         AGE
pvc-manila   Pending                                      csi-manila-default   97m

$ oc get pvc pvc-manila -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"pvc-manila","namespace":"manila-test"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}},"storageClassName":"csi-manila-default"}}
    volume.beta.kubernetes.io/storage-provisioner: manila.csi.openstack.org
    volume.kubernetes.io/storage-provisioner: manila.csi.openstack.org
  creationTimestamp: "2022-05-09T06:36:55Z"
  finalizers:
  - kubernetes.io/pvc-protection
  name: pvc-manila
  namespace: manila-test
  resourceVersion: "58900"
  uid: 35090e06-2f4e-466d-9125-1c986c5bc44d
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-manila-default
  volumeMode: Filesystem
status:
  phase: Pending

$ oc describe pvc pvc-manila
Name:          pvc-manila
Namespace:     manila-test
StorageClass:  csi-manila-default
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: manila.csi.openstack.org
               volume.kubernetes.io/storage-provisioner: manila.csi.openstack.org
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       demo-59c59d6d55-554sw
               demo-59c59d6d55-rfptf
Events:
  Type     Reason                Age                    From                                                                                                                  Message
  ----     ------                ----                   ----                                                                                                                  -------
  Warning  ProvisioningFailed    76m (x14 over 97m)     manila.csi.openstack.org_openstack-manila-csi-controllerplugin-6bb98796c8-zrcqj_22d16e05-2a02-48c3-ad09-ec80263ba656  failed to provision volume with StorageClass "csi-manila-default": rpc error: code = InvalidArgument desc = invalid OpenStack secrets: parameter 'os-authURL' requires exactly one of [os-password os-trustID os-applicationCredentialSecret os-clientCertPath] parameters
  Normal   ExternalProvisioning  2m28s (x393 over 97m)  persistentvolume-controller                                                                                           waiting for a volume to be created, either by external provisioner "manila.csi.openstack.org" or manually created by system administrator
  Normal   Provisioning          100s (x35 over 97m)    manila.csi.openstack.org_openstack-manila-csi-controllerplugin-6bb98796c8-zrcqj_22d16e05-2a02-48c3-ad09-ec80263ba656  External provisioner is provisioning volume for claim "manila-test/pvc-manila"

```


Actual results:
Unable to use application credentials for pvc-manila creation on OpenStack.


Expected results:
Successful in creating a pvc-manila.

Comment 2 Pierre Prinetti 2022-05-09 13:39:45 UTC
CSI issues reproduced in the Prow CI with Cinder: https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_release/23126/rehearse-23126-periodic-ci-shiftstack-shiftstack-ci-main-periodic-4.11-e2e-openstack-appcreds/1522586239746707456


Failing tests:

[sig-arch][bz-etcd][Late] Alerts alert/etcdMemberCommunicationSlow should not be at or above info [Suite:openshift/conformance/parallel]
[sig-cluster-lifecycle] TestAdminAck should succeed [Suite:openshift/conformance/parallel]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (block volmode)] volumes should store data [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (Always)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup applied to the volume contents [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, new pod fsgroup applied to volume contents [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with different fsgroup applied to the volume contents [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] fsgroupchangepolicy (OnRootMismatch)[LinuxOnly], pod created with an initial fsgroup, volume contents ownership changed via chgrp in first pod, new pod with same fsgroup skips ownership changes to the volume contents [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] subPath should be able to unmount after the subpath directory is deleted [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] subPath should support existing directories when readOnly specified in the volumeSource [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] subPath should support existing directory [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] subPath should support existing single file [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] subPath should support file as subpath [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] subPath should support non-existent path [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] subPath should support readOnly directory specified in the volumeMount [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] subPath should support readOnly file specified in the volumeMount [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] volumes should allow exec of files on the volume [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (default fs)] volumes should store data [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (delayed binding)] topology should provision a volume and schedule a pod with AllowedTopologies [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Dynamic PV (immediate binding)] topology should provision a volume and schedule a pod with AllowedTopologies [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral should create read/write inline ephemeral volume [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral should support multiple inline ephemeral volumes [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Generic Ephemeral-volume (block volmode) (late-binding)] ephemeral should support two pods which have the same volume definition [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should create read-only inline ephemeral volume [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should create read/write inline ephemeral volume [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Generic Ephemeral-volume (default fs) (immediate-binding)] ephemeral should support two pods which have the same volume definition [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should create read-only inline ephemeral volume [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should create read/write inline ephemeral volume [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should support multiple inline ephemeral volumes [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should support two pods which have the same volume definition [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Inline-volume (default fs)] volumes should allow exec of files on the volume [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Inline-volume (default fs)] volumes should store data [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Pre-provisioned PV (block volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Pre-provisioned PV (block volmode)] volumes should store data [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Pre-provisioned PV (default fs)] volumes should allow exec of files on the volume [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Pre-provisioned PV (default fs)] volumes should store data [Suite:openshift/conformance/parallel] [Suite:k8s]
[sig-storage] In-tree Volumes [Driver: cinder] [Testpattern: Pre-provisioned PV (filesystem volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly] [Suite:openshift/conformance/parallel] [Suite:k8s]

Setting medium priority as the external cloud provider, which is a dependency, is not set for GA in 4.11.

Comment 3 Pierre Prinetti 2022-05-10 12:01:56 UTC
Flagged blocker- as application credentials depends on the external cloud provider, which is not expected to go GA in 4.11 any more.

Comment 6 Itay Matza 2022-06-07 08:47:43 UTC
Verified with OCP 4.11.0-0.nightly-2022-06-06-025509 on top of RHOS-16.1-RHEL-8-20220329.n.1 with OpenshiftSDN.


Verification steps:
1. Install OCP 4.11 with eCCM enabled on featureGate and application credentials on top of OSP:
```
$ source shiftstackrc && openstack application credential list
+----------------------------------+--------------------+----------------------------------+-----------------------+------------+
| ID                               | Name               | Project ID                       | Description           | Expires At |
+----------------------------------+--------------------+----------------------------------+-----------------------+------------+
| bac8ef3636c1403ea4f9089c0d5c4183 | AppCredsshiftstack | c8c72dacabe54ccf80816bed8cae137b | App Creds - All roles | None       |
+----------------------------------+--------------------+----------------------------------+-----------------------+------------+

$ grep -A 10 shiftstack clouds.yaml
#BEGIN shiftstack PARAMETERS
 shiftstack:
    auth:
        application_credential_id: bac8ef3636c1403ea4f9089c0d5c4183
        application_credential_secret: <omited_as_this_is_secret>
        auth_url: https://10.46.44.140:13000
    auth_type: v3applicationcredential
    identity_api_version: '3'                      
    region_name: regionOne
#END shiftstack PARAMETERS

$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.0-0.nightly-2022-06-06-025509   True        False         43m     Cluster version is 4.11.0-0.nightly-2022-06-06-025509

$ oc get featureGate/cluster -o yaml
apiVersion: config.openshift.io/v1
kind: FeatureGate
metadata:
  annotations:
    include.release.openshift.io/self-managed-high-availability: "true"
    include.release.openshift.io/single-node-developer: "true"
    release.openshift.io/create-only: "true"
  creationTimestamp: "2022-06-07T06:37:37Z"
  generation: 1
  name: cluster
  resourceVersion: "1214"
  uid: 77a85e07-cc08-4707-8626-74abaa88217b
spec:
  customNoUpgrade:
    enabled:
    - ExternalCloudProvider
  featureSet: CustomNoUpgrade
```

2. Make sure manila is present on the OSP installation:
```
$ source ~/shiftstackrc && openstack catalog show manila -c name
+-------+--------+
| Field | Value  |
+-------+--------+                                                                                                                                                                                                
| name  | manila |                                                                                                                                                                                                 +-------+--------+
```

3. Apply manila manifests for a namespace:
```
$ cat manila.manifests.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: "pvc-manila"
  namespace: "manila-test"
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-manila-default
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo
  namespace: "manila-test"
spec:
  replicas: 2
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      containers:
      - name: demo
        image: quay.io/kuryr/demo
        ports:
        - containerPort: 80
          protocol: TCP
        volumeMounts:
          - mountPath: /var/lib/www/data
            name: mydata
      volumes:
        - name: mydata
          persistentVolumeClaim:
            claimName: pvc-manila
            readOnly: false

$ oc apply -f manila.manifests.yaml
```

4. The pvc-manila creation success to use application credentials:
```
$ oc get pvc -n manila-test
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS         AGE
pvc-manila   Bound    pvc-bb7e0034-8c2d-4788-bc7c-370dedcc3cee   1Gi        RWO            csi-manila-default   56m

$ oc describe pvc -n manila-test pvc-manila
Name:          pvc-manila
Namespace:     manila-test
StorageClass:  csi-manila-default
Status:        Bound
Volume:        pvc-bb7e0034-8c2d-4788-bc7c-370dedcc3cee
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: manila.csi.openstack.org
               volume.kubernetes.io/storage-provisioner: manila.csi.openstack.org
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      1Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       demo-54b84f78df-927qq
               demo-54b84f78df-zvk6v
Events:
  Type    Reason                 Age   From                                                                                                                  Message
  ----    ------                 ----  ----                                                                                                                  -------
  Normal  ExternalProvisioning   57m   persistentvolume-controller                                                                                           waiting for a volume to be created, either by external provisioner "manila.csi.openstack.org" or manually created by system administrator
  Normal  Provisioning           57m   manila.csi.openstack.org_openstack-manila-csi-controllerplugin-75dcfd4b6d-qdhnp_e4a753e7-55d6-4276-bc8e-cff7f6a6dbc5  External provisioner is provisioning volume for claim "manila-test/pvc-manila"
  Normal  ProvisioningSucceeded  57m   manila.csi.openstack.org_openstack-manila-csi-controllerplugin-75dcfd4b6d-qdhnp_e4a753e7-55d6-4276-bc8e-cff7f6a6dbc5  Successfully provisioned volume pvc-bb7e0034-8c2d-4788-bc7c-370dedcc3cee
```

Comment 8 errata-xmlrpc 2022-08-10 11:10:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069


Note You need to log in before you can comment on or make changes to this bug.