Bug 1908998 - [cinder-csi-driver] doesn't detect the credentials change
Summary: [cinder-csi-driver] doesn't detect the credentials change
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 4.7
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.7.0
Assignee: Mike Fedosin
QA Contact: Wei Duan
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-12-18 06:45 UTC by Wei Duan
Modified: 2021-02-24 15:46 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-02-24 15:46:32 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift openstack-cinder-csi-driver-operator pull 24 0 None closed Bug 1908998: add secret hash annotation hook to the controller 2021-02-03 09:59:02 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:46:52 UTC

Description Wei Duan 2020-12-18 06:45:07 UTC
Description of problem:
After changing the credential with invalid password, cinder csi driver still works fine, and no related log found that it detected the secret change. 

Version-Release number of selected component (if applicable):
4.7.0-0.nightly-2020-12-14-080124

How reproducible:
Always

Steps to Reproduce:
1. Update the secret with invalid password:
oc -n kube-system edit secret openstack-credentials
...

2. check the secret is reverted
oc -n openshift-cluster-csi-drivers get secret openstack-cloud-credentials

3. Wait some time(10 minutes) and check driver log, no 
oc -n openshift-cluster-csi-drivers logs openstack-cinder-csi-driver-controller-68f4bccb58-6rbp7 -c csi-driver

4. create pod and pvc provisioned by csi driver still works
$ oc get pod,pvc
NAME          READY   STATUS    RESTARTS   AGE
pod/mypod03   1/1     Running   0          14s

NAME                            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/mypvc03   Bound    pvc-6b100cb6-3fff-4bb7-b4fa-a451c3bae0c7   1Gi        RWO            standard-csi   15s

Actual results:
cinder-csi-driver doesn't detect the credentials changes, csi driver should not work after changing to invalid password

Expected results:
cinder-csi-driver should detect the credentials changes

Master Log:

Node Log (of failed PODs):

PV Dump:
  
PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:
$ oc -n openshift-cluster-csi-drivers logs openstack-cinder-csi-driver-controller-68f4bccb58-6rbp7 -c csi-driver
I1217 01:05:26.199744       1 driver.go:69] Driver: cinder.csi.openstack.org
I1217 01:05:26.199869       1 driver.go:70] Driver version: 1.2.1@
I1217 01:05:26.199873       1 driver.go:71] CSI Spec version: 1.2.0
I1217 01:05:26.199881       1 driver.go:100] Enabling controller service capability: LIST_VOLUMES
I1217 01:05:26.199886       1 driver.go:100] Enabling controller service capability: CREATE_DELETE_VOLUME
I1217 01:05:26.199889       1 driver.go:100] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME
I1217 01:05:26.199892       1 driver.go:100] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I1217 01:05:26.199895       1 driver.go:100] Enabling controller service capability: LIST_SNAPSHOTS
I1217 01:05:26.199898       1 driver.go:100] Enabling controller service capability: EXPAND_VOLUME
I1217 01:05:26.199900       1 driver.go:100] Enabling controller service capability: CLONE_VOLUME
I1217 01:05:26.199903       1 driver.go:100] Enabling controller service capability: LIST_VOLUMES_PUBLISHED_NODES
I1217 01:05:26.199907       1 driver.go:112] Enabling volume access mode: SINGLE_NODE_WRITER
I1217 01:05:26.199911       1 driver.go:122] Enabling node service capability: STAGE_UNSTAGE_VOLUME
I1217 01:05:26.199914       1 driver.go:122] Enabling node service capability: EXPAND_VOLUME
I1217 01:05:26.199917       1 driver.go:122] Enabling node service capability: GET_VOLUME_STATS
I1217 01:05:26.200966       1 openstack.go:88] Block storage opts: {0 false false}
I1217 01:05:26.289431       1 server.go:108] Listening for connections on address: &net.UnixAddr{Name:"/csi/csi.sock", Net:"unix"}
$

Comment 2 Mike Fedosin 2021-01-12 10:05:07 UTC
Hello! The reason why I reassigned this bz is because we don't sync credentials directly, we just create a standard Credentials Request and expect that secrets will be synced across all namespaces automatically: https://github.com/openshift/cluster-storage-operator/blob/master/manifests/03_credentials_request_cinder.yaml

Comment 3 Devan Goodwin 2021-01-12 11:48:53 UTC
Mike to clarify the bug, QE found that the storage operator appears to not be detecting a change in that credential, it appeared to continue to use the old one. We now need credentials to be able to be seamlessly rotated. All other operators tested were handling this well (detecting the change, restarting if necessary, etc), but the storage operator did not appear to do this. 

The CredentialsOperator does populate the Secrets, but the issue is it may need to update that Secret.

Does this make sense?

Comment 6 Wei Duan 2021-01-28 09:36:16 UTC
Verified pass on 4.7.0-0.nightly-2021-01-27-213348.

Comment 7 Wei Duan 2021-02-03 10:01:07 UTC
I'd like to change the QA Contact but not assign, changed back.

Comment 10 errata-xmlrpc 2021-02-24 15:46:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.