Bug 1948448

Summary: [vSphere csi driver operator] vmware-vsphere-csi-driver-operator pod doesn't restart and doesn't run as expected log level after changing the storage.spec.logLevel
Product: OpenShift Container Platform Reporter: Wei Duan <wduan>
Component: StorageAssignee: Fabio Bertinatto <fbertina>
Storage sub component: Operators QA Contact: Wei Duan <wduan>
Status: CLOSED NOTABUG Docs Contact:
Severity: medium    
Priority: unspecified CC: aos-bugs, fbertina
Version: 4.8   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-08-25 12:01:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Wei Duan 2021-04-12 08:14:30 UTC
Description of problem:
Updating the storage.spec.logLevel doesn't take effective, the vmware-vsphere-csi-driver-operator pod doesn't restart and doesn't run as expected logLevel

Version-Release number of selected component (if applicable):
4.8.0-0.nightly-2021-04-09-222447

How reproducible:
Always

Steps to Reproduce:
1. Set Up an OCP4.8 cluster on vSphere
2. Enable TechPreviewNoUpgrade featureset
3. Edit the logLevel for storage.cluster to TraceAll
$ oc get storage cluster -o json | jq .spec
{
  "logLevel": "TraceAll",
  "managementState": "Managed",
  "operatorLogLevel": "Debug"
}
4. Check vmware-vsphere-csi-driver-operator pod 

Actual results:
The vmware-vsphere-csi-driver-operator pod doesn't not restart
And the clustercsidriver.spec.operatorLogLevel is "Normal"
$ oc get clustercsidriver csi.vsphere.vmware.com -o json | jq .spec.operatorLogLevel
"Normal"

Expected results:
The vmware-vsphere-csi-driver-operator pod should restart and run as TraceAll log level

Comment 2 Wei Duan 2021-04-15 10:57:29 UTC
After changing the storage.spec.logLevel to "TraceAll", the vsphere-csi-driver-operator did not run with TraceAll logLevel. Change status to "Assigned" back.

$ oc get storage cluster -o json | jq .spec
{
  "logLevel": "TraceAll",
  "managementState": "Managed",
  "operatorLogLevel": "Normal"
}

$ oc get clustercsidriver csi.vsphere.vmware.com -o json | jq .spec.operatorLogLevel
"Normal"


$ oc -n openshift-cluster-csi-drivers rsh vmware-vsphere-csi-driver-operator-55c8ddfdb7-c87pv
sh-4.4$ ps -aux 
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
1000270+       1  0.7  0.4 1742704 80348 ?       Ssl  10:44   0:04 /usr/bin/vmware-vsphere-csi-driver-operator start
1000270+      20  0.1  0.0  12028  3192 pts/0    Ss   10:52   0:00 /bin/sh
1000270+      26  0.0  0.0  47548  3468 pts/0    R+   10:52   0:00 ps -aux

Comment 3 Fabio Bertinatto 2021-04-22 12:31:20 UTC
@Wei, since PR 131 [1] we no longer cascade the log level from storage CR down to clustercsidriver CR.

CSO only creates the clustercsidriver CR if it doesn't exist, but it doesn't change it once it has been created. If the user wants to edit the clustercsidriver CR (to change the log level, for instance), they have to change the clustercsidriver directly. 

This change was done to accommodate the CSI Config Observer Controller into CSO (as stated in the PR description), but we'll also need this behavior to allow users to configure CSI operators individually (for opting-out of features or removing the operators completely, for example).

https://github.com/openshift/cluster-storage-operator/pull/131

Comment 4 Wei Duan 2021-04-26 09:07:16 UTC
Hi Fabio,

Yes, we tested that if update storage.spec.logLevel -> the csi driver operator pod log level changed, but clustercsidriver.spec.operatorLoglevel NOT changes.
We are okay to the change and close this bz.

Comment 6 Fabio Bertinatto 2021-08-25 12:01:30 UTC
Closing as per comments #3 and #4.