Bug 1948448 - [vSphere csi driver operator] vmware-vsphere-csi-driver-operator pod doesn't restart and doesn't run as expected log level after changing the storage.spec.logLevel
Summary: [vSphere csi driver operator] vmware-vsphere-csi-driver-operator pod doesn't ...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 4.8
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: ---
Assignee: Fabio Bertinatto
QA Contact: Wei Duan
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-04-12 08:14 UTC by Wei Duan
Modified: 2021-08-25 12:01 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-08-25 12:01:30 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-storage-operator pull 158 0 None open Bug 1948448: Add log level to vSphere operator 2021-04-13 11:39:14 UTC

Description Wei Duan 2021-04-12 08:14:30 UTC
Description of problem:
Updating the storage.spec.logLevel doesn't take effective, the vmware-vsphere-csi-driver-operator pod doesn't restart and doesn't run as expected logLevel

Version-Release number of selected component (if applicable):
4.8.0-0.nightly-2021-04-09-222447

How reproducible:
Always

Steps to Reproduce:
1. Set Up an OCP4.8 cluster on vSphere
2. Enable TechPreviewNoUpgrade featureset
3. Edit the logLevel for storage.cluster to TraceAll
$ oc get storage cluster -o json | jq .spec
{
  "logLevel": "TraceAll",
  "managementState": "Managed",
  "operatorLogLevel": "Debug"
}
4. Check vmware-vsphere-csi-driver-operator pod 

Actual results:
The vmware-vsphere-csi-driver-operator pod doesn't not restart
And the clustercsidriver.spec.operatorLogLevel is "Normal"
$ oc get clustercsidriver csi.vsphere.vmware.com -o json | jq .spec.operatorLogLevel
"Normal"

Expected results:
The vmware-vsphere-csi-driver-operator pod should restart and run as TraceAll log level

Comment 2 Wei Duan 2021-04-15 10:57:29 UTC
After changing the storage.spec.logLevel to "TraceAll", the vsphere-csi-driver-operator did not run with TraceAll logLevel. Change status to "Assigned" back.

$ oc get storage cluster -o json | jq .spec
{
  "logLevel": "TraceAll",
  "managementState": "Managed",
  "operatorLogLevel": "Normal"
}

$ oc get clustercsidriver csi.vsphere.vmware.com -o json | jq .spec.operatorLogLevel
"Normal"


$ oc -n openshift-cluster-csi-drivers rsh vmware-vsphere-csi-driver-operator-55c8ddfdb7-c87pv
sh-4.4$ ps -aux 
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
1000270+       1  0.7  0.4 1742704 80348 ?       Ssl  10:44   0:04 /usr/bin/vmware-vsphere-csi-driver-operator start
1000270+      20  0.1  0.0  12028  3192 pts/0    Ss   10:52   0:00 /bin/sh
1000270+      26  0.0  0.0  47548  3468 pts/0    R+   10:52   0:00 ps -aux

Comment 3 Fabio Bertinatto 2021-04-22 12:31:20 UTC
@Wei, since PR 131 [1] we no longer cascade the log level from storage CR down to clustercsidriver CR.

CSO only creates the clustercsidriver CR if it doesn't exist, but it doesn't change it once it has been created. If the user wants to edit the clustercsidriver CR (to change the log level, for instance), they have to change the clustercsidriver directly. 

This change was done to accommodate the CSI Config Observer Controller into CSO (as stated in the PR description), but we'll also need this behavior to allow users to configure CSI operators individually (for opting-out of features or removing the operators completely, for example).

https://github.com/openshift/cluster-storage-operator/pull/131

Comment 4 Wei Duan 2021-04-26 09:07:16 UTC
Hi Fabio,

Yes, we tested that if update storage.spec.logLevel -> the csi driver operator pod log level changed, but clustercsidriver.spec.operatorLoglevel NOT changes.
We are okay to the change and close this bz.

Comment 6 Fabio Bertinatto 2021-08-25 12:01:30 UTC
Closing as per comments #3 and #4.


Note You need to log in before you can comment on or make changes to this bug.