Bug 2130841

Summary: Sidecar logs for ODF 4.12 cluster with CSI_SIDECAR_LOG_LEVEL : 5 is not as much as for ODF 4.11 clusters.
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Amrita Mahapatra <ammahapa>
Component: ocs-operatorAssignee: Humble Chirammal <hchiramm>
Status: CLOSED NOTABUG QA Contact: Martin Bukatovic <mbukatov>
Severity: high Docs Contact:
Priority: high    
Version: 4.12CC: madam, mparida, muagarwa, ocs-bugs, odf-bz-bot, sostapov
Target Milestone: ---Flags: mparida: needinfo? (ammahapa)
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-10-19 03:30:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Amrita Mahapatra 2022-09-29 06:52:21 UTC
Description of problem (please be detailed as possible and provide log
snippests): The sidecar logs for ODF 4.12 cluster with CSI_SIDECAR_LOG_LEVEL : 5 is not as much as for ODF 4.11 clusters.
And for ODF 4.12 cluster there are no entry for the sidecar actions like creating a pvc, resizing pvc when CSI_SIDECAR_LOG_LEVEL: "5" is set however the logs triggered for CSI_SIDECAR_LOG_LEVEL: "1".

For ODF 4.12 cluster----

[ammahapa@ammahapa ~]$ oc get cm rook-ceph-operator-config -o yaml
apiVersion: v1
data:
  CSI_ENABLE_METADATA: "true"
  CSI_SIDECAR_LOG_LEVEL: "5"
kind: ConfigMap
metadata:
  creationTimestamp: "2022-09-20T08:00:17Z"
  name: rook-ceph-operator-config
  namespace: openshift-storage
  resourceVersion: "17446064"
  uid: 27620482-cc7e-4117-9a26-41a902faa17c


For ODF 4.11 cluster----

[ammahapa@ammahapa ~]$ oc get cm rook-ceph-operator-config -o yaml
apiVersion: v1
data:
  CSI_ENABLE_CSIADDONS: "true"
  CSI_ENABLE_METADATA: "true"
  CSI_LOG_LEVEL: "5"
  CSI_PLUGIN_TOLERATIONS: |2-

    - key: node.ocs.openshift.io/storage
      operator: Equal
      value: "true"
      effect: NoSchedule
  CSI_PROVISIONER_TOLERATIONS: |2-

    - key: node.ocs.openshift.io/storage
      operator: Equal
      value: "true"
      effect: NoSchedule
kind: ConfigMap
metadata:
  creationTimestamp: "2022-09-22T10:40:32Z"
  name: rook-ceph-operator-config
  namespace: openshift-storage
  resourceVersion: "1912448"
  uid: c404fd07-7749-424a-915d-f8c162f8689a

Version of all relevant components (if applicable):
For ODF 4.12 cluster:
OCP --- 4.12.0-0.nightly-2022-09-19-200722
ODF --- 4.12.0-59

For ODF 4.11 cluster:
OCP --- 4.11.0-0.nightly-2022-09-20-234850
ODF --- 4.11.2-1

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)? Its not blocking the feature validation.


Is there any workaround available to the best of your knowledge? No


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)? 4


Can this issue reproducible? Yes


Can this issue reproduce from the UI? Yes


If this is a regression, please provide more details to justify this: NA


Steps to Reproduce:
1. Update CSI_SIDECAR_LOG_LEVEL: "5" in rook-ceph-operator-config configmap for ODF 4.12 cluster and for ODF 4.11 default value is there, CSI_LOG_LEVEL: "5"

2. Check the sidecar logs for ODF 4.12 cluster and ODF 4.11 cluster and compare
3. Provision a PVC for ODF 4.12 cluster and check sidecar log for csi-provisioner log stream.



Actual results: The sidecar logs for ODF 4.12 cluster with CSI_SIDECAR_LOG_LEVEL : 5 is not as much as for ODF 4.11 clusters with default log level set, CSI_LOG_LEVEL: "5". 
And for ODF 4.12 cluster sidecar logs did not trigger for sidecar actions like creating a pvc, resizing pvc when CSI_SIDECAR_LOG_LEVEL: "5" is set.

Expected results: The sidecar logs for ODF 4.12 cluster with CSI_SIDECAR_LOG_LEVEL : 5 should be as much as for ODF 4.11 clusters with default log level set, CSI_LOG_LEVEL: "5". 
And for ODF 4.12 cluster sidecar logs should trigger for sidecar actions like creating a pvc, resizing pvc when CSI_SIDECAR_LOG_LEVEL: "5" is set.


Additional info:

Comment 6 Humble Chirammal 2022-09-30 06:50:22 UTC
I have accessed the cluster and can confirm the feature is working as expected!

I have created a PVC ( humble-test3) , Cloned, Snapshotted, Resized ..etc on this cluster at log level set to "1" for SIDECAR. I can see the 'required' logs popping up !

Comment 11 Malay Kumar parida 2022-09-30 08:31:30 UTC
The number of lines of logs on 4.11 & 4.12 is becoming so different due to something else here. 
Basically, if you see the logs in 4.11 the message 
"leader election.go:278] successfully renewed lease openshift-storage/openshift-storage-rbd-csi-ceph-com"
gets printed every 5 seconds.

but the same is done in 4.12 every 26 seconds.

Basically, these are logs of leader election & the difference in the interval is because 
in 4.12 has upstream k8s change to leader election method.

So given enough time, there will be a huge difference in the number of lines of logs due to this.

Comment 12 Malay Kumar parida 2022-10-11 15:05:40 UTC
Hi Amrita, Do we still need this?