Bug 1871879

Summary:	OCS 4.5 : Difficult to debug CSI issues as CSI log level is set to 0
Product:	[Red Hat Storage] Red Hat OpenShift Container Storage	Reporter:	Madhu Rajanna <mrajanna>
Component:	ocs-operator	Assignee:	Mudit Agarwal <muagarwa>
Status:	ASSIGNED ---	QA Contact:	Elad <ebenahar>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	4.5	CC:	bkunal, kelwhite, muagarwa, nberry, sostapov
Target Milestone:	---	Keywords:	AutomationBackLog
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Madhu Rajanna 2020-08-24 14:24:03 UTC

Description of problem (please be detailed as possible and provide log
snippests):

In ocs 4.5 the log level of all CSI components are set to `0` which is default in upstem(rook) which will print only error logs, In ocs 4.2 4.3 and 4.4 the log level as `5` which prints all the info,debug and error logs.

see log level (--v=0) in http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-vuf1cs33-t1/jnk-vuf1cs33-t1_20200813T232008/logs/failed_testcase_ocs_logs_1597365034/test_change_reclaim_policy_of_pv%5bCephBlockPool-Retain%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-191e1a9fadc5b379104a64cc6516b8712acaf72f7c1ec31ad80263f5a3ba8128/ceph/namespaces/openshift-storage/pods/csi-cephfsplugin-provisioner-c748c89bf-7mvcx/csi-cephfsplugin-provisioner-c748c89bf-7mvcx.yaml

Version of all relevant components (if applicable):

ocs 4.5

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?

As the CSI log level is set to `0` it will log only the error messages not the debug or info logs which will make life very difficult to analyze the BZ, it would be good we can set the log level to `5` so it prints all logs and helps for debugging/analyzing issues, 

Is there any workaround available to the best of your knowledge?

yes manually adding the CSI_LOG_LEVEL: "5" to the rook config map.

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?

1

Can this issue reproducible?

Deploy ocs 4.5 and check log level of CSI pods

Can this issue reproduce from the UI?

Yes

If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1.Deploy ocs 4.5
2.Check log level in CSI pods
3.


Actual results:
CSI log level is be set to `0`

Expected results:

CSI log level is be set to `5`

Additional info:

in ocs-operator log level is set to `5` in master branch  see https://github.com/openshift/ocs-operator/pull/668

Comment 2 Neha Berry 2020-08-24 14:29:29 UTC

Thanks Madhu for pointing this out to us.

As Seen in Comment#0, the log level used to be 5 in OCS 4.4. So what was the reason for changing it in OCS 4.5 ? Even OCS 4.6 has log level set to 5


This could result in incomplete data for log analysis from field once OCS 4.5 is released.

Please help in reviewing this BZ .

@bipin FYI

Comment 3 Mudit Agarwal 2020-08-24 15:04:03 UTC

Needs to be backported in 4.5.z

Comment 4 Mudit Agarwal 2020-08-25 07:49:18 UTC

Removing the blocker flag and moving it to 4.5 based on an offline discussion.

Summary:
We need a consensus before we change this value again. Ideally it should print error and info messages. Log level to debug can be changed when needed.

Comment 6 Neha Berry 2020-09-14 14:39:47 UTC

Having log level 0 means we only get following logs in the plugin pods


2020-08-28T10:22:07.338868866Z E0828 10:22:07.338768       1 cephcsi.go:147] Failed to set new PID limit to -1: open /sys/fs/cgroup/pids/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cdd7d3b_55fa_45a7_bfd6_b411c1ac870b.slice/crio-ff9e1c3088fbc412d1e42a9b5cd2c3a770e772d6e016d1b46f8d1c10aa3f6ebf.scope/pids.max: permission denied
2020-08-28T10:22:07.351915128Z E0828 10:22:07.351857       1 volumemounter.go:171] failed to run ceph-fuse exec: "ceph-fuse": executable file not found in $PATH
2020-08-28T10:22:07.352113612Z W0828 10:22:07.352063       1 driver.go:164] EnableGRPCMetrics is deprecated



As seen in [1], if someone wishes to verify mount issues from plugin pods, he/she gets nothing :( 


[1] http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-ai3c33-ua/jnk-ai3c33-ua_20200912T182702/logs/failed_testcase_ocs_logs_1599946063/test_must_gather_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-b0627919feaa87b3b6ee521c4c6e8c496281c2834ed1180aa72d95feb01009aa/ceph/namespaces/openshift-storage/pods/csi-cephfsplugin-4bf2l/csi-cephfsplugin/csi-cephfsplugin/logs/current.log

Comment 13 kelwhite 2023-03-06 15:11:36 UTC

Has this been fixed in more recent versions of OCS/ODF? In ODF 4.10, I was unable to find 'CSI_LOG_LEVEL' in the configmap.