Description of problem (please be detailed as possible and provide log snippests): In ocs 4.5 the log level of all CSI components are set to `0` which is default in upstem(rook) which will print only error logs, In ocs 4.2 4.3 and 4.4 the log level as `5` which prints all the info,debug and error logs. see log level (--v=0) in http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-vuf1cs33-t1/jnk-vuf1cs33-t1_20200813T232008/logs/failed_testcase_ocs_logs_1597365034/test_change_reclaim_policy_of_pv%5bCephBlockPool-Retain%5d_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-191e1a9fadc5b379104a64cc6516b8712acaf72f7c1ec31ad80263f5a3ba8128/ceph/namespaces/openshift-storage/pods/csi-cephfsplugin-provisioner-c748c89bf-7mvcx/csi-cephfsplugin-provisioner-c748c89bf-7mvcx.yaml Version of all relevant components (if applicable): ocs 4.5 Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? As the CSI log level is set to `0` it will log only the error messages not the debug or info logs which will make life very difficult to analyze the BZ, it would be good we can set the log level to `5` so it prints all logs and helps for debugging/analyzing issues, Is there any workaround available to the best of your knowledge? yes manually adding the CSI_LOG_LEVEL: "5" to the rook config map. Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 1 Can this issue reproducible? Deploy ocs 4.5 and check log level of CSI pods Can this issue reproduce from the UI? Yes If this is a regression, please provide more details to justify this: Steps to Reproduce: 1.Deploy ocs 4.5 2.Check log level in CSI pods 3. Actual results: CSI log level is be set to `0` Expected results: CSI log level is be set to `5` Additional info: in ocs-operator log level is set to `5` in master branch see https://github.com/openshift/ocs-operator/pull/668
Thanks Madhu for pointing this out to us. As Seen in Comment#0, the log level used to be 5 in OCS 4.4. So what was the reason for changing it in OCS 4.5 ? Even OCS 4.6 has log level set to 5 This could result in incomplete data for log analysis from field once OCS 4.5 is released. Please help in reviewing this BZ . @bipin FYI
Needs to be backported in 4.5.z
Removing the blocker flag and moving it to 4.5 based on an offline discussion. Summary: We need a consensus before we change this value again. Ideally it should print error and info messages. Log level to debug can be changed when needed.
Having log level 0 means we only get following logs in the plugin pods 2020-08-28T10:22:07.338868866Z E0828 10:22:07.338768 1 cephcsi.go:147] Failed to set new PID limit to -1: open /sys/fs/cgroup/pids/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cdd7d3b_55fa_45a7_bfd6_b411c1ac870b.slice/crio-ff9e1c3088fbc412d1e42a9b5cd2c3a770e772d6e016d1b46f8d1c10aa3f6ebf.scope/pids.max: permission denied 2020-08-28T10:22:07.351915128Z E0828 10:22:07.351857 1 volumemounter.go:171] failed to run ceph-fuse exec: "ceph-fuse": executable file not found in $PATH 2020-08-28T10:22:07.352113612Z W0828 10:22:07.352063 1 driver.go:164] EnableGRPCMetrics is deprecated As seen in [1], if someone wishes to verify mount issues from plugin pods, he/she gets nothing :( [1] http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/jnk-ai3c33-ua/jnk-ai3c33-ua_20200912T182702/logs/failed_testcase_ocs_logs_1599946063/test_must_gather_ocs_logs/ocs_must_gather/quay-io-rhceph-dev-ocs-must-gather-sha256-b0627919feaa87b3b6ee521c4c6e8c496281c2834ed1180aa72d95feb01009aa/ceph/namespaces/openshift-storage/pods/csi-cephfsplugin-4bf2l/csi-cephfsplugin/csi-cephfsplugin/logs/current.log
Has this been fixed in more recent versions of OCS/ODF? In ODF 4.10, I was unable to find 'CSI_LOG_LEVEL' in the configmap.