Bug 2353378

Summary: [NFS-Ganesha] The QoS-related logs are flooding the ganesha.log, causing the log scrolling to freeze on the scaled cluster due to increased log size.
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Manisha Saini <msaini>
Component: NFS-GaneshaAssignee: Naresh <nchillar>
Status: CLOSED ERRATA QA Contact: Manisha Saini <msaini>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 8.0CC: cephqe-warriors, kkeithle, mbenjamin, nchillar, rpollack, tserlin
Target Milestone: ---Flags: msaini: needinfo? (tserlin)
Target Release: 8.0z3   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: nfs-ganesha-6.5-9.el9cp Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2025-04-07 15:27:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Manisha Saini 2025-03-19 16:04:56 UTC 
Description of problem:
=============

On a scale cluster, create 2000 exports and set the QoS limits (BW and OPS) on all 2000 exports and at cluster level. 

# cephadm logs --name nfs.nfsganesha.0.0.cali015.vlvnfq > /root/ganesha1.log
# du -sh /root/ganesha1.log
779M	/root/ganesha1.log

With the latest NFS builds,The ganesha.log files are observed to be flooded with excessive QoS-related print statements. This continuous logging resulting in a noticeable hang while scrolling through the log file. The excessive volume of log entries hampers efficient troubleshooting and monitoring of other critical logs.

Log file took 779M of file space.

This issue becomes particularly prominent when the number of exports is large.



Version-Release number of selected component (if applicable):
-------
# ceph --version
ceph version 19.2.0-114.el9cp (ded81e05f6ece06e39d5564535edd25ec76f7198) squid (stable)

# rpm -qa | grep nfs
libnfsidmap-2.5.4-27.el9.x86_64
nfs-utils-2.5.4-27.el9.x86_64
nfs-ganesha-selinux-6.5-8.el9cp.noarch
nfs-ganesha-6.5-8.el9cp.x86_64
nfs-ganesha-ceph-6.5-8.el9cp.x86_64
nfs-ganesha-rados-grace-6.5-8.el9cp.x86_64
nfs-ganesha-rados-urls-6.5-8.el9cp.x86_64
nfs-ganesha-rgw-6.5-8.el9cp.x86_64
nfs-ganesha-utils-6.5-8.el9cp.x86_64


How reproducible:
============
2/2


Steps to Reproduce:
==========
1. Create NFS Ganesha cluster
2. Create 2000 NFS exports
3. Enable the QoS (BW and OPS) at both cluster level and export level for 2000 exports

Actual results:
==========

The ganesha.log files are being flooded with excessive QoS-related log entries taking upto 779M of space. This logging causes a noticeable delay or hang when scrolling through the logs



Expected results:
==========
The logging experience should remain manageable even when handling a large number of exports.


Additional info:
=============

nesha.nfsd-2[sigmgr] setNode_pe :QOS :EVENT :Added new config for :1093
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_class_values :QOS :EVENT :debugdp SI:1093 s_wio:0
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_class_values :QOS :EVENT :bw_e:1 t_e:0 iops_e:1 c_bw:0 c_t:0 c_iops:1
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_bucket_values :QOS :EVENT :wio:0 bw:3000000000 bw_ldct:0 mat:0 tc:0 trt:0 ltct:0
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_bucket_values :QOS :EVENT :max_iops:0 iops_ldct:0 iops_consumed:0
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_bucket_values :QOS :EVENT :wio:0 bw:3000000000 bw_ldct:0 mat:0 tc:0 trt:0 ltct:0
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_bucket_values :QOS :EVENT :max_iops:14000 iops_ldct:0 iops_consumed:0
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] setNode_pe :QOS :EVENT :Added new config for :1094
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_class_values :QOS :EVENT :debugdp SI:1094 s_wio:0
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_class_values :QOS :EVENT :bw_e:1 t_e:0 iops_e:1 c_bw:0 c_t:0 c_iops:1
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_bucket_values :QOS :EVENT :wio:0 bw:3000000000 bw_ldct:0 mat:0 tc:0 trt:0 ltct:0
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_bucket_values :QOS :EVENT :max_iops:0 iops_ldct:0 iops_consumed:0
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_bucket_values :QOS :EVENT :wio:0 bw:3000000000 bw_ldct:0 mat:0 tc:0 trt:0 ltct:0
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_bucket_values :QOS :EVENT :wio:0 bw:3000000000 bw_ldct:0 mat:0 tc:0 trt:0 ltct:0
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_bucket_values :QOS :EVENT :max_iops:14000 iops_ldct:0 iops_consumed:0
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] setNode_pe :QOS :EVENT :Added new config for :1138
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_class_values :QOS :EVENT :debugdp SI:1138 s_wio:0
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_class_values :QOS :EVENT :bw_e:1 t_e:0 iops_e:1 c_bw:0 c_t:0 c_iops:1
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_bucket_values :QOS :EVENT :wio:0 bw:3000000000 bw_ldct:0 mat:0 tc:0 trt:0 ltct:0
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_bucket_values :QOS :EVENT :max_iops:0 iops_ldct:0 iops_consumed:0
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_bucket_values :QOS :EVENT :wio:0 bw:3000000000 bw_ldct:0 mat:0 tc:0 trt:0 ltct:0
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_bucket_values :QOS :EVENT :max_iops:14000 iops_ldct:0 iops_consumed:0
Comment 10 Naresh 2025-04-01 10:49:13 UTC 
Documentation update is not required for this Bug.
Comment 11 Naresh 2025-04-01 10:50:02 UTC 
Documentation update is not required for this Bug.
Comment 13 errata-xmlrpc 2025-04-07 15:27:33 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Ceph Storage 8.0 security, bug fix, and enhancement updates), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2025:3635