Bug 2353378 - [NFS-Ganesha] The QoS-related logs are flooding the ganesha.log, causing the log scrolling to freeze on the scaled cluster due to increased log size. [NEEDINFO]
Summary: [NFS-Ganesha] The QoS-related logs are flooding the ganesha.log, causing the ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: NFS-Ganesha
Version: 8.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 8.0z3
Assignee: Naresh
QA Contact: Manisha Saini
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2025-03-19 16:04 UTC by Manisha Saini
Modified: 2025-04-07 15:27 UTC (History)
6 users (show)

Fixed In Version: nfs-ganesha-6.5-9.el9cp
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2025-04-07 15:27:33 UTC
Embargoed:
msaini: needinfo? (tserlin)

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHCEPH-10904 0 None None None 2025-03-19 16:06:46 UTC
Red Hat Product Errata RHSA-2025:3635 0 None None None 2025-04-07 15:27:36 UTC

Description Manisha Saini 2025-03-19 16:04:56 UTC 
Description of problem:
=============

On a scale cluster, create 2000 exports and set the QoS limits (BW and OPS) on all 2000 exports and at cluster level. 

# cephadm logs --name nfs.nfsganesha.0.0.cali015.vlvnfq > /root/ganesha1.log
# du -sh /root/ganesha1.log
779M	/root/ganesha1.log

With the latest NFS builds,The ganesha.log files are observed to be flooded with excessive QoS-related print statements. This continuous logging resulting in a noticeable hang while scrolling through the log file. The excessive volume of log entries hampers efficient troubleshooting and monitoring of other critical logs.

Log file took 779M of file space.

This issue becomes particularly prominent when the number of exports is large.



Version-Release number of selected component (if applicable):
-------
# ceph --version
ceph version 19.2.0-114.el9cp (ded81e05f6ece06e39d5564535edd25ec76f7198) squid (stable)

# rpm -qa | grep nfs
libnfsidmap-2.5.4-27.el9.x86_64
nfs-utils-2.5.4-27.el9.x86_64
nfs-ganesha-selinux-6.5-8.el9cp.noarch
nfs-ganesha-6.5-8.el9cp.x86_64
nfs-ganesha-ceph-6.5-8.el9cp.x86_64
nfs-ganesha-rados-grace-6.5-8.el9cp.x86_64
nfs-ganesha-rados-urls-6.5-8.el9cp.x86_64
nfs-ganesha-rgw-6.5-8.el9cp.x86_64
nfs-ganesha-utils-6.5-8.el9cp.x86_64


How reproducible:
============
2/2


Steps to Reproduce:
==========
1. Create NFS Ganesha cluster
2. Create 2000 NFS exports
3. Enable the QoS (BW and OPS) at both cluster level and export level for 2000 exports

Actual results:
==========

The ganesha.log files are being flooded with excessive QoS-related log entries taking upto 779M of space. This logging causes a noticeable delay or hang when scrolling through the logs



Expected results:
==========
The logging experience should remain manageable even when handling a large number of exports.


Additional info:
=============

nesha.nfsd-2[sigmgr] setNode_pe :QOS :EVENT :Added new config for :1093
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_class_values :QOS :EVENT :debugdp SI:1093 s_wio:0
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_class_values :QOS :EVENT :bw_e:1 t_e:0 iops_e:1 c_bw:0 c_t:0 c_iops:1
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_bucket_values :QOS :EVENT :wio:0 bw:3000000000 bw_ldct:0 mat:0 tc:0 trt:0 ltct:0
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_bucket_values :QOS :EVENT :max_iops:0 iops_ldct:0 iops_consumed:0
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_bucket_values :QOS :EVENT :wio:0 bw:3000000000 bw_ldct:0 mat:0 tc:0 trt:0 ltct:0
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_bucket_values :QOS :EVENT :max_iops:14000 iops_ldct:0 iops_consumed:0
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] setNode_pe :QOS :EVENT :Added new config for :1094
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_class_values :QOS :EVENT :debugdp SI:1094 s_wio:0
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_class_values :QOS :EVENT :bw_e:1 t_e:0 iops_e:1 c_bw:0 c_t:0 c_iops:1
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_bucket_values :QOS :EVENT :wio:0 bw:3000000000 bw_ldct:0 mat:0 tc:0 trt:0 ltct:0
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_bucket_values :QOS :EVENT :max_iops:0 iops_ldct:0 iops_consumed:0
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_bucket_values :QOS :EVENT :wio:0 bw:3000000000 bw_ldct:0 mat:0 tc:0 trt:0 ltct:0
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_bucket_values :QOS :EVENT :wio:0 bw:3000000000 bw_ldct:0 mat:0 tc:0 trt:0 ltct:0
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_bucket_values :QOS :EVENT :max_iops:14000 iops_ldct:0 iops_consumed:0
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] setNode_pe :QOS :EVENT :Added new config for :1138
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_class_values :QOS :EVENT :debugdp SI:1138 s_wio:0
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_class_values :QOS :EVENT :bw_e:1 t_e:0 iops_e:1 c_bw:0 c_t:0 c_iops:1
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_bucket_values :QOS :EVENT :wio:0 bw:3000000000 bw_ldct:0 mat:0 tc:0 trt:0 ltct:0
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_bucket_values :QOS :EVENT :max_iops:0 iops_ldct:0 iops_consumed:0
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_bucket_values :QOS :EVENT :wio:0 bw:3000000000 bw_ldct:0 mat:0 tc:0 trt:0 ltct:0
Mar 19 14:47:30 cali015 ceph-1c080210-e417-11ef-ae33-b49691cee574-nfs-nfsganesha-0-0-cali015-vlvnfq[1402788]: 19/03/2025 14:47:30 : epoch 67dac56c : cali015 : ganesha.nfsd-2[sigmgr] print_bucket_values :QOS :EVENT :max_iops:14000 iops_ldct:0 iops_consumed:0
Comment 10 Naresh 2025-04-01 10:49:13 UTC 
Documentation update is not required for this Bug.
Comment 11 Naresh 2025-04-01 10:50:02 UTC 
Documentation update is not required for this Bug.
Comment 13 errata-xmlrpc 2025-04-07 15:27:33 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Ceph Storage 8.0 security, bug fix, and enhancement updates), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2025:3635
Note You need to log in before you can comment on or make changes to this bug.