Description of problem: ----------------------- 2 Node Ganesha cluster. Ran Bonnie,iozone,dbench,finds,rms from three v4 mounts. Ganesha logs are flooded with these messages : <snip> 14/06/2017 07:17:52 : epoch 50420000 : gqas007.sbu.lab.eng.bos.redhat.com : ganesha.nfsd-26891[work-154] nfs_dupreq_finish :DUPREQ :WARN :DRC retire entries: unable to find reclaimable dupreq LRU entry after 5 tries on DRC=0x7f46a40008c0, drc->size=2467 14/06/2017 07:17:52 : epoch 50420000 : gqas007.sbu.lab.eng.bos.redhat.com : ganesha.nfsd-26891[work-90] nfs_dupreq_finish :DUPREQ :WARN :DRC retire entries: unable to find reclaimable dupreq LRU entry after 5 tries on DRC=0x7f46a40008c0, drc->size=2466 14/06/2017 07:17:52 : epoch 50420000 : gqas007.sbu.lab.eng.bos.redhat.com : ganesha.nfsd-26891[work-223] nfs_dupreq_finish :DUPREQ :WARN :DRC retire entries: unable to find reclaimable dupreq LRU entry after 5 tries on DRC=0x7f46a40008c0, drc->size=2462 14/06/2017 07:17:52 : epoch 50420000 : gqas007.sbu.lab.eng.bos.redhat.com : ganesha.nfsd-26891[work-92] nfs_dupreq_finish :DUPREQ :WARN :DRC retire entries: unable to find reclaimable dupreq LRU entry after 5 tries on DRC=0x7f4650002a80, drc->size=1048 14/06/2017 07:19:06 : epoch 50420000 : gqas007.sbu.lab.eng.bos.redhat.com : ganesha.nfsd-26891[work-236] nfs_dupreq_finish :DUPREQ :WARN :DRC retire entries: unable to find reclaimable dupreq LRU entry after 5 tries on DRC=0x7f4650002a80, drc->size=1026 14/06/2017 07:19:21 : epoch 50420000 : gqas007.sbu.lab.eng.bos.redhat.com : ganesha.nfsd-26891[work-143] nfs_dupreq_finish :DUPREQ :WARN :DRC retire entries: unable to find reclaimable dupreq LRU entry after 5 tries on DRC=0x7f4650002a80, drc->size=1050 14/06/2017 07:19:21 : epoch 50420000 : gqas007.sbu.lab.eng.bos.redhat.com : ganesha.nfsd-26891[work-58] nfs_dupreq_finish :DUPREQ :WARN :DRC retire entries: unable to find reclaimable dupreq LRU entry after 5 tries on DRC=0x7f46a40008c0, drc->size=1026 14/06/2017 07:19:21 : epoch 50420000 : gqas007.sbu.lab.eng.bos.redhat.com : ganesha.nfsd-26891[work-14] nfs_dupreq_finish :DUPREQ :WARN :DRC retire entries: unable to find reclaimable dupreq LRU entry after 5 tries on DRC=0x7f46a40008c0, drc->size=1025 14/06/2017 07:25:59 : epoch 50420000 : gqas007.sbu.lab.eng.bos.redhat.com : ganesha.nfsd-26891[work-203] nfs_dupreq_finish :DUPREQ :WARN :DRC retire entries: unable to find reclaimable dupreq LRU entry after 5 tries on DRC=0x7f46a40008c0, drc->size=1028 </snip> [root@gqas007 ~]# cat /var/log/ganesha.log |grep -i "unable to find reclaimable"|wc -l 2927 [root@gqas007 ~]# [root@gqas007 ~]# [root@gqas007 ~]# ll -h /var/log/ganesha.log -rw-r--r-- 1 root root 814K Jun 14 11:51 /var/log/ganesha.log [root@gqas007 ~]# Version-Release number of selected component (if applicable): ------------------------------------------------------------- nfs-ganesha-2.4.4-8.el7rhgs.x86_64 glusterfs-ganesha-3.8.4-25.el7rhgs.x86_64 How reproducible: ----------------- Fairly. Additional info: ----------------- Volume Name: testvol Type: Distributed-Replicate Volume ID: 3b04b36a-1837-48e8-b437-fbc091b2f992 Status: Started Snapshot Count: 0 Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: gqas007.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick0 Brick2: gqas009.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick1 Brick3: gqas007.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick2 Brick4: gqas009.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick3 Options Reconfigured: ganesha.enable: on features.cache-invalidation: on server.allow-insecure: on performance.stat-prefetch: off transport.address-family: inet nfs.disable: on nfs-ganesha: enable cluster.enable-shared-storage: enable [root@gqas007 ~]#
A simple code fix to this could be to just change the log level of that message.
(In reply to Frank Filz from comment #9) > A simple code fix to this could be to just change the log level of that > message. But then will you not actually be suppressing the log for a genuine failure as well?
(In reply to Atin Mukherjee from comment #10) > (In reply to Frank Filz from comment #9) > > A simple code fix to this could be to just change the log level of that > > message. > > But then will you not actually be suppressing the log for a genuine failure > as well? Hmm, considering what Matt said, it is actually warning about something that is really happening, so I guess the config fix is the best workaround, so never mind about my idea for a quick code fix...
You can also turn logging down for the DUPREQ component like this: LOG { Components { DUPREQ = CRIT; } } (That log line is at the WARN level, so CRIT or lower will disable that warning).
POST with rebase to nfs-ganesha-2.5.x
Is this intended to be a doctext for 3.4? Because 3.4 is base on 2.5, which has this fixed, and doesn't even have that log message in it anymore. This should just be closed fixed with no doctext in 3.4.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:2610