Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: The glustershd log is getting spammed with messages like these: [2020-06-02 15:11:38.945198] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-HS_High_Resiliency-client-22: remote operation failed [Invalid argument] [2020-06-02 15:11:39.103001] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-HS_High_Resiliency-client-13: remote operation failed [Invalid argument] [2020-06-02 15:11:39.121600] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-HS_High_Resiliency-client-14: remote operation failed [Invalid argument] [2020-06-02 15:11:39.137356] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-HS_High_Resiliency-client-23: remote operation failed [Invalid argument] [2020-06-02 15:11:39.160914] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-HS_High_Resiliency-client-19: remote operation failed [Invalid argument] [2020-06-02 15:11:39.220582] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-HS_High_Resiliency-client-21: remote operation failed [Invalid argument] [2020-06-02 15:11:39.403613] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-HS_High_Resiliency-client-15: remote operation failed [Invalid argument] [2020-06-02 15:11:39.406699] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-HS_High_Resiliency-client-17: remote operation failed [Invalid argument] [2020-06-02 15:11:39.527867] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-HS_High_Resiliency-client-18: remote operation failed [Invalid argument] [2020-06-02 15:11:39.530420] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-HS_High_Resiliency-client-20: remote operation failed [Invalid argument] The spamming is so prolific, it slows down 'gluster volume heal' operations including the actual healing and the listing of the files in healing. The customer sent the current set of logs, and this is the current counts of these messages in the logs: <node>92-glustershd.log: 69,815,054 <node>93-glustershd.log: 81,046,086 <node>94-glustershd.log: 80,786,780 <node>95-glustershd.log: 18,223,843 <node>96-glustershd.log: 48,505,734 <node>97-glustershd.log: 22,290 Version-Release number of selected component (if applicable): Red Hat Enterprise Linux Server release 7.7 (Maipo) glusterfs-3.12.2-47.5.el7rhgs.x86_64 glusterfs-api-3.12.2-47.5.el7rhgs.x86_64 glusterfs-client-xlators-3.12.2-47.5.el7rhgs.x86_64 glusterfs-cli-3.12.2-47.5.el7rhgs.x86_64 glusterfs-events-3.12.2-47.5.el7rhgs.x86_64 glusterfs-fuse-3.12.2-47.5.el7rhgs.x86_64 glusterfs-geo-replication-3.12.2-47.5.el7rhgs.x86_64 glusterfs-libs-3.12.2-47.5.el7rhgs.x86_64 glusterfs-rdma-3.12.2-47.5.el7rhgs.x86_64 glusterfs-server-3.12.2-47.5.el7rhgs.x86_64 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: This bug appears to be related to Bug 1750211 - [GSS] glustershd logs are fillinf up with error client-rpc-fops.c:1513:client3_3_inodelk_cbk: remote operation failed [Invalid argument] https://bugzilla.redhat.com/show_bug.cgi?id=1750211
*** This bug has been marked as a duplicate of bug 1750211 ***