Bug 1843670 - [Healing] glustershd.log spammed with "remote operation failed [Invalid argument]" messages
Summary: [Healing] glustershd.log spammed with "remote operation failed [Invalid argum...
Keywords:
Status: CLOSED DUPLICATE of bug 1750211
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: disperse
Version: rhgs-3.5
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: ---
: ---
Assignee: Ashish Pandey
QA Contact: Nag Pavan Chilakam
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-06-03 19:09 UTC by Andrew Robinson
Modified: 2020-07-15 07:00 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-06-11 05:37:53 UTC
Embargoed:


Attachments (Terms of Use)

Description Andrew Robinson 2020-06-03 19:09:21 UTC
Description of problem:

The glustershd log is getting spammed with messages like these:

[2020-06-02 15:11:38.945198] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-HS_High_Resiliency-client-22: remote operation failed [Invalid argument]
[2020-06-02 15:11:39.103001] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-HS_High_Resiliency-client-13: remote operation failed [Invalid argument]
[2020-06-02 15:11:39.121600] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-HS_High_Resiliency-client-14: remote operation failed [Invalid argument]
[2020-06-02 15:11:39.137356] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-HS_High_Resiliency-client-23: remote operation failed [Invalid argument]
[2020-06-02 15:11:39.160914] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-HS_High_Resiliency-client-19: remote operation failed [Invalid argument]
[2020-06-02 15:11:39.220582] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-HS_High_Resiliency-client-21: remote operation failed [Invalid argument]
[2020-06-02 15:11:39.403613] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-HS_High_Resiliency-client-15: remote operation failed [Invalid argument]
[2020-06-02 15:11:39.406699] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-HS_High_Resiliency-client-17: remote operation failed [Invalid argument]
[2020-06-02 15:11:39.527867] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-HS_High_Resiliency-client-18: remote operation failed [Invalid argument]
[2020-06-02 15:11:39.530420] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-HS_High_Resiliency-client-20: remote operation failed [Invalid argument]

The spamming is so prolific, it slows down 'gluster volume heal' operations including the actual healing and the listing of the files in healing.

The customer sent the current set of logs, and this is the current counts of these messages in the logs:

<node>92-glustershd.log: 69,815,054
<node>93-glustershd.log: 81,046,086
<node>94-glustershd.log: 80,786,780
<node>95-glustershd.log: 18,223,843
<node>96-glustershd.log: 48,505,734
<node>97-glustershd.log:     22,290


Version-Release number of selected component (if applicable):

Red Hat Enterprise Linux Server release 7.7 (Maipo)
glusterfs-3.12.2-47.5.el7rhgs.x86_64
glusterfs-api-3.12.2-47.5.el7rhgs.x86_64                    
glusterfs-client-xlators-3.12.2-47.5.el7rhgs.x86_64         
glusterfs-cli-3.12.2-47.5.el7rhgs.x86_64                    
glusterfs-events-3.12.2-47.5.el7rhgs.x86_64                 
glusterfs-fuse-3.12.2-47.5.el7rhgs.x86_64                   
glusterfs-geo-replication-3.12.2-47.5.el7rhgs.x86_64        
glusterfs-libs-3.12.2-47.5.el7rhgs.x86_64                   
glusterfs-rdma-3.12.2-47.5.el7rhgs.x86_64                   
glusterfs-server-3.12.2-47.5.el7rhgs.x86_64                 


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

This bug appears to be related to 

Bug 1750211 - [GSS] glustershd logs are fillinf up with error client-rpc-fops.c:1513:client3_3_inodelk_cbk: remote operation failed [Invalid argument]
https://bugzilla.redhat.com/show_bug.cgi?id=1750211

Comment 15 Sunil Kumar Acharya 2020-06-11 05:37:53 UTC

*** This bug has been marked as a duplicate of bug 1750211 ***


Note You need to log in before you can comment on or make changes to this bug.