Bug 1843670

Summary:	[Healing] glustershd.log spammed with "remote operation failed [Invalid argument]" messages
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Andrew Robinson <anrobins>
Component:	disperse	Assignee:	Ashish Pandey <aspandey>
Status:	CLOSED DUPLICATE	QA Contact:	Nag Pavan Chilakam <nchilaka>
Severity:	urgent	Docs Contact:
Priority:	high
Version:	rhgs-3.5	CC:	aspandey, cswanson, rhs-bugs, sheggodu, storage-qa-internal, svaughn
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-06-11 05:37:53 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Andrew Robinson 2020-06-03 19:09:21 UTC

Description of problem:

The glustershd log is getting spammed with messages like these:

[2020-06-02 15:11:38.945198] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-HS_High_Resiliency-client-22: remote operation failed [Invalid argument]
[2020-06-02 15:11:39.103001] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-HS_High_Resiliency-client-13: remote operation failed [Invalid argument]
[2020-06-02 15:11:39.121600] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-HS_High_Resiliency-client-14: remote operation failed [Invalid argument]
[2020-06-02 15:11:39.137356] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-HS_High_Resiliency-client-23: remote operation failed [Invalid argument]
[2020-06-02 15:11:39.160914] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-HS_High_Resiliency-client-19: remote operation failed [Invalid argument]
[2020-06-02 15:11:39.220582] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-HS_High_Resiliency-client-21: remote operation failed [Invalid argument]
[2020-06-02 15:11:39.403613] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-HS_High_Resiliency-client-15: remote operation failed [Invalid argument]
[2020-06-02 15:11:39.406699] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-HS_High_Resiliency-client-17: remote operation failed [Invalid argument]
[2020-06-02 15:11:39.527867] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-HS_High_Resiliency-client-18: remote operation failed [Invalid argument]
[2020-06-02 15:11:39.530420] E [MSGID: 114031] [client-rpc-fops_v2.c:1345:client4_0_inodelk_cbk] 0-HS_High_Resiliency-client-20: remote operation failed [Invalid argument]

The spamming is so prolific, it slows down 'gluster volume heal' operations including the actual healing and the listing of the files in healing.

The customer sent the current set of logs, and this is the current counts of these messages in the logs:

<node>92-glustershd.log: 69,815,054
<node>93-glustershd.log: 81,046,086
<node>94-glustershd.log: 80,786,780
<node>95-glustershd.log: 18,223,843
<node>96-glustershd.log: 48,505,734
<node>97-glustershd.log:     22,290


Version-Release number of selected component (if applicable):

Red Hat Enterprise Linux Server release 7.7 (Maipo)
glusterfs-3.12.2-47.5.el7rhgs.x86_64
glusterfs-api-3.12.2-47.5.el7rhgs.x86_64                    
glusterfs-client-xlators-3.12.2-47.5.el7rhgs.x86_64         
glusterfs-cli-3.12.2-47.5.el7rhgs.x86_64                    
glusterfs-events-3.12.2-47.5.el7rhgs.x86_64                 
glusterfs-fuse-3.12.2-47.5.el7rhgs.x86_64                   
glusterfs-geo-replication-3.12.2-47.5.el7rhgs.x86_64        
glusterfs-libs-3.12.2-47.5.el7rhgs.x86_64                   
glusterfs-rdma-3.12.2-47.5.el7rhgs.x86_64                   
glusterfs-server-3.12.2-47.5.el7rhgs.x86_64                 


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

This bug appears to be related to 

Bug 1750211 - [GSS] glustershd logs are fillinf up with error client-rpc-fops.c:1513:client3_3_inodelk_cbk: remote operation failed [Invalid argument]
https://bugzilla.redhat.com/show_bug.cgi?id=1750211

Comment 15 Sunil Kumar Acharya 2020-06-11 05:37:53 UTC


*** This bug has been marked as a duplicate of bug 1750211 ***