Bug 1412602

Summary:	[Scale] : getxatrr failures in rebalance logs - "Numerical result out of range"
Product:	[Red Hat Storage] Red Hat Gluster Storage	Reporter:	Ambarish <asoman>
Component:	distribute	Assignee:	Nithya Balachandran <nbalacha>
Status:	CLOSED WORKSFORME	QA Contact:	Prasad Desala <tdesala>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	rhgs-3.2	CC:	amukherj, asoman, bturner, rhinduja, rhs-bugs, storage-qa-internal
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-02-28 09:29:49 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Ambarish 2017-01-12 12:19:39 UTC

Description of problem:
-----------------------
The intent was to scale from 1*2 to 6*2 and then back to 1*2 amidst continuous I/O from FUSE mounts.

The rebalance logs have a couple of getxattr failures :

<snip>

butcher-rebalance.log:[2017-01-11 18:47:53.842869] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac016.sbu.lab.eng.bos.redhat.com/thrd_05/d_002/d_000/_05_2068_ (b58d0fd6-ade7-4bf4-8ea4-5267ab82e923). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 18:49:32.386091] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-2: remote operation failed. Path: /file_dstdir/gqac016.sbu.lab.eng.bos.redhat.com/thrd_06/d_002/d_000/_06_2059_ (8afc3b39-80d6-4a78-96a6-6b0b229643bf). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 19:00:59.996604] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-2: remote operation failed. Path: /file_dstdir/gqac028.sbu.lab.eng.bos.redhat.com/thrd_06/d_004/d_000/_06_4033_ (ea63ac97-240c-4ece-91b5-c99da4a03c89). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 19:10:56.888079] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac028.sbu.lab.eng.bos.redhat.com/thrd_07/d_005/d_008/_07_5839_ (22b350f8-3e2d-48b8-b15e-162efc940685). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 19:11:29.767269] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac028.sbu.lab.eng.bos.redhat.com/thrd_07/d_006/d_005/_07_6568_ (7b8c9911-95af-49c4-bc0b-8c633cd51cdb). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 19:17:13.321748] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac028.sbu.lab.eng.bos.redhat.com/thrd_00/d_005/d_007/_00_5741_ (a1f7769c-42fb-4f85-a7cb-b2e85202b8cb). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 19:33:41.191712] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac005.sbu.lab.eng.bos.redhat.com/thrd_02/d_004/d_009/_02_4968_ (8527906b-0507-42ab-8e24-7a0556056624). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 19:39:27.007562] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac005.sbu.lab.eng.bos.redhat.com/thrd_01/d_005/d_000/_01_5079_ (b02e63d6-7c2e-4d99-8f2d-48269fa74d65). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 19:51:29.721946] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac010.sbu.lab.eng.bos.redhat.com/thrd_05/d_006/d_008/_05_6850_ (1ae2637d-7a2d-4bc2-b87e-98e5b11b81a0). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 19:53:59.911108] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac010.sbu.lab.eng.bos.redhat.com/thrd_03/d_003/d_003/_03_3320_ (1bbe8bd5-65fc-47d8-bc35-7cc9c344a79c). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 19:54:21.713989] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac010.sbu.lab.eng.bos.redhat.com/thrd_03/d_004/d_002/_03_4279_ (293ae3b5-6895-4fd1-bcb0-8143fa85c1aa). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 19:56:46.180999] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-2: remote operation failed. Path: /file_dstdir/gqac010.sbu.lab.eng.bos.redhat.com/thrd_07/d_009/d_006/_07_9650_ (e9b5607b-a31e-4b56-b327-429d56d647f4). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 19:57:34.920859] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac010.sbu.lab.eng.bos.redhat.com/thrd_07/d_006/d_009/_07_6945_ (ad2288e2-3e2a-4e1c-a68c-7cd8a873ebcd). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 19:59:09.116049] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac010.sbu.lab.eng.bos.redhat.com/thrd_07/d_004/d_002/_07_4296_ (8938d741-fcdc-421d-931f-fbb21f485d33). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 19:59:42.039437] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac010.sbu.lab.eng.bos.redhat.com/thrd_07/d_007/d_006/_07_7618_ (2ba41971-eb6f-422f-821a-3947e61e73e0). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 20:00:02.228801] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac010.sbu.lab.eng.bos.redhat.com/thrd_07/d_008/d_001/_07_8183_ (bddd5fb1-e60a-4762-af6b-1f5229102d9f). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 20:00:24.405253] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-2: remote operation failed. Path: /file_dstdir/gqac010.sbu.lab.eng.bos.redhat.com/thrd_04/d_001/d_006/_04_1697_ (52ddce55-4d53-4136-b10c-58e7eb6a7cf3). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 20:02:40.923958] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac010.sbu.lab.eng.bos.redhat.com/thrd_04/d_002/d_001/_04_2180_ (d03765c7-f531-4a83-b5a2-71f29d15cde8). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 20:04:46.796442] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac010.sbu.lab.eng.bos.redhat.com/thrd_01/d_005/d_001/_01_5145_ (d6e94666-1109-49f7-b86c-319836382fca). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 20:09:45.395056] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac010.sbu.lab.eng.bos.redhat.com/thrd_00/d_003/d_009/_00_3976_ (1031214c-6ec7-4360-849a-2ed2a7b65810). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 20:10:14.028826] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac010.sbu.lab.eng.bos.redhat.com/thrd_00/d_004/d_009/_00_4934_ (108ce0a8-1c62-4054-a21f-39399956c349). Key: (null) [Numerical result out of range]
[root@gqas010 glusterfs]# 

<snip>

Version-Release number of selected component (if applicable):
-------------------------------------------------------------
glusterfs-3.8.4-11.el7rhgs.x86_64

How reproducible:
-----------------

1/1


Actual results:
---------------

getxattr fails in rebal logs.

Expected results:
-----------------

No getxattr failures.

Additional info:
----------------
Client and Server OS :RHEL 7.3

*Vol Config* :

[root@gqas009 ~]# gluster v status
Status of volume: butcher
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick gqas010.sbu.lab.eng.bos.redhat.com:/b
ricks1/A                                    49152     0          Y       23269
Brick gqas009.sbu.lab.eng.bos.redhat.com:/b
ricks1/A                                    49152     0          Y       23170
Brick gqas010.sbu.lab.eng.bos.redhat.com:/b
ricks2/A                                    49153     0          Y       23466
Brick gqas009.sbu.lab.eng.bos.redhat.com:/b
ricks2/A                                    49153     0          Y       23380
Brick gqas010.sbu.lab.eng.bos.redhat.com:/b
ricks3/A                                    49154     0          Y       24074
Brick gqas009.sbu.lab.eng.bos.redhat.com:/b
ricks3/A                                    49154     0          Y       24472
Brick gqas010.sbu.lab.eng.bos.redhat.com:/b
ricks4/A                                    49155     0          Y       24872
Brick gqas009.sbu.lab.eng.bos.redhat.com:/b
ricks4/A                                    49155     0          Y       25346
Self-heal Daemon on localhost               N/A       N/A        Y       27002
Quota Daemon on localhost                   N/A       N/A        Y       27010
Self-heal Daemon on gqas015.sbu.lab.eng.bos
.redhat.com                                 N/A       N/A        Y       25917
Quota Daemon on gqas015.sbu.lab.eng.bos.red
hat.com                                     N/A       N/A        Y       25925
Self-heal Daemon on gqas014.sbu.lab.eng.bos
.redhat.com                                 N/A       N/A        Y       25484
Quota Daemon on gqas014.sbu.lab.eng.bos.red
hat.com                                     N/A       N/A        Y       25492
Self-heal Daemon on gqas010.sbu.lab.eng.bos
.redhat.com                                 N/A       N/A        Y       26554
Quota Daemon on gqas010.sbu.lab.eng.bos.red
hat.com                                     N/A       N/A        Y       26562
 
Task Status of Volume butcher
------------------------------------------------------------------------------
Task                 : Rebalance           
ID                   : 86df50c3-00fc-409c-aac8-02c64dd5faa5
Status               : completed           
 
[root@gqas009 ~]#

Comment 3 Ambarish 2017-01-12 12:21:07 UTC

**************
EXACT WORKLOAD
**************

Client 1 : dd in loop 

Client 2 : Bonnie++

Client 3 : tarball untar

Client 4:  finds and fileop

Comment 4 Pranith Kumar K 2017-01-13 04:48:47 UTC

Afr does either lookup/fstat populating xdata to get afr xattrs. The only time afr does extra getxattrs is when it performs heals but in rebalance heals are disabled. Considering these are coming only in rebalance logs as per the bz description assigning the bug to distribute to take a first look.