Bug 1412602 - [Scale] : getxatrr failures in rebalance logs - "Numerical result out of range"
Summary: [Scale] : getxatrr failures in rebalance logs - "Numerical result out of range"
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: distribute
Version: rhgs-3.2
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: ---
Assignee: Nithya Balachandran
QA Contact: Prasad Desala
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-12 12:19 UTC by Ambarish
Modified: 2017-02-28 09:29 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-02-28 09:29:49 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Ambarish 2017-01-12 12:19:39 UTC
Description of problem:
-----------------------
The intent was to scale from 1*2 to 6*2 and then back to 1*2 amidst continuous I/O from FUSE mounts.

The rebalance logs have a couple of getxattr failures :

<snip>

butcher-rebalance.log:[2017-01-11 18:47:53.842869] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac016.sbu.lab.eng.bos.redhat.com/thrd_05/d_002/d_000/_05_2068_ (b58d0fd6-ade7-4bf4-8ea4-5267ab82e923). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 18:49:32.386091] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-2: remote operation failed. Path: /file_dstdir/gqac016.sbu.lab.eng.bos.redhat.com/thrd_06/d_002/d_000/_06_2059_ (8afc3b39-80d6-4a78-96a6-6b0b229643bf). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 19:00:59.996604] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-2: remote operation failed. Path: /file_dstdir/gqac028.sbu.lab.eng.bos.redhat.com/thrd_06/d_004/d_000/_06_4033_ (ea63ac97-240c-4ece-91b5-c99da4a03c89). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 19:10:56.888079] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac028.sbu.lab.eng.bos.redhat.com/thrd_07/d_005/d_008/_07_5839_ (22b350f8-3e2d-48b8-b15e-162efc940685). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 19:11:29.767269] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac028.sbu.lab.eng.bos.redhat.com/thrd_07/d_006/d_005/_07_6568_ (7b8c9911-95af-49c4-bc0b-8c633cd51cdb). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 19:17:13.321748] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac028.sbu.lab.eng.bos.redhat.com/thrd_00/d_005/d_007/_00_5741_ (a1f7769c-42fb-4f85-a7cb-b2e85202b8cb). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 19:33:41.191712] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac005.sbu.lab.eng.bos.redhat.com/thrd_02/d_004/d_009/_02_4968_ (8527906b-0507-42ab-8e24-7a0556056624). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 19:39:27.007562] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac005.sbu.lab.eng.bos.redhat.com/thrd_01/d_005/d_000/_01_5079_ (b02e63d6-7c2e-4d99-8f2d-48269fa74d65). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 19:51:29.721946] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac010.sbu.lab.eng.bos.redhat.com/thrd_05/d_006/d_008/_05_6850_ (1ae2637d-7a2d-4bc2-b87e-98e5b11b81a0). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 19:53:59.911108] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac010.sbu.lab.eng.bos.redhat.com/thrd_03/d_003/d_003/_03_3320_ (1bbe8bd5-65fc-47d8-bc35-7cc9c344a79c). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 19:54:21.713989] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac010.sbu.lab.eng.bos.redhat.com/thrd_03/d_004/d_002/_03_4279_ (293ae3b5-6895-4fd1-bcb0-8143fa85c1aa). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 19:56:46.180999] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-2: remote operation failed. Path: /file_dstdir/gqac010.sbu.lab.eng.bos.redhat.com/thrd_07/d_009/d_006/_07_9650_ (e9b5607b-a31e-4b56-b327-429d56d647f4). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 19:57:34.920859] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac010.sbu.lab.eng.bos.redhat.com/thrd_07/d_006/d_009/_07_6945_ (ad2288e2-3e2a-4e1c-a68c-7cd8a873ebcd). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 19:59:09.116049] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac010.sbu.lab.eng.bos.redhat.com/thrd_07/d_004/d_002/_07_4296_ (8938d741-fcdc-421d-931f-fbb21f485d33). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 19:59:42.039437] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac010.sbu.lab.eng.bos.redhat.com/thrd_07/d_007/d_006/_07_7618_ (2ba41971-eb6f-422f-821a-3947e61e73e0). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 20:00:02.228801] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac010.sbu.lab.eng.bos.redhat.com/thrd_07/d_008/d_001/_07_8183_ (bddd5fb1-e60a-4762-af6b-1f5229102d9f). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 20:00:24.405253] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-2: remote operation failed. Path: /file_dstdir/gqac010.sbu.lab.eng.bos.redhat.com/thrd_04/d_001/d_006/_04_1697_ (52ddce55-4d53-4136-b10c-58e7eb6a7cf3). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 20:02:40.923958] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac010.sbu.lab.eng.bos.redhat.com/thrd_04/d_002/d_001/_04_2180_ (d03765c7-f531-4a83-b5a2-71f29d15cde8). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 20:04:46.796442] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac010.sbu.lab.eng.bos.redhat.com/thrd_01/d_005/d_001/_01_5145_ (d6e94666-1109-49f7-b86c-319836382fca). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 20:09:45.395056] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac010.sbu.lab.eng.bos.redhat.com/thrd_00/d_003/d_009/_00_3976_ (1031214c-6ec7-4360-849a-2ed2a7b65810). Key: (null) [Numerical result out of range]
butcher-rebalance.log:[2017-01-11 20:10:14.028826] W [MSGID: 114031] [client-rpc-fops.c:1102:client3_3_getxattr_cbk] 0-butcher-client-0: remote operation failed. Path: /file_dstdir/gqac010.sbu.lab.eng.bos.redhat.com/thrd_00/d_004/d_009/_00_4934_ (108ce0a8-1c62-4054-a21f-39399956c349). Key: (null) [Numerical result out of range]
[root@gqas010 glusterfs]# 

<snip>

Version-Release number of selected component (if applicable):
-------------------------------------------------------------
glusterfs-3.8.4-11.el7rhgs.x86_64

How reproducible:
-----------------

1/1


Actual results:
---------------

getxattr fails in rebal logs.

Expected results:
-----------------

No getxattr failures.

Additional info:
----------------
Client and Server OS :RHEL 7.3

*Vol Config* :

[root@gqas009 ~]# gluster v status
Status of volume: butcher
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick gqas010.sbu.lab.eng.bos.redhat.com:/b
ricks1/A                                    49152     0          Y       23269
Brick gqas009.sbu.lab.eng.bos.redhat.com:/b
ricks1/A                                    49152     0          Y       23170
Brick gqas010.sbu.lab.eng.bos.redhat.com:/b
ricks2/A                                    49153     0          Y       23466
Brick gqas009.sbu.lab.eng.bos.redhat.com:/b
ricks2/A                                    49153     0          Y       23380
Brick gqas010.sbu.lab.eng.bos.redhat.com:/b
ricks3/A                                    49154     0          Y       24074
Brick gqas009.sbu.lab.eng.bos.redhat.com:/b
ricks3/A                                    49154     0          Y       24472
Brick gqas010.sbu.lab.eng.bos.redhat.com:/b
ricks4/A                                    49155     0          Y       24872
Brick gqas009.sbu.lab.eng.bos.redhat.com:/b
ricks4/A                                    49155     0          Y       25346
Self-heal Daemon on localhost               N/A       N/A        Y       27002
Quota Daemon on localhost                   N/A       N/A        Y       27010
Self-heal Daemon on gqas015.sbu.lab.eng.bos
.redhat.com                                 N/A       N/A        Y       25917
Quota Daemon on gqas015.sbu.lab.eng.bos.red
hat.com                                     N/A       N/A        Y       25925
Self-heal Daemon on gqas014.sbu.lab.eng.bos
.redhat.com                                 N/A       N/A        Y       25484
Quota Daemon on gqas014.sbu.lab.eng.bos.red
hat.com                                     N/A       N/A        Y       25492
Self-heal Daemon on gqas010.sbu.lab.eng.bos
.redhat.com                                 N/A       N/A        Y       26554
Quota Daemon on gqas010.sbu.lab.eng.bos.red
hat.com                                     N/A       N/A        Y       26562
 
Task Status of Volume butcher
------------------------------------------------------------------------------
Task                 : Rebalance           
ID                   : 86df50c3-00fc-409c-aac8-02c64dd5faa5
Status               : completed           
 
[root@gqas009 ~]#

Comment 3 Ambarish 2017-01-12 12:21:07 UTC
**************
EXACT WORKLOAD
**************

Client 1 : dd in loop 

Client 2 : Bonnie++

Client 3 : tarball untar

Client 4:  finds and fileop

Comment 4 Pranith Kumar K 2017-01-13 04:48:47 UTC
Afr does either lookup/fstat populating xdata to get afr xattrs. The only time afr does extra getxattrs is when it performs heals but in rebalance heals are disabled. Considering these are coming only in rebalance logs as per the bz description assigning the bug to distribute to take a first look.


Note You need to log in before you can comment on or make changes to this bug.