Bug 1224159

Summary: data tiering:detach-tier start command fails with "Commit failed on localhost"
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Nag Pavan Chilakam <nchilaka>
Component: tierAssignee: Mohammed Rafi KC <rkavunga>
Status: CLOSED ERRATA QA Contact: Nag Pavan Chilakam <nchilaka>
Severity: high Docs Contact:
Priority: medium    
Version: rhgs-3.1CC: josferna, rhs-bugs, storage-qa-internal, trao
Target Milestone: ---   
Target Release: RHGS 3.1.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-07-29 04:48:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1202842    

Description Nag Pavan Chilakam 2015-05-22 09:35:44 UTC 
Description of problem:
=======================
Detach tier start command fails with Commit failed on localhost. error

Version-Release number of selected component (if applicable):
=======================
root@zod glusterfs]# gluster --version
glusterfs 3.7.0 built on May 15 2015 01:33:40
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.
[root@zod glusterfs]# rpm -qa|grep gluster
glusterfs-debuginfo-3.7.0-2.el7rhs.x86_64
glusterfs-geo-replication-3.7.0-2.el7rhs.x86_64
glusterfs-client-xlators-3.7.0-2.el7rhs.x86_64
glusterfs-cli-3.7.0-2.el7rhs.x86_64
glusterfs-libs-3.7.0-2.el7rhs.x86_64
glusterfs-api-3.7.0-2.el7rhs.x86_64
glusterfs-server-3.7.0-2.el7rhs.x86_64
glusterfs-resource-agents-3.7.0-2.el7rhs.noarch
glusterfs-rdma-3.7.0-2.el7rhs.x86_64
glusterfs-devel-3.7.0-2.el7rhs.x86_64
glusterfs-api-devel-3.7.0-2.el7rhs.x86_64
glusterfs-3.7.0-2.el7rhs.x86_64
glusterfs-fuse-3.7.0-2.el7rhs.x86_64



Steps to Reproduce:
==================
1.create tiered vol and start it
2.do a detach-tier start 
3.this fails

bash-4.3$ ssh root@zod
root@zod's password: 
Last login: Wed May 20 15:09:24 2015 from 10.10.51.217
[root@zod ~]# gluster v status
Status of volume: vol1
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Hot Bricks:
Brick yarrow:/ssdbricks_75G_1/vol1          49154     0          Y       22011
Brick 10.70.35.144:/ssdbricks_75G_1/vol1    49154     0          Y       2416 
Brick 10.70.35.144:/brick_200G_1/vol1       49152     0          Y       2187 
Brick yarrow:/brick_200G_1/vol1             49152     0          Y       21883
Brick 10.70.35.144:/brick_200G_2/vol1       49153     0          Y       2206 
Brick yarrow:/brick_200G_2/vol1             49153     0          Y       21902
NFS Server on localhost                     N/A       N/A        N       N/A  
NFS Server on yarrow                        N/A       N/A        N       N/A  
 
Task Status of Volume vol1
------------------------------------------------------------------------------
Task                 : Rebalance           
ID                   : 9dd4b419-4436-493f-9553-c39cc52a6582
Status               : in progress         
 
Status of volume: vol2
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Hot Bricks:
Brick 10.70.35.144:/ssdbricks_75G_1/vol2    49157     0          Y       5742 
Brick yarrow:/ssdbricks_75G_1/vol2          49157     0          Y       25289
Brick 10.70.35.144:/brick_200G_1/vol2       49155     0          Y       5618 
Brick yarrow:/brick_200G_1/vol2             49155     0          Y       25173
Brick 10.70.35.144:/brick_200G_2/vol2       49156     0          Y       5636 
Brick yarrow:/brick_200G_2/vol2             49156     0          Y       25192
NFS Server on localhost                     N/A       N/A        N       N/A  
NFS Server on yarrow                        N/A       N/A        N       N/A  
 
Task Status of Volume vol2
------------------------------------------------------------------------------
Task                 : Rebalance           
ID                   : aaea82ea-ddd5-4ae5-ad68-5343da3a29c8
Status               : in progress         
 
[root@zod ~]# gluster v info
 
Volume Name: vol1
Type: Tier
Volume ID: 23dba7de-a94a-49c2-80f1-6d97b0ab1309
Status: Started
Number of Bricks: 6
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distribute
Number of Bricks: 2
Brick1: yarrow:/ssdbricks_75G_1/vol1
Brick2: 10.70.35.144:/ssdbricks_75G_1/vol1
Cold Bricks:
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick3: 10.70.35.144:/brick_200G_1/vol1
Brick4: yarrow:/brick_200G_1/vol1
Brick5: 10.70.35.144:/brick_200G_2/vol1
Brick6: yarrow:/brick_200G_2/vol1
Options Reconfigured:
cluster.tier-demote-frequency: 10
features.record-counters: on
features.ctr-enabled: on
performance.readdir-ahead: on
 
Volume Name: vol2
Type: Tier
Volume ID: 858ae0b9-0cc9-41a9-b89b-d42e6791e2d7
Status: Started
Number of Bricks: 6
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distribute
Number of Bricks: 2
Brick1: 10.70.35.144:/ssdbricks_75G_1/vol2
Brick2: yarrow:/ssdbricks_75G_1/vol2
Cold Bricks:
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick3: 10.70.35.144:/brick_200G_1/vol2
Brick4: yarrow:/brick_200G_1/vol2
Brick5: 10.70.35.144:/brick_200G_2/vol2
Brick6: yarrow:/brick_200G_2/vol2
Options Reconfigured:
performance.readdir-ahead: on
[root@zod ~]# gluster v detach-tier 
Usage: volume detach-tier <VOLNAME>  <start|stop|status|commit|[force]>
[root@zod ~]# gluster v detach-tier  vol2
Usage: volume detach-tier <VOLNAME>  <start|stop|status|commit|[force]>
[root@zod ~]# gluster v detach-tier  vol2 start
volume detach-tier start: failed: Commit failed on localhost. Please check the log file for more details.
[root@zod ~]# gluster v info vol2
 
Volume Name: vol2
Type: Tier
Volume ID: 858ae0b9-0cc9-41a9-b89b-d42e6791e2d7
Status: Started
Number of Bricks: 6
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distribute
Number of Bricks: 2
Brick1: 10.70.35.144:/ssdbricks_75G_1/vol2
Brick2: yarrow:/ssdbricks_75G_1/vol2
Cold Bricks:
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick3: 10.70.35.144:/brick_200G_1/vol2
Brick4: yarrow:/brick_200G_1/vol2
Brick5: 10.70.35.144:/brick_200G_2/vol2
Brick6: yarrow:/brick_200G_2/vol2
Options Reconfigured:
performance.readdir-ahead: on
[root@zod ~]# gluster v detach-tier  vol2 start force
volume detach-tier start: failed: An earlier remove-brick task exists for volume vol2. Either commit it or stop it before starting a new task.
[root@zod ~]# gluster v info
 
Volume Name: vol1
Type: Tier
Volume ID: 23dba7de-a94a-49c2-80f1-6d97b0ab1309
Status: Started
Number of Bricks: 6
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distribute
Number of Bricks: 2
Brick1: yarrow:/ssdbricks_75G_1/vol1
Brick2: 10.70.35.144:/ssdbricks_75G_1/vol1
Cold Bricks:
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick3: 10.70.35.144:/brick_200G_1/vol1
Brick4: yarrow:/brick_200G_1/vol1
Brick5: 10.70.35.144:/brick_200G_2/vol1
Brick6: yarrow:/brick_200G_2/vol1
Options Reconfigured:
cluster.tier-demote-frequency: 10
features.record-counters: on
features.ctr-enabled: on
performance.readdir-ahead: on
 
Volume Name: vol2
Type: Tier
Volume ID: 858ae0b9-0cc9-41a9-b89b-d42e6791e2d7
Status: Started
Number of Bricks: 6
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distribute
Number of Bricks: 2
Brick1: 10.70.35.144:/ssdbricks_75G_1/vol2
Brick2: yarrow:/ssdbricks_75G_1/vol2
Cold Bricks:
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick3: 10.70.35.144:/brick_200G_1/vol2
Brick4: yarrow:/brick_200G_1/vol2
Brick5: 10.70.35.144:/brick_200G_2/vol2
Brick6: yarrow:/brick_200G_2/vol2
Options Reconfigured:
performance.readdir-ahead: on
[root@zod ~]# gluster v info vol1 
 
Volume Name: vol1
Type: Tier
Volume ID: 23dba7de-a94a-49c2-80f1-6d97b0ab1309
Status: Started
Number of Bricks: 6
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distribute
Number of Bricks: 2
Brick1: yarrow:/ssdbricks_75G_1/vol1
Brick2: 10.70.35.144:/ssdbricks_75G_1/vol1
Cold Bricks:
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick3: 10.70.35.144:/brick_200G_1/vol1
Brick4: yarrow:/brick_200G_1/vol1
Brick5: 10.70.35.144:/brick_200G_2/vol1
Brick6: yarrow:/brick_200G_2/vol1
Options Reconfigured:
cluster.tier-demote-frequency: 10
features.record-counters: on
features.ctr-enabled: on
performance.readdir-ahead: on
[root@zod ~]# gluster v detach-tier vol1 start
volume detach-tier start: failed: Commit failed on localhost. Please check the log file for more details.
[root@zod ~]# cd /var/log/glusterfs/
[root@zod glusterfs]# ls
bricks   cmd_history.log           etc-glusterfs-glusterd.vol.log  geo-replication-slaves  glustershd.log-20150521  nfs.log-20150521  vol1-rebalance.log
cli.log  cmd_history.log-20150521  geo-replication                 glustershd.log          nfs.log                  snaps             vol2-rebalance.log
[root@zod glusterfs]# less etc-glusterfs-glusterd.vol.log 
[root@zod glusterfs]# gluster v--vers
unrecognized word: v--vers (position 0)
[root@zod glusterfs]# gluster --version
glusterfs 3.7.0 built on May 15 2015 01:33:40
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.
[root@zod glusterfs]# rpm -qa|grep gluster
glusterfs-debuginfo-3.7.0-2.el7rhs.x86_64
glusterfs-geo-replication-3.7.0-2.el7rhs.x86_64
glusterfs-client-xlators-3.7.0-2.el7rhs.x86_64
glusterfs-cli-3.7.0-2.el7rhs.x86_64
glusterfs-libs-3.7.0-2.el7rhs.x86_64
glusterfs-api-3.7.0-2.el7rhs.x86_64
glusterfs-server-3.7.0-2.el7rhs.x86_64
glusterfs-resource-agents-3.7.0-2.el7rhs.noarch
glusterfs-rdma-3.7.0-2.el7rhs.x86_64
glusterfs-devel-3.7.0-2.el7rhs.x86_64
glusterfs-api-devel-3.7.0-2.el7rhs.x86_64
glusterfs-3.7.0-2.el7rhs.x86_64
glusterfs-fuse-3.7.0-2.el7rhs.x86_64
[root@zod glusterfs]# 






bash-4.3$ ssh root@zod
root@zod's password: 
Last login: Fri May 22 14:51:13 2015 from vpn-61-236.rdu2.redhat.com
[root@zod ~]# gluster v status vol1
Status of volume: vol1
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Hot Bricks:
Brick yarrow:/ssdbricks_75G_1/vol1          49154     0          Y       22011
Brick 10.70.35.144:/ssdbricks_75G_1/vol1    49154     0          Y       2416 
Brick 10.70.35.144:/brick_200G_1/vol1       49152     0          Y       2187 
Brick yarrow:/brick_200G_1/vol1             49152     0          Y       21883
Brick 10.70.35.144:/brick_200G_2/vol1       49153     0          Y       2206 
Brick yarrow:/brick_200G_2/vol1             49153     0          Y       21902
NFS Server on localhost                     N/A       N/A        N       N/A  
NFS Server on yarrow                        N/A       N/A        N       N/A  
 
Task Status of Volume vol1
------------------------------------------------------------------------------
Task                 : Remove brick        
ID                   : b41122e6-3707-4f23-bff1-b383ec2d1c6b
Removed bricks:     
10.70.35.144:/ssdbricks_75G_1/vol1
yarrow:/ssdbricks_75G_1/vol1
Status               : completed           
 
[root@zod ~]# gluster v detach-tier vol1 status
volume detach-tier unknown: success
[root@zod ~]#  gluster  v 
unrecognized command
[root@zod ~]#  gluster 
gluster> ?
unrecognized word: ? (position 0)
gluster> ?
unrecognized word: ? (position 0)
gluster> helpo
unrecognized word: helpo (position 0)
gluster> help
volume info [all|<VOLNAME>] - list information of all volumes
volume create <NEW-VOLNAME> [stripe <COUNT>] [replica <COUNT> [arbiter <COUNT>]] [disperse [<COUNT>]] [disperse-data <COUNT>] [redundancy <COUNT>] [transport <tcp|rdma|tcp,rdma>] <NEW-BRICK>?<vg_name>... [force] - create a new volume of specified type with mentioned bricks
volume delete <VOLNAME> - delete volume specified by <VOLNAME>
volume start <VOLNAME> [force] - start volume specified by <VOLNAME>
volume stop <VOLNAME> [force] - stop volume specified by <VOLNAME>
volume attach-tier <VOLNAME> [<replica COUNT>] <NEW-BRICK>... - attach tier to volume <VOLNAME>
volume detach-tier <VOLNAME>  <start|stop|status|commit|[force]> - detach tier from volume <VOLNAME>
volume add-brick <VOLNAME> [<stripe|replica> <COUNT>] <NEW-BRICK> ... [force] - add brick to volume <VOLNAME>
volume remove-brick <VOLNAME> [replica <COUNT>] <BRICK> ... <start|stop|status|commit|force> - remove brick from volume <VOLNAME>
volume rebalance <VOLNAME> {{fix-layout start} | {start [force]|stop|status}} - rebalance operations
volume replace-brick <VOLNAME> <SOURCE-BRICK> <NEW-BRICK> {commit force} - replace-brick operations
volume set <VOLNAME> <KEY> <VALUE> - set options for volume <VOLNAME>
volume help - display help for the volume command
volume log <VOLNAME> rotate [BRICK] - rotate the log file for corresponding volume/brick
volume log rotate <VOLNAME> [BRICK] - rotate the log file for corresponding volume/brick NOTE: This is an old syntax, will be deprecated from next release.
volume sync <HOSTNAME> [all|<VOLNAME>] - sync the volume information from a peer
volume reset <VOLNAME> [option] [force] - reset all the reconfigured options
volume geo-replication [<VOLNAME>] [<SLAVE-URL>] {create [[no-verify]|[push-pem]] [force]|start [force]|stop [force]|pause [force]|resume [force]|config|status [detail]|delete} [options...] - Geo-sync operations
volume profile <VOLNAME> {start|info [peek|incremental [peek]|cumulative|clear]|stop} [nfs] - volume profile operations
volume quota <VOLNAME> {enable|disable|list [<path> ...]| list-objects [<path> ...] | remove <path>| remove-objects <path> | default-soft-limit <percent>} |
volume quota <VOLNAME> {limit-usage <path> <size> [<percent>]} |
volume quota <VOLNAME> {limit-objects <path> <number> [<percent>]} |
volume quota <VOLNAME> {alert-time|soft-timeout|hard-timeout} {<time>} - quota translator specific operations
volume inode-quota <VOLNAME> enable - quota translator specific operations
volume top <VOLNAME> {open|read|write|opendir|readdir|clear} [nfs|brick <brick>] [list-cnt <value>] |
volume top <VOLNAME> {read-perf|write-perf} [bs <size> count <count>] [brick <brick>] [list-cnt <value>] - volume top operations
volume status [all | <VOLNAME> [nfs|shd|<BRICK>|quotad]] [detail|clients|mem|inode|fd|callpool|tasks] - display status of all or specified volume(s)/brick
volume heal <VOLNAME> [enable | disable | full |statistics [heal-count [replica <HOSTNAME:BRICKNAME>]] |info [healed | heal-failed | split-brain] |split-brain {bigger-file <FILE> |source-brick <HOSTNAME:BRICKNAME> [<FILE>]}] - self-heal commands on volume specified by <VOLNAME>
volume statedump <VOLNAME> [nfs|quotad] [all|mem|iobuf|callpool|priv|fd|inode|history]... - perform statedump on bricks
volume list - list all volumes in cluster
volume clear-locks <VOLNAME> <path> kind {blocked|granted|all}{inode [range]|entry [basename]|posix [range]} - Clear locks held on path
volume barrier <VOLNAME> {enable|disable} - Barrier/unbarrier file operations on a volume
volume get <VOLNAME> <key|all> - Get the value of the all options or given option for volume <VOLNAME>
volume bitrot <VOLNAME> {enable|disable} |
volume bitrot <volname> scrub-throttle {lazy|normal|aggressive} |
volume bitrot <volname> scrub-frequency {daily|weekly|biweekly|monthly} |
volume bitrot <volname> scrub {pause|resume} - Bitrot translator specific operation. For more information about bitrot command type  'man gluster'
peer probe <HOSTNAME> - probe peer specified by <HOSTNAME>
peer detach <HOSTNAME> [force] - detach peer specified by <HOSTNAME>
peer status - list status of peers
peer help - Help command for peer 
pool list - list all the nodes in the pool (including localhost)
quit - quit
help - display command options
exit - exit
snapshot help - display help for snapshot commands
snapshot create <snapname> <volname> [no-timestamp] [description <description>] [force] - Snapshot Create.
snapshot clone <clonename> <snapname> - Snapshot Clone.
snapshot restore <snapname> - Snapshot Restore.
snapshot status [(snapname | volume <volname>)] - Snapshot Status.
snapshot info [(snapname | volume <volname>)] - Snapshot Info.
snapshot list [volname] - Snapshot List.
snapshot config [volname] ([snap-max-hard-limit <count>] [snap-max-soft-limit <percent>]) | ([auto-delete <enable|disable>])| ([activate-on-create <enable|disable>]) - Snapshot Config.
snapshot delete (all | snapname | volume <volname>) - Snapshot Delete.
snapshot activate <snapname> [force] - Activate snapshot volume.
snapshot deactivate <snapname> - Deactivate snapshot volume.
global help - list global commands
nfs-ganesha {enable| disable}  - Enable/disable NFS-Ganesha support
gluster> volume detach-tier  status vol1
Usage: volume detach-tier <VOLNAME>  <start|stop|status|commit|[force]>
gluster> volume detach-tier vol1 status
volume detach-tier unknown: success
gluster>
Comment 2 Nag Pavan Chilakam 2015-05-22 09:37:06 UTC 



[2015-05-22 09:25:16.695688] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:25:19.696025] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:25:22.696578] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:25:25.696955] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:25:28.697341] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:25:31.697721] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:25:34.698140] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:25:37.698539] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:25:40.698893] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:25:43.699259] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:25:46.699673] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:25:49.700046] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:25:52.700441] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:25:55.700855] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:25:58.701212] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:26:01.701612] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:26:04.701964] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:26:07.702345] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:26:10.702735] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:26:13.703112] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:26:16.703489] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:26:19.703877] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:26:22.704347] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:26:25.704732] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:26:28.705113] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:26:31.706102] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:26:32.434082] W [xlator.c:141:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/3.7.0/xlator/rpc-transport/socket.so: cannot open shared object file: No such file or directory
[2015-05-22 09:26:32.434135] W [xlator.c:141:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/3.7.0/xlator/rpc-transport/socket.so: cannot open shared object file: No such file or directory
[2015-05-22 09:26:32.479252] W [xlator.c:141:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/3.7.0/xlator/configuration.so: cannot open shared object file: No such file or directory
[2015-05-22 09:26:32.479285] W [xlator.c:141:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/3.7.0/xlator/configuration.so: cannot open shared object file: No such file or directory
[2015-05-22 09:26:32.480983] W [xlator.c:141:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/3.7.0/xlator/cluster/tier.so: undefined symbol: dht_methods
[2015-05-22 09:26:32.481107] W [xlator.c:141:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/3.7.0/xlator/cluster/tier.so: undefined symbol: dht_methods
[2015-05-22 09:26:32.481219] W [xlator.c:141:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/3.7.0/xlator/cluster/tier.so: undefined symbol: dht_methods
[2015-05-22 09:26:32.481329] W [xlator.c:141:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/3.7.0/xlator/cluster/tier.so: undefined symbol: dht_methods
[2015-05-22 09:26:34.706874] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:26:37.707260] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:26:40.707671] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:26:43.708043] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:26:46.708413] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:26:49.708758] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:26:52.709126] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:26:53.734648] I [glusterd-handler.c:4057:__glusterd_handle_status_volume] 0-management: Received status volume req for volume vol1
[2015-05-22 09:26:53.743487] E [glusterd-utils.c:7478:glusterd_volume_status_aggregate_tasks_status] 0-management: Could not find matching local task for task 9dd4b419-4436-493f-9553-c39cc52a6582
[2015-05-22 09:26:55.709817] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:26:58.710155] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:27:01.710546] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
The message "I [MSGID: 106006] [glusterd-svc-mgmt.c:330:glusterd_svc_common_rpc_notify] 0-management: nfs has disconnected from glusterd." repeated 39 times between [2015-05-22 09:25:04.693905] and [2015-05-22 09:27:01.710593]
[2015-05-22 09:27:04.711038] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:27:04.711077] I [MSGID: 106006] [glusterd-svc-mgmt.c:330:glusterd_svc_common_rpc_notify] 0-management: nfs has disconnected from glusterd.
[2015-05-22 09:27:07.711424] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:27:10.711793] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:27:13.712176] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:27:16.712557] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:27:19.712935] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:27:22.713534] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:27:25.713875] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:27:28.714259] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:27:31.714646] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:27:34.715045] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:27:36.243989] I [glusterd-brick-ops.c:770:__glusterd_handle_remove_brick] 0-management: Received rem brick req
[2015-05-22 09:27:36.403724] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick /ssdbricks_75G_1/vol1 on port 49154
[2015-05-22 09:27:36.405564] W [socket.c:642:__socket_rwv] 0-socket.management: writev on 10.70.35.144:1018 failed (Broken pipe)
[2015-05-22 09:27:36.405601] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (No data available)
[2015-05-22 09:27:36.407665] I [MSGID: 106005] [glusterd-handler.c:4665:__glusterd_brick_rpc_notify] 0-management: Brick 10.70.35.144:/ssdbricks_75G_1/vol1 has disconnected from glusterd.
[2015-05-22 09:27:37.715796] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:27:37.718230] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:27:40.716224] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:27:40.718892] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:27:43.716653] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:27:43.719059] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:27:46.717095] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:27:46.719784] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:27:49.717808] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:27:49.720282] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:27:52.718225] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:27:52.721732] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:27:55.718777] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:27:55.721376] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:27:58.719166] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:27:58.721265] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:28:01.719566] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:28:01.721770] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:28:04.720009] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:28:04.722675] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:28:07.720424] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:28:07.722787] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:28:10.720857] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:28:10.723492] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:28:13.721270] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:28:13.722974] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:28:16.721687] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:28:16.724449] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:28:19.722093] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:28:19.723671] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:28:22.722705] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:28:22.724443] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:28:25.723216] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:28:25.725606] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:28:28.723662] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:28:28.725986] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:28:31.724103] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:28:31.726457] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:28:34.724539] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:28:34.726810] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:28:37.724954] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:28:37.727257] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:28:40.725346] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:28:40.727483] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:28:43.725742] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:28:43.727911] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:28:46.726176] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:28:46.728843] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:28:49.726599] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:28:49.728808] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:28:51.184035] I [glusterd-brick-ops.c:770:__glusterd_handle_remove_brick] 0-management: Received rem brick req
[2015-05-22 09:28:51.195751] W [dict.c:584:dict_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7fb8551a9f16] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_op_remove_brick+0x704)[0x7fb84a009504] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_op_commit_perform+0x69b)[0x7fb849f996db] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(gd_commit_op_phase+0xb6)[0x7fb84a00e8e6] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(gd_sync_task_begin+0x75a)[0x7fb84a00fd1a] ))))) 0-dict: dict is NULL
[2015-05-22 09:28:52.727262] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:28:52.729994] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:28:55.727652] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:28:55.734901] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:28:58.728075] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:28:58.730347] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:29:01.728496] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:29:01.731055] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
The message "I [MSGID: 106006] [glusterd-svc-mgmt.c:330:glusterd_svc_common_rpc_notify] 0-management: nfs has disconnected from glusterd." repeated 39 times between [2015-05-22 09:27:04.711077] and [2015-05-22 09:29:01.728569]
The message "I [MSGID: 106005] [glusterd-handler.c:4665:__glusterd_brick_rpc_notify] 0-management: Brick 10.70.35.144:/ssdbricks_75G_1/vol1 has disconnected from glusterd." repeated 29 times between [2015-05-22 09:27:36.407665] and [2015-05-22 09:29:01.733321]
[2015-05-22 09:29:04.729066] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:29:04.729110] I [MSGID: 106006] [glusterd-svc-mgmt.c:330:glusterd_svc_common_rpc_notify] 0-management: nfs has disconnected from glusterd.
[2015-05-22 09:29:04.731490] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:29:04.734506] I [MSGID: 106005] [glusterd-handler.c:4665:__glusterd_brick_rpc_notify] 0-management: Brick 10.70.35.144:/ssdbricks_75G_1/vol1 has disconnected from glusterd.
[2015-05-22 09:29:07.729507] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:29:07.731859] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:29:10.729913] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:29:10.732275] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:29:13.730346] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:29:13.732266] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:29:16.730794] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:29:16.732457] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:29:19.731196] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:29:19.732929] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:29:22.731919] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:29:22.734103] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:29:25.732310] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:29:25.734729] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:29:28.732715] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:29:28.734735] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:29:31.733102] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:29:31.735467] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:29:34.733483] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:29:34.736214] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:29:37.733889] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:29:37.736323] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:29:40.734286] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:29:40.736588] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:29:43.734676] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:29:43.736752] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:29:46.735080] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:29:46.737538] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:29:49.735470] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:29:49.737638] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:29:52.735855] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:29:52.738283] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:29:55.736245] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:29:55.738455] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:29:58.736635] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:29:58.739074] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:30:01.737010] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:30:01.739649] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:30:04.737733] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:30:04.740346] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:30:07.738098] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:30:07.740633] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:30:10.738483] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:30:10.740453] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:30:13.738871] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:30:13.741138] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:30:16.739268] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:30:16.741882] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:30:19.739668] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:30:19.741569] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:30:22.740271] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:30:22.743253] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:30:25.740661] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:30:25.743451] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:30:28.741053] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:30:28.743354] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:30:31.741432] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:30:31.743932] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:30:34.741809] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:30:34.744140] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:30:37.742261] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:30:37.744847] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:30:40.742646] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:30:40.744350] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:30:43.743043] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:30:43.745725] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:30:46.743447] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:30:46.745778] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:30:49.743840] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:30:49.745506] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:30:52.744234] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:30:52.746945] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:30:55.744662] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:30:55.746985] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:30:58.745065] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:30:58.747572] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:31:01.745452] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:31:01.747978] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
The message "I [MSGID: 106006] [glusterd-svc-mgmt.c:330:glusterd_svc_common_rpc_notify] 0-management: nfs has disconnected from glusterd." repeated 39 times between [2015-05-22 09:29:04.729110] and [2015-05-22 09:31:01.745500]
The message "I [MSGID: 106005] [glusterd-handler.c:4665:__glusterd_brick_rpc_notify] 0-management: Brick 10.70.35.144:/ssdbricks_75G_1/vol1 has disconnected from glusterd." repeated 39 times between [2015-05-22 09:29:04.734506] and [2015-05-22 09:31:01.750155]
[2015-05-22 09:31:04.746504] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:31:04.746583] I [MSGID: 106006] [glusterd-svc-mgmt.c:330:glusterd_svc_common_rpc_notify] 0-management: nfs has disconnected from glusterd.
[2015-05-22 09:31:04.749227] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:31:04.751308] I [MSGID: 106005] [glusterd-handler.c:4665:__glusterd_brick_rpc_notify] 0-management: Brick 10.70.35.144:/ssdbricks_75G_1/vol1 has disconnected from glusterd.
[2015-05-22 09:31:07.747103] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:31:07.749879] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:31:10.747502] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:31:10.750133] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:31:13.747922] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:31:13.749553] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:31:16.748311] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:31:16.750602] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:31:19.748722] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:31:19.750313] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:31:22.749373] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:31:22.751232] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:31:25.749855] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:31:25.751928] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:31:28.750267] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:31:28.752220] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:31:31.750657] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:31:31.752608] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:31:34.751034] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:31:34.752691] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:31:37.751437] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:31:37.753732] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:31:40.751825] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:31:40.753394] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:31:43.752218] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:31:43.754020] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:31:46.752610] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:31:46.754352] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:31:49.752997] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:31:49.754797] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:31:52.753378] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:31:52.755611] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:31:55.753769] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:31:55.756510] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:31:58.754185] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:31:58.756576] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:32:01.754570] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:32:01.757295] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:32:04.754972] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:32:04.757133] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:32:07.755408] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:32:07.762486] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:32:10.755782] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:32:10.758595] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:32:13.756164] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:32:13.758007] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:32:16.756557] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:32:16.758194] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:32:19.756974] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:32:19.758651] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:32:22.757432] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:32:22.759230] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:32:25.757822] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:32:25.760007] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:32:28.758195] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:32:28.761507] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:32:31.758621] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:32:31.761255] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:32:34.759033] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:32:34.761218] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:32:37.759446] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:32:37.763146] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:32:40.759861] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:32:40.762119] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:32:43.760303] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:32:43.762709] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:32:46.760705] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:32:46.763119] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:32:49.761099] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:32:49.763476] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:32:52.761486] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:32:52.764923] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:32:55.761916] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:32:55.764254] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:32:58.762322] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:32:58.764472] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:33:01.762723] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:33:01.765503] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
The message "I [MSGID: 106006] [glusterd-svc-mgmt.c:330:glusterd_svc_common_rpc_notify] 0-management: nfs has disconnected from glusterd." repeated 39 times between [2015-05-22 09:31:04.746583] and [2015-05-22 09:33:01.762779]
The message "I [MSGID: 106005] [glusterd-handler.c:4665:__glusterd_brick_rpc_notify] 0-management: Brick 10.70.35.144:/ssdbricks_75G_1/vol1 has disconnected from glusterd." repeated 39 times between [2015-05-22 09:31:04.751308] and [2015-05-22 09:33:01.767842]
[2015-05-22 09:33:04.763186] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:33:04.763221] I [MSGID: 106006] [glusterd-svc-mgmt.c:330:glusterd_svc_common_rpc_notify] 0-management: nfs has disconnected from glusterd.
[2015-05-22 09:33:04.765625] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[2015-05-22 09:33:04.767809] I [MSGID: 106005] [glusterd-handler.c:4665:__glusterd_brick_rpc_notify] 0-management: Brick 10.70.35.144:/ssdbricks_75G_1/vol1 has disconnected from glusterd.
[2015-05-22 09:33:07.763604] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument)
[2015-05-22 09:33:07.765846] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument)
[root@zod glusterfs]#
Comment 3 Triveni Rao 2015-06-11 17:29:40 UTC 
this bug is verified and found no issues.

[root@rhsqa14-vm1 ~]# gluster v detach-tier mix start
volume detach-tier start: success
ID: 5f08c911-0007-4fd5-b88f-f8ba6b3aefa2
[root@rhsqa14-vm1 ~]# 
[root@rhsqa14-vm1 ~]# gluster v detach-tier mix status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in s
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     -----------
                               localhost                0        0Bytes             0             0             0            completed               0
                            10.70.47.163                0        0Bytes             0             0             0            completed               0
[root@rhsqa14-vm1 ~]# gluster v detach-tier mix commit
volume detach-tier commit: success
Check the detached bricks to ensure all files are migrated.
If files with data are found on the brick path, copy them via a gluster mount point before re-purposing the removed brick. 
[root@rhsqa14-vm1 ~]#
Comment 4 Triveni Rao 2015-06-12 11:12:13 UTC 
build information was missing on which this bug was verified:

[root@rhsqa14-vm1 ~]# glusterfs --version
glusterfs 3.7.1 built on Jun  9 2015 02:31:54
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.
[root@rhsqa14-vm1 ~]# rpm -qa | grep gluster
glusterfs-3.7.1-1.el6rhs.x86_64
glusterfs-cli-3.7.1-1.el6rhs.x86_64
glusterfs-libs-3.7.1-1.el6rhs.x86_64
glusterfs-client-xlators-3.7.1-1.el6rhs.x86_64
glusterfs-fuse-3.7.1-1.el6rhs.x86_64
glusterfs-server-3.7.1-1.el6rhs.x86_64
glusterfs-api-3.7.1-1.el6rhs.x86_64
[root@rhsqa14-vm1 ~]#
Comment 7 errata-xmlrpc 2015-07-29 04:48:12 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-1495.html