Description of problem: ======================= Detach tier start command fails with Commit failed on localhost. error Version-Release number of selected component (if applicable): ======================= root@zod glusterfs]# gluster --version glusterfs 3.7.0 built on May 15 2015 01:33:40 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com> GlusterFS comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GlusterFS under the terms of the GNU General Public License. [root@zod glusterfs]# rpm -qa|grep gluster glusterfs-debuginfo-3.7.0-2.el7rhs.x86_64 glusterfs-geo-replication-3.7.0-2.el7rhs.x86_64 glusterfs-client-xlators-3.7.0-2.el7rhs.x86_64 glusterfs-cli-3.7.0-2.el7rhs.x86_64 glusterfs-libs-3.7.0-2.el7rhs.x86_64 glusterfs-api-3.7.0-2.el7rhs.x86_64 glusterfs-server-3.7.0-2.el7rhs.x86_64 glusterfs-resource-agents-3.7.0-2.el7rhs.noarch glusterfs-rdma-3.7.0-2.el7rhs.x86_64 glusterfs-devel-3.7.0-2.el7rhs.x86_64 glusterfs-api-devel-3.7.0-2.el7rhs.x86_64 glusterfs-3.7.0-2.el7rhs.x86_64 glusterfs-fuse-3.7.0-2.el7rhs.x86_64 Steps to Reproduce: ================== 1.create tiered vol and start it 2.do a detach-tier start 3.this fails bash-4.3$ ssh root@zod root@zod's password: Last login: Wed May 20 15:09:24 2015 from 10.10.51.217 [root@zod ~]# gluster v status Status of volume: vol1 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Hot Bricks: Brick yarrow:/ssdbricks_75G_1/vol1 49154 0 Y 22011 Brick 10.70.35.144:/ssdbricks_75G_1/vol1 49154 0 Y 2416 Brick 10.70.35.144:/brick_200G_1/vol1 49152 0 Y 2187 Brick yarrow:/brick_200G_1/vol1 49152 0 Y 21883 Brick 10.70.35.144:/brick_200G_2/vol1 49153 0 Y 2206 Brick yarrow:/brick_200G_2/vol1 49153 0 Y 21902 NFS Server on localhost N/A N/A N N/A NFS Server on yarrow N/A N/A N N/A Task Status of Volume vol1 ------------------------------------------------------------------------------ Task : Rebalance ID : 9dd4b419-4436-493f-9553-c39cc52a6582 Status : in progress Status of volume: vol2 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Hot Bricks: Brick 10.70.35.144:/ssdbricks_75G_1/vol2 49157 0 Y 5742 Brick yarrow:/ssdbricks_75G_1/vol2 49157 0 Y 25289 Brick 10.70.35.144:/brick_200G_1/vol2 49155 0 Y 5618 Brick yarrow:/brick_200G_1/vol2 49155 0 Y 25173 Brick 10.70.35.144:/brick_200G_2/vol2 49156 0 Y 5636 Brick yarrow:/brick_200G_2/vol2 49156 0 Y 25192 NFS Server on localhost N/A N/A N N/A NFS Server on yarrow N/A N/A N N/A Task Status of Volume vol2 ------------------------------------------------------------------------------ Task : Rebalance ID : aaea82ea-ddd5-4ae5-ad68-5343da3a29c8 Status : in progress [root@zod ~]# gluster v info Volume Name: vol1 Type: Tier Volume ID: 23dba7de-a94a-49c2-80f1-6d97b0ab1309 Status: Started Number of Bricks: 6 Transport-type: tcp Hot Tier : Hot Tier Type : Distribute Number of Bricks: 2 Brick1: yarrow:/ssdbricks_75G_1/vol1 Brick2: 10.70.35.144:/ssdbricks_75G_1/vol1 Cold Bricks: Cold Tier Type : Distributed-Replicate Number of Bricks: 2 x 2 = 4 Brick3: 10.70.35.144:/brick_200G_1/vol1 Brick4: yarrow:/brick_200G_1/vol1 Brick5: 10.70.35.144:/brick_200G_2/vol1 Brick6: yarrow:/brick_200G_2/vol1 Options Reconfigured: cluster.tier-demote-frequency: 10 features.record-counters: on features.ctr-enabled: on performance.readdir-ahead: on Volume Name: vol2 Type: Tier Volume ID: 858ae0b9-0cc9-41a9-b89b-d42e6791e2d7 Status: Started Number of Bricks: 6 Transport-type: tcp Hot Tier : Hot Tier Type : Distribute Number of Bricks: 2 Brick1: 10.70.35.144:/ssdbricks_75G_1/vol2 Brick2: yarrow:/ssdbricks_75G_1/vol2 Cold Bricks: Cold Tier Type : Distributed-Replicate Number of Bricks: 2 x 2 = 4 Brick3: 10.70.35.144:/brick_200G_1/vol2 Brick4: yarrow:/brick_200G_1/vol2 Brick5: 10.70.35.144:/brick_200G_2/vol2 Brick6: yarrow:/brick_200G_2/vol2 Options Reconfigured: performance.readdir-ahead: on [root@zod ~]# gluster v detach-tier Usage: volume detach-tier <VOLNAME> <start|stop|status|commit|[force]> [root@zod ~]# gluster v detach-tier vol2 Usage: volume detach-tier <VOLNAME> <start|stop|status|commit|[force]> [root@zod ~]# gluster v detach-tier vol2 start volume detach-tier start: failed: Commit failed on localhost. Please check the log file for more details. [root@zod ~]# gluster v info vol2 Volume Name: vol2 Type: Tier Volume ID: 858ae0b9-0cc9-41a9-b89b-d42e6791e2d7 Status: Started Number of Bricks: 6 Transport-type: tcp Hot Tier : Hot Tier Type : Distribute Number of Bricks: 2 Brick1: 10.70.35.144:/ssdbricks_75G_1/vol2 Brick2: yarrow:/ssdbricks_75G_1/vol2 Cold Bricks: Cold Tier Type : Distributed-Replicate Number of Bricks: 2 x 2 = 4 Brick3: 10.70.35.144:/brick_200G_1/vol2 Brick4: yarrow:/brick_200G_1/vol2 Brick5: 10.70.35.144:/brick_200G_2/vol2 Brick6: yarrow:/brick_200G_2/vol2 Options Reconfigured: performance.readdir-ahead: on [root@zod ~]# gluster v detach-tier vol2 start force volume detach-tier start: failed: An earlier remove-brick task exists for volume vol2. Either commit it or stop it before starting a new task. [root@zod ~]# gluster v info Volume Name: vol1 Type: Tier Volume ID: 23dba7de-a94a-49c2-80f1-6d97b0ab1309 Status: Started Number of Bricks: 6 Transport-type: tcp Hot Tier : Hot Tier Type : Distribute Number of Bricks: 2 Brick1: yarrow:/ssdbricks_75G_1/vol1 Brick2: 10.70.35.144:/ssdbricks_75G_1/vol1 Cold Bricks: Cold Tier Type : Distributed-Replicate Number of Bricks: 2 x 2 = 4 Brick3: 10.70.35.144:/brick_200G_1/vol1 Brick4: yarrow:/brick_200G_1/vol1 Brick5: 10.70.35.144:/brick_200G_2/vol1 Brick6: yarrow:/brick_200G_2/vol1 Options Reconfigured: cluster.tier-demote-frequency: 10 features.record-counters: on features.ctr-enabled: on performance.readdir-ahead: on Volume Name: vol2 Type: Tier Volume ID: 858ae0b9-0cc9-41a9-b89b-d42e6791e2d7 Status: Started Number of Bricks: 6 Transport-type: tcp Hot Tier : Hot Tier Type : Distribute Number of Bricks: 2 Brick1: 10.70.35.144:/ssdbricks_75G_1/vol2 Brick2: yarrow:/ssdbricks_75G_1/vol2 Cold Bricks: Cold Tier Type : Distributed-Replicate Number of Bricks: 2 x 2 = 4 Brick3: 10.70.35.144:/brick_200G_1/vol2 Brick4: yarrow:/brick_200G_1/vol2 Brick5: 10.70.35.144:/brick_200G_2/vol2 Brick6: yarrow:/brick_200G_2/vol2 Options Reconfigured: performance.readdir-ahead: on [root@zod ~]# gluster v info vol1 Volume Name: vol1 Type: Tier Volume ID: 23dba7de-a94a-49c2-80f1-6d97b0ab1309 Status: Started Number of Bricks: 6 Transport-type: tcp Hot Tier : Hot Tier Type : Distribute Number of Bricks: 2 Brick1: yarrow:/ssdbricks_75G_1/vol1 Brick2: 10.70.35.144:/ssdbricks_75G_1/vol1 Cold Bricks: Cold Tier Type : Distributed-Replicate Number of Bricks: 2 x 2 = 4 Brick3: 10.70.35.144:/brick_200G_1/vol1 Brick4: yarrow:/brick_200G_1/vol1 Brick5: 10.70.35.144:/brick_200G_2/vol1 Brick6: yarrow:/brick_200G_2/vol1 Options Reconfigured: cluster.tier-demote-frequency: 10 features.record-counters: on features.ctr-enabled: on performance.readdir-ahead: on [root@zod ~]# gluster v detach-tier vol1 start volume detach-tier start: failed: Commit failed on localhost. Please check the log file for more details. [root@zod ~]# cd /var/log/glusterfs/ [root@zod glusterfs]# ls bricks cmd_history.log etc-glusterfs-glusterd.vol.log geo-replication-slaves glustershd.log-20150521 nfs.log-20150521 vol1-rebalance.log cli.log cmd_history.log-20150521 geo-replication glustershd.log nfs.log snaps vol2-rebalance.log [root@zod glusterfs]# less etc-glusterfs-glusterd.vol.log [root@zod glusterfs]# gluster v--vers unrecognized word: v--vers (position 0) [root@zod glusterfs]# gluster --version glusterfs 3.7.0 built on May 15 2015 01:33:40 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com> GlusterFS comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of GlusterFS under the terms of the GNU General Public License. [root@zod glusterfs]# rpm -qa|grep gluster glusterfs-debuginfo-3.7.0-2.el7rhs.x86_64 glusterfs-geo-replication-3.7.0-2.el7rhs.x86_64 glusterfs-client-xlators-3.7.0-2.el7rhs.x86_64 glusterfs-cli-3.7.0-2.el7rhs.x86_64 glusterfs-libs-3.7.0-2.el7rhs.x86_64 glusterfs-api-3.7.0-2.el7rhs.x86_64 glusterfs-server-3.7.0-2.el7rhs.x86_64 glusterfs-resource-agents-3.7.0-2.el7rhs.noarch glusterfs-rdma-3.7.0-2.el7rhs.x86_64 glusterfs-devel-3.7.0-2.el7rhs.x86_64 glusterfs-api-devel-3.7.0-2.el7rhs.x86_64 glusterfs-3.7.0-2.el7rhs.x86_64 glusterfs-fuse-3.7.0-2.el7rhs.x86_64 [root@zod glusterfs]# bash-4.3$ ssh root@zod root@zod's password: Last login: Fri May 22 14:51:13 2015 from vpn-61-236.rdu2.redhat.com [root@zod ~]# gluster v status vol1 Status of volume: vol1 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Hot Bricks: Brick yarrow:/ssdbricks_75G_1/vol1 49154 0 Y 22011 Brick 10.70.35.144:/ssdbricks_75G_1/vol1 49154 0 Y 2416 Brick 10.70.35.144:/brick_200G_1/vol1 49152 0 Y 2187 Brick yarrow:/brick_200G_1/vol1 49152 0 Y 21883 Brick 10.70.35.144:/brick_200G_2/vol1 49153 0 Y 2206 Brick yarrow:/brick_200G_2/vol1 49153 0 Y 21902 NFS Server on localhost N/A N/A N N/A NFS Server on yarrow N/A N/A N N/A Task Status of Volume vol1 ------------------------------------------------------------------------------ Task : Remove brick ID : b41122e6-3707-4f23-bff1-b383ec2d1c6b Removed bricks: 10.70.35.144:/ssdbricks_75G_1/vol1 yarrow:/ssdbricks_75G_1/vol1 Status : completed [root@zod ~]# gluster v detach-tier vol1 status volume detach-tier unknown: success [root@zod ~]# gluster v unrecognized command [root@zod ~]# gluster gluster> ? unrecognized word: ? (position 0) gluster> ? unrecognized word: ? (position 0) gluster> helpo unrecognized word: helpo (position 0) gluster> help volume info [all|<VOLNAME>] - list information of all volumes volume create <NEW-VOLNAME> [stripe <COUNT>] [replica <COUNT> [arbiter <COUNT>]] [disperse [<COUNT>]] [disperse-data <COUNT>] [redundancy <COUNT>] [transport <tcp|rdma|tcp,rdma>] <NEW-BRICK>?<vg_name>... [force] - create a new volume of specified type with mentioned bricks volume delete <VOLNAME> - delete volume specified by <VOLNAME> volume start <VOLNAME> [force] - start volume specified by <VOLNAME> volume stop <VOLNAME> [force] - stop volume specified by <VOLNAME> volume attach-tier <VOLNAME> [<replica COUNT>] <NEW-BRICK>... - attach tier to volume <VOLNAME> volume detach-tier <VOLNAME> <start|stop|status|commit|[force]> - detach tier from volume <VOLNAME> volume add-brick <VOLNAME> [<stripe|replica> <COUNT>] <NEW-BRICK> ... [force] - add brick to volume <VOLNAME> volume remove-brick <VOLNAME> [replica <COUNT>] <BRICK> ... <start|stop|status|commit|force> - remove brick from volume <VOLNAME> volume rebalance <VOLNAME> {{fix-layout start} | {start [force]|stop|status}} - rebalance operations volume replace-brick <VOLNAME> <SOURCE-BRICK> <NEW-BRICK> {commit force} - replace-brick operations volume set <VOLNAME> <KEY> <VALUE> - set options for volume <VOLNAME> volume help - display help for the volume command volume log <VOLNAME> rotate [BRICK] - rotate the log file for corresponding volume/brick volume log rotate <VOLNAME> [BRICK] - rotate the log file for corresponding volume/brick NOTE: This is an old syntax, will be deprecated from next release. volume sync <HOSTNAME> [all|<VOLNAME>] - sync the volume information from a peer volume reset <VOLNAME> [option] [force] - reset all the reconfigured options volume geo-replication [<VOLNAME>] [<SLAVE-URL>] {create [[no-verify]|[push-pem]] [force]|start [force]|stop [force]|pause [force]|resume [force]|config|status [detail]|delete} [options...] - Geo-sync operations volume profile <VOLNAME> {start|info [peek|incremental [peek]|cumulative|clear]|stop} [nfs] - volume profile operations volume quota <VOLNAME> {enable|disable|list [<path> ...]| list-objects [<path> ...] | remove <path>| remove-objects <path> | default-soft-limit <percent>} | volume quota <VOLNAME> {limit-usage <path> <size> [<percent>]} | volume quota <VOLNAME> {limit-objects <path> <number> [<percent>]} | volume quota <VOLNAME> {alert-time|soft-timeout|hard-timeout} {<time>} - quota translator specific operations volume inode-quota <VOLNAME> enable - quota translator specific operations volume top <VOLNAME> {open|read|write|opendir|readdir|clear} [nfs|brick <brick>] [list-cnt <value>] | volume top <VOLNAME> {read-perf|write-perf} [bs <size> count <count>] [brick <brick>] [list-cnt <value>] - volume top operations volume status [all | <VOLNAME> [nfs|shd|<BRICK>|quotad]] [detail|clients|mem|inode|fd|callpool|tasks] - display status of all or specified volume(s)/brick volume heal <VOLNAME> [enable | disable | full |statistics [heal-count [replica <HOSTNAME:BRICKNAME>]] |info [healed | heal-failed | split-brain] |split-brain {bigger-file <FILE> |source-brick <HOSTNAME:BRICKNAME> [<FILE>]}] - self-heal commands on volume specified by <VOLNAME> volume statedump <VOLNAME> [nfs|quotad] [all|mem|iobuf|callpool|priv|fd|inode|history]... - perform statedump on bricks volume list - list all volumes in cluster volume clear-locks <VOLNAME> <path> kind {blocked|granted|all}{inode [range]|entry [basename]|posix [range]} - Clear locks held on path volume barrier <VOLNAME> {enable|disable} - Barrier/unbarrier file operations on a volume volume get <VOLNAME> <key|all> - Get the value of the all options or given option for volume <VOLNAME> volume bitrot <VOLNAME> {enable|disable} | volume bitrot <volname> scrub-throttle {lazy|normal|aggressive} | volume bitrot <volname> scrub-frequency {daily|weekly|biweekly|monthly} | volume bitrot <volname> scrub {pause|resume} - Bitrot translator specific operation. For more information about bitrot command type 'man gluster' peer probe <HOSTNAME> - probe peer specified by <HOSTNAME> peer detach <HOSTNAME> [force] - detach peer specified by <HOSTNAME> peer status - list status of peers peer help - Help command for peer pool list - list all the nodes in the pool (including localhost) quit - quit help - display command options exit - exit snapshot help - display help for snapshot commands snapshot create <snapname> <volname> [no-timestamp] [description <description>] [force] - Snapshot Create. snapshot clone <clonename> <snapname> - Snapshot Clone. snapshot restore <snapname> - Snapshot Restore. snapshot status [(snapname | volume <volname>)] - Snapshot Status. snapshot info [(snapname | volume <volname>)] - Snapshot Info. snapshot list [volname] - Snapshot List. snapshot config [volname] ([snap-max-hard-limit <count>] [snap-max-soft-limit <percent>]) | ([auto-delete <enable|disable>])| ([activate-on-create <enable|disable>]) - Snapshot Config. snapshot delete (all | snapname | volume <volname>) - Snapshot Delete. snapshot activate <snapname> [force] - Activate snapshot volume. snapshot deactivate <snapname> - Deactivate snapshot volume. global help - list global commands nfs-ganesha {enable| disable} - Enable/disable NFS-Ganesha support gluster> volume detach-tier status vol1 Usage: volume detach-tier <VOLNAME> <start|stop|status|commit|[force]> gluster> volume detach-tier vol1 status volume detach-tier unknown: success gluster>
[2015-05-22 09:25:16.695688] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:25:19.696025] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:25:22.696578] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:25:25.696955] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:25:28.697341] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:25:31.697721] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:25:34.698140] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:25:37.698539] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:25:40.698893] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:25:43.699259] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:25:46.699673] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:25:49.700046] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:25:52.700441] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:25:55.700855] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:25:58.701212] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:26:01.701612] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:26:04.701964] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:26:07.702345] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:26:10.702735] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:26:13.703112] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:26:16.703489] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:26:19.703877] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:26:22.704347] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:26:25.704732] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:26:28.705113] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:26:31.706102] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:26:32.434082] W [xlator.c:141:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/3.7.0/xlator/rpc-transport/socket.so: cannot open shared object file: No such file or directory [2015-05-22 09:26:32.434135] W [xlator.c:141:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/3.7.0/xlator/rpc-transport/socket.so: cannot open shared object file: No such file or directory [2015-05-22 09:26:32.479252] W [xlator.c:141:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/3.7.0/xlator/configuration.so: cannot open shared object file: No such file or directory [2015-05-22 09:26:32.479285] W [xlator.c:141:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/3.7.0/xlator/configuration.so: cannot open shared object file: No such file or directory [2015-05-22 09:26:32.480983] W [xlator.c:141:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/3.7.0/xlator/cluster/tier.so: undefined symbol: dht_methods [2015-05-22 09:26:32.481107] W [xlator.c:141:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/3.7.0/xlator/cluster/tier.so: undefined symbol: dht_methods [2015-05-22 09:26:32.481219] W [xlator.c:141:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/3.7.0/xlator/cluster/tier.so: undefined symbol: dht_methods [2015-05-22 09:26:32.481329] W [xlator.c:141:xlator_volopt_dynload] 0-xlator: /usr/lib64/glusterfs/3.7.0/xlator/cluster/tier.so: undefined symbol: dht_methods [2015-05-22 09:26:34.706874] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:26:37.707260] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:26:40.707671] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:26:43.708043] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:26:46.708413] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:26:49.708758] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:26:52.709126] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:26:53.734648] I [glusterd-handler.c:4057:__glusterd_handle_status_volume] 0-management: Received status volume req for volume vol1 [2015-05-22 09:26:53.743487] E [glusterd-utils.c:7478:glusterd_volume_status_aggregate_tasks_status] 0-management: Could not find matching local task for task 9dd4b419-4436-493f-9553-c39cc52a6582 [2015-05-22 09:26:55.709817] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:26:58.710155] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:27:01.710546] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) The message "I [MSGID: 106006] [glusterd-svc-mgmt.c:330:glusterd_svc_common_rpc_notify] 0-management: nfs has disconnected from glusterd." repeated 39 times between [2015-05-22 09:25:04.693905] and [2015-05-22 09:27:01.710593] [2015-05-22 09:27:04.711038] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:27:04.711077] I [MSGID: 106006] [glusterd-svc-mgmt.c:330:glusterd_svc_common_rpc_notify] 0-management: nfs has disconnected from glusterd. [2015-05-22 09:27:07.711424] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:27:10.711793] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:27:13.712176] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:27:16.712557] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:27:19.712935] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:27:22.713534] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:27:25.713875] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:27:28.714259] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:27:31.714646] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:27:34.715045] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:27:36.243989] I [glusterd-brick-ops.c:770:__glusterd_handle_remove_brick] 0-management: Received rem brick req [2015-05-22 09:27:36.403724] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick /ssdbricks_75G_1/vol1 on port 49154 [2015-05-22 09:27:36.405564] W [socket.c:642:__socket_rwv] 0-socket.management: writev on 10.70.35.144:1018 failed (Broken pipe) [2015-05-22 09:27:36.405601] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (No data available) [2015-05-22 09:27:36.407665] I [MSGID: 106005] [glusterd-handler.c:4665:__glusterd_brick_rpc_notify] 0-management: Brick 10.70.35.144:/ssdbricks_75G_1/vol1 has disconnected from glusterd. [2015-05-22 09:27:37.715796] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:27:37.718230] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:27:40.716224] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:27:40.718892] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:27:43.716653] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:27:43.719059] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:27:46.717095] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:27:46.719784] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:27:49.717808] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:27:49.720282] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:27:52.718225] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:27:52.721732] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:27:55.718777] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:27:55.721376] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:27:58.719166] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:27:58.721265] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:28:01.719566] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:28:01.721770] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:28:04.720009] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:28:04.722675] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:28:07.720424] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:28:07.722787] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:28:10.720857] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:28:10.723492] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:28:13.721270] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:28:13.722974] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:28:16.721687] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:28:16.724449] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:28:19.722093] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:28:19.723671] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:28:22.722705] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:28:22.724443] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:28:25.723216] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:28:25.725606] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:28:28.723662] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:28:28.725986] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:28:31.724103] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:28:31.726457] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:28:34.724539] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:28:34.726810] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:28:37.724954] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:28:37.727257] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:28:40.725346] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:28:40.727483] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:28:43.725742] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:28:43.727911] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:28:46.726176] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:28:46.728843] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:28:49.726599] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:28:49.728808] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:28:51.184035] I [glusterd-brick-ops.c:770:__glusterd_handle_remove_brick] 0-management: Received rem brick req [2015-05-22 09:28:51.195751] W [dict.c:584:dict_unref] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7fb8551a9f16] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_op_remove_brick+0x704)[0x7fb84a009504] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(glusterd_op_commit_perform+0x69b)[0x7fb849f996db] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(gd_commit_op_phase+0xb6)[0x7fb84a00e8e6] (--> /usr/lib64/glusterfs/3.7.0/xlator/mgmt/glusterd.so(gd_sync_task_begin+0x75a)[0x7fb84a00fd1a] ))))) 0-dict: dict is NULL [2015-05-22 09:28:52.727262] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:28:52.729994] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:28:55.727652] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:28:55.734901] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:28:58.728075] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:28:58.730347] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:29:01.728496] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:29:01.731055] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) The message "I [MSGID: 106006] [glusterd-svc-mgmt.c:330:glusterd_svc_common_rpc_notify] 0-management: nfs has disconnected from glusterd." repeated 39 times between [2015-05-22 09:27:04.711077] and [2015-05-22 09:29:01.728569] The message "I [MSGID: 106005] [glusterd-handler.c:4665:__glusterd_brick_rpc_notify] 0-management: Brick 10.70.35.144:/ssdbricks_75G_1/vol1 has disconnected from glusterd." repeated 29 times between [2015-05-22 09:27:36.407665] and [2015-05-22 09:29:01.733321] [2015-05-22 09:29:04.729066] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:29:04.729110] I [MSGID: 106006] [glusterd-svc-mgmt.c:330:glusterd_svc_common_rpc_notify] 0-management: nfs has disconnected from glusterd. [2015-05-22 09:29:04.731490] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:29:04.734506] I [MSGID: 106005] [glusterd-handler.c:4665:__glusterd_brick_rpc_notify] 0-management: Brick 10.70.35.144:/ssdbricks_75G_1/vol1 has disconnected from glusterd. [2015-05-22 09:29:07.729507] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:29:07.731859] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:29:10.729913] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:29:10.732275] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:29:13.730346] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:29:13.732266] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:29:16.730794] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:29:16.732457] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:29:19.731196] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:29:19.732929] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:29:22.731919] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:29:22.734103] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:29:25.732310] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:29:25.734729] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:29:28.732715] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:29:28.734735] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:29:31.733102] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:29:31.735467] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:29:34.733483] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:29:34.736214] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:29:37.733889] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:29:37.736323] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:29:40.734286] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:29:40.736588] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:29:43.734676] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:29:43.736752] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:29:46.735080] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:29:46.737538] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:29:49.735470] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:29:49.737638] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:29:52.735855] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:29:52.738283] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:29:55.736245] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:29:55.738455] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:29:58.736635] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:29:58.739074] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:30:01.737010] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:30:01.739649] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:30:04.737733] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:30:04.740346] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:30:07.738098] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:30:07.740633] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:30:10.738483] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:30:10.740453] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:30:13.738871] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:30:13.741138] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:30:16.739268] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:30:16.741882] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:30:19.739668] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:30:19.741569] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:30:22.740271] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:30:22.743253] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:30:25.740661] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:30:25.743451] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:30:28.741053] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:30:28.743354] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:30:31.741432] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:30:31.743932] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:30:34.741809] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:30:34.744140] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:30:37.742261] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:30:37.744847] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:30:40.742646] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:30:40.744350] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:30:43.743043] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:30:43.745725] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:30:46.743447] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:30:46.745778] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:30:49.743840] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:30:49.745506] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:30:52.744234] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:30:52.746945] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:30:55.744662] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:30:55.746985] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:30:58.745065] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:30:58.747572] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:31:01.745452] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:31:01.747978] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) The message "I [MSGID: 106006] [glusterd-svc-mgmt.c:330:glusterd_svc_common_rpc_notify] 0-management: nfs has disconnected from glusterd." repeated 39 times between [2015-05-22 09:29:04.729110] and [2015-05-22 09:31:01.745500] The message "I [MSGID: 106005] [glusterd-handler.c:4665:__glusterd_brick_rpc_notify] 0-management: Brick 10.70.35.144:/ssdbricks_75G_1/vol1 has disconnected from glusterd." repeated 39 times between [2015-05-22 09:29:04.734506] and [2015-05-22 09:31:01.750155] [2015-05-22 09:31:04.746504] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:31:04.746583] I [MSGID: 106006] [glusterd-svc-mgmt.c:330:glusterd_svc_common_rpc_notify] 0-management: nfs has disconnected from glusterd. [2015-05-22 09:31:04.749227] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:31:04.751308] I [MSGID: 106005] [glusterd-handler.c:4665:__glusterd_brick_rpc_notify] 0-management: Brick 10.70.35.144:/ssdbricks_75G_1/vol1 has disconnected from glusterd. [2015-05-22 09:31:07.747103] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:31:07.749879] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:31:10.747502] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:31:10.750133] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:31:13.747922] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:31:13.749553] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:31:16.748311] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:31:16.750602] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:31:19.748722] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:31:19.750313] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:31:22.749373] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:31:22.751232] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:31:25.749855] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:31:25.751928] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:31:28.750267] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:31:28.752220] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:31:31.750657] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:31:31.752608] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:31:34.751034] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:31:34.752691] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:31:37.751437] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:31:37.753732] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:31:40.751825] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:31:40.753394] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:31:43.752218] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:31:43.754020] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:31:46.752610] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:31:46.754352] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:31:49.752997] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:31:49.754797] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:31:52.753378] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:31:52.755611] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:31:55.753769] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:31:55.756510] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:31:58.754185] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:31:58.756576] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:32:01.754570] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:32:01.757295] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:32:04.754972] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:32:04.757133] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:32:07.755408] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:32:07.762486] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:32:10.755782] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:32:10.758595] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:32:13.756164] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:32:13.758007] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:32:16.756557] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:32:16.758194] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:32:19.756974] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:32:19.758651] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:32:22.757432] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:32:22.759230] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:32:25.757822] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:32:25.760007] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:32:28.758195] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:32:28.761507] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:32:31.758621] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:32:31.761255] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:32:34.759033] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:32:34.761218] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:32:37.759446] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:32:37.763146] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:32:40.759861] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:32:40.762119] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:32:43.760303] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:32:43.762709] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:32:46.760705] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:32:46.763119] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:32:49.761099] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:32:49.763476] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:32:52.761486] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:32:52.764923] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:32:55.761916] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:32:55.764254] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:32:58.762322] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:32:58.764472] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:33:01.762723] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:33:01.765503] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) The message "I [MSGID: 106006] [glusterd-svc-mgmt.c:330:glusterd_svc_common_rpc_notify] 0-management: nfs has disconnected from glusterd." repeated 39 times between [2015-05-22 09:31:04.746583] and [2015-05-22 09:33:01.762779] The message "I [MSGID: 106005] [glusterd-handler.c:4665:__glusterd_brick_rpc_notify] 0-management: Brick 10.70.35.144:/ssdbricks_75G_1/vol1 has disconnected from glusterd." repeated 39 times between [2015-05-22 09:31:04.751308] and [2015-05-22 09:33:01.767842] [2015-05-22 09:33:04.763186] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:33:04.763221] I [MSGID: 106006] [glusterd-svc-mgmt.c:330:glusterd_svc_common_rpc_notify] 0-management: nfs has disconnected from glusterd. [2015-05-22 09:33:04.765625] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [2015-05-22 09:33:04.767809] I [MSGID: 106005] [glusterd-handler.c:4665:__glusterd_brick_rpc_notify] 0-management: Brick 10.70.35.144:/ssdbricks_75G_1/vol1 has disconnected from glusterd. [2015-05-22 09:33:07.763604] W [socket.c:642:__socket_rwv] 0-nfs: readv on /var/run/gluster/973d40aebdb16d981666433bafdeba16.socket failed (Invalid argument) [2015-05-22 09:33:07.765846] W [socket.c:642:__socket_rwv] 0-management: readv on /var/run/gluster/719c583c33dfe7913794ae3c40ffc191.socket failed (Invalid argument) [root@zod glusterfs]#
this bug is verified and found no issues. [root@rhsqa14-vm1 ~]# gluster v detach-tier mix start volume detach-tier start: success ID: 5f08c911-0007-4fd5-b88f-f8ba6b3aefa2 [root@rhsqa14-vm1 ~]# [root@rhsqa14-vm1 ~]# gluster v detach-tier mix status Node Rebalanced-files size scanned failures skipped status run time in s --------- ----------- ----------- ----------- ----------- ----------- ------------ ----------- localhost 0 0Bytes 0 0 0 completed 0 10.70.47.163 0 0Bytes 0 0 0 completed 0 [root@rhsqa14-vm1 ~]# gluster v detach-tier mix commit volume detach-tier commit: success Check the detached bricks to ensure all files are migrated. If files with data are found on the brick path, copy them via a gluster mount point before re-purposing the removed brick. [root@rhsqa14-vm1 ~]#
build information was missing on which this bug was verified: [root@rhsqa14-vm1 ~]# glusterfs --version glusterfs 3.7.1 built on Jun 9 2015 02:31:54 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/> GlusterFS comes with ABSOLUTELY NO WARRANTY. It is licensed to you under your choice of the GNU Lesser General Public License, version 3 or any later version (LGPLv3 or later), or the GNU General Public License, version 2 (GPLv2), in all cases as published by the Free Software Foundation. [root@rhsqa14-vm1 ~]# rpm -qa | grep gluster glusterfs-3.7.1-1.el6rhs.x86_64 glusterfs-cli-3.7.1-1.el6rhs.x86_64 glusterfs-libs-3.7.1-1.el6rhs.x86_64 glusterfs-client-xlators-3.7.1-1.el6rhs.x86_64 glusterfs-fuse-3.7.1-1.el6rhs.x86_64 glusterfs-server-3.7.1-1.el6rhs.x86_64 glusterfs-api-3.7.1-1.el6rhs.x86_64 [root@rhsqa14-vm1 ~]#
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-1495.html