Bug 1205128

Summary: Disperse volume: "df -h" on a cifs mount throws IO error and no file systems processed message
Product: [Community] GlusterFS Reporter: Bhaskarakiran <byarlaga>
Component: disperseAssignee: Ashish Pandey <aspandey>
Status: CLOSED WORKSFORME QA Contact:
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: mainlineCC: annair, aspandey, bugs, byarlaga, gluster-bugs, mzywusko, pkarampu
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1224145 (view as bug list) Environment:
Last Closed: 2015-09-18 02:21:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1186580, 1224145    

Description Bhaskarakiran 2015-03-24 10:05:55 UTC 
Description of problem:
=======================

Configured samba on one of the server, mounted a 2x(8+3) disperse volume on linux using cifs. Ran IO operations for some time, like files and directory creates / deletes and did "df -h ." from one of the directory of the mount.

[root@dhcp37-61 truncation]# df -h .
df: `.': Input/output error
df: no file systems processed
[root@dhcp37-61 truncation]# 

[root@dhcp37-61 truncation]# mount
/dev/mapper/vg_dhcp3761-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")
/dev/vda1 on /boot type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
ninja:/dispersevol on /mnt/fuse1 type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)
\\\\vertigo\\gluster-testvol on /mnt/cifs type cifs (rw)

Version-Release number of selected component (if applicable):
=============================================================
[root@vertigo ~]# gluster --version
glusterfs 3.7dev built on Mar 17 2015 01:06:35
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.

How reproducible:
=================
100%

Steps to Reproduce:
1. Create a distributed disperse volume 2x(8+3)
2. configure samba server and mount on linux client using "mount -t cifs"
3. Run IO, create files / directories and do some deletes.
4. Run df -h . on a directory in the mount.

Actual results:
Input / Output error and no file systems processed errors.

Expected results:


Additional info:


Gluster v status and info :
===========================
[root@vertigo ~]# gluster v status testvol
Status of volume: testvol
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick vertigo:/rhs/brick1/testvol-1         49184     0          Y       28052
Brick ninja:/rhs/brick1/testvol-2           49183     0          Y       28545
Brick vertigo:/rhs/brick2/testvol-3         49185     0          Y       28067
Brick ninja:/rhs/brick2/testvol-4           49184     0          Y       28560
Brick vertigo:/rhs/brick3/testvol-5         49186     0          Y       28082
Brick ninja:/rhs/brick3/testvol-6           49185     0          Y       28575
Brick vertigo:/rhs/brick4/testvol-7         49187     0          Y       28097
Brick ninja:/rhs/brick4/testvol-8           49186     0          Y       3641 
Brick vertigo:/rhs/brick1/testvol-9         49188     0          Y       28112
Brick ninja:/rhs/brick1/testvol-10          49187     0          Y       3647 
Brick vertigo:/rhs/brick2/testvol-11        49189     0          Y       28127
Brick ninja:/rhs/brick2/testvol-12          49188     0          Y       3673 
Brick vertigo:/rhs/brick3/testvol-13        49190     0          Y       3684 
Brick ninja:/rhs/brick3/testvol-14          49189     0          Y       3678 
Brick vertigo:/rhs/brick4/testvol-15        49191     0          Y       3665 
Brick ninja:/rhs/brick4/testvol-16          49190     0          Y       3690 
Brick vertigo:/rhs/brick1/testvol-17        49192     0          Y       3710 
Brick ninja:/rhs/brick1/testvol-18          49191     0          Y       3689 
Brick vertigo:/rhs/brick2/testvol-19        49193     0          Y       3698 
Brick ninja:/rhs/brick2/testvol-20          49192     0          Y       3679 
Brick vertigo:/rhs/brick3/testvol-21        49194     0          Y       3716 
Brick ninja:/rhs/brick3/testvol-22          49193     0          Y       3691 
Snapshot Daemon on localhost                49195     0          Y       4041 
NFS Server on localhost                     2049      0          Y       29601
Quota Daemon on localhost                   N/A       N/A        N       N/A  
Snapshot Daemon on ninja                    49194     0          Y       4029 
NFS Server on ninja                         2049      0          Y       28591
Quota Daemon on ninja                       N/A       N/A        Y       28610
 
Task Status of Volume testvol
------------------------------------------------------------------------------
Task                 : Rebalance           
ID                   : b41c6131-5c3d-4335-a6f8-ce54cfdb720b
Status               : completed           
 
[root@vertigo ~]# gluster v info testvol
 
Volume Name: testvol
Type: Distributed-Disperse
Volume ID: 8d541198-bd77-4aaa-8b19-a29880cabea5
Status: Started
Number of Bricks: 2 x (8 + 3) = 22
Transport-type: tcp
Bricks:
Brick1: vertigo:/rhs/brick1/testvol-1
Brick2: ninja:/rhs/brick1/testvol-2
Brick3: vertigo:/rhs/brick2/testvol-3
Brick4: ninja:/rhs/brick2/testvol-4
Brick5: vertigo:/rhs/brick3/testvol-5
Brick6: ninja:/rhs/brick3/testvol-6
Brick7: vertigo:/rhs/brick4/testvol-7
Brick8: ninja:/rhs/brick4/testvol-8
Brick9: vertigo:/rhs/brick1/testvol-9
Brick10: ninja:/rhs/brick1/testvol-10
Brick11: vertigo:/rhs/brick2/testvol-11
Brick12: ninja:/rhs/brick2/testvol-12
Brick13: vertigo:/rhs/brick3/testvol-13
Brick14: ninja:/rhs/brick3/testvol-14
Brick15: vertigo:/rhs/brick4/testvol-15
Brick16: ninja:/rhs/brick4/testvol-16
Brick17: vertigo:/rhs/brick1/testvol-17
Brick18: ninja:/rhs/brick1/testvol-18
Brick19: vertigo:/rhs/brick2/testvol-19
Brick20: ninja:/rhs/brick2/testvol-20
Brick21: vertigo:/rhs/brick3/testvol-21
Brick22: ninja:/rhs/brick3/testvol-22
Options Reconfigured:
cluster.disperse-self-heal-daemon: enable
client.event-threads: 4
server.event-threads: 4
features.quota: on
features.uss: on

Log snippet from server :
========================
[2015-03-24 09:44:54.968748] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2015-03-24 09:44:54.987699] I [graph.c:269:gf_add_cmdline_options] 0-testvol-md-cache: adding option 'cache-posix-acl' for volume 'testvol-md-cache' with value 'true'
[2015-03-24 09:44:54.988553] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2015-03-24 09:44:54.988641] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 3
[2015-03-24 09:44:54.988695] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 4
[2015-03-24 09:44:54.988713] I [client.c:2391:client_check_remote_host] 0-testvol-snapd-client: Remote host is not set. Assuming the volfile server as remote host.
[2015-03-24 09:44:54.989665] I [dht-shared.c:337:dht_init_regex] 0-testvol-dht: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
[2015-03-24 09:44:55.006666] I [glfs-master.c:93:notify] 0-gfapi: New graph 76657274-6967-6f2e-6c61-622e656e672e (0) coming up
[2015-03-24 09:44:55.006709] I [client.c:2350:notify] 0-testvol-client-0: parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.013811] I [client.c:2350:notify] 0-testvol-client-1: parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.014208] I [rpc-clnt.c:1806:rpc_clnt_reconfig] 0-testvol-client-0: changing port to 49184 (from 0)
[2015-03-24 09:44:55.019587] I [client.c:2350:notify] 0-testvol-client-2: parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.020014] I [rpc-clnt.c:1806:rpc_clnt_reconfig] 0-testvol-client-1: changing port to 49183 (from 0)
[2015-03-24 09:44:55.022412] I [client-handshake.c:1412:select_server_supported_programs] 0-testvol-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-03-24 09:44:55.022684] I [client-handshake.c:1200:client_setvolume_cbk] 0-testvol-client-0: Connected to testvol-client-0, attached to remote volume '/rhs/brick1/testvol-1'.
[2015-03-24 09:44:55.022713] I [client-handshake.c:1210:client_setvolume_cbk] 0-testvol-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2015-03-24 09:44:55.022845] I [client-handshake.c:187:client_set_lk_version_cbk] 0-testvol-client-0: Server lk version = 1
[2015-03-24 09:44:55.024963] I [client.c:2350:notify] 0-testvol-client-3: parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.025196] I [rpc-clnt.c:1806:rpc_clnt_reconfig] 0-testvol-client-2: changing port to 49185 (from 0)
[2015-03-24 09:44:55.028642] I [client-handshake.c:1412:select_server_supported_programs] 0-testvol-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-03-24 09:44:55.029376] I [client-handshake.c:1200:client_setvolume_cbk] 0-testvol-client-1: Connected to testvol-client-1, attached to remote volume '/rhs/brick1/testvol-2'.
[2015-03-24 09:44:55.029417] I [client-handshake.c:1210:client_setvolume_cbk] 0-testvol-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2015-03-24 09:44:55.029592] I [client-handshake.c:187:client_set_lk_version_cbk] 0-testvol-client-1: Server lk version = 1
[2015-03-24 09:44:55.031706] I [client.c:2350:notify] 0-testvol-client-4: parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.032175] I [rpc-clnt.c:1806:rpc_clnt_reconfig] 0-testvol-client-3: changing port to 49184 (from 0)
[2015-03-24 09:44:55.035424] I [client-handshake.c:1412:select_server_supported_programs] 0-testvol-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-03-24 09:44:55.035735] I [client-handshake.c:1200:client_setvolume_cbk] 0-testvol-client-2: Connected to testvol-client-2, attached to remote volume '/rhs/brick2/testvol-3'.
[2015-03-24 09:44:55.035764] I [client-handshake.c:1210:client_setvolume_cbk] 0-testvol-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2015-03-24 09:44:55.035872] I [client-handshake.c:187:client_set_lk_version_cbk] 0-testvol-client-2: Server lk version = 1
[2015-03-24 09:44:55.038987] I [client.c:2350:notify] 0-testvol-client-5: parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.039275] I [rpc-clnt.c:1806:rpc_clnt_reconfig] 0-testvol-client-4: changing port to 49186 (from 0)
[2015-03-24 09:44:55.044907] I [client-handshake.c:1412:select_server_supported_programs] 0-testvol-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-03-24 09:44:55.045297] I [client-handshake.c:1200:client_setvolume_cbk] 0-testvol-client-3: Connected to testvol-client-3, attached to remote volume '/rhs/brick2/testvol-4'.
[2015-03-24 09:44:55.045326] I [client-handshake.c:1210:client_setvolume_cbk] 0-testvol-client-3: Server and Client lk-version numbers are not same, reopening the fds
[2015-03-24 09:44:55.045485] I [client-handshake.c:187:client_set_lk_version_cbk] 0-testvol-client-3: Server lk version = 1
[2015-03-24 09:44:55.048855] I [client.c:2350:notify] 0-testvol-client-6: parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.049145] W [socket.c:636:__socket_rwv] 0-testvol-client-5: readv on 10.70.34.68:24007 failed (No data available)
[2015-03-24 09:44:55.049849] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] ))))) 0-testvol-client-5: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2015-03-24 09:44:55.049014 (xid=0x1)
[2015-03-24 09:44:55.049888] W [client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-5: received RPC status error
[2015-03-24 09:44:55.049905] I [client.c:2282:client_rpc_notify] 0-testvol-client-5: disconnected from testvol-client-5. Client process will keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.051424] W [socket.c:636:__socket_rwv] 0-testvol-client-4: readv on 10.70.34.56:49186 failed (No data available)
[2015-03-24 09:44:55.051640] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] ))))) 0-testvol-client-4: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2015-03-24 09:44:55.051347 (xid=0x3)
[2015-03-24 09:44:55.051666] W [client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-4: received RPC status error
[2015-03-24 09:44:55.051676] I [client.c:2282:client_rpc_notify] 0-testvol-client-4: disconnected from testvol-client-4. Client process will keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.056473] I [client.c:2350:notify] 0-testvol-client-7: parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.056658] W [socket.c:636:__socket_rwv] 0-testvol-client-6: readv on 10.70.34.56:24007 failed (No data available)
[2015-03-24 09:44:55.056914] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] ))))) 0-testvol-client-6: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2015-03-24 09:44:55.056576 (xid=0x1)
[2015-03-24 09:44:55.056953] W [client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-6: received RPC status error
[2015-03-24 09:44:55.056971] I [client.c:2282:client_rpc_notify] 0-testvol-client-6: disconnected from testvol-client-6. Client process will keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.061408] I [client.c:2350:notify] 0-testvol-client-8: parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.061738] W [socket.c:636:__socket_rwv] 0-testvol-client-7: readv on 10.70.34.68:24007 failed (No data available)
[2015-03-24 09:44:55.061882] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] ))))) 0-testvol-client-7: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2015-03-24 09:44:55.061523 (xid=0x1)
[2015-03-24 09:44:55.061901] W [client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-7: received RPC status error
[2015-03-24 09:44:55.061912] I [client.c:2282:client_rpc_notify] 0-testvol-client-7: disconnected from testvol-client-7. Client process will keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.065247] I [client.c:2350:notify] 0-testvol-client-9: parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.065372] W [socket.c:636:__socket_rwv] 0-testvol-client-8: readv on 10.70.34.56:24007 failed (No data available)
[2015-03-24 09:44:55.065504] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] ))))) 0-testvol-client-8: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2015-03-24 09:44:55.065278 (xid=0x1)
[2015-03-24 09:44:55.065529] W [client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-8: received RPC status error
[2015-03-24 09:44:55.065544] I [client.c:2282:client_rpc_notify] 0-testvol-client-8: disconnected from testvol-client-8. Client process will keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.068921] I [client.c:2350:notify] 0-testvol-client-10: parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.069235] W [socket.c:636:__socket_rwv] 0-testvol-client-9: readv on 10.70.34.68:24007 failed (No data available)
[2015-03-24 09:44:55.069459] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] ))))) 0-testvol-client-9: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2015-03-24 09:44:55.069091 (xid=0x1)
[2015-03-24 09:44:55.069491] W [client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-9: received RPC status error
[2015-03-24 09:44:55.069507] I [client.c:2282:client_rpc_notify] 0-testvol-client-9: disconnected from testvol-client-9. Client process will keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.072761] I [client.c:2350:notify] 0-testvol-client-11: parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.072899] W [socket.c:636:__socket_rwv] 0-testvol-client-10: readv on 10.70.34.56:24007 failed (No data available)
[2015-03-24 09:44:55.073153] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] ))))) 0-testvol-client-10: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2015-03-24 09:44:55.072831 (xid=0x1)
[2015-03-24 09:44:55.073186] W [client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-10: received RPC status error
[2015-03-24 09:44:55.073202] I [client.c:2282:client_rpc_notify] 0-testvol-client-10: disconnected from testvol-client-10. Client process will keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.077104] I [client.c:2350:notify] 0-testvol-client-12: parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.077423] W [socket.c:636:__socket_rwv] 0-testvol-client-11: readv on 10.70.34.68:24007 failed (No data available)
[2015-03-24 09:44:55.077645] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] ))))) 0-testvol-client-11: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2015-03-24 09:44:55.077253 (xid=0x1)
[2015-03-24 09:44:55.077676] W [client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-11: received RPC status error
[2015-03-24 09:44:55.077694] I [client.c:2282:client_rpc_notify] 0-testvol-client-11: disconnected from testvol-client-11. Client process will keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.080986] I [client.c:2350:notify] 0-testvol-client-13: parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.081115] W [socket.c:636:__socket_rwv] 0-testvol-client-12: readv on 10.70.34.56:24007 failed (No data available)
[2015-03-24 09:44:55.081354] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] ))))) 0-testvol-client-12: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2015-03-24 09:44:55.081041 (xid=0x1)
[2015-03-24 09:44:55.081386] W [client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-12: received RPC status error
[2015-03-24 09:44:55.081408] I [client.c:2282:client_rpc_notify] 0-testvol-client-12: disconnected from testvol-client-12. Client process will keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.085114] I [client.c:2350:notify] 0-testvol-client-14: parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.085351] W [socket.c:636:__socket_rwv] 0-testvol-client-13: readv on 10.70.34.68:24007 failed (No data available)
[2015-03-24 09:44:55.085596] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] ))))) 0-testvol-client-13: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2015-03-24 09:44:55.085220 (xid=0x1)
[2015-03-24 09:44:55.085627] W [client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-13: received RPC status error
[2015-03-24 09:44:55.085645] I [client.c:2282:client_rpc_notify] 0-testvol-client-13: disconnected from testvol-client-13. Client process will keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.089322] I [client.c:2350:notify] 0-testvol-client-15: parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.089477] W [socket.c:636:__socket_rwv] 0-testvol-client-14: readv on 10.70.34.56:24007 failed (No data available)
[2015-03-24 09:44:55.089700] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] ))))) 0-testvol-client-14: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2015-03-24 09:44:55.089415 (xid=0x1)
[2015-03-24 09:44:55.089765] W [client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-14: received RPC status error
[2015-03-24 09:44:55.089793] I [client.c:2282:client_rpc_notify] 0-testvol-client-14: disconnected from testvol-client-14. Client process will keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.093698] I [client.c:2350:notify] 0-testvol-client-16: parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.094060] W [socket.c:636:__socket_rwv] 0-testvol-client-15: readv on 10.70.34.68:24007 failed (No data available)
[2015-03-24 09:44:55.094314] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] ))))) 0-testvol-client-15: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2015-03-24 09:44:55.093817 (xid=0x1)
[2015-03-24 09:44:55.094346] W [client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-15: received RPC status error
[2015-03-24 09:44:55.094363] I [client.c:2282:client_rpc_notify] 0-testvol-client-15: disconnected from testvol-client-15. Client process will keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.098442] I [client.c:2350:notify] 0-testvol-client-17: parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.098636] W [socket.c:636:__socket_rwv] 0-testvol-client-16: readv on 10.70.34.56:24007 failed (No data available)
[2015-03-24 09:44:55.098799] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] ))))) 0-testvol-client-16: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2015-03-24 09:44:55.098517 (xid=0x1)
[2015-03-24 09:44:55.098821] W [client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-16: received RPC status error
[2015-03-24 09:44:55.098832] I [client.c:2282:client_rpc_notify] 0-testvol-client-16: disconnected from testvol-client-16. Client process will keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.101610] E [snapview-client.c:309:svc_lookup_cbk] 0-testvol-snapview-client: Lookup failed on normal graph with error Transport endpoint is not connected
[2015-03-24 09:44:55.101666] I [glfs-resolve.c:795:__glfs_active_subvol] 0-testvol: first lookup on graph 76657274-6967-6f2e-6c61-622e656e672e (0) failed (Transport endpoint is not connected)
[2015-03-24 09:44:55.103740] I [client.c:2350:notify] 0-testvol-client-18: parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.104011] W [socket.c:636:__socket_rwv] 0-testvol-client-17: readv on 10.70.34.68:24007 failed (No data available)
[2015-03-24 09:44:55.104319] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] ))))) 0-testvol-client-17: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2015-03-24 09:44:55.103840 (xid=0x1)
[2015-03-24 09:44:55.104364] W [client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-17: received RPC status error
[2015-03-24 09:44:55.104389] I [client.c:2282:client_rpc_notify] 0-testvol-client-17: disconnected from testvol-client-17. Client process will keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.108992] I [client.c:2350:notify] 0-testvol-client-19: parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.109205] W [socket.c:636:__socket_rwv] 0-testvol-client-18: readv on 10.70.34.56:24007 failed (No data available)
[2015-03-24 09:44:55.109446] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] ))))) 0-testvol-client-18: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2015-03-24 09:44:55.109072 (xid=0x1)
[2015-03-24 09:44:55.109477] W [client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-18: received RPC status error
[2015-03-24 09:44:55.109493] I [client.c:2282:client_rpc_notify] 0-testvol-client-18: disconnected from testvol-client-18. Client process will keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.114609] I [client.c:2350:notify] 0-testvol-client-20: parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.114929] W [socket.c:636:__socket_rwv] 0-testvol-client-19: readv on 10.70.34.68:24007 failed (No data available)
[2015-03-24 09:44:55.115259] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] ))))) 0-testvol-client-19: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2015-03-24 09:44:55.114780 (xid=0x1)
[2015-03-24 09:44:55.115312] W [client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-19: received RPC status error
[2015-03-24 09:44:55.115342] I [client.c:2282:client_rpc_notify] 0-testvol-client-19: disconnected from testvol-client-19. Client process will keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.119475] I [client.c:2350:notify] 0-testvol-client-21: parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.119644] W [socket.c:636:__socket_rwv] 0-testvol-client-20: readv on 10.70.34.56:24007 failed (No data available)
[2015-03-24 09:44:55.119901] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] ))))) 0-testvol-client-20: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2015-03-24 09:44:55.119547 (xid=0x1)
[2015-03-24 09:44:55.119963] W [client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-20: received RPC status error
[2015-03-24 09:44:55.119989] I [client.c:2282:client_rpc_notify] 0-testvol-client-20: disconnected from testvol-client-20. Client process will keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.124589] I [client.c:2350:notify] 0-testvol-snapd-client: parent translators are ready, attempting connect on transport
[2015-03-24 09:44:55.124880] W [socket.c:636:__socket_rwv] 0-testvol-client-21: readv on 10.70.34.68:24007 failed (No data available)
[2015-03-24 09:44:55.125169] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] ))))) 0-testvol-client-21: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2015-03-24 09:44:55.124728 (xid=0x1)
[2015-03-24 09:44:55.125204] W [client-handshake.c:1587:client_dump_version_cbk] 0-testvol-client-21: received RPC status error
[2015-03-24 09:44:55.125221] I [client.c:2282:client_rpc_notify] 0-testvol-client-21: disconnected from testvol-client-21. Client process will keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.131850] W [socket.c:636:__socket_rwv] 0-testvol-snapd-client: readv on 127.0.0.1:24007 failed (No data available)
[2015-03-24 09:44:55.132118] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] ))))) 0-testvol-snapd-client: forced unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2015-03-24 09:44:55.131786 (xid=0x1)
[2015-03-24 09:44:55.132149] W [client-handshake.c:1587:client_dump_version_cbk] 0-testvol-snapd-client: received RPC status error
[2015-03-24 09:44:55.132166] I [client.c:2282:client_rpc_notify] 0-testvol-snapd-client: disconnected from testvol-snapd-client. Client process will keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.201978] E [snapview-client.c:309:svc_lookup_cbk] 0-testvol-snapview-client: Lookup failed on normal graph with error Transport endpoint is not connected
[2015-03-24 09:44:55.202040] I [glfs-resolve.c:795:__glfs_active_subvol] 0-testvol: first lookup on graph 76657274-6967-6f2e-6c61-622e656e672e (0) failed (Transport endpoint is not connected)
[2015-03-24 09:44:55.202279] I [mem-pool.c:557:mem_pool_destroy] 0-gfapi: size=84 max=0 total=0
[2015-03-24 09:44:55.202863] I [mem-pool.c:557:mem_pool_destroy] 0-gfapi: size=156 max=1 total=1
[2015-03-24 09:44:55.203221] I [mem-pool.c:557:mem_pool_destroy] 0-gfapi: size=108 max=0 total=0
[2015-03-24 09:44:55.203280] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-0: size=1300 max=0 total=0
[2015-03-24 09:44:55.203299] I [client.c:2282:client_rpc_notify] 0-testvol-client-0: disconnected from testvol-client-0. Client process will keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.203322] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-1: size=1300 max=0 total=0
[2015-03-24 09:44:55.203335] I [client.c:2282:client_rpc_notify] 0-testvol-client-1: disconnected from testvol-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.203383] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-2: size=1300 max=0 total=0
[2015-03-24 09:44:55.203386] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-1: size=588 max=3 total=6
[2015-03-24 09:44:55.203401] I [client.c:2282:client_rpc_notify] 0-testvol-client-2: disconnected from testvol-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.203422] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-3: size=1300 max=0 total=0
[2015-03-24 09:44:55.203457] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-2: size=588 max=3 total=6
[2015-03-24 09:44:55.203467] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-1: size=124 max=3 total=6
[2015-03-24 09:44:55.203442] I [client.c:2282:client_rpc_notify] 0-testvol-client-3: disconnected from testvol-client-3. Client process will keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:55.203477] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-4: size=588 max=2 total=3
[2015-03-24 09:44:55.203516] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-2: size=124 max=3 total=6
[2015-03-24 09:44:55.203352] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-0: size=588 max=3 total=6
[2015-03-24 09:44:55.203543] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-3: size=588 max=3 total=6
[2015-03-24 09:44:55.203588] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-4: size=124 max=2 total=3
[2015-03-24 09:44:55.203637] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-4: size=1300 max=0 total=0
[2015-03-24 09:44:55.203654] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-0: size=124 max=3 total=6
[2015-03-24 09:44:55.203661] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-5: size=588 max=1 total=1
[2015-03-24 09:44:55.203746] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-3: size=124 max=3 total=6
[2015-03-24 09:44:55.203789] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-5: size=124 max=1 total=1
[2015-03-24 09:44:55.203813] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-5: size=1300 max=0 total=0
[2015-03-24 09:44:55.203838] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-6: size=588 max=1 total=1
[2015-03-24 09:44:55.203897] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-6: size=124 max=1 total=1
[2015-03-24 09:44:55.203917] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-6: size=1300 max=0 total=0
[2015-03-24 09:44:55.203937] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-7: size=588 max=1 total=1
[2015-03-24 09:44:55.204001] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-7: size=124 max=1 total=1
[2015-03-24 09:44:55.204025] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-7: size=1300 max=0 total=0
[2015-03-24 09:44:55.204045] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-8: size=588 max=1 total=1
[2015-03-24 09:44:55.204108] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-8: size=124 max=1 total=1
[2015-03-24 09:44:55.204134] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-8: size=1300 max=0 total=0
[2015-03-24 09:44:55.204153] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-9: size=588 max=1 total=1
[2015-03-24 09:44:55.204212] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-9: size=124 max=1 total=1
[2015-03-24 09:44:55.204233] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-9: size=1300 max=0 total=0
[2015-03-24 09:44:55.204251] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-10: size=588 max=1 total=1
[2015-03-24 09:44:55.204305] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-10: size=124 max=1 total=1
[2015-03-24 09:44:55.204325] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-10: size=1300 max=0 total=0
[2015-03-24 09:44:57.204410] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-disperse-0: size=1764 max=1 total=2
[2015-03-24 09:44:57.204705] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-disperse-0: size=1932 max=0 total=0
[2015-03-24 09:44:57.204964] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-disperse-0: size=1260 max=0 total=0
[2015-03-24 09:44:57.205162] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-11: size=588 max=1 total=1
[2015-03-24 09:44:57.205228] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-11: size=124 max=1 total=1
[2015-03-24 09:44:57.205257] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-11: size=1300 max=0 total=0
[2015-03-24 09:44:57.205277] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-12: size=588 max=1 total=1
[2015-03-24 09:44:57.205336] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-12: size=124 max=1 total=1
[2015-03-24 09:44:57.205357] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-12: size=1300 max=0 total=0
[2015-03-24 09:44:57.205375] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-13: size=588 max=1 total=1
[2015-03-24 09:44:57.205432] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-13: size=124 max=1 total=1
[2015-03-24 09:44:57.205452] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-13: size=1300 max=0 total=0
[2015-03-24 09:44:57.205469] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-14: size=588 max=1 total=1
[2015-03-24 09:44:57.205525] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-14: size=124 max=1 total=1
[2015-03-24 09:44:57.205545] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-14: size=1300 max=0 total=0
[2015-03-24 09:44:57.205564] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-15: size=588 max=1 total=1
[2015-03-24 09:44:57.205619] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-15: size=124 max=1 total=1
[2015-03-24 09:44:57.205639] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-15: size=1300 max=0 total=0
[2015-03-24 09:44:57.205658] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-16: size=588 max=1 total=1
[2015-03-24 09:44:57.205717] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-16: size=124 max=1 total=1
[2015-03-24 09:44:57.205736] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-16: size=1300 max=0 total=0
[2015-03-24 09:44:57.205754] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-17: size=588 max=1 total=1
[2015-03-24 09:44:57.205810] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-17: size=124 max=1 total=1
[2015-03-24 09:44:57.205835] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-17: size=1300 max=0 total=0
[2015-03-24 09:44:57.205862] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-18: size=588 max=1 total=1
[2015-03-24 09:44:57.205928] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-18: size=124 max=1 total=1
[2015-03-24 09:44:57.205949] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-18: size=1300 max=0 total=0
[2015-03-24 09:44:57.205967] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-19: size=588 max=1 total=1
[2015-03-24 09:44:57.206026] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-19: size=124 max=1 total=1
[2015-03-24 09:44:57.206062] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-19: size=1300 max=0 total=0
[2015-03-24 09:44:57.206081] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-20: size=588 max=1 total=1
[2015-03-24 09:44:57.206142] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-20: size=124 max=1 total=1
[2015-03-24 09:44:57.206163] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-20: size=1300 max=0 total=0
[2015-03-24 09:44:57.206186] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-21: size=588 max=1 total=1
[2015-03-24 09:44:57.206249] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-21: size=124 max=1 total=1
[2015-03-24 09:44:57.206270] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-client-21: size=1300 max=0 total=0
[2015-03-24 09:44:58.973320] I [timer.c:44:gf_timer_call_after] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (--> /usr/lib64/libglusterfs.so.0(gf_timer_call_after+0x1a9)[0x7ff40bc01f79] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_reconnect+0x107)[0x7ff40c08c5f7] (--> /usr/lib64/libglusterfs.so.0(gf_timer_proc+0xc8)[0x7ff40bc01b98] (--> /lib64/libpthread.so.0(+0x3d2fc079d1)[0x7ff4223909d1] ))))) 0-timer: ctx cleanup started
[2015-03-24 09:44:58.973694] I [timer.c:44:gf_timer_call_after] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (--> /usr/lib64/libglusterfs.so.0(gf_timer_call_after+0x1a9)[0x7ff40bc01f79] (--> /usr/lib64/libgfrpc.so.0(__save_frame+0x6e)[0x7ff40c08c7ae] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x3e2)[0x7ff40c08d102] (--> /usr/lib64/glusterfs/3.7dev/xlator/protocol/client.so(client_submit_request+0x196)[0x7ff402761e26] ))))) 0-timer: ctx cleanup started
[2015-03-24 09:44:58.974020] I [timer.c:44:gf_timer_call_after] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (--> /usr/lib64/libglusterfs.so.0(gf_timer_call_after+0x1a9)[0x7ff40bc01f79] (--> /usr/lib64/libgfrpc.so.0(__save_frame+0x6e)[0x7ff40c08c7ae] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x3e2)[0x7ff40c08d102] (--> /usr/lib64/glusterfs/3.7dev/xlator/protocol/client.so(client_submit_request+0x196)[0x7ff402761e26] ))))) 0-timer: ctx cleanup started
[2015-03-24 09:44:58.974078] I [rpc-clnt.c:1806:rpc_clnt_reconfig] 0-testvol-snapd-client: changing port to 49195 (from 0)
[2015-03-24 09:44:58.974425] I [timer.c:44:gf_timer_call_after] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (--> /usr/lib64/libglusterfs.so.0(gf_timer_call_after+0x1a9)[0x7ff40bc01f79] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x1e7)[0x7ff40c08e157] (--> /usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x28)[0x7ff40c0897a8] (--> /usr/lib64/glusterfs/3.7dev/rpc-transport/socket.so(+0xb221)[0x7ff4080de221] ))))) 0-timer: ctx cleanup started
[2015-03-24 09:44:58.984287] I [timer.c:44:gf_timer_call_after] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (--> /usr/lib64/libglusterfs.so.0(gf_timer_call_after+0x1a9)[0x7ff40bc01f79] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_reconnect+0x107)[0x7ff40c08c5f7] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_start+0x17)[0x7ff40c08c617] (--> /usr/lib64/glusterfs/3.7dev/xlator/protocol/client.so(client_rpc_notify+0x3d2)[0x7ff4027628a2] ))))) 0-timer: ctx cleanup started
[2015-03-24 09:44:58.984588] I [timer.c:44:gf_timer_call_after] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (--> /usr/lib64/libglusterfs.so.0(gf_timer_call_after+0x1a9)[0x7ff40bc01f79] (--> /usr/lib64/libgfrpc.so.0(__save_frame+0x6e)[0x7ff40c08c7ae] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x3e2)[0x7ff40c08d102] (--> /usr/lib64/glusterfs/3.7dev/xlator/protocol/client.so(client_submit_request+0x196)[0x7ff402761e26] ))))) 0-timer: ctx cleanup started
[2015-03-24 09:44:58.984650] I [client-handshake.c:1412:select_server_supported_programs] 0-testvol-snapd-client: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-03-24 09:44:58.984957] I [timer.c:44:gf_timer_call_after] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (--> /usr/lib64/libglusterfs.so.0(gf_timer_call_after+0x1a9)[0x7ff40bc01f79] (--> /usr/lib64/libgfrpc.so.0(__save_frame+0x6e)[0x7ff40c08c7ae] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x3e2)[0x7ff40c08d102] (--> /usr/lib64/glusterfs/3.7dev/xlator/protocol/client.so(client_submit_request+0x196)[0x7ff402761e26] ))))) 0-timer: ctx cleanup started
[2015-03-24 09:44:58.985029] I [client-handshake.c:1200:client_setvolume_cbk] 0-testvol-snapd-client: Connected to testvol-snapd-client, attached to remote volume 'snapd-testvol'.
[2015-03-24 09:44:58.985056] I [client-handshake.c:1210:client_setvolume_cbk] 0-testvol-snapd-client: Server and Client lk-version numbers are not same, reopening the fds
[2015-03-24 09:44:58.985313] I [timer.c:44:gf_timer_call_after] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (--> /usr/lib64/libglusterfs.so.0(gf_timer_call_after+0x1a9)[0x7ff40bc01f79] (--> /usr/lib64/libgfrpc.so.0(__save_frame+0x6e)[0x7ff40c08c7ae] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x3e2)[0x7ff40c08d102] (--> /usr/lib64/glusterfs/3.7dev/xlator/protocol/client.so(client_submit_request+0x196)[0x7ff402761e26] ))))) 0-timer: ctx cleanup started
[2015-03-24 09:44:58.985527] I [timer.c:44:gf_timer_call_after] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (--> /usr/lib64/libglusterfs.so.0(gf_timer_call_after+0x1a9)[0x7ff40bc01f79] (--> /usr/lib64/libgfrpc.so.0(+0x3d318128a8)[0x7ff40c0908a8] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_submit+0x390)[0x7ff40c08d0b0] (--> /usr/lib64/glusterfs/3.7dev/xlator/protocol/client.so(client_submit_request+0x196)[0x7ff402761e26] ))))) 0-timer: ctx cleanup started
[2015-03-24 09:44:58.985556] W [rpc-clnt-ping.c:296:rpc_clnt_start_ping] 0-testvol-snapd-client: unable to setup ping timer
[2015-03-24 09:44:58.985600] I [client-handshake.c:187:client_set_lk_version_cbk] 0-testvol-snapd-client: Server lk version = 1
[2015-03-24 09:44:59.206359] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-disperse-1: size=1764 max=1 total=2
[2015-03-24 09:44:59.206661] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-disperse-1: size=1932 max=0 total=0
[2015-03-24 09:44:59.207173] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-disperse-1: size=1260 max=0 total=0
[2015-03-24 09:44:59.207373] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-dht: size=1148 max=0 total=0
[2015-03-24 09:44:59.207482] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-dht: size=2148 max=1 total=2
[2015-03-24 09:44:59.207646] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-read-ahead: size=188 max=0 total=0
[2015-03-24 09:44:59.207668] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-io-cache: size=68 max=0 total=0
[2015-03-24 09:44:59.207683] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-io-cache: size=252 max=1 total=2
[2015-03-24 09:44:59.207732] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-snapd-client: size=1300 max=0 total=0
[2015-03-24 09:44:59.207772] I [client.c:2282:client_rpc_notify] 0-testvol-snapd-client: disconnected from testvol-snapd-client. Client process will keep trying to connect to glusterd until brick's port is available
[2015-03-24 09:44:59.207825] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-snapd-client: size=588 max=2 total=6
[2015-03-24 09:44:59.207954] I [mem-pool.c:557:mem_pool_destroy] 0-testvol-snapd-client: size=124 max=2 total=6
[2015-03-24 09:44:59.208134] E [mem-pool.c:250:__gf_free] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (--> /usr/lib64/libglusterfs.so.0(__gf_free+0xb4)[0x7ff40bc13be4] (--> /usr/lib64/glusterfs/3.7dev/xlator/features/snapview-client.so(fini+0x2e)[0x7ff4014408ae] (--> /usr/lib64/libglusterfs.so.0(+0x3d3141f341)[0x7ff40bbe2341] (--> /usr/lib64/libglusterfs.so.0(+0x3d3141f2d5)[0x7ff40bbe22d5] ))))) 0-: Assertion failed: GF_MEM_HEADER_MAGIC == *(uint32_t *)ptr
[2015-03-24 09:45:09.433579] I [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2015-03-24 09:45:09.441878] W [socket.c:636:__socket_rwv] 0-gfapi: readv on 127.0.0.1:24007 failed (No data available)
[2015-03-24 09:45:09.442464] E [rpc-clnt.c:362:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7ff40bbe4820] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e7)[0x7ff40c08d927] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff40c08da3e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7ff40c08db0b] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7ff40c08e0cf] ))))) 0-gfapi: forced unwinding frame type(GlusterFS Handshake) op(GETSPEC(2)) called at 2015-03-24 09:45:09.441697 (xid=0x1)
[2015-03-24 09:45:09.442510] E [glfs-mgmt.c:605:glfs_mgmt_getspec_cbk] 0-glfs-mgmt: failed to fetch volume file (key:testvol)
[2015-03-24 09:45:09.442531] E [glfs-mgmt.c:702:mgmt_rpc_notify] 0-glfs-mgmt: failed to connect with remote-host: localhost (No data available)
[2015-03-24 09:45:09.442545] I [glfs-mgmt.c:707:mgmt_rpc_notify] 0-glfs-mgmt: Exhausted all volfile servers
[2015-03-24 09:45:09.442833] I [event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread with index 1

==============================================
sosreport of the server will be attached.
Comment 2 Pranith Kumar K 2015-09-18 02:21:02 UTC 
As per Bhaskar this is not happening anymore. So closing the bug.