Bug 763435 (GLUSTER-1703)

Summary:	df hangs during add-brick,remove-brick loop
Product:	[Community] GlusterFS	Reporter:	Lakshmipathi G <lakshmipathi>
Component:	replicate	Assignee:	shishir gowda <sgowda>
Status:	CLOSED NOTABUG	QA Contact:
Severity:	medium	Docs Contact:
Priority:	low
Version:	3.1-alpha	CC:	amarts, gluster-bugs, nsathyan, vijay
Target Milestone:	---
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	---
Regression:	RTP	Mount Type:	nfs
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Amar Tumballi 2010-09-25 03:21:26 UTC

As of now, Replicate has issues with 4 subvolumes.. Its known to crash, hence can you reduce the severity? and also, no need to run so many tests on it.. lets address that once we are done with bugs related to 2/3 subvolumes of AFR.

Comment 1 Lakshmipathi G 2010-09-25 03:36:38 UTC

ok.moved it to "major"

Comment 2 Lakshmipathi G 2010-09-25 06:16:40 UTC

running bonnie on client mount-pt with 4 afr and  doing add-brick remove-brick in a loop - on client df command hangs-
+------------------------------------------------------------------------------+
[2010-09-24 07:23:21.366539] E [afr-common.c:2654:afr_notify] AFRbeta-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
[2010-09-24 07:23:21.366699] E [afr-common.c:2654:afr_notify] AFRbeta-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
[2010-09-24 07:23:21.366726] E [afr-common.c:2654:afr_notify] AFRbeta-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
[2010-09-24 07:23:21.366793] E [afr-common.c:2654:afr_notify] AFRbeta-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
[2010-09-24 07:23:21.637234] E [rpc-clnt.c:332:saved_frames_unwind] rpc-clnt: forced unwinding frame type(GlusterFS 3.1) op(INODELK(29)) called at 2010-09-24 07:23:21.197605
pending frames:
pending frames:

patchset: v3.1.0qa7-389-gc5a5fea
signal received: 11
time of crash: 2010-09-24 07:23:21
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.1.0beta

patchset: v3.1.0qa7-389-gc5a5fea
signal received: 11
time of crash: 2010-09-24 07:23:21
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.1.0beta
/lib64/libc.so.6[0x34172302d0]
/usr/local/lib/glusterfs/3.1.0beta/xlator/protocol/client.so(client_release+0x64)[0x2aaaabea6f43]
/usr/local/lib/libglusterfs.so.0[0x2aaaaaaea04d]
/usr/local/lib/libglusterfs.so.0(fd_unref+0xad)[0x2aaaaaaea172]
/usr/local/lib/glusterfs/3.1.0beta/xlator/cluster/replicate.so(afr_sh_data_done+0x76)[0x2aaaac0f5ac6]
/usr/local/lib/glusterfs/3.1.0beta/xlator/cluster/replicate.so(afr_sh_data_flush_cbk+0x120)[0x2aaaac0f5f8b]
/usr/local/lib/glusterfs/3.1.0beta/xlator/cluster/replicate.so(afr_sh_data_setattr_cbk+0x39)[0x2aaaac0f5fcb]
/usr/local/lib/glusterfs/3.1.0beta/xlator/protocol/client.so(client3_1_setattr_cbk+0x2af)[0x2aaaabeb4ff3]
/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x1e5)[0x2aaaaad31002]
/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x28d)[0x2aaaaad3133d]
/usr/local/lib/libgfrpc.so.0(rpc_transport_notify+0xeb)[0x2aaaaad2eb91]
/usr/local/lib/glusterfs/3.1.0beta/rpc-transport/socket.so(socket_event_poll_in+0x4b)[0x2aaab05373fe]
/usr/local/lib/glusterfs/3.1.0beta/rpc-transport/socket.so(socket_event_handler+0xfc)[0x2aaab053776f]
/usr/local/lib/libglusterfs.so.0[0x2aaaaaaed4b0]
/usr/local/lib/libglusterfs.so.0[0x2aaaaaaed69f]
/usr/local/lib/libglusterfs.so.0(event_dispatch+0x81)[0x2aaaaaaed9fb]
/usr/local/sbin/glusterfs(main+0xec)[0x405d1a]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x341721d994]
/usr/local/sbin/glusterfs[0x402d39]
---------
[2010-09-24 07:23:25.355568] I [client-handshake.c:660:select_server_supported_programs] AFRbeta-client-2: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-24 07:23:25.355807] I [client-handshake.c:496:client_setvolume_cbk] AFRbeta-client-2: Connected to 172.24.21.134:6971, attached to remote volume '/mnt/test1'.
[2010-09-24 07:23:25.355843] I [afr-common.c:2624:afr_notify] AFRbeta-replicate-0: Subvolume 'AFRbeta-client-2' came back up; going online.
[2010-09-24 07:23:25.356424] I [afr-common.c:827:afr_fresh_lookup_cbk] AFRbeta-replicate-0: added root inode
[2010-09-24 07:23:25.356461] I [nfs.c:315:__nfs_subvolume_start] nfs: All exports up
[2010-09-24 07:23:25.358945] I [client-handshake.c:660:select_server_supported_programs] AFRbeta-client-3: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-24 07:23:25.360638] I [client-handshake.c:660:select_server_supported_programs] AFRbeta-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2010-09-24 07:23:25.360936] I [client-handshake.c:496:client_setvolume_cbk] AFRbeta-client-0: Connected to 172.24.21.131:6971, attached to remote volume '/mnt/test1'.
[2010-09-24 07:23:25.362617] E [socket.c:1657:socket_connect_finish] AFRbeta-client-1: connection to 172.24.21.132:6969 failed (Connection refused)
[2010-09-24 07:23:25.411249] I [client-handshake.c:496:client_setvolume_cbk] AFRbeta-client-3: Connected to 172.24.21.133:6995, attached to remote volume '/mnt/test1'.
[2010-09-24 07:23:31.190462] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.008
[2010-09-24 07:23:31.195906] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.008
[2010-09-24 07:23:31.196176] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.008
[2010-09-24 07:23:31.196307] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.008
[2010-09-24 07:23:31.196700] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.008
[2010-09-24 07:23:31.196821] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.008
[2010-09-24 07:23:31.197140] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.008
[2010-09-24 07:23:31.197363] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.008
[2010-09-24 07:23:31.197617] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.008
[2010-09-24 07:23:31.197813] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.008
[2010-09-24 07:23:31.198016] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.008
[2010-09-24 07:23:31.198308] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.008
[2010-09-24 07:23:31.198463] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.008
[2010-09-24 07:23:31.198655] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.008
[2010-09-24 07:23:31.198849] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.008
[2010-09-24 07:23:31.198979] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.008
[2010-09-24 07:23:31.200519] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.008
[2010-09-24 07:23:31.200838] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.008
[2010-09-24 07:23:31.201042] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.008
[2010-09-24 07:23:31.201307] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.008
[2010-09-24 07:23:31.201462] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.008
[2010-09-24 07:23:31.201698] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.008
[2010-09-24 07:23:31.201872] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.008
[2010-09-24 07:23:31.202117] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.008
[2010-09-24 07:23:31.202296] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.008
[2010-09-24 07:23:31.202510] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.008
[2010-09-24 07:23:31.202699] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.008
[2010-09-24 07:23:31.202948] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.008
[2010-09-24 07:23:31.203139] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.008
[2010-09-24 07:23:31.203398] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.008
[2010-09-24 07:23:31.203704] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.008
[2010-09-24 07:23:31.208011] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.008
[2010-09-24 07:23:31.231490] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.011
[2010-09-24 07:23:31.232325] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.011
[2010-09-24 07:23:31.232741] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.011
[2010-09-24 07:23:31.233032] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.011
[2010-09-24 07:23:31.233308] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.011
[2010-09-24 07:23:31.233554] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.011
[2010-09-24 07:23:31.233889] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.011
[2010-09-24 07:23:31.234050] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.011
[2010-09-24 07:23:31.234226] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.011
[2010-09-24 07:23:31.234416] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.011
[2010-09-24 07:23:31.234642] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.011
[2010-09-24 07:23:31.238019] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.011
[2010-09-24 07:23:31.238693] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.011
[2010-09-24 07:23:31.238895] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.011
[2010-09-24 07:23:31.239152] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.011
[2010-09-24 07:23:31.239362] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.011
[2010-09-24 07:23:31.239818] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.011
[2010-09-24 07:23:31.240034] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.011
[2010-09-24 07:23:31.240289] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.011
[2010-09-24 07:23:31.240484] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.011
[2010-09-24 07:23:31.240720] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.011
[2010-09-24 07:23:32.970061] W [afr-self-heal-common.c:584:afr_sh_pending_to_delta] afr_sh_pending_to_delta: Unable to get dict value.
[2010-09-24 07:23:32.970143] W [afr-self-heal-common.c:584:afr_sh_pending_to_delta] afr_sh_pending_to_delta: Unable to get dict value.
[2010-09-24 07:23:32.970167] W [afr-self-heal-common.c:584:afr_sh_pending_to_delta] afr_sh_pending_to_delta: Unable to get dict value.
[2010-09-24 07:23:32.970179] W [afr-self-heal-common.c:584:afr_sh_pending_to_delta] afr_sh_pending_to_delta: Unable to get dict value.
[2010-09-24 07:23:32.975631] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.011
[2010-09-24 07:23:32.977749] E [afr-transaction.c:699:afr_changelog_pre_op_cbk] AFRbeta-replicate-0: xattrop failed on child AFRbeta-client-2: Bad file descriptor
[2010-09-24 07:23:32.980750] E [afr-transaction.c:699:afr_changelog_pre_op_cbk] AFRbeta-replicate-0: xattrop failed on child AFRbeta-client-3: Bad file descriptor
[2010-09-24 07:23:32.984182] E [afr-transaction.c:699:afr_changelog_pre_op_cbk] AFRbeta-replicate-0: xattrop failed on child AFRbeta-client-0: Bad file descriptor
[2010-09-24 07:23:33.2265] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.010
[2010-09-24 07:23:33.2481] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.010
[2010-09-24 07:23:33.2970] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.010
[2010-09-24 07:23:33.3152] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.010
[2010-09-24 07:23:33.3366] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.010
[2010-09-24 07:23:33.3538] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.010
[2010-09-24 07:23:33.4813] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.010
[2010-09-24 07:23:33.5083] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.010
[2010-09-24 07:23:33.6163] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.010
[2010-09-24 07:23:33.6657] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.010
[2010-09-24 07:23:33.6835] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.010
[2010-09-24 07:23:33.6988] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.010
[2010-09-24 07:23:33.7716] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.010
[2010-09-24 07:23:33.7899] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.010
[2010-09-24 07:23:33.8170] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.010
[2010-09-24 07:23:33.8601] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.010
[2010-09-24 07:23:33.8866] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.010
[2010-09-24 07:23:33.9065] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.010
[2010-09-24 07:23:33.9330] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.010
[2010-09-24 07:23:33.9771] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.010
[2010-09-24 07:23:33.10225] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.010
[2010-09-24 07:23:33.10946] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.010
[2010-09-24 07:23:33.11802] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.010
[2010-09-24 07:23:33.11996] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.010
[2010-09-24 07:23:33.12223] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.010
[2010-09-24 07:23:33.12895] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.010
[2010-09-24 07:23:33.13239] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.010
[2010-09-24 07:23:33.16261] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.010
[2010-09-24 07:23:33.39267] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.009
[2010-09-24 07:23:33.39648] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.009
[2010-09-24 07:23:33.41769] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.009
[2010-09-24 07:23:33.41959] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.009
[2010-09-24 07:23:33.42153] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.009
[2010-09-24 07:23:33.43059] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.009
[2010-09-24 07:23:33.43346] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.009
[2010-09-24 07:23:33.43538] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.009
[2010-09-24 07:23:33.43692] I [afr-common.c:724:afr_lookup_done] AFRbeta-replicate-0: background  data self-heal triggered. path: /Bonnie.27814.009
[2010-09-24 07:23:33.44415] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.009
[2010-09-24 07:23:33.46053] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.009
[2010-09-24 07:23:33.46387] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.009
[2010-09-24 07:23:33.46621] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.009
[2010-09-24 07:23:33.47887] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.009
[2010-09-24 07:23:33.48161] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.009
[2010-09-24 07:23:33.48456] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.009
[2010-09-24 07:23:33.48808] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.009
[2010-09-24 07:23:34.560238] W [afr-self-heal-common.c:584:afr_sh_pending_to_delta] afr_sh_pending_to_delta: Unable to get dict value.
[2010-09-24 07:23:34.560354] W [afr-self-heal-common.c:584:afr_sh_pending_to_delta] afr_sh_pending_to_delta: Unable to get dict value.
[2010-09-24 07:23:34.560367] W [afr-self-heal-common.c:584:afr_sh_pending_to_delta] afr_sh_pending_to_delta: Unable to get dict value.
[2010-09-24 07:23:34.560379] W [afr-self-heal-common.c:584:afr_sh_pending_to_delta] afr_sh_pending_to_delta: Unable to get dict value.
[2010-09-24 07:23:34.565463] I [afr-self-heal-common.c:1583:afr_self_heal_completion_cbk] AFRbeta-replicate-0: background  data self-heal completed on /Bonnie.27814.009
[2010-09-24 07:23:34.567775] E [afr-transaction.c:699:afr_changelog_pre_op_cbk] AFRbeta-replicate-0: xattrop failed on child AFRbeta-client-2: Bad file descriptor
[2010-09-24 07:23:34.567831] E [afr-transaction.c:699:afr_changelog_pre_op_cbk] AFRbeta-replicate-0: xattrop failed on child AFRbeta-client-2: Bad file descriptor
[2010-09-24 07:23:34.567911] E [afr-transaction.c:699:afr_changelog_pre_op_cbk] AFRbeta-replicate-0: xattrop failed on child AFRbeta-client-2: Bad file descriptor
[2010-09-24 07:23:34.567947] E [afr-transaction.c:699:afr_changelog_pre_op_cbk] AFRbeta-replicate-0: xattrop failed on child AFRbeta-client-2: Bad file descriptor
[2010-09-24 07:23:34.569271] E [afr-transaction.c:699:afr_changelog_pre_op_cbk] AFRbeta-replicate-0: xattrop failed on child AFRbeta-client-3: Bad file descriptor
[2010-09-24 07:23:34.569301] E [afr-transaction.c:699:afr_changelog_pre_op_cbk] AFRbeta-replicate-0: xattrop failed on child AFRbeta-client-0: Bad file descriptor
[2010-09-24 07:23:34.569480] E [afr-transaction.c:699:afr_changelog_pre_op_cbk] AFRbeta-replicate-0: xattrop failed on child AFRbeta-client-3: Bad file descriptor
[2010-09-24 07:23:34.569505] E [afr-transaction.c:699:afr_changelog_pre_op_cbk] AFRbeta-replicate-0: xattrop failed on child AFRbeta-client-0: Bad file descriptor
[2010-09-24 07:23:34.571139] E [afr-transaction.c:699:afr_changelog_pre_op_cbk] AFRbeta-replicate-0: xattrop failed on child AFRbeta-client-0: Bad file descriptor
[2010-09-24 07:23:34.571234] E [afr-transaction.c:699:afr_changelog_pre_op_cbk] AFRbeta-replicate-0: xattrop failed on child AFRbeta-client-3: Bad file descriptor
[2010-09-24 07:23:34.571935] E [afr-transaction.c:699:afr_changelog_pre_op_cbk] AFRbeta-replicate-0: xattrop failed on child AFRbeta-client-3: Bad file descriptor
[2010-09-24 07:23:34.572695] E [afr-transaction.c:699:afr_changelog_pre_op_cbk] AFRbeta-replicate-0: xattrop failed on child AFRbeta-client-0: Bad file descriptor

Comment 3 shishir gowda 2011-02-02 08:11:01 UTC

We dont support afr with 4 bricks. Closing the bug. Please file a new bug if found in 2 brick config