Bug 763514 (GLUSTER-1782)

Summary: add-brick says unsuccessful when its successful - qa39
Product: [Community] GlusterFS Reporter: Lakshmipathi G <lakshmipathi>
Component: cliAssignee: Pranith Kumar K <pkarampu>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: medium Docs Contact:
Priority: low    
Version: 3.1-alphaCC: amarts, gluster-bugs
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: RTP Mount Type: All
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Lakshmipathi G 2010-10-03 10:00:08 UTC
+++++++++++
log file of added-brick3
++++++++++++

10.214.231.112#cat /usr/local/var/log/glusterfs/usr-local-etc-glusterfs-glusterd.vol.log
[2010-10-03 08:46:29.866158] I [glusterd.c:276:init] management: Using /etc/glusterd as working directory
[2010-10-03 08:46:29.877279] E [rpc-transport.c:897:rpc_transport_load] rpc-transport: /usr/local/lib/glusterfs/3.1.0qa39/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
[2010-10-03 08:46:29.877299] E [rpc-transport.c:901:rpc_transport_load] rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine
[2010-10-03 08:46:29.877375] I [glusterd.c:95:glusterd_uuid_init] glusterd: generated UUID: 71042154-2a52-4796-a246-1120731a2fa4
Given volfile:
+------------------------------------------------------------------------------+
  1: volume management
  2:     type mgmt/glusterd
  3:     option working-directory /etc/glusterd
  4:     option transport-type socket,rdma
  5:     option transport.socket.listen-port 6969
  6:     option transport.rdma.listen-port 6968
  7:     # option listen-port 6969
  8: end-volume
  9: 

+------------------------------------------------------------------------------+
[2010-10-03 08:47:55.123454] I [glusterd-handler.c:2249:glusterd_handle_probe_query] glusterd: Received probe from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:47:55.123537] I [glusterd-handler.c:354:glusterd_friend_find] glusterd: Unable to find peer by uuid
[2010-10-03 08:47:55.124922] I [glusterd-handler.c:366:glusterd_friend_find] glusterd: Unable to find hostname: 10.192.141.187
[2010-10-03 08:47:55.124995] I [glusterd-handler.c:2272:glusterd_handle_probe_query] glusterd: Responded to 10.214.231.112, op_ret: 0, op_errno: 0, ret: 0
[2010-10-03 08:47:55.125022] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:47:55.125582] I [glusterd-handler.c:2028:glusterd_handle_incoming_friend_req] glusterd: Received probe from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:47:55.125628] I [glusterd-handler.c:354:glusterd_friend_find] glusterd: Unable to find peer by uuid
[2010-10-03 08:47:55.126092] I [glusterd-handler.c:366:glusterd_friend_find] glusterd: Unable to find hostname: 10.192.141.187
[2010-10-03 08:47:55.126111] I [glusterd-handler.c:119:glusterd_handle_friend_req] glusterd:  for host: 10.192.141.187 (6969)
[2010-10-03 08:47:55.128604] W [rpc-transport.c:841:rpc_transport_load] rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2010-10-03 08:47:55.130949] I [glusterd-handler.c:2419:glusterd_friend_add] glusterd: connect returned 0
[2010-10-03 08:47:55.130973] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 7
[2010-10-03 08:47:55.132038] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 4
[2010-10-03 08:47:55.132079] I [glusterd-handler.c:2569:glusterd_xfer_friend_add_resp] glusterd: Responded to 10.214.231.112 (0), ret: 0
[2010-10-03 08:47:55.132100] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 0 to 2
[2010-10-03 08:47:55.132191] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 1
[2010-10-03 08:47:55.132207] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 2 to 2
[2010-10-03 08:47:55.132297] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.192.141.187 found.. state: 2
[2010-10-03 08:47:55.132347] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 2 to 2
[2010-10-03 08:47:55.133306] I [glusterd3_1-mops.c:82:glusterd3_1_probe_cbk] glusterd: Received probe resp from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe, host: 10.192.141.187
[2010-10-03 08:47:55.133324] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 2
[2010-10-03 08:47:55.133340] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 2
[2010-10-03 08:47:55.133409] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 2 to 5
[2010-10-03 08:47:55.133490] I [glusterd3_1-mops.c:139:glusterd3_1_probe_cbk] glusterd: Received resp to probe req
[2010-10-03 08:47:55.134035] I [glusterd3_1-mops.c:186:glusterd3_1_friend_add_cbk] glusterd: Received ACC from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe, host: 10.192.141.187, port: 0
[2010-10-03 08:47:55.134053] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 5
[2010-10-03 08:47:55.134069] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:47:55.134092] I [glusterd3_1-mops.c:964:glusterd3_1_friend_update] : Added uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe, host: 10.192.141.187
[2010-10-03 08:47:55.134129] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 5 to 3
[2010-10-03 08:47:55.134291] I [glusterd-handler.c:2146:glusterd_handle_friend_update] glusterd: Received friend update from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:47:55.134321] I [glusterd-handler.c:2191:glusterd_handle_friend_update] : Received uuid: bd364a85-e8b3-43d6-8a88-363df38be63c, hostname:10.192.134.144
[2010-10-03 08:47:55.134337] I [glusterd-handler.c:354:glusterd_friend_find] glusterd: Unable to find peer by uuid
[2010-10-03 08:47:55.134829] I [glusterd-handler.c:366:glusterd_friend_find] glusterd: Unable to find hostname: 10.192.134.144
[2010-10-03 08:47:55.137360] W [rpc-transport.c:841:rpc_transport_load] rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2010-10-03 08:47:55.139526] I [glusterd-handler.c:2419:glusterd_friend_add] glusterd: connect returned 0
[2010-10-03 08:47:55.139547] I [glusterd-handler.c:2191:glusterd_handle_friend_update] : Received uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e, hostname:10.198.110.16
[2010-10-03 08:47:55.139563] I [glusterd-handler.c:354:glusterd_friend_find] glusterd: Unable to find peer by uuid
[2010-10-03 08:47:55.160065] I [glusterd-handler.c:366:glusterd_friend_find] glusterd: Unable to find hostname: 10.198.110.16
[2010-10-03 08:47:55.162677] W [rpc-transport.c:841:rpc_transport_load] rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2010-10-03 08:47:55.164843] I [glusterd-handler.c:2419:glusterd_friend_add] glusterd: connect returned 0
[2010-10-03 08:47:55.164864] I [glusterd-handler.c:2191:glusterd_handle_friend_update] : Received uuid: 71042154-2a52-4796-a246-1120731a2fa4, hostname:10.214.231.112
[2010-10-03 08:47:55.164880] I [glusterd-handler.c:2194:glusterd_handle_friend_update] : Received my uuid as Friend
[2010-10-03 08:47:55.164934] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:47:55.165027] I [glusterd3_1-mops.c:371:glusterd3_1_friend_update_cbk] glusterd: Received ACC from uuid: 
[2010-10-03 08:48:02.719468] I [glusterd-handler.c:394:glusterd_handle_cluster_lock] glusterd: Received LOCK from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:02.719542] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 2
[2010-10-03 08:48:02.719572] I [glusterd-handler.c:410:glusterd_handle_cluster_lock] : Returning 0
[2010-10-03 08:48:02.719596] I [glusterd-utils.c:222:glusterd_lock] glusterd: Cluster lock held by 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:02.719650] I [glusterd-handler.c:1884:glusterd_op_lock_send_resp] glusterd: Responded, ret: 0
[2010-10-03 08:48:02.719667] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 0 to 2 due to event 2
[2010-10-03 08:48:02.719684] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:02.720382] I [glusterd-handler.c:433:glusterd_handle_stage_op] glusterd: Received stage op from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:02.720400] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 8
[2010-10-03 08:48:02.720437] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.198.110.16:/mnt/oct3
[2010-10-03 08:48:02.720457] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 3
[2010-10-03 08:48:02.720474] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.192.134.144 found.. state: 3
[2010-10-03 08:48:02.720513] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.192.141.187 found.. state: 3
[2010-10-03 08:48:02.720529] E [glusterd-op-sm.c:718:glusterd_op_stage_add_brick] : Adding duplicate brick: 10.198.110.16:/mnt/oct3
[2010-10-03 08:48:02.720547] E [glusterd-op-sm.c:4370:glusterd_op_ac_stage_op] : Validate failed: -1
[2010-10-03 08:48:02.720584] I [glusterd-handler.c:1966:glusterd_op_stage_send_resp] glusterd: Responded to stage, ret: 0
[2010-10-03 08:48:02.720600] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 2 to 4 due to event 8
[2010-10-03 08:48:02.720620] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:02.721887] I [glusterd-handler.c:1928:glusterd_handle_cluster_unlock] glusterd: Received UNLOCK from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:02.721904] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 10
[2010-10-03 08:48:02.721935] I [glusterd-handler.c:1904:glusterd_op_unlock_send_resp] glusterd: Responded to unlock, ret: 0
[2010-10-03 08:48:02.721951] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 4 to 0 due to event 10
[2010-10-03 08:48:02.721972] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:04.176608] I [glusterd-handler.c:394:glusterd_handle_cluster_lock] glusterd: Received LOCK from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:04.176673] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 2
[2010-10-03 08:48:04.176689] I [glusterd-handler.c:410:glusterd_handle_cluster_lock] : Returning 0
[2010-10-03 08:48:04.176709] I [glusterd-utils.c:222:glusterd_lock] glusterd: Cluster lock held by 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:04.176760] I [glusterd-handler.c:1884:glusterd_op_lock_send_resp] glusterd: Responded, ret: 0
[2010-10-03 08:48:04.176777] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 0 to 2 due to event 2
[2010-10-03 08:48:04.176794] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:04.178598] I [glusterd-handler.c:433:glusterd_handle_stage_op] glusterd: Received stage op from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:04.178616] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 8
[2010-10-03 08:48:04.178645] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.198.110.16:/mnt/oct3
[2010-10-03 08:48:04.178665] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 3
[2010-10-03 08:48:04.178691] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 3
[2010-10-03 08:48:04.178707] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.214.231.112:/mnt/oct3
[2010-10-03 08:48:04.180401] I [glusterd-handler.c:1966:glusterd_op_stage_send_resp] glusterd: Responded to stage, ret: 0
[2010-10-03 08:48:04.180454] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 2 to 4 due to event 8
[2010-10-03 08:48:04.180473] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:04.183634] I [rpc-clnt.c:691:rpc_clnt_handle_cbk] rpc-clnt: recieved rpc message (XID: 0x2a, Ver: 2, Program: 52743234, ProgVers: 1, Proc: 1) from rpc-transport (management)
[2010-10-03 08:48:05.306287] I [glusterd-handler.c:483:glusterd_handle_commit_op] glusterd: Received commit op from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:05.306321] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 9
[2010-10-03 08:48:05.306360] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 3
[2010-10-03 08:48:05.307983] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.198.110.16:/mnt/oct3
[2010-10-03 08:48:05.308004] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 3
[2010-10-03 08:48:05.308037] I [glusterd-utils.c:696:glusterd_brickinfo_get] : Found brick
[2010-10-03 08:48:05.308072] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.214.231.112:/mnt/oct3
[2010-10-03 08:48:05.308659] I [glusterd-utils.c:696:glusterd_brickinfo_get] : Found brick
[2010-10-03 08:48:05.884572] I [glusterd-utils.c:906:glusterd_volume_start_glusterfs] : About to start glusterfs for brick 10.214.231.112:/mnt/oct3
[2010-10-03 08:48:05.955017] E [glusterd-store.c:215:glusterd_store_delete_brick] : Unlink failed on /etc/glusterd/vols/dhtqa39/bricks/10.198.110.16:-mnt-oct3
[2010-10-03 08:48:05.955079] E [glusterd-store.c:215:glusterd_store_delete_brick] : Unlink failed on /etc/glusterd/vols/dhtqa39/bricks/10.214.231.112:-mnt-oct3
[2010-10-03 08:48:06.21384] I [glusterd-handler.c:2003:glusterd_op_commit_send_resp] glusterd: Responded to commit, ret: 0
[2010-10-03 08:48:06.21430] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 4 to 6 due to event 9
[2010-10-03 08:48:06.21470] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:06.21571] I [rpc-clnt.c:691:rpc_clnt_handle_cbk] rpc-clnt: recieved rpc message (XID: 0x2a, Ver: 2, Program: 52743234, ProgVers: 1, Proc: 1) from rpc-transport (management)
[2010-10-03 08:48:06.21604] I [rpc-clnt.c:691:rpc_clnt_handle_cbk] rpc-clnt: recieved rpc message (XID: 0x2a, Ver: 2, Program: 52743234, ProgVers: 1, Proc: 1) from rpc-transport (management)
[2010-10-03 08:48:06.24515] I [glusterd-pmap.c:237:pmap_registry_bind] pmap: adding brick /mnt/oct3 on port 6971
[2010-10-03 08:48:06.448820] I [glusterd-handler.c:1928:glusterd_handle_cluster_unlock] glusterd: Received UNLOCK from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:06.448852] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 10
[2010-10-03 08:48:06.448889] I [glusterd-handler.c:1904:glusterd_op_unlock_send_resp] glusterd: Responded to unlock, ret: 0
[2010-10-03 08:48:06.448922] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 6 to 0 due to event 10
[2010-10-03 08:48:06.448943] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:08.442951] I [glusterd-handler.c:394:glusterd_handle_cluster_lock] glusterd: Received LOCK from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:08.443004] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 2
[2010-10-03 08:48:08.443020] I [glusterd-handler.c:410:glusterd_handle_cluster_lock] : Returning 0
[2010-10-03 08:48:08.443040] I [glusterd-utils.c:222:glusterd_lock] glusterd: Cluster lock held by 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:08.443093] I [glusterd-handler.c:1884:glusterd_op_lock_send_resp] glusterd: Responded, ret: 0
[2010-10-03 08:48:08.443111] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 0 to 2 due to event 2
[2010-10-03 08:48:08.443128] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:08.443956] I [glusterd-handler.c:1928:glusterd_handle_cluster_unlock] glusterd: Received UNLOCK from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:08.443974] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 10
[2010-10-03 08:48:08.444006] I [glusterd-handler.c:1904:glusterd_op_unlock_send_resp] glusterd: Responded to unlock, ret: 0
[2010-10-03 08:48:08.444022] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 2 to 0 due to event 10
[2010-10-03 08:48:08.444042] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
=====
+++++++++++
log file of added-brick3
++++++++++++
[root@domU-12-31-39-10-6D-E2 ~]# ps aux | grep gluster
root     23398  0.0  0.0   6064   604 pts/2    R+   08:34   0:00 grep gluster

root@domU-12-31-39-10-6D-E2 ~]# PS1="10.198.110.16#"

10.198.110.16#glusterd

10.198.110.16#cat /usr/local/var/log/glusterfs/usr-local-etc-glusterfs-glusterd.vol.log
[2010-10-03 08:46:35.361429] I [glusterd.c:276:init] management: Using /etc/glusterd as working directory
[2010-10-03 08:46:35.362537] E [rpc-transport.c:897:rpc_transport_load] rpc-transport: /usr/local/lib/glusterfs/3.1.0qa39/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
[2010-10-03 08:46:35.362558] E [rpc-transport.c:901:rpc_transport_load] rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine
[2010-10-03 08:46:35.362635] I [glusterd.c:95:glusterd_uuid_init] glusterd: generated UUID: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e
Given volfile:
+------------------------------------------------------------------------------+
  1: volume management
  2:     type mgmt/glusterd
  3:     option working-directory /etc/glusterd
  4:     option transport-type socket,rdma
  5:     option transport.socket.listen-port 6969
  6:     option transport.rdma.listen-port 6968
  7:     # option listen-port 6969
  8: end-volume
  9: 

+------------------------------------------------------------------------------+
[2010-10-03 08:47:48.357579] I [glusterd-handler.c:2249:glusterd_handle_probe_query] glusterd: Received probe from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:47:48.357670] I [glusterd-handler.c:354:glusterd_friend_find] glusterd: Unable to find peer by uuid
[2010-10-03 08:47:48.359831] I [glusterd-handler.c:366:glusterd_friend_find] glusterd: Unable to find hostname: 10.192.141.187
[2010-10-03 08:47:48.359984] I [glusterd-handler.c:2272:glusterd_handle_probe_query] glusterd: Responded to 10.198.110.16, op_ret: 0, op_errno: 0, ret: 0
[2010-10-03 08:47:48.360013] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:47:48.361790] I [glusterd-handler.c:2028:glusterd_handle_incoming_friend_req] glusterd: Received probe from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:47:48.361847] I [glusterd-handler.c:354:glusterd_friend_find] glusterd: Unable to find peer by uuid
[2010-10-03 08:47:48.362355] I [glusterd-handler.c:366:glusterd_friend_find] glusterd: Unable to find hostname: 10.192.141.187
[2010-10-03 08:47:48.362376] I [glusterd-handler.c:119:glusterd_handle_friend_req] glusterd:  for host: 10.192.141.187 (6969)
[2010-10-03 08:47:48.364811] W [rpc-transport.c:841:rpc_transport_load] rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2010-10-03 08:47:48.367143] I [glusterd-handler.c:2419:glusterd_friend_add] glusterd: connect returned 0
[2010-10-03 08:47:48.367167] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 7
[2010-10-03 08:47:48.378108] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 4
[2010-10-03 08:47:48.378151] I [glusterd-handler.c:2569:glusterd_xfer_friend_add_resp] glusterd: Responded to 10.198.110.16 (0), ret: 0
[2010-10-03 08:47:48.378173] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 0 to 2
[2010-10-03 08:47:48.378262] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 1
[2010-10-03 08:47:48.378279] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 2 to 2
[2010-10-03 08:47:48.378370] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.192.141.187 found.. state: 2
[2010-10-03 08:47:48.378420] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 2 to 2
[2010-10-03 08:47:48.380136] I [glusterd3_1-mops.c:82:glusterd3_1_probe_cbk] glusterd: Received probe resp from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe, host: 10.192.141.187
[2010-10-03 08:47:48.380155] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 2
[2010-10-03 08:47:48.380171] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 2
[2010-10-03 08:47:48.380240] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 2 to 5
[2010-10-03 08:47:48.380321] I [glusterd3_1-mops.c:139:glusterd3_1_probe_cbk] glusterd: Received resp to probe req
[2010-10-03 08:47:48.382310] I [glusterd-handler.c:2146:glusterd_handle_friend_update] glusterd: Received friend update from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:47:48.382339] I [glusterd-handler.c:2191:glusterd_handle_friend_update] : Received uuid: bd364a85-e8b3-43d6-8a88-363df38be63c, hostname:10.192.134.144
[2010-10-03 08:47:48.382355] I [glusterd-handler.c:354:glusterd_friend_find] glusterd: Unable to find peer by uuid
[2010-10-03 08:47:48.383843] I [glusterd-handler.c:366:glusterd_friend_find] glusterd: Unable to find hostname: 10.192.134.144
[2010-10-03 08:47:48.386207] W [rpc-transport.c:841:rpc_transport_load] rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2010-10-03 08:47:48.388461] I [glusterd-handler.c:2419:glusterd_friend_add] glusterd: connect returned 0
[2010-10-03 08:47:48.388483] I [glusterd-handler.c:2191:glusterd_handle_friend_update] : Received uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e, hostname:10.198.110.16
[2010-10-03 08:47:48.388499] I [glusterd-handler.c:2194:glusterd_handle_friend_update] : Received my uuid as Friend
[2010-10-03 08:47:48.388550] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:47:48.399191] I [glusterd3_1-mops.c:186:glusterd3_1_friend_add_cbk] glusterd: Received ACC from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe, host: 10.192.141.187, port: 0
[2010-10-03 08:47:48.399209] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 5
[2010-10-03 08:47:48.399226] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:47:48.399251] I [glusterd3_1-mops.c:964:glusterd3_1_friend_update] : Added uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe, host: 10.192.141.187
[2010-10-03 08:47:48.399271] I [glusterd3_1-mops.c:964:glusterd3_1_friend_update] : Added uuid: bd364a85-e8b3-43d6-8a88-363df38be63c, host: 10.192.134.144
[2010-10-03 08:47:48.399322] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 5 to 3
[2010-10-03 08:47:48.400121] I [glusterd3_1-mops.c:371:glusterd3_1_friend_update_cbk] glusterd: Received ACC from uuid: 
[2010-10-03 08:47:55.80015] I [glusterd-handler.c:2146:glusterd_handle_friend_update] glusterd: Received friend update from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:47:55.80101] I [glusterd-handler.c:2191:glusterd_handle_friend_update] : Received uuid: bd364a85-e8b3-43d6-8a88-363df38be63c, hostname:10.192.134.144
[2010-10-03 08:47:55.80739] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:47:55.80806] I [glusterd-handler.c:2191:glusterd_handle_friend_update] : Received uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e, hostname:10.198.110.16
[2010-10-03 08:47:55.80824] I [glusterd-handler.c:2194:glusterd_handle_friend_update] : Received my uuid as Friend
[2010-10-03 08:47:55.80841] I [glusterd-handler.c:2191:glusterd_handle_friend_update] : Received uuid: 71042154-2a52-4796-a246-1120731a2fa4, hostname:10.214.231.112
[2010-10-03 08:47:55.80856] I [glusterd-handler.c:354:glusterd_friend_find] glusterd: Unable to find peer by uuid
[2010-10-03 08:47:55.82358] I [glusterd-handler.c:366:glusterd_friend_find] glusterd: Unable to find hostname: 10.214.231.112
[2010-10-03 08:47:55.84715] W [rpc-transport.c:841:rpc_transport_load] rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2010-10-03 08:47:55.86981] I [glusterd-handler.c:2419:glusterd_friend_add] glusterd: connect returned 0
[2010-10-03 08:47:55.87046] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:02.664091] I [glusterd-handler.c:394:glusterd_handle_cluster_lock] glusterd: Received LOCK from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:02.664161] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 2
[2010-10-03 08:48:02.664191] I [glusterd-handler.c:410:glusterd_handle_cluster_lock] : Returning 0
[2010-10-03 08:48:02.664215] I [glusterd-utils.c:222:glusterd_lock] glusterd: Cluster lock held by 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:02.664296] I [glusterd-handler.c:1884:glusterd_op_lock_send_resp] glusterd: Responded, ret: 0
[2010-10-03 08:48:02.664315] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 0 to 2 due to event 2
[2010-10-03 08:48:02.664333] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:02.665965] I [glusterd-handler.c:433:glusterd_handle_stage_op] glusterd: Received stage op from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:02.665983] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 8
[2010-10-03 08:48:02.666021] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.198.110.16:/mnt/oct3
[2010-10-03 08:48:02.668227] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.192.134.144 found.. state: 3
[2010-10-03 08:48:02.668291] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.192.141.187 found.. state: 3
[2010-10-03 08:48:02.668308] E [glusterd-op-sm.c:718:glusterd_op_stage_add_brick] : Adding duplicate brick: 10.198.110.16:/mnt/oct3
[2010-10-03 08:48:02.668330] E [glusterd-op-sm.c:4370:glusterd_op_ac_stage_op] : Validate failed: -1
[2010-10-03 08:48:02.668401] I [glusterd-handler.c:1966:glusterd_op_stage_send_resp] glusterd: Responded to stage, ret: 0
[2010-10-03 08:48:02.668423] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 2 to 4 due to event 8
[2010-10-03 08:48:02.668441] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:02.669370] I [glusterd-handler.c:1928:glusterd_handle_cluster_unlock] glusterd: Received UNLOCK from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:02.669388] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 10
[2010-10-03 08:48:02.669421] I [glusterd-handler.c:1904:glusterd_op_unlock_send_resp] glusterd: Responded to unlock, ret: 0
[2010-10-03 08:48:02.669441] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 4 to 0 due to event 10
[2010-10-03 08:48:02.669457] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:04.122211] I [glusterd-handler.c:394:glusterd_handle_cluster_lock] glusterd: Received LOCK from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:04.122277] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 2
[2010-10-03 08:48:04.122293] I [glusterd-handler.c:410:glusterd_handle_cluster_lock] : Returning 0
[2010-10-03 08:48:04.122314] I [glusterd-utils.c:222:glusterd_lock] glusterd: Cluster lock held by 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:04.122365] I [glusterd-handler.c:1884:glusterd_op_lock_send_resp] glusterd: Responded, ret: 0
[2010-10-03 08:48:04.122383] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 0 to 2 due to event 2
[2010-10-03 08:48:04.122402] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:04.124202] I [glusterd-handler.c:433:glusterd_handle_stage_op] glusterd: Received stage op from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:04.124220] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 8
[2010-10-03 08:48:04.124250] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.198.110.16:/mnt/oct3
[2010-10-03 08:48:04.125792] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.214.231.112:/mnt/oct3
[2010-10-03 08:48:04.125836] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.214.231.112 found.. state: 3
[2010-10-03 08:48:04.125857] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.214.231.112 found.. state: 3
[2010-10-03 08:48:04.125919] I [glusterd-handler.c:1966:glusterd_op_stage_send_resp] glusterd: Responded to stage, ret: 0
[2010-10-03 08:48:04.125940] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 2 to 4 due to event 8
[2010-10-03 08:48:04.125980] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:04.129230] I [rpc-clnt.c:691:rpc_clnt_handle_cbk] rpc-clnt: recieved rpc message (XID: 0x2a, Ver: 2, Program: 52743234, ProgVers: 1, Proc: 1) from rpc-transport (management)
[2010-10-03 08:48:05.251887] I [glusterd-handler.c:483:glusterd_handle_commit_op] glusterd: Received commit op from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:05.251938] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 9
[2010-10-03 08:48:05.252630] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.214.231.112 found.. state: 3
[2010-10-03 08:48:05.253498] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.198.110.16:/mnt/oct3
[2010-10-03 08:48:05.254054] I [glusterd-utils.c:696:glusterd_brickinfo_get] : Found brick
[2010-10-03 08:48:05.836077] I [glusterd-utils.c:906:glusterd_volume_start_glusterfs] : About to start glusterfs for brick 10.198.110.16:/mnt/oct3
[2010-10-03 08:48:05.912227] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.214.231.112:/mnt/oct3
[2010-10-03 08:48:05.912281] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.214.231.112 found.. state: 3
[2010-10-03 08:48:05.912301] I [glusterd-utils.c:696:glusterd_brickinfo_get] : Found brick
[2010-10-03 08:48:05.912408] E [glusterd-store.c:215:glusterd_store_delete_brick] : Unlink failed on /etc/glusterd/vols/dhtqa39/bricks/10.198.110.16:-mnt-oct3
[2010-10-03 08:48:05.912433] E [glusterd-store.c:215:glusterd_store_delete_brick] : Unlink failed on /etc/glusterd/vols/dhtqa39/bricks/10.214.231.112:-mnt-oct3
[2010-10-03 08:48:05.986914] I [glusterd-handler.c:2003:glusterd_op_commit_send_resp] glusterd: Responded to commit, ret: 0
[2010-10-03 08:48:05.987002] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 4 to 6 due to event 9
[2010-10-03 08:48:05.987046] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:05.987160] I [rpc-clnt.c:691:rpc_clnt_handle_cbk] rpc-clnt: recieved rpc message (XID: 0x2a, Ver: 2, Program: 52743234, ProgVers: 1, Proc: 1) from rpc-transport (management)
[2010-10-03 08:48:05.987197] I [rpc-clnt.c:691:rpc_clnt_handle_cbk] rpc-clnt: recieved rpc message (XID: 0x2a, Ver: 2, Program: 52743234, ProgVers: 1, Proc: 1) from rpc-transport (management)
[2010-10-03 08:48:05.990518] I [glusterd-pmap.c:237:pmap_registry_bind] pmap: adding brick /mnt/oct3 on port 6971
[2010-10-03 08:48:06.394516] I [glusterd-handler.c:1928:glusterd_handle_cluster_unlock] glusterd: Received UNLOCK from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:06.394552] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 10
[2010-10-03 08:48:06.394590] I [glusterd-handler.c:1904:glusterd_op_unlock_send_resp] glusterd: Responded to unlock, ret: 0
[2010-10-03 08:48:06.394611] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 6 to 0 due to event 10
[2010-10-03 08:48:06.394628] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:08.388484] I [glusterd-handler.c:394:glusterd_handle_cluster_lock] glusterd: Received LOCK from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:08.388546] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 2
[2010-10-03 08:48:08.388563] I [glusterd-handler.c:410:glusterd_handle_cluster_lock] : Returning 0
[2010-10-03 08:48:08.388588] I [glusterd-utils.c:222:glusterd_lock] glusterd: Cluster lock held by 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:08.388641] I [glusterd-handler.c:1884:glusterd_op_lock_send_resp] glusterd: Responded, ret: 0
[2010-10-03 08:48:08.388661] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 0 to 2 due to event 2
[2010-10-03 08:48:08.388679] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:08.389618] I [glusterd-handler.c:1928:glusterd_handle_cluster_unlock] glusterd: Received UNLOCK from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:08.389656] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 10
[2010-10-03 08:48:08.389690] I [glusterd-handler.c:1904:glusterd_op_unlock_send_resp] glusterd: Responded to unlock, ret: 0
[2010-10-03 08:48:08.389707] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 2 to 0 due to event 10
[2010-10-03 08:48:08.389725] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0


10.198.110.16#ps aux | grep gluster
root     23424  0.1  0.2  70288 17256 ?        Ssl  08:46   0:00 glusterd
root     23428  0.0  0.7 131948 59564 ?        Ssl  08:48   0:00 /usr/local/sbin/glusterfsd --xlator-option dhtqa39-server.listen-port=6971 -s localhost --volfile-id dhtqa39.10.198.110.16.mnt-oct3 -p /etc/glusterd/vols/dhtqa39/run/10.198.110.16-mnt-oct3.pid --brick-name /mnt/oct3 --brick-port 6971 -l /etc/glusterd/logs/bricks/mnt-oct3.log
root     23432  0.0  0.8 134572 69344 ?        Ssl  08:48   0:00 /usr/local/sbin/glusterfs -f /etc/glusterd/nfs/nfs-server.vol -p /etc/glusterd/nfs/run/nfs.pid -l /etc/glusterd/logs/nfs.log
root     23442  0.0  0.0   6064   616 pts/2    S+   08:51   0:00 grep gluster

Comment 1 Lakshmipathi G 2010-10-03 12:59:52 UTC
add-brick said unsuccessful when its successful ,when tried to add 2 bricks to existing 2dht cluster.  

[ unlike this - http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1773
this time no previous glusterfs process was running  ]

=========================
#gluster volume info

Volume Name: dhtqa39
Type: Distribute
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: 10.192.134.144:/mnt/oct3
Brick2: 10.192.141.187:/mnt/oct3

10.192.141.187#gluster peer probe 10.198.110.16
Probe successful
10.192.141.187#gluster peer probe  10.214.231.112
Probe successful
10.192.141.187#gluster volume add-brick dhtqa39 10.198.110.16:/mnt/oct3 10.214.231.112:/mnt/oct3
Add Brick unsuccessful

10.192.141.187#gluster volume add-brick dhtqa39 10.198.110.16:/mnt/oct3 10.214.231.112:/mnt/oct3
Add Brick unsuccessful

10.192.141.187#gluster volume add-brick dhtqa39 10.198.110.16:/mnt/oct3 10.214.231.112:/mnt/oct3
Add Brick unsuccessful

10.192.141.187#gluster volume info

Volume Name: dhtqa39
Type: Distribute
Status: Started
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: 10.192.134.144:/mnt/oct3
Brick2: 10.192.141.187:/mnt/oct3
Brick3: 10.198.110.16:/mnt/oct3
Brick4: 10.214.231.112:/mnt/oct3


10.192.141.187#cat /usr/local/var/log/glusterfs/usr-local-etc-glusterfs-glusterd.vol.log 
[2010-10-03 08:39:42.786179] I [glusterd.c:276:init] management: Using /etc/glusterd as working directory
[2010-10-03 08:39:42.787280] E [rpc-transport.c:897:rpc_transport_load] rpc-transport: /usr/local/lib/glusterfs/3.1.0qa39/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
[2010-10-03 08:39:42.787300] E [rpc-transport.c:901:rpc_transport_load] rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine
[2010-10-03 08:39:42.787369] I [glusterd.c:95:glusterd_uuid_init] glusterd: generated UUID: 7c9a6b04-3436-4e9c-812e-91be61c407fe
Given volfile:
+------------------------------------------------------------------------------+
  1: volume management
  2:     type mgmt/glusterd
  3:     option working-directory /etc/glusterd
  4:     option transport-type socket,rdma
  5:     option transport.socket.listen-port 6969
  6:     option transport.rdma.listen-port 6968
  7:     # option listen-port 6969
  8: end-volume
  9: 

+------------------------------------------------------------------------------+
[2010-10-03 08:39:55.936290] I [glusterd-handler.c:531:glusterd_handle_cli_probe] glusterd: Received CLI probe req 10.192.134.144 6969
[2010-10-03 08:39:55.948120] I [glusterd-handler.c:366:glusterd_friend_find] glusterd: Unable to find hostname: 10.192.134.144
[2010-10-03 08:39:55.948140] I [glusterd-handler.c:2438:glusterd_probe_begin] glusterd: Unable to find peerinfo for host: 10.192.134.144 (6969)
[2010-10-03 08:39:55.950808] W [rpc-transport.c:841:rpc_transport_load] rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2010-10-03 08:39:55.953130] I [glusterd-handler.c:2419:glusterd_friend_add] glusterd: connect returned 0
[2010-10-03 08:39:55.953606] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 11
[2010-10-03 08:39:55.953677] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.192.134.144 found.. state: 0
[2010-10-03 08:39:55.953757] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 0 to 0
[2010-10-03 08:39:55.955851] I [glusterd3_1-mops.c:82:glusterd3_1_probe_cbk] glusterd: Received probe resp from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c, host: 10.192.134.144
[2010-10-03 08:39:55.955881] I [glusterd-handler.c:354:glusterd_friend_find] glusterd: Unable to find peer by uuid
[2010-10-03 08:39:55.955902] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.192.134.144 found.. state: 0
[2010-10-03 08:39:55.955923] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 2
[2010-10-03 08:39:55.955982] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 0 to 1
[2010-10-03 08:39:55.956106] I [glusterd3_1-mops.c:139:glusterd3_1_probe_cbk] glusterd: Received resp to probe req
[2010-10-03 08:39:55.962006] I [glusterd3_1-mops.c:186:glusterd3_1_friend_add_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c, host: 10.192.134.144, port: 0
[2010-10-03 08:39:55.962025] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 1
[2010-10-03 08:39:55.962042] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:39:55.962093] I [glusterd-handler.c:2592:glusterd_xfer_cli_probe_resp] glusterd: Responded to CLI, ret: 0
[2010-10-03 08:39:55.962110] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 1 to 4
[2010-10-03 08:39:55.962420] I [glusterd-handler.c:2249:glusterd_handle_probe_query] glusterd: Received probe from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:39:55.962443] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 4
[2010-10-03 08:39:55.962473] I [glusterd-handler.c:2272:glusterd_handle_probe_query] glusterd: Responded to 10.192.141.187, op_ret: 0, op_errno: 0, ret: 0
[2010-10-03 08:39:55.962503] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:39:55.970199] I [glusterd-handler.c:2028:glusterd_handle_incoming_friend_req] glusterd: Received probe from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:39:55.970263] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 4
[2010-10-03 08:39:55.970292] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 7
[2010-10-03 08:39:55.970310] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 4
[2010-10-03 08:39:55.970341] I [glusterd-handler.c:2569:glusterd_xfer_friend_add_resp] glusterd: Responded to 10.192.141.187 (0), ret: 0
[2010-10-03 08:39:55.970358] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 4 to 4
[2010-10-03 08:39:55.970467] I [glusterd3_1-mops.c:964:glusterd3_1_friend_update] : Added uuid: bd364a85-e8b3-43d6-8a88-363df38be63c, host: 10.192.134.144
[2010-10-03 08:39:55.970509] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 4 to 3
[2010-10-03 08:39:55.970595] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:39:55.970756] I [glusterd-handler.c:2146:glusterd_handle_friend_update] glusterd: Received friend update from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:39:55.970790] I [glusterd-handler.c:2191:glusterd_handle_friend_update] : Received uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe, hostname:10.192.141.187
[2010-10-03 08:39:55.970810] I [glusterd-handler.c:2194:glusterd_handle_friend_update] : Received my uuid as Friend
[2010-10-03 08:39:55.970851] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:39:55.971026] I [glusterd3_1-mops.c:371:glusterd3_1_friend_update_cbk] glusterd: Received ACC from uuid: 
[2010-10-03 08:39:59.22074] I [glusterd-handler.c:633:glusterd_handle_cli_list_friends] glusterd: Received cli list req
[2010-10-03 08:39:59.22217] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:40:56.354775] I [glusterd-handler.c:675:glusterd_handle_cli_get_volume] glusterd: Received get vol req
[2010-10-03 08:40:56.354852] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:41:24.17419] I [glusterd-op-sm.c:4901:glusterd_op_set_cli_op] : Returning 0
[2010-10-03 08:41:24.17475] I [glusterd-handler.c:745:glusterd_handle_create_volume] glusterd: Received create volume req
[2010-10-03 08:41:24.17660] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.192.134.144 found.. state: 3
[2010-10-03 08:41:24.17794] I [glusterd-utils.c:222:glusterd_lock] glusterd: Cluster lock held by 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:41:24.17811] I [glusterd-handler.c:2636:glusterd_op_txn_begin] glusterd: Acquired local lock
[2010-10-03 08:41:24.17826] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 1
[2010-10-03 08:41:24.17851] I [glusterd-handler.c:2640:glusterd_op_txn_begin] glusterd: Returning 0
[2010-10-03 08:41:24.17923] I [glusterd3_1-mops.c:1054:glusterd3_1_cluster_lock] glusterd: Sent lock req to 1 peers
[2010-10-03 08:41:24.17940] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 0 to 1 due to event 1
[2010-10-03 08:41:24.17966] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:41:24.18908] I [glusterd3_1-mops.c:410:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:41:24.18945] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:41:24.18965] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:41:24.18981] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 4
[2010-10-03 08:41:24.18995] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 1 due to event 3
[2010-10-03 08:41:24.19043] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.192.134.144 found.. state: 3
[2010-10-03 08:41:24.19834] I [glusterd3_1-mops.c:1195:glusterd3_1_stage_op] glusterd: Sent op req to 1 peers
[2010-10-03 08:41:24.19889] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 3 due to event 4
[2010-10-03 08:41:24.21347] I [glusterd3_1-mops.c:533:glusterd3_1_stage_op_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:41:24.21371] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:41:24.21392] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:41:24.21413] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 5
[2010-10-03 08:41:24.21432] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 3 to 3 due to event 3
[2010-10-03 08:41:24.21493] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.192.134.144 found.. state: 3
[2010-10-03 08:41:24.23005] I [glusterd3_1-mops.c:1285:glusterd3_1_commit_op] glusterd: Sent op req to 1 peers
[2010-10-03 08:41:24.23034] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 3 to 5 due to event 5
[2010-10-03 08:41:24.24790] I [rpc-clnt.c:691:rpc_clnt_handle_cbk] rpc-clnt: recieved rpc message (XID: 0x2a, Ver: 2, Program: 52743234, ProgVers: 1, Proc: 1) from rpc-transport (management)
[2010-10-03 08:41:24.24919] I [glusterd3_1-mops.c:680:glusterd3_1_commit_op_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:41:24.24947] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:41:24.24968] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:41:24.24993] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 6
[2010-10-03 08:41:24.25013] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 5 to 5 due to event 3
[2010-10-03 08:41:24.25066] I [glusterd3_1-mops.c:1108:glusterd3_1_cluster_unlock] glusterd: Sent unlock req to 1 peers
[2010-10-03 08:41:24.25090] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 5 to 7 due to event 6
[2010-10-03 08:41:24.25413] I [glusterd3_1-mops.c:471:glusterd3_1_cluster_unlock_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:41:24.25441] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:41:24.25462] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:41:24.25482] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 4
[2010-10-03 08:41:24.25501] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 7 due to event 3
[2010-10-03 08:41:24.25524] I [glusterd-op-sm.c:4277:glusterd_op_txn_complete] glusterd: Cleared local lock
[2010-10-03 08:41:24.25574] I [glusterd-op-sm.c:4241:glusterd_op_send_cli_response] : Returning 0
[2010-10-03 08:41:24.25593] I [glusterd-op-sm.c:4320:glusterd_op_txn_complete] glusterd: Returning 0
[2010-10-03 08:41:24.25608] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 0 due to event 4
[2010-10-03 08:41:31.822112] I [glusterd-op-sm.c:4901:glusterd_op_set_cli_op] : Returning 0
[2010-10-03 08:41:31.822159] I [glusterd-handler.c:928:glusterd_handle_cli_start_volume] glusterd: Received start vol reqfor volume dhtqa39
[2010-10-03 08:41:31.822190] I [glusterd-utils.c:222:glusterd_lock] glusterd: Cluster lock held by 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:41:31.822206] I [glusterd-handler.c:2636:glusterd_op_txn_begin] glusterd: Acquired local lock
[2010-10-03 08:41:31.822222] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 1
[2010-10-03 08:41:31.822237] I [glusterd-handler.c:2640:glusterd_op_txn_begin] glusterd: Returning 0
[2010-10-03 08:41:31.822313] I [glusterd3_1-mops.c:1054:glusterd3_1_cluster_lock] glusterd: Sent lock req to 1 peers
[2010-10-03 08:41:31.822365] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 0 to 1 due to event 1
[2010-10-03 08:41:31.822385] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:41:31.823041] I [glusterd3_1-mops.c:410:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:41:31.823073] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:41:31.823089] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:41:31.823104] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 4
[2010-10-03 08:41:31.823119] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 1 due to event 3
[2010-10-03 08:41:31.823161] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.192.134.144 found.. state: 3
[2010-10-03 08:41:31.824124] I [glusterd3_1-mops.c:1195:glusterd3_1_stage_op] glusterd: Sent op req to 1 peers
[2010-10-03 08:41:31.824144] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 3 due to event 4
[2010-10-03 08:41:31.825633] I [glusterd3_1-mops.c:533:glusterd3_1_stage_op_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:41:31.825657] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:41:31.825680] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:41:31.825695] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 5
[2010-10-03 08:41:31.825710] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 3 to 3 due to event 3
[2010-10-03 08:41:32.394764] I [glusterd-utils.c:906:glusterd_volume_start_glusterfs] : About to start glusterfs for brick 10.192.141.187:/mnt/oct3
[2010-10-03 08:41:32.501777] I [glusterd3_1-mops.c:1285:glusterd3_1_commit_op] glusterd: Sent op req to 1 peers
[2010-10-03 08:41:32.501833] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 3 to 5 due to event 5
[2010-10-03 08:41:32.507442] I [glusterd-pmap.c:237:pmap_registry_bind] pmap: adding brick /mnt/oct3 on port 6971
[2010-10-03 08:41:33.222034] I [glusterd3_1-mops.c:680:glusterd3_1_commit_op_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:41:33.222075] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:41:33.222093] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:41:33.222111] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 6
[2010-10-03 08:41:33.222128] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 5 to 5 due to event 3
[2010-10-03 08:41:33.222170] I [glusterd3_1-mops.c:1108:glusterd3_1_cluster_unlock] glusterd: Sent unlock req to 1 peers
[2010-10-03 08:41:33.222191] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 5 to 7 due to event 6
[2010-10-03 08:41:33.222666] I [glusterd3_1-mops.c:471:glusterd3_1_cluster_unlock_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:41:33.222684] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:41:33.222703] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:41:33.222718] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 4
[2010-10-03 08:41:33.222733] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 7 due to event 3
[2010-10-03 08:41:33.222751] I [glusterd-op-sm.c:4277:glusterd_op_txn_complete] glusterd: Cleared local lock
[2010-10-03 08:41:33.222796] I [glusterd-op-sm.c:4241:glusterd_op_send_cli_response] : Returning 0
[2010-10-03 08:41:33.222836] I [glusterd-op-sm.c:4320:glusterd_op_txn_complete] glusterd: Returning 0
[2010-10-03 08:41:33.222852] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 0 due to event 4
[2010-10-03 08:42:05.2094] I [glusterd-handler.c:675:glusterd_handle_cli_get_volume] glusterd: Received get vol req
[2010-10-03 08:42:05.2240] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:42:05.2509] I [glusterd-handler.c:675:glusterd_handle_cli_get_volume] glusterd: Received get vol req
[2010-10-03 08:42:05.2561] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:47:48.614346] I [glusterd-handler.c:531:glusterd_handle_cli_probe] glusterd: Received CLI probe req 10.198.110.16 6969
[2010-10-03 08:47:48.619484] I [glusterd-handler.c:366:glusterd_friend_find] glusterd: Unable to find hostname: 10.198.110.16
[2010-10-03 08:47:48.619516] I [glusterd-handler.c:2438:glusterd_probe_begin] glusterd: Unable to find peerinfo for host: 10.198.110.16 (6969)
[2010-10-03 08:47:48.622682] W [rpc-transport.c:841:rpc_transport_load] rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2010-10-03 08:47:48.624994] I [glusterd-handler.c:2419:glusterd_friend_add] glusterd: connect returned 0
[2010-10-03 08:47:48.625823] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 11
[2010-10-03 08:47:48.625855] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 0
[2010-10-03 08:47:48.625905] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 0 to 0
[2010-10-03 08:47:48.630522] I [glusterd3_1-mops.c:82:glusterd3_1_probe_cbk] glusterd: Received probe resp from uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e, host: 10.198.110.16
[2010-10-03 08:47:48.630571] I [glusterd-handler.c:354:glusterd_friend_find] glusterd: Unable to find peer by uuid
[2010-10-03 08:47:48.630587] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 0
[2010-10-03 08:47:48.630605] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 2
[2010-10-03 08:47:48.630701] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 0 to 1
[2010-10-03 08:47:48.630837] I [glusterd3_1-mops.c:139:glusterd3_1_probe_cbk] glusterd: Received resp to probe req
[2010-10-03 08:47:48.648461] I [rpc-clnt.c:691:rpc_clnt_handle_cbk] rpc-clnt: recieved rpc message (XID: 0x2a, Ver: 2, Program: 52743234, ProgVers: 1, Proc: 1) from rpc-transport (management)
[2010-10-03 08:47:48.648991] I [glusterd3_1-mops.c:186:glusterd3_1_friend_add_cbk] glusterd: Received ACC from uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e, host: 10.198.110.16, port: 0
[2010-10-03 08:47:48.649024] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 1
[2010-10-03 08:47:48.649042] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:47:48.649094] I [glusterd-handler.c:2592:glusterd_xfer_cli_probe_resp] glusterd: Responded to CLI, ret: 0
[2010-10-03 08:47:48.649113] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 1 to 4
[2010-10-03 08:47:48.649983] I [glusterd-handler.c:2249:glusterd_handle_probe_query] glusterd: Received probe from uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e
[2010-10-03 08:47:48.650009] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 4
[2010-10-03 08:47:48.650044] I [glusterd-handler.c:2272:glusterd_handle_probe_query] glusterd: Responded to 10.192.141.187, op_ret: 0, op_errno: 0, ret: 0
[2010-10-03 08:47:48.650063] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:47:48.651662] I [glusterd-handler.c:2028:glusterd_handle_incoming_friend_req] glusterd: Received probe from uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e
[2010-10-03 08:47:48.651704] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 4
[2010-10-03 08:47:48.651743] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 7
[2010-10-03 08:47:48.651769] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 4
[2010-10-03 08:47:48.651799] I [glusterd-handler.c:2569:glusterd_xfer_friend_add_resp] glusterd: Responded to 10.192.141.187 (0), ret: 0
[2010-10-03 08:47:48.651840] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 4 to 4
[2010-10-03 08:47:48.651968] I [glusterd3_1-mops.c:964:glusterd3_1_friend_update] : Added uuid: bd364a85-e8b3-43d6-8a88-363df38be63c, host: 10.192.134.144
[2010-10-03 08:47:48.651989] I [glusterd3_1-mops.c:964:glusterd3_1_friend_update] : Added uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e, host: 10.198.110.16
[2010-10-03 08:47:48.652053] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 4 to 3
[2010-10-03 08:47:48.652136] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:47:48.659452] I [glusterd3_1-mops.c:371:glusterd3_1_friend_update_cbk] glusterd: Received ACC from uuid: 
[2010-10-03 08:47:48.669955] I [glusterd-handler.c:2146:glusterd_handle_friend_update] glusterd: Received friend update from uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e
[2010-10-03 08:47:48.670020] I [glusterd-handler.c:2191:glusterd_handle_friend_update] : Received uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe, hostname:10.192.141.187
[2010-10-03 08:47:48.670040] I [glusterd-handler.c:2194:glusterd_handle_friend_update] : Received my uuid as Friend
[2010-10-03 08:47:48.670057] I [glusterd-handler.c:2191:glusterd_handle_friend_update] : Received uuid: bd364a85-e8b3-43d6-8a88-363df38be63c, hostname:10.192.134.144
[2010-10-03 08:47:48.670072] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:47:48.670110] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:47:48.670906] I [glusterd3_1-mops.c:371:glusterd3_1_friend_update_cbk] glusterd: Received ACC from uuid: 
[2010-10-03 08:47:55.331443] I [glusterd-handler.c:531:glusterd_handle_cli_probe] glusterd: Received CLI probe req 10.214.231.112 6969
[2010-10-03 08:47:55.333106] I [glusterd-handler.c:366:glusterd_friend_find] glusterd: Unable to find hostname: 10.214.231.112
[2010-10-03 08:47:55.333126] I [glusterd-handler.c:2438:glusterd_probe_begin] glusterd: Unable to find peerinfo for host: 10.214.231.112 (6969)
[2010-10-03 08:47:55.335673] W [rpc-transport.c:841:rpc_transport_load] rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2010-10-03 08:47:55.338047] I [glusterd-handler.c:2419:glusterd_friend_add] glusterd: connect returned 0
[2010-10-03 08:47:55.338491] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 11
[2010-10-03 08:47:55.338523] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.214.231.112 found.. state: 0
[2010-10-03 08:47:55.338565] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 0 to 0
[2010-10-03 08:47:55.341155] I [glusterd3_1-mops.c:82:glusterd3_1_probe_cbk] glusterd: Received probe resp from uuid: 71042154-2a52-4796-a246-1120731a2fa4, host: 10.214.231.112
[2010-10-03 08:47:55.341180] I [glusterd-handler.c:354:glusterd_friend_find] glusterd: Unable to find peer by uuid
[2010-10-03 08:47:55.341195] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.214.231.112 found.. state: 0
[2010-10-03 08:47:55.341212] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 2
[2010-10-03 08:47:55.341291] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 0 to 1
[2010-10-03 08:47:55.341409] I [glusterd3_1-mops.c:139:glusterd3_1_probe_cbk] glusterd: Received resp to probe req
[2010-10-03 08:47:55.348316] I [rpc-clnt.c:691:rpc_clnt_handle_cbk] rpc-clnt: recieved rpc message (XID: 0x2a, Ver: 2, Program: 52743234, ProgVers: 1, Proc: 1) from rpc-transport (management)
[2010-10-03 08:47:55.348380] I [glusterd3_1-mops.c:186:glusterd3_1_friend_add_cbk] glusterd: Received ACC from uuid: 71042154-2a52-4796-a246-1120731a2fa4, host: 10.214.231.112, port: 0
[2010-10-03 08:47:55.348421] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 1
[2010-10-03 08:47:55.348439] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:47:55.348491] I [glusterd-handler.c:2592:glusterd_xfer_cli_probe_resp] glusterd: Responded to CLI, ret: 0
[2010-10-03 08:47:55.348509] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 1 to 4
[2010-10-03 08:47:55.349018] I [glusterd-handler.c:2249:glusterd_handle_probe_query] glusterd: Received probe from uuid: 71042154-2a52-4796-a246-1120731a2fa4
[2010-10-03 08:47:55.349041] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 4
[2010-10-03 08:47:55.349075] I [glusterd-handler.c:2272:glusterd_handle_probe_query] glusterd: Responded to 10.192.141.187, op_ret: 0, op_errno: 0, ret: 0
[2010-10-03 08:47:55.349098] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:47:55.349626] I [glusterd-handler.c:2028:glusterd_handle_incoming_friend_req] glusterd: Received probe from uuid: 71042154-2a52-4796-a246-1120731a2fa4
[2010-10-03 08:47:55.349692] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 4
[2010-10-03 08:47:55.349714] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 7
[2010-10-03 08:47:55.349738] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 4
[2010-10-03 08:47:55.349770] I [glusterd-handler.c:2569:glusterd_xfer_friend_add_resp] glusterd: Responded to 10.192.141.187 (0), ret: 0
[2010-10-03 08:47:55.349790] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 4 to 4
[2010-10-03 08:47:55.349884] I [glusterd3_1-mops.c:964:glusterd3_1_friend_update] : Added uuid: bd364a85-e8b3-43d6-8a88-363df38be63c, host: 10.192.134.144
[2010-10-03 08:47:55.349905] I [glusterd3_1-mops.c:964:glusterd3_1_friend_update] : Added uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e, host: 10.198.110.16
[2010-10-03 08:47:55.349924] I [glusterd3_1-mops.c:964:glusterd3_1_friend_update] : Added uuid: 71042154-2a52-4796-a246-1120731a2fa4, host: 10.214.231.112
[2010-10-03 08:47:55.350006] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 4 to 3
[2010-10-03 08:47:55.350092] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:47:55.350305] I [glusterd-handler.c:2146:glusterd_handle_friend_update] glusterd: Received friend update from uuid: 71042154-2a52-4796-a246-1120731a2fa4
[2010-10-03 08:47:55.350333] I [glusterd-handler.c:2191:glusterd_handle_friend_update] : Received uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe, hostname:10.192.141.187
[2010-10-03 08:47:55.350354] I [glusterd-handler.c:2194:glusterd_handle_friend_update] : Received my uuid as Friend
[2010-10-03 08:47:55.350385] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:47:55.356573] I [glusterd3_1-mops.c:371:glusterd3_1_friend_update_cbk] glusterd: Received ACC from uuid: 
[2010-10-03 08:47:55.366917] I [glusterd3_1-mops.c:371:glusterd3_1_friend_update_cbk] glusterd: Received ACC from uuid: 
[2010-10-03 08:47:55.387203] I [glusterd3_1-mops.c:371:glusterd3_1_friend_update_cbk] glusterd: Received ACC from uuid: 
[2010-10-03 08:48:02.933690] I [glusterd-op-sm.c:4901:glusterd_op_set_cli_op] : Returning 0
[2010-10-03 08:48:02.933871] I [glusterd-handler.c:1043:glusterd_handle_add_brick] glusterd: Received add brick req
[2010-10-03 08:48:02.934014] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 3
[2010-10-03 08:48:02.934145] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.214.231.112 found.. state: 3
[2010-10-03 08:48:02.934269] I [glusterd-utils.c:222:glusterd_lock] glusterd: Cluster lock held by 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:02.934310] I [glusterd-handler.c:2636:glusterd_op_txn_begin] glusterd: Acquired local lock
[2010-10-03 08:48:02.934328] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 1
[2010-10-03 08:48:02.934343] I [glusterd-handler.c:2640:glusterd_op_txn_begin] glusterd: Returning 0
[2010-10-03 08:48:02.934696] I [glusterd3_1-mops.c:1054:glusterd3_1_cluster_lock] glusterd: Sent lock req to 3 peers
[2010-10-03 08:48:02.934715] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 0 to 1 due to event 1
[2010-10-03 08:48:02.934732] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:02.935421] I [glusterd3_1-mops.c:410:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:48:02.935440] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:02.935456] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:02.935482] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 1 due to event 3
[2010-10-03 08:48:02.935511] I [glusterd3_1-mops.c:410:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e
[2010-10-03 08:48:02.935530] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:02.935545] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:02.935577] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 1 due to event 3
[2010-10-03 08:48:02.936172] I [glusterd3_1-mops.c:410:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: 71042154-2a52-4796-a246-1120731a2fa4
[2010-10-03 08:48:02.936200] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:02.936215] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:02.936231] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 4
[2010-10-03 08:48:02.936245] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 1 due to event 3
[2010-10-03 08:48:02.936271] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.198.110.16:/mnt/oct3
[2010-10-03 08:48:02.936288] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 3
[2010-10-03 08:48:02.936308] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 3
[2010-10-03 08:48:02.936324] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.214.231.112:/mnt/oct3
[2010-10-03 08:48:02.936340] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.214.231.112 found.. state: 3
[2010-10-03 08:48:02.936359] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.214.231.112 found.. state: 3
[2010-10-03 08:48:02.936426] I [glusterd3_1-mops.c:1195:glusterd3_1_stage_op] glusterd: Sent op req to 3 peers
[2010-10-03 08:48:02.936443] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 3 due to event 4
[2010-10-03 08:48:02.937554] I [glusterd3_1-mops.c:533:glusterd3_1_stage_op_cbk] glusterd: Received RJT from uuid: 71042154-2a52-4796-a246-1120731a2fa4
[2010-10-03 08:48:02.937574] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:02.937590] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 7
[2010-10-03 08:48:02.937666] I [glusterd3_1-mops.c:1108:glusterd3_1_cluster_unlock] glusterd: Sent unlock req to 3 peers
[2010-10-03 08:48:02.937682] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 3 to 7 due to event 7
[2010-10-03 08:48:02.937715] I [glusterd3_1-mops.c:533:glusterd3_1_stage_op_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:48:02.937742] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:02.937762] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:02.937777] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 7 due to event 3
[2010-10-03 08:48:02.938747] I [glusterd3_1-mops.c:471:glusterd3_1_cluster_unlock_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:48:02.938775] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:02.938791] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:02.938806] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 7 due to event 3
[2010-10-03 08:48:02.938834] I [glusterd3_1-mops.c:471:glusterd3_1_cluster_unlock_cbk] glusterd: Received ACC from uuid: 71042154-2a52-4796-a246-1120731a2fa4
[2010-10-03 08:48:02.938850] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:02.938865] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:02.938880] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 4
[2010-10-03 08:48:02.938895] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 7 due to event 3
[2010-10-03 08:48:02.938912] I [glusterd-op-sm.c:4277:glusterd_op_txn_complete] glusterd: Cleared local lock
[2010-10-03 08:48:02.938947] I [glusterd-op-sm.c:4241:glusterd_op_send_cli_response] : Returning 0
[2010-10-03 08:48:02.938966] I [glusterd-op-sm.c:4320:glusterd_op_txn_complete] glusterd: Returning 0
[2010-10-03 08:48:02.938981] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 0 due to event 4
[2010-10-03 08:48:02.939360] I [glusterd3_1-mops.c:533:glusterd3_1_stage_op_cbk] glusterd: Received RJT from uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e
[2010-10-03 08:48:02.939379] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:02.939396] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 7
[2010-10-03 08:48:02.939446] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 0 to 0 due to event 7
[2010-10-03 08:48:02.941048] I [glusterd3_1-mops.c:471:glusterd3_1_cluster_unlock_cbk] glusterd: Received ACC from uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e
[2010-10-03 08:48:02.941069] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:02.941085] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:02.941112] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 0 to 0 due to event 3
[2010-10-03 08:48:04.391591] I [glusterd-op-sm.c:4901:glusterd_op_set_cli_op] : Returning 0
[2010-10-03 08:48:04.391670] I [glusterd-handler.c:1043:glusterd_handle_add_brick] glusterd: Received add brick req
[2010-10-03 08:48:04.391930] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 3
[2010-10-03 08:48:04.392068] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.214.231.112 found.. state: 3
[2010-10-03 08:48:04.392360] I [glusterd-utils.c:222:glusterd_lock] glusterd: Cluster lock held by 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:04.392380] I [glusterd-handler.c:2636:glusterd_op_txn_begin] glusterd: Acquired local lock
[2010-10-03 08:48:04.392438] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 1
[2010-10-03 08:48:04.392454] I [glusterd-handler.c:2640:glusterd_op_txn_begin] glusterd: Returning 0
[2010-10-03 08:48:04.392595] I [glusterd3_1-mops.c:1054:glusterd3_1_cluster_lock] glusterd: Sent lock req to 3 peers
[2010-10-03 08:48:04.392628] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 0 to 1 due to event 1
[2010-10-03 08:48:04.392666] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:04.393937] I [glusterd3_1-mops.c:410:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e
[2010-10-03 08:48:04.393960] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:04.393976] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:04.393992] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 1 due to event 3
[2010-10-03 08:48:04.394107] I [glusterd3_1-mops.c:410:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: 71042154-2a52-4796-a246-1120731a2fa4
[2010-10-03 08:48:04.394124] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:04.394140] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:04.394155] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 1 due to event 3
[2010-10-03 08:48:04.394182] I [glusterd3_1-mops.c:410:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:48:04.394293] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:04.394309] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:04.394324] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 4
[2010-10-03 08:48:04.394339] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 1 due to event 3
[2010-10-03 08:48:04.394365] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.198.110.16:/mnt/oct3
[2010-10-03 08:48:04.394392] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 3
[2010-10-03 08:48:04.394418] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 3
[2010-10-03 08:48:04.394434] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.214.231.112:/mnt/oct3
[2010-10-03 08:48:04.394450] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.214.231.112 found.. state: 3
[2010-10-03 08:48:04.394469] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.214.231.112 found.. state: 3
[2010-10-03 08:48:04.394536] I [glusterd3_1-mops.c:1195:glusterd3_1_stage_op] glusterd: Sent op req to 3 peers
[2010-10-03 08:48:04.394577] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 3 due to event 4
[2010-10-03 08:48:04.396090] I [glusterd3_1-mops.c:533:glusterd3_1_stage_op_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:48:04.396111] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:04.396129] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:04.396145] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 3 to 3 due to event 3
[2010-10-03 08:48:04.398039] I [glusterd3_1-mops.c:533:glusterd3_1_stage_op_cbk] glusterd: Received ACC from uuid: 71042154-2a52-4796-a246-1120731a2fa4
[2010-10-03 08:48:04.398061] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:04.398078] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:04.398094] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 3 to 3 due to event 3
[2010-10-03 08:48:04.398123] I [glusterd3_1-mops.c:533:glusterd3_1_stage_op_cbk] glusterd: Received ACC from uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e
[2010-10-03 08:48:04.398140] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:04.398168] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:04.398185] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 5
[2010-10-03 08:48:04.398200] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 3 to 3 due to event 3
[2010-10-03 08:48:04.398232] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 3
[2010-10-03 08:48:04.398259] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.214.231.112 found.. state: 3
[2010-10-03 08:48:04.399657] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.198.110.16:/mnt/oct3
[2010-10-03 08:48:04.399685] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 3
[2010-10-03 08:48:04.399702] I [glusterd-utils.c:696:glusterd_brickinfo_get] : Found brick
[2010-10-03 08:48:04.399717] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.214.231.112:/mnt/oct3
[2010-10-03 08:48:04.399733] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.214.231.112 found.. state: 3
[2010-10-03 08:48:04.399749] I [glusterd-utils.c:696:glusterd_brickinfo_get] : Found brick
[2010-10-03 08:48:04.399817] E [glusterd-store.c:215:glusterd_store_delete_brick] : Unlink failed on /etc/glusterd/vols/dhtqa39/bricks/10.198.110.16:-mnt-oct3
[2010-10-03 08:48:04.399841] E [glusterd-store.c:215:glusterd_store_delete_brick] : Unlink failed on /etc/glusterd/vols/dhtqa39/bricks/10.214.231.112:-mnt-oct3
[2010-10-03 08:48:04.400862] I [glusterd-utils.c:805:glusterd_service_stop] : Stopping gluster nfsd running in pid: 17528
[2010-10-03 08:48:05.522319] I [glusterd3_1-mops.c:1285:glusterd3_1_commit_op] glusterd: Sent op req to 3 peers
[2010-10-03 08:48:05.522419] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 3 to 5 due to event 5
[2010-10-03 08:48:05.523700] I [rpc-clnt.c:691:rpc_clnt_handle_cbk] rpc-clnt: recieved rpc message (XID: 0x2a, Ver: 2, Program: 52743234, ProgVers: 1, Proc: 1) from rpc-transport (management)
[2010-10-03 08:48:05.524361] I [rpc-clnt.c:691:rpc_clnt_handle_cbk] rpc-clnt: recieved rpc message (XID: 0x2a, Ver: 2, Program: 52743234, ProgVers: 1, Proc: 1) from rpc-transport (management)
[2010-10-03 08:48:05.524491] I [rpc-clnt.c:691:rpc_clnt_handle_cbk] rpc-clnt: recieved rpc message (XID: 0x2a, Ver: 2, Program: 52743234, ProgVers: 1, Proc: 1) from rpc-transport (management)
[2010-10-03 08:48:06.238008] I [glusterd3_1-mops.c:680:glusterd3_1_commit_op_cbk] glusterd: Received ACC from uuid: 71042154-2a52-4796-a246-1120731a2fa4
[2010-10-03 08:48:06.238050] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:06.238068] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:06.238086] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 5 to 5 due to event 3
[2010-10-03 08:48:06.257916] I [glusterd3_1-mops.c:680:glusterd3_1_commit_op_cbk] glusterd: Received ACC from uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e
[2010-10-03 08:48:06.257950] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:06.257966] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:06.257981] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 5 to 5 due to event 3
[2010-10-03 08:48:06.664684] I [glusterd3_1-mops.c:680:glusterd3_1_commit_op_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:48:06.664753] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:06.664770] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:06.664788] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 6
[2010-10-03 08:48:06.664823] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 5 to 5 due to event 3
[2010-10-03 08:48:06.664917] I [glusterd3_1-mops.c:1108:glusterd3_1_cluster_unlock] glusterd: Sent unlock req to 3 peers
[2010-10-03 08:48:06.664934] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 5 to 7 due to event 6
[2010-10-03 08:48:06.665269] I [glusterd3_1-mops.c:471:glusterd3_1_cluster_unlock_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:48:06.665293] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:06.665323] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:06.665339] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 7 due to event 3
[2010-10-03 08:48:06.665407] I [glusterd3_1-mops.c:471:glusterd3_1_cluster_unlock_cbk] glusterd: Received ACC from uuid: 71042154-2a52-4796-a246-1120731a2fa4
[2010-10-03 08:48:06.665429] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:06.665449] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:06.665465] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 7 due to event 3
[2010-10-03 08:48:06.665519] I [glusterd3_1-mops.c:471:glusterd3_1_cluster_unlock_cbk] glusterd: Received ACC from uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e
[2010-10-03 08:48:06.665540] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:06.665555] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:06.665571] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 4
[2010-10-03 08:48:06.665585] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 7 due to event 3
[2010-10-03 08:48:06.665604] I [glusterd-op-sm.c:4277:glusterd_op_txn_complete] glusterd: Cleared local lock
[2010-10-03 08:48:06.665662] I [glusterd-op-sm.c:4241:glusterd_op_send_cli_response] : Returning 0
[2010-10-03 08:48:06.665682] I [glusterd-op-sm.c:4320:glusterd_op_txn_complete] glusterd: Returning 0
[2010-10-03 08:48:06.665698] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 0 due to event 4
[2010-10-03 08:48:08.658121] I [glusterd-op-sm.c:4901:glusterd_op_set_cli_op] : Returning 0
[2010-10-03 08:48:08.658238] I [glusterd-handler.c:1043:glusterd_handle_add_brick] glusterd: Received add brick req
[2010-10-03 08:48:08.658393] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 3
[2010-10-03 08:48:08.658592] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.214.231.112 found.. state: 3
[2010-10-03 08:48:08.658786] I [glusterd-utils.c:222:glusterd_lock] glusterd: Cluster lock held by 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:08.658802] I [glusterd-handler.c:2636:glusterd_op_txn_begin] glusterd: Acquired local lock
[2010-10-03 08:48:08.658818] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 1
[2010-10-03 08:48:08.658833] I [glusterd-handler.c:2640:glusterd_op_txn_begin] glusterd: Returning 0
[2010-10-03 08:48:08.658967] I [glusterd3_1-mops.c:1054:glusterd3_1_cluster_lock] glusterd: Sent lock req to 3 peers
[2010-10-03 08:48:08.658992] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 0 to 1 due to event 1
[2010-10-03 08:48:08.659026] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:08.659683] I [glusterd3_1-mops.c:410:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e
[2010-10-03 08:48:08.659701] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:08.659720] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:08.659801] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 1 due to event 3
[2010-10-03 08:48:08.659846] I [glusterd3_1-mops.c:410:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:48:08.659862] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:08.659877] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:08.659892] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 1 due to event 3
[2010-10-03 08:48:08.659920] I [glusterd3_1-mops.c:410:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: 71042154-2a52-4796-a246-1120731a2fa4
[2010-10-03 08:48:08.659936] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:08.659950] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:08.659965] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 4
[2010-10-03 08:48:08.659980] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 1 due to event 3
[2010-10-03 08:48:08.660007] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.198.110.16:/mnt/oct3
[2010-10-03 08:48:08.660025] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 3
[2010-10-03 08:48:08.660041] I [glusterd-utils.c:696:glusterd_brickinfo_get] : Found brick
[2010-10-03 08:48:08.660056] E [glusterd-op-sm.c:718:glusterd_op_stage_add_brick] : Adding duplicate brick: 10.198.110.16:/mnt/oct3
[2010-10-03 08:48:08.660073] E [glusterd3_1-mops.c:1164:glusterd3_1_stage_op] : Staging failed
[2010-10-03 08:48:08.660088] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 7
[2010-10-03 08:48:08.660103] E [glusterd-op-sm.c:4819:glusterd_op_sm] glusterd: handler returned: -1
[2010-10-03 08:48:08.660166] I [glusterd3_1-mops.c:1108:glusterd3_1_cluster_unlock] glusterd: Sent unlock req to 3 peers
[2010-10-03 08:48:08.660183] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 7 due to event 7
[2010-10-03 08:48:08.660542] I [glusterd3_1-mops.c:471:glusterd3_1_cluster_unlock_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:48:08.660568] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:08.660589] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:08.660648] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 7 due to event 3
[2010-10-03 08:48:08.660679] I [glusterd3_1-mops.c:471:glusterd3_1_cluster_unlock_cbk] glusterd: Received ACC from uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e
[2010-10-03 08:48:08.660694] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:08.660710] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:08.660725] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 7 due to event 3
[2010-10-03 08:48:08.660752] I [glusterd3_1-mops.c:471:glusterd3_1_cluster_unlock_cbk] glusterd: Received ACC from uuid: 71042154-2a52-4796-a246-1120731a2fa4
[2010-10-03 08:48:08.660767] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:08.660782] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:08.660797] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 4
[2010-10-03 08:48:08.660812] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 7 due to event 3
[2010-10-03 08:48:08.660830] I [glusterd-op-sm.c:4277:glusterd_op_txn_complete] glusterd: Cleared local lock
[2010-10-03 08:48:08.660891] I [glusterd-op-sm.c:4241:glusterd_op_send_cli_response] : Returning 0
[2010-10-03 08:48:08.660910] I [glusterd-op-sm.c:4320:glusterd_op_txn_complete] glusterd: Returning 0
[2010-10-03 08:48:08.660925] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 0 due to event 4
[2010-10-03 08:48:56.789130] I [glusterd-handler.c:675:glusterd_handle_cli_get_volume] glusterd: Received get vol req
[2010-10-03 08:48:56.789250] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:56.789422] I [glusterd-handler.c:675:glusterd_handle_cli_get_volume] glusterd: Received get vol req
[2010-10-03 08:48:56.789471] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
=============================

Comment 2 Amar Tumballi 2010-10-04 02:14:57 UTC
Shishir, Can you make out anything from the logs? Lets work on this today, as 'add-brick' is the most highlighted feature of 3.1 release.

Comment 3 Amar Tumballi 2010-10-04 03:59:49 UTC
Below log is the exact reason why its failed..

[glusterd-op-sm.c:718:glusterd_op_stage_add_brick] : Adding duplicate brick:
10.198.110.16:/mnt/oct3
[2010-10-03 08:48:02.720547] E [glusterd-op-sm.c:4370:glusterd_op_ac_stage_op]
: Validate failed: -1


Need to see why its saying a duplicate brick.

Amar

Comment 4 Pranith Kumar K 2010-10-05 03:19:08 UTC
fixed as the commit 2420fd1f0cc5c357776d3766884b86859f3c3781. Not sure why it is not updated in the bug.

Moving to resolved.