Bug 763514 - (GLUSTER-1782) add-brick says unsuccessful when its successful - qa39
add-brick says unsuccessful when its successful - qa39
Status: CLOSED CURRENTRELEASE
Product: GlusterFS
Classification: Community
Component: cli (Show other bugs)
3.1-alpha
All Linux
low Severity medium
: ---
: ---
Assigned To: Pranith Kumar K
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2010-10-03 08:59 EDT by Lakshmipathi G
Modified: 2015-12-01 11:45 EST (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: ---
Regression: RTP
Mount Type: All
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Lakshmipathi G 2010-10-03 06:00:08 EDT
+++++++++++
log file of added-brick3
++++++++++++

10.214.231.112#cat /usr/local/var/log/glusterfs/usr-local-etc-glusterfs-glusterd.vol.log
[2010-10-03 08:46:29.866158] I [glusterd.c:276:init] management: Using /etc/glusterd as working directory
[2010-10-03 08:46:29.877279] E [rpc-transport.c:897:rpc_transport_load] rpc-transport: /usr/local/lib/glusterfs/3.1.0qa39/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
[2010-10-03 08:46:29.877299] E [rpc-transport.c:901:rpc_transport_load] rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine
[2010-10-03 08:46:29.877375] I [glusterd.c:95:glusterd_uuid_init] glusterd: generated UUID: 71042154-2a52-4796-a246-1120731a2fa4
Given volfile:
+------------------------------------------------------------------------------+
  1: volume management
  2:     type mgmt/glusterd
  3:     option working-directory /etc/glusterd
  4:     option transport-type socket,rdma
  5:     option transport.socket.listen-port 6969
  6:     option transport.rdma.listen-port 6968
  7:     # option listen-port 6969
  8: end-volume
  9: 

+------------------------------------------------------------------------------+
[2010-10-03 08:47:55.123454] I [glusterd-handler.c:2249:glusterd_handle_probe_query] glusterd: Received probe from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:47:55.123537] I [glusterd-handler.c:354:glusterd_friend_find] glusterd: Unable to find peer by uuid
[2010-10-03 08:47:55.124922] I [glusterd-handler.c:366:glusterd_friend_find] glusterd: Unable to find hostname: 10.192.141.187
[2010-10-03 08:47:55.124995] I [glusterd-handler.c:2272:glusterd_handle_probe_query] glusterd: Responded to 10.214.231.112, op_ret: 0, op_errno: 0, ret: 0
[2010-10-03 08:47:55.125022] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:47:55.125582] I [glusterd-handler.c:2028:glusterd_handle_incoming_friend_req] glusterd: Received probe from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:47:55.125628] I [glusterd-handler.c:354:glusterd_friend_find] glusterd: Unable to find peer by uuid
[2010-10-03 08:47:55.126092] I [glusterd-handler.c:366:glusterd_friend_find] glusterd: Unable to find hostname: 10.192.141.187
[2010-10-03 08:47:55.126111] I [glusterd-handler.c:119:glusterd_handle_friend_req] glusterd:  for host: 10.192.141.187 (6969)
[2010-10-03 08:47:55.128604] W [rpc-transport.c:841:rpc_transport_load] rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2010-10-03 08:47:55.130949] I [glusterd-handler.c:2419:glusterd_friend_add] glusterd: connect returned 0
[2010-10-03 08:47:55.130973] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 7
[2010-10-03 08:47:55.132038] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 4
[2010-10-03 08:47:55.132079] I [glusterd-handler.c:2569:glusterd_xfer_friend_add_resp] glusterd: Responded to 10.214.231.112 (0), ret: 0
[2010-10-03 08:47:55.132100] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 0 to 2
[2010-10-03 08:47:55.132191] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 1
[2010-10-03 08:47:55.132207] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 2 to 2
[2010-10-03 08:47:55.132297] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.192.141.187 found.. state: 2
[2010-10-03 08:47:55.132347] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 2 to 2
[2010-10-03 08:47:55.133306] I [glusterd3_1-mops.c:82:glusterd3_1_probe_cbk] glusterd: Received probe resp from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe, host: 10.192.141.187
[2010-10-03 08:47:55.133324] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 2
[2010-10-03 08:47:55.133340] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 2
[2010-10-03 08:47:55.133409] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 2 to 5
[2010-10-03 08:47:55.133490] I [glusterd3_1-mops.c:139:glusterd3_1_probe_cbk] glusterd: Received resp to probe req
[2010-10-03 08:47:55.134035] I [glusterd3_1-mops.c:186:glusterd3_1_friend_add_cbk] glusterd: Received ACC from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe, host: 10.192.141.187, port: 0
[2010-10-03 08:47:55.134053] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 5
[2010-10-03 08:47:55.134069] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:47:55.134092] I [glusterd3_1-mops.c:964:glusterd3_1_friend_update] : Added uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe, host: 10.192.141.187
[2010-10-03 08:47:55.134129] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 5 to 3
[2010-10-03 08:47:55.134291] I [glusterd-handler.c:2146:glusterd_handle_friend_update] glusterd: Received friend update from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:47:55.134321] I [glusterd-handler.c:2191:glusterd_handle_friend_update] : Received uuid: bd364a85-e8b3-43d6-8a88-363df38be63c, hostname:10.192.134.144
[2010-10-03 08:47:55.134337] I [glusterd-handler.c:354:glusterd_friend_find] glusterd: Unable to find peer by uuid
[2010-10-03 08:47:55.134829] I [glusterd-handler.c:366:glusterd_friend_find] glusterd: Unable to find hostname: 10.192.134.144
[2010-10-03 08:47:55.137360] W [rpc-transport.c:841:rpc_transport_load] rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2010-10-03 08:47:55.139526] I [glusterd-handler.c:2419:glusterd_friend_add] glusterd: connect returned 0
[2010-10-03 08:47:55.139547] I [glusterd-handler.c:2191:glusterd_handle_friend_update] : Received uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e, hostname:10.198.110.16
[2010-10-03 08:47:55.139563] I [glusterd-handler.c:354:glusterd_friend_find] glusterd: Unable to find peer by uuid
[2010-10-03 08:47:55.160065] I [glusterd-handler.c:366:glusterd_friend_find] glusterd: Unable to find hostname: 10.198.110.16
[2010-10-03 08:47:55.162677] W [rpc-transport.c:841:rpc_transport_load] rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2010-10-03 08:47:55.164843] I [glusterd-handler.c:2419:glusterd_friend_add] glusterd: connect returned 0
[2010-10-03 08:47:55.164864] I [glusterd-handler.c:2191:glusterd_handle_friend_update] : Received uuid: 71042154-2a52-4796-a246-1120731a2fa4, hostname:10.214.231.112
[2010-10-03 08:47:55.164880] I [glusterd-handler.c:2194:glusterd_handle_friend_update] : Received my uuid as Friend
[2010-10-03 08:47:55.164934] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:47:55.165027] I [glusterd3_1-mops.c:371:glusterd3_1_friend_update_cbk] glusterd: Received ACC from uuid: 
[2010-10-03 08:48:02.719468] I [glusterd-handler.c:394:glusterd_handle_cluster_lock] glusterd: Received LOCK from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:02.719542] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 2
[2010-10-03 08:48:02.719572] I [glusterd-handler.c:410:glusterd_handle_cluster_lock] : Returning 0
[2010-10-03 08:48:02.719596] I [glusterd-utils.c:222:glusterd_lock] glusterd: Cluster lock held by 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:02.719650] I [glusterd-handler.c:1884:glusterd_op_lock_send_resp] glusterd: Responded, ret: 0
[2010-10-03 08:48:02.719667] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 0 to 2 due to event 2
[2010-10-03 08:48:02.719684] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:02.720382] I [glusterd-handler.c:433:glusterd_handle_stage_op] glusterd: Received stage op from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:02.720400] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 8
[2010-10-03 08:48:02.720437] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.198.110.16:/mnt/oct3
[2010-10-03 08:48:02.720457] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 3
[2010-10-03 08:48:02.720474] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.192.134.144 found.. state: 3
[2010-10-03 08:48:02.720513] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.192.141.187 found.. state: 3
[2010-10-03 08:48:02.720529] E [glusterd-op-sm.c:718:glusterd_op_stage_add_brick] : Adding duplicate brick: 10.198.110.16:/mnt/oct3
[2010-10-03 08:48:02.720547] E [glusterd-op-sm.c:4370:glusterd_op_ac_stage_op] : Validate failed: -1
[2010-10-03 08:48:02.720584] I [glusterd-handler.c:1966:glusterd_op_stage_send_resp] glusterd: Responded to stage, ret: 0
[2010-10-03 08:48:02.720600] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 2 to 4 due to event 8
[2010-10-03 08:48:02.720620] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:02.721887] I [glusterd-handler.c:1928:glusterd_handle_cluster_unlock] glusterd: Received UNLOCK from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:02.721904] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 10
[2010-10-03 08:48:02.721935] I [glusterd-handler.c:1904:glusterd_op_unlock_send_resp] glusterd: Responded to unlock, ret: 0
[2010-10-03 08:48:02.721951] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 4 to 0 due to event 10
[2010-10-03 08:48:02.721972] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:04.176608] I [glusterd-handler.c:394:glusterd_handle_cluster_lock] glusterd: Received LOCK from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:04.176673] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 2
[2010-10-03 08:48:04.176689] I [glusterd-handler.c:410:glusterd_handle_cluster_lock] : Returning 0
[2010-10-03 08:48:04.176709] I [glusterd-utils.c:222:glusterd_lock] glusterd: Cluster lock held by 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:04.176760] I [glusterd-handler.c:1884:glusterd_op_lock_send_resp] glusterd: Responded, ret: 0
[2010-10-03 08:48:04.176777] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 0 to 2 due to event 2
[2010-10-03 08:48:04.176794] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:04.178598] I [glusterd-handler.c:433:glusterd_handle_stage_op] glusterd: Received stage op from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:04.178616] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 8
[2010-10-03 08:48:04.178645] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.198.110.16:/mnt/oct3
[2010-10-03 08:48:04.178665] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 3
[2010-10-03 08:48:04.178691] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 3
[2010-10-03 08:48:04.178707] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.214.231.112:/mnt/oct3
[2010-10-03 08:48:04.180401] I [glusterd-handler.c:1966:glusterd_op_stage_send_resp] glusterd: Responded to stage, ret: 0
[2010-10-03 08:48:04.180454] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 2 to 4 due to event 8
[2010-10-03 08:48:04.180473] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:04.183634] I [rpc-clnt.c:691:rpc_clnt_handle_cbk] rpc-clnt: recieved rpc message (XID: 0x2a, Ver: 2, Program: 52743234, ProgVers: 1, Proc: 1) from rpc-transport (management)
[2010-10-03 08:48:05.306287] I [glusterd-handler.c:483:glusterd_handle_commit_op] glusterd: Received commit op from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:05.306321] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 9
[2010-10-03 08:48:05.306360] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 3
[2010-10-03 08:48:05.307983] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.198.110.16:/mnt/oct3
[2010-10-03 08:48:05.308004] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 3
[2010-10-03 08:48:05.308037] I [glusterd-utils.c:696:glusterd_brickinfo_get] : Found brick
[2010-10-03 08:48:05.308072] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.214.231.112:/mnt/oct3
[2010-10-03 08:48:05.308659] I [glusterd-utils.c:696:glusterd_brickinfo_get] : Found brick
[2010-10-03 08:48:05.884572] I [glusterd-utils.c:906:glusterd_volume_start_glusterfs] : About to start glusterfs for brick 10.214.231.112:/mnt/oct3
[2010-10-03 08:48:05.955017] E [glusterd-store.c:215:glusterd_store_delete_brick] : Unlink failed on /etc/glusterd/vols/dhtqa39/bricks/10.198.110.16:-mnt-oct3
[2010-10-03 08:48:05.955079] E [glusterd-store.c:215:glusterd_store_delete_brick] : Unlink failed on /etc/glusterd/vols/dhtqa39/bricks/10.214.231.112:-mnt-oct3
[2010-10-03 08:48:06.21384] I [glusterd-handler.c:2003:glusterd_op_commit_send_resp] glusterd: Responded to commit, ret: 0
[2010-10-03 08:48:06.21430] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 4 to 6 due to event 9
[2010-10-03 08:48:06.21470] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:06.21571] I [rpc-clnt.c:691:rpc_clnt_handle_cbk] rpc-clnt: recieved rpc message (XID: 0x2a, Ver: 2, Program: 52743234, ProgVers: 1, Proc: 1) from rpc-transport (management)
[2010-10-03 08:48:06.21604] I [rpc-clnt.c:691:rpc_clnt_handle_cbk] rpc-clnt: recieved rpc message (XID: 0x2a, Ver: 2, Program: 52743234, ProgVers: 1, Proc: 1) from rpc-transport (management)
[2010-10-03 08:48:06.24515] I [glusterd-pmap.c:237:pmap_registry_bind] pmap: adding brick /mnt/oct3 on port 6971
[2010-10-03 08:48:06.448820] I [glusterd-handler.c:1928:glusterd_handle_cluster_unlock] glusterd: Received UNLOCK from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:06.448852] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 10
[2010-10-03 08:48:06.448889] I [glusterd-handler.c:1904:glusterd_op_unlock_send_resp] glusterd: Responded to unlock, ret: 0
[2010-10-03 08:48:06.448922] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 6 to 0 due to event 10
[2010-10-03 08:48:06.448943] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:08.442951] I [glusterd-handler.c:394:glusterd_handle_cluster_lock] glusterd: Received LOCK from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:08.443004] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 2
[2010-10-03 08:48:08.443020] I [glusterd-handler.c:410:glusterd_handle_cluster_lock] : Returning 0
[2010-10-03 08:48:08.443040] I [glusterd-utils.c:222:glusterd_lock] glusterd: Cluster lock held by 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:08.443093] I [glusterd-handler.c:1884:glusterd_op_lock_send_resp] glusterd: Responded, ret: 0
[2010-10-03 08:48:08.443111] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 0 to 2 due to event 2
[2010-10-03 08:48:08.443128] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:08.443956] I [glusterd-handler.c:1928:glusterd_handle_cluster_unlock] glusterd: Received UNLOCK from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:08.443974] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 10
[2010-10-03 08:48:08.444006] I [glusterd-handler.c:1904:glusterd_op_unlock_send_resp] glusterd: Responded to unlock, ret: 0
[2010-10-03 08:48:08.444022] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 2 to 0 due to event 10
[2010-10-03 08:48:08.444042] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
=====
+++++++++++
log file of added-brick3
++++++++++++
[root@domU-12-31-39-10-6D-E2 ~]# ps aux | grep gluster
root     23398  0.0  0.0   6064   604 pts/2    R+   08:34   0:00 grep gluster

root@domU-12-31-39-10-6D-E2 ~]# PS1="10.198.110.16#"

10.198.110.16#glusterd

10.198.110.16#cat /usr/local/var/log/glusterfs/usr-local-etc-glusterfs-glusterd.vol.log
[2010-10-03 08:46:35.361429] I [glusterd.c:276:init] management: Using /etc/glusterd as working directory
[2010-10-03 08:46:35.362537] E [rpc-transport.c:897:rpc_transport_load] rpc-transport: /usr/local/lib/glusterfs/3.1.0qa39/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
[2010-10-03 08:46:35.362558] E [rpc-transport.c:901:rpc_transport_load] rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine
[2010-10-03 08:46:35.362635] I [glusterd.c:95:glusterd_uuid_init] glusterd: generated UUID: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e
Given volfile:
+------------------------------------------------------------------------------+
  1: volume management
  2:     type mgmt/glusterd
  3:     option working-directory /etc/glusterd
  4:     option transport-type socket,rdma
  5:     option transport.socket.listen-port 6969
  6:     option transport.rdma.listen-port 6968
  7:     # option listen-port 6969
  8: end-volume
  9: 

+------------------------------------------------------------------------------+
[2010-10-03 08:47:48.357579] I [glusterd-handler.c:2249:glusterd_handle_probe_query] glusterd: Received probe from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:47:48.357670] I [glusterd-handler.c:354:glusterd_friend_find] glusterd: Unable to find peer by uuid
[2010-10-03 08:47:48.359831] I [glusterd-handler.c:366:glusterd_friend_find] glusterd: Unable to find hostname: 10.192.141.187
[2010-10-03 08:47:48.359984] I [glusterd-handler.c:2272:glusterd_handle_probe_query] glusterd: Responded to 10.198.110.16, op_ret: 0, op_errno: 0, ret: 0
[2010-10-03 08:47:48.360013] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:47:48.361790] I [glusterd-handler.c:2028:glusterd_handle_incoming_friend_req] glusterd: Received probe from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:47:48.361847] I [glusterd-handler.c:354:glusterd_friend_find] glusterd: Unable to find peer by uuid
[2010-10-03 08:47:48.362355] I [glusterd-handler.c:366:glusterd_friend_find] glusterd: Unable to find hostname: 10.192.141.187
[2010-10-03 08:47:48.362376] I [glusterd-handler.c:119:glusterd_handle_friend_req] glusterd:  for host: 10.192.141.187 (6969)
[2010-10-03 08:47:48.364811] W [rpc-transport.c:841:rpc_transport_load] rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2010-10-03 08:47:48.367143] I [glusterd-handler.c:2419:glusterd_friend_add] glusterd: connect returned 0
[2010-10-03 08:47:48.367167] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 7
[2010-10-03 08:47:48.378108] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 4
[2010-10-03 08:47:48.378151] I [glusterd-handler.c:2569:glusterd_xfer_friend_add_resp] glusterd: Responded to 10.198.110.16 (0), ret: 0
[2010-10-03 08:47:48.378173] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 0 to 2
[2010-10-03 08:47:48.378262] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 1
[2010-10-03 08:47:48.378279] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 2 to 2
[2010-10-03 08:47:48.378370] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.192.141.187 found.. state: 2
[2010-10-03 08:47:48.378420] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 2 to 2
[2010-10-03 08:47:48.380136] I [glusterd3_1-mops.c:82:glusterd3_1_probe_cbk] glusterd: Received probe resp from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe, host: 10.192.141.187
[2010-10-03 08:47:48.380155] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 2
[2010-10-03 08:47:48.380171] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 2
[2010-10-03 08:47:48.380240] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 2 to 5
[2010-10-03 08:47:48.380321] I [glusterd3_1-mops.c:139:glusterd3_1_probe_cbk] glusterd: Received resp to probe req
[2010-10-03 08:47:48.382310] I [glusterd-handler.c:2146:glusterd_handle_friend_update] glusterd: Received friend update from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:47:48.382339] I [glusterd-handler.c:2191:glusterd_handle_friend_update] : Received uuid: bd364a85-e8b3-43d6-8a88-363df38be63c, hostname:10.192.134.144
[2010-10-03 08:47:48.382355] I [glusterd-handler.c:354:glusterd_friend_find] glusterd: Unable to find peer by uuid
[2010-10-03 08:47:48.383843] I [glusterd-handler.c:366:glusterd_friend_find] glusterd: Unable to find hostname: 10.192.134.144
[2010-10-03 08:47:48.386207] W [rpc-transport.c:841:rpc_transport_load] rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2010-10-03 08:47:48.388461] I [glusterd-handler.c:2419:glusterd_friend_add] glusterd: connect returned 0
[2010-10-03 08:47:48.388483] I [glusterd-handler.c:2191:glusterd_handle_friend_update] : Received uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e, hostname:10.198.110.16
[2010-10-03 08:47:48.388499] I [glusterd-handler.c:2194:glusterd_handle_friend_update] : Received my uuid as Friend
[2010-10-03 08:47:48.388550] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:47:48.399191] I [glusterd3_1-mops.c:186:glusterd3_1_friend_add_cbk] glusterd: Received ACC from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe, host: 10.192.141.187, port: 0
[2010-10-03 08:47:48.399209] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 5
[2010-10-03 08:47:48.399226] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:47:48.399251] I [glusterd3_1-mops.c:964:glusterd3_1_friend_update] : Added uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe, host: 10.192.141.187
[2010-10-03 08:47:48.399271] I [glusterd3_1-mops.c:964:glusterd3_1_friend_update] : Added uuid: bd364a85-e8b3-43d6-8a88-363df38be63c, host: 10.192.134.144
[2010-10-03 08:47:48.399322] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 5 to 3
[2010-10-03 08:47:48.400121] I [glusterd3_1-mops.c:371:glusterd3_1_friend_update_cbk] glusterd: Received ACC from uuid: 
[2010-10-03 08:47:55.80015] I [glusterd-handler.c:2146:glusterd_handle_friend_update] glusterd: Received friend update from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:47:55.80101] I [glusterd-handler.c:2191:glusterd_handle_friend_update] : Received uuid: bd364a85-e8b3-43d6-8a88-363df38be63c, hostname:10.192.134.144
[2010-10-03 08:47:55.80739] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:47:55.80806] I [glusterd-handler.c:2191:glusterd_handle_friend_update] : Received uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e, hostname:10.198.110.16
[2010-10-03 08:47:55.80824] I [glusterd-handler.c:2194:glusterd_handle_friend_update] : Received my uuid as Friend
[2010-10-03 08:47:55.80841] I [glusterd-handler.c:2191:glusterd_handle_friend_update] : Received uuid: 71042154-2a52-4796-a246-1120731a2fa4, hostname:10.214.231.112
[2010-10-03 08:47:55.80856] I [glusterd-handler.c:354:glusterd_friend_find] glusterd: Unable to find peer by uuid
[2010-10-03 08:47:55.82358] I [glusterd-handler.c:366:glusterd_friend_find] glusterd: Unable to find hostname: 10.214.231.112
[2010-10-03 08:47:55.84715] W [rpc-transport.c:841:rpc_transport_load] rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2010-10-03 08:47:55.86981] I [glusterd-handler.c:2419:glusterd_friend_add] glusterd: connect returned 0
[2010-10-03 08:47:55.87046] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:02.664091] I [glusterd-handler.c:394:glusterd_handle_cluster_lock] glusterd: Received LOCK from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:02.664161] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 2
[2010-10-03 08:48:02.664191] I [glusterd-handler.c:410:glusterd_handle_cluster_lock] : Returning 0
[2010-10-03 08:48:02.664215] I [glusterd-utils.c:222:glusterd_lock] glusterd: Cluster lock held by 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:02.664296] I [glusterd-handler.c:1884:glusterd_op_lock_send_resp] glusterd: Responded, ret: 0
[2010-10-03 08:48:02.664315] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 0 to 2 due to event 2
[2010-10-03 08:48:02.664333] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:02.665965] I [glusterd-handler.c:433:glusterd_handle_stage_op] glusterd: Received stage op from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:02.665983] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 8
[2010-10-03 08:48:02.666021] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.198.110.16:/mnt/oct3
[2010-10-03 08:48:02.668227] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.192.134.144 found.. state: 3
[2010-10-03 08:48:02.668291] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.192.141.187 found.. state: 3
[2010-10-03 08:48:02.668308] E [glusterd-op-sm.c:718:glusterd_op_stage_add_brick] : Adding duplicate brick: 10.198.110.16:/mnt/oct3
[2010-10-03 08:48:02.668330] E [glusterd-op-sm.c:4370:glusterd_op_ac_stage_op] : Validate failed: -1
[2010-10-03 08:48:02.668401] I [glusterd-handler.c:1966:glusterd_op_stage_send_resp] glusterd: Responded to stage, ret: 0
[2010-10-03 08:48:02.668423] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 2 to 4 due to event 8
[2010-10-03 08:48:02.668441] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:02.669370] I [glusterd-handler.c:1928:glusterd_handle_cluster_unlock] glusterd: Received UNLOCK from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:02.669388] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 10
[2010-10-03 08:48:02.669421] I [glusterd-handler.c:1904:glusterd_op_unlock_send_resp] glusterd: Responded to unlock, ret: 0
[2010-10-03 08:48:02.669441] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 4 to 0 due to event 10
[2010-10-03 08:48:02.669457] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:04.122211] I [glusterd-handler.c:394:glusterd_handle_cluster_lock] glusterd: Received LOCK from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:04.122277] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 2
[2010-10-03 08:48:04.122293] I [glusterd-handler.c:410:glusterd_handle_cluster_lock] : Returning 0
[2010-10-03 08:48:04.122314] I [glusterd-utils.c:222:glusterd_lock] glusterd: Cluster lock held by 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:04.122365] I [glusterd-handler.c:1884:glusterd_op_lock_send_resp] glusterd: Responded, ret: 0
[2010-10-03 08:48:04.122383] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 0 to 2 due to event 2
[2010-10-03 08:48:04.122402] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:04.124202] I [glusterd-handler.c:433:glusterd_handle_stage_op] glusterd: Received stage op from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:04.124220] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 8
[2010-10-03 08:48:04.124250] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.198.110.16:/mnt/oct3
[2010-10-03 08:48:04.125792] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.214.231.112:/mnt/oct3
[2010-10-03 08:48:04.125836] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.214.231.112 found.. state: 3
[2010-10-03 08:48:04.125857] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.214.231.112 found.. state: 3
[2010-10-03 08:48:04.125919] I [glusterd-handler.c:1966:glusterd_op_stage_send_resp] glusterd: Responded to stage, ret: 0
[2010-10-03 08:48:04.125940] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 2 to 4 due to event 8
[2010-10-03 08:48:04.125980] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:04.129230] I [rpc-clnt.c:691:rpc_clnt_handle_cbk] rpc-clnt: recieved rpc message (XID: 0x2a, Ver: 2, Program: 52743234, ProgVers: 1, Proc: 1) from rpc-transport (management)
[2010-10-03 08:48:05.251887] I [glusterd-handler.c:483:glusterd_handle_commit_op] glusterd: Received commit op from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:05.251938] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 9
[2010-10-03 08:48:05.252630] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.214.231.112 found.. state: 3
[2010-10-03 08:48:05.253498] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.198.110.16:/mnt/oct3
[2010-10-03 08:48:05.254054] I [glusterd-utils.c:696:glusterd_brickinfo_get] : Found brick
[2010-10-03 08:48:05.836077] I [glusterd-utils.c:906:glusterd_volume_start_glusterfs] : About to start glusterfs for brick 10.198.110.16:/mnt/oct3
[2010-10-03 08:48:05.912227] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.214.231.112:/mnt/oct3
[2010-10-03 08:48:05.912281] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.214.231.112 found.. state: 3
[2010-10-03 08:48:05.912301] I [glusterd-utils.c:696:glusterd_brickinfo_get] : Found brick
[2010-10-03 08:48:05.912408] E [glusterd-store.c:215:glusterd_store_delete_brick] : Unlink failed on /etc/glusterd/vols/dhtqa39/bricks/10.198.110.16:-mnt-oct3
[2010-10-03 08:48:05.912433] E [glusterd-store.c:215:glusterd_store_delete_brick] : Unlink failed on /etc/glusterd/vols/dhtqa39/bricks/10.214.231.112:-mnt-oct3
[2010-10-03 08:48:05.986914] I [glusterd-handler.c:2003:glusterd_op_commit_send_resp] glusterd: Responded to commit, ret: 0
[2010-10-03 08:48:05.987002] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 4 to 6 due to event 9
[2010-10-03 08:48:05.987046] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:05.987160] I [rpc-clnt.c:691:rpc_clnt_handle_cbk] rpc-clnt: recieved rpc message (XID: 0x2a, Ver: 2, Program: 52743234, ProgVers: 1, Proc: 1) from rpc-transport (management)
[2010-10-03 08:48:05.987197] I [rpc-clnt.c:691:rpc_clnt_handle_cbk] rpc-clnt: recieved rpc message (XID: 0x2a, Ver: 2, Program: 52743234, ProgVers: 1, Proc: 1) from rpc-transport (management)
[2010-10-03 08:48:05.990518] I [glusterd-pmap.c:237:pmap_registry_bind] pmap: adding brick /mnt/oct3 on port 6971
[2010-10-03 08:48:06.394516] I [glusterd-handler.c:1928:glusterd_handle_cluster_unlock] glusterd: Received UNLOCK from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:06.394552] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 10
[2010-10-03 08:48:06.394590] I [glusterd-handler.c:1904:glusterd_op_unlock_send_resp] glusterd: Responded to unlock, ret: 0
[2010-10-03 08:48:06.394611] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 6 to 0 due to event 10
[2010-10-03 08:48:06.394628] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:08.388484] I [glusterd-handler.c:394:glusterd_handle_cluster_lock] glusterd: Received LOCK from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:08.388546] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 2
[2010-10-03 08:48:08.388563] I [glusterd-handler.c:410:glusterd_handle_cluster_lock] : Returning 0
[2010-10-03 08:48:08.388588] I [glusterd-utils.c:222:glusterd_lock] glusterd: Cluster lock held by 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:08.388641] I [glusterd-handler.c:1884:glusterd_op_lock_send_resp] glusterd: Responded, ret: 0
[2010-10-03 08:48:08.388661] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 0 to 2 due to event 2
[2010-10-03 08:48:08.388679] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:08.389618] I [glusterd-handler.c:1928:glusterd_handle_cluster_unlock] glusterd: Received UNLOCK from uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:08.389656] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 10
[2010-10-03 08:48:08.389690] I [glusterd-handler.c:1904:glusterd_op_unlock_send_resp] glusterd: Responded to unlock, ret: 0
[2010-10-03 08:48:08.389707] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 2 to 0 due to event 10
[2010-10-03 08:48:08.389725] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0


10.198.110.16#ps aux | grep gluster
root     23424  0.1  0.2  70288 17256 ?        Ssl  08:46   0:00 glusterd
root     23428  0.0  0.7 131948 59564 ?        Ssl  08:48   0:00 /usr/local/sbin/glusterfsd --xlator-option dhtqa39-server.listen-port=6971 -s localhost --volfile-id dhtqa39.10.198.110.16.mnt-oct3 -p /etc/glusterd/vols/dhtqa39/run/10.198.110.16-mnt-oct3.pid --brick-name /mnt/oct3 --brick-port 6971 -l /etc/glusterd/logs/bricks/mnt-oct3.log
root     23432  0.0  0.8 134572 69344 ?        Ssl  08:48   0:00 /usr/local/sbin/glusterfs -f /etc/glusterd/nfs/nfs-server.vol -p /etc/glusterd/nfs/run/nfs.pid -l /etc/glusterd/logs/nfs.log
root     23442  0.0  0.0   6064   616 pts/2    S+   08:51   0:00 grep gluster
Comment 1 Lakshmipathi G 2010-10-03 08:59:52 EDT
add-brick said unsuccessful when its successful ,when tried to add 2 bricks to existing 2dht cluster.  

[ unlike this - http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=1773
this time no previous glusterfs process was running  ]

=========================
#gluster volume info

Volume Name: dhtqa39
Type: Distribute
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: 10.192.134.144:/mnt/oct3
Brick2: 10.192.141.187:/mnt/oct3

10.192.141.187#gluster peer probe 10.198.110.16
Probe successful
10.192.141.187#gluster peer probe  10.214.231.112
Probe successful
10.192.141.187#gluster volume add-brick dhtqa39 10.198.110.16:/mnt/oct3 10.214.231.112:/mnt/oct3
Add Brick unsuccessful

10.192.141.187#gluster volume add-brick dhtqa39 10.198.110.16:/mnt/oct3 10.214.231.112:/mnt/oct3
Add Brick unsuccessful

10.192.141.187#gluster volume add-brick dhtqa39 10.198.110.16:/mnt/oct3 10.214.231.112:/mnt/oct3
Add Brick unsuccessful

10.192.141.187#gluster volume info

Volume Name: dhtqa39
Type: Distribute
Status: Started
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: 10.192.134.144:/mnt/oct3
Brick2: 10.192.141.187:/mnt/oct3
Brick3: 10.198.110.16:/mnt/oct3
Brick4: 10.214.231.112:/mnt/oct3


10.192.141.187#cat /usr/local/var/log/glusterfs/usr-local-etc-glusterfs-glusterd.vol.log 
[2010-10-03 08:39:42.786179] I [glusterd.c:276:init] management: Using /etc/glusterd as working directory
[2010-10-03 08:39:42.787280] E [rpc-transport.c:897:rpc_transport_load] rpc-transport: /usr/local/lib/glusterfs/3.1.0qa39/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
[2010-10-03 08:39:42.787300] E [rpc-transport.c:901:rpc_transport_load] rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine
[2010-10-03 08:39:42.787369] I [glusterd.c:95:glusterd_uuid_init] glusterd: generated UUID: 7c9a6b04-3436-4e9c-812e-91be61c407fe
Given volfile:
+------------------------------------------------------------------------------+
  1: volume management
  2:     type mgmt/glusterd
  3:     option working-directory /etc/glusterd
  4:     option transport-type socket,rdma
  5:     option transport.socket.listen-port 6969
  6:     option transport.rdma.listen-port 6968
  7:     # option listen-port 6969
  8: end-volume
  9: 

+------------------------------------------------------------------------------+
[2010-10-03 08:39:55.936290] I [glusterd-handler.c:531:glusterd_handle_cli_probe] glusterd: Received CLI probe req 10.192.134.144 6969
[2010-10-03 08:39:55.948120] I [glusterd-handler.c:366:glusterd_friend_find] glusterd: Unable to find hostname: 10.192.134.144
[2010-10-03 08:39:55.948140] I [glusterd-handler.c:2438:glusterd_probe_begin] glusterd: Unable to find peerinfo for host: 10.192.134.144 (6969)
[2010-10-03 08:39:55.950808] W [rpc-transport.c:841:rpc_transport_load] rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2010-10-03 08:39:55.953130] I [glusterd-handler.c:2419:glusterd_friend_add] glusterd: connect returned 0
[2010-10-03 08:39:55.953606] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 11
[2010-10-03 08:39:55.953677] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.192.134.144 found.. state: 0
[2010-10-03 08:39:55.953757] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 0 to 0
[2010-10-03 08:39:55.955851] I [glusterd3_1-mops.c:82:glusterd3_1_probe_cbk] glusterd: Received probe resp from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c, host: 10.192.134.144
[2010-10-03 08:39:55.955881] I [glusterd-handler.c:354:glusterd_friend_find] glusterd: Unable to find peer by uuid
[2010-10-03 08:39:55.955902] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.192.134.144 found.. state: 0
[2010-10-03 08:39:55.955923] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 2
[2010-10-03 08:39:55.955982] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 0 to 1
[2010-10-03 08:39:55.956106] I [glusterd3_1-mops.c:139:glusterd3_1_probe_cbk] glusterd: Received resp to probe req
[2010-10-03 08:39:55.962006] I [glusterd3_1-mops.c:186:glusterd3_1_friend_add_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c, host: 10.192.134.144, port: 0
[2010-10-03 08:39:55.962025] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 1
[2010-10-03 08:39:55.962042] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:39:55.962093] I [glusterd-handler.c:2592:glusterd_xfer_cli_probe_resp] glusterd: Responded to CLI, ret: 0
[2010-10-03 08:39:55.962110] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 1 to 4
[2010-10-03 08:39:55.962420] I [glusterd-handler.c:2249:glusterd_handle_probe_query] glusterd: Received probe from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:39:55.962443] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 4
[2010-10-03 08:39:55.962473] I [glusterd-handler.c:2272:glusterd_handle_probe_query] glusterd: Responded to 10.192.141.187, op_ret: 0, op_errno: 0, ret: 0
[2010-10-03 08:39:55.962503] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:39:55.970199] I [glusterd-handler.c:2028:glusterd_handle_incoming_friend_req] glusterd: Received probe from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:39:55.970263] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 4
[2010-10-03 08:39:55.970292] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 7
[2010-10-03 08:39:55.970310] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 4
[2010-10-03 08:39:55.970341] I [glusterd-handler.c:2569:glusterd_xfer_friend_add_resp] glusterd: Responded to 10.192.141.187 (0), ret: 0
[2010-10-03 08:39:55.970358] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 4 to 4
[2010-10-03 08:39:55.970467] I [glusterd3_1-mops.c:964:glusterd3_1_friend_update] : Added uuid: bd364a85-e8b3-43d6-8a88-363df38be63c, host: 10.192.134.144
[2010-10-03 08:39:55.970509] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 4 to 3
[2010-10-03 08:39:55.970595] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:39:55.970756] I [glusterd-handler.c:2146:glusterd_handle_friend_update] glusterd: Received friend update from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:39:55.970790] I [glusterd-handler.c:2191:glusterd_handle_friend_update] : Received uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe, hostname:10.192.141.187
[2010-10-03 08:39:55.970810] I [glusterd-handler.c:2194:glusterd_handle_friend_update] : Received my uuid as Friend
[2010-10-03 08:39:55.970851] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:39:55.971026] I [glusterd3_1-mops.c:371:glusterd3_1_friend_update_cbk] glusterd: Received ACC from uuid: 
[2010-10-03 08:39:59.22074] I [glusterd-handler.c:633:glusterd_handle_cli_list_friends] glusterd: Received cli list req
[2010-10-03 08:39:59.22217] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:40:56.354775] I [glusterd-handler.c:675:glusterd_handle_cli_get_volume] glusterd: Received get vol req
[2010-10-03 08:40:56.354852] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:41:24.17419] I [glusterd-op-sm.c:4901:glusterd_op_set_cli_op] : Returning 0
[2010-10-03 08:41:24.17475] I [glusterd-handler.c:745:glusterd_handle_create_volume] glusterd: Received create volume req
[2010-10-03 08:41:24.17660] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.192.134.144 found.. state: 3
[2010-10-03 08:41:24.17794] I [glusterd-utils.c:222:glusterd_lock] glusterd: Cluster lock held by 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:41:24.17811] I [glusterd-handler.c:2636:glusterd_op_txn_begin] glusterd: Acquired local lock
[2010-10-03 08:41:24.17826] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 1
[2010-10-03 08:41:24.17851] I [glusterd-handler.c:2640:glusterd_op_txn_begin] glusterd: Returning 0
[2010-10-03 08:41:24.17923] I [glusterd3_1-mops.c:1054:glusterd3_1_cluster_lock] glusterd: Sent lock req to 1 peers
[2010-10-03 08:41:24.17940] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 0 to 1 due to event 1
[2010-10-03 08:41:24.17966] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:41:24.18908] I [glusterd3_1-mops.c:410:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:41:24.18945] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:41:24.18965] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:41:24.18981] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 4
[2010-10-03 08:41:24.18995] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 1 due to event 3
[2010-10-03 08:41:24.19043] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.192.134.144 found.. state: 3
[2010-10-03 08:41:24.19834] I [glusterd3_1-mops.c:1195:glusterd3_1_stage_op] glusterd: Sent op req to 1 peers
[2010-10-03 08:41:24.19889] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 3 due to event 4
[2010-10-03 08:41:24.21347] I [glusterd3_1-mops.c:533:glusterd3_1_stage_op_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:41:24.21371] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:41:24.21392] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:41:24.21413] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 5
[2010-10-03 08:41:24.21432] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 3 to 3 due to event 3
[2010-10-03 08:41:24.21493] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.192.134.144 found.. state: 3
[2010-10-03 08:41:24.23005] I [glusterd3_1-mops.c:1285:glusterd3_1_commit_op] glusterd: Sent op req to 1 peers
[2010-10-03 08:41:24.23034] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 3 to 5 due to event 5
[2010-10-03 08:41:24.24790] I [rpc-clnt.c:691:rpc_clnt_handle_cbk] rpc-clnt: recieved rpc message (XID: 0x2a, Ver: 2, Program: 52743234, ProgVers: 1, Proc: 1) from rpc-transport (management)
[2010-10-03 08:41:24.24919] I [glusterd3_1-mops.c:680:glusterd3_1_commit_op_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:41:24.24947] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:41:24.24968] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:41:24.24993] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 6
[2010-10-03 08:41:24.25013] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 5 to 5 due to event 3
[2010-10-03 08:41:24.25066] I [glusterd3_1-mops.c:1108:glusterd3_1_cluster_unlock] glusterd: Sent unlock req to 1 peers
[2010-10-03 08:41:24.25090] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 5 to 7 due to event 6
[2010-10-03 08:41:24.25413] I [glusterd3_1-mops.c:471:glusterd3_1_cluster_unlock_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:41:24.25441] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:41:24.25462] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:41:24.25482] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 4
[2010-10-03 08:41:24.25501] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 7 due to event 3
[2010-10-03 08:41:24.25524] I [glusterd-op-sm.c:4277:glusterd_op_txn_complete] glusterd: Cleared local lock
[2010-10-03 08:41:24.25574] I [glusterd-op-sm.c:4241:glusterd_op_send_cli_response] : Returning 0
[2010-10-03 08:41:24.25593] I [glusterd-op-sm.c:4320:glusterd_op_txn_complete] glusterd: Returning 0
[2010-10-03 08:41:24.25608] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 0 due to event 4
[2010-10-03 08:41:31.822112] I [glusterd-op-sm.c:4901:glusterd_op_set_cli_op] : Returning 0
[2010-10-03 08:41:31.822159] I [glusterd-handler.c:928:glusterd_handle_cli_start_volume] glusterd: Received start vol reqfor volume dhtqa39
[2010-10-03 08:41:31.822190] I [glusterd-utils.c:222:glusterd_lock] glusterd: Cluster lock held by 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:41:31.822206] I [glusterd-handler.c:2636:glusterd_op_txn_begin] glusterd: Acquired local lock
[2010-10-03 08:41:31.822222] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 1
[2010-10-03 08:41:31.822237] I [glusterd-handler.c:2640:glusterd_op_txn_begin] glusterd: Returning 0
[2010-10-03 08:41:31.822313] I [glusterd3_1-mops.c:1054:glusterd3_1_cluster_lock] glusterd: Sent lock req to 1 peers
[2010-10-03 08:41:31.822365] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 0 to 1 due to event 1
[2010-10-03 08:41:31.822385] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:41:31.823041] I [glusterd3_1-mops.c:410:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:41:31.823073] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:41:31.823089] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:41:31.823104] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 4
[2010-10-03 08:41:31.823119] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 1 due to event 3
[2010-10-03 08:41:31.823161] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.192.134.144 found.. state: 3
[2010-10-03 08:41:31.824124] I [glusterd3_1-mops.c:1195:glusterd3_1_stage_op] glusterd: Sent op req to 1 peers
[2010-10-03 08:41:31.824144] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 3 due to event 4
[2010-10-03 08:41:31.825633] I [glusterd3_1-mops.c:533:glusterd3_1_stage_op_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:41:31.825657] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:41:31.825680] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:41:31.825695] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 5
[2010-10-03 08:41:31.825710] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 3 to 3 due to event 3
[2010-10-03 08:41:32.394764] I [glusterd-utils.c:906:glusterd_volume_start_glusterfs] : About to start glusterfs for brick 10.192.141.187:/mnt/oct3
[2010-10-03 08:41:32.501777] I [glusterd3_1-mops.c:1285:glusterd3_1_commit_op] glusterd: Sent op req to 1 peers
[2010-10-03 08:41:32.501833] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 3 to 5 due to event 5
[2010-10-03 08:41:32.507442] I [glusterd-pmap.c:237:pmap_registry_bind] pmap: adding brick /mnt/oct3 on port 6971
[2010-10-03 08:41:33.222034] I [glusterd3_1-mops.c:680:glusterd3_1_commit_op_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:41:33.222075] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:41:33.222093] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:41:33.222111] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 6
[2010-10-03 08:41:33.222128] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 5 to 5 due to event 3
[2010-10-03 08:41:33.222170] I [glusterd3_1-mops.c:1108:glusterd3_1_cluster_unlock] glusterd: Sent unlock req to 1 peers
[2010-10-03 08:41:33.222191] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 5 to 7 due to event 6
[2010-10-03 08:41:33.222666] I [glusterd3_1-mops.c:471:glusterd3_1_cluster_unlock_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:41:33.222684] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:41:33.222703] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:41:33.222718] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 4
[2010-10-03 08:41:33.222733] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 7 due to event 3
[2010-10-03 08:41:33.222751] I [glusterd-op-sm.c:4277:glusterd_op_txn_complete] glusterd: Cleared local lock
[2010-10-03 08:41:33.222796] I [glusterd-op-sm.c:4241:glusterd_op_send_cli_response] : Returning 0
[2010-10-03 08:41:33.222836] I [glusterd-op-sm.c:4320:glusterd_op_txn_complete] glusterd: Returning 0
[2010-10-03 08:41:33.222852] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 0 due to event 4
[2010-10-03 08:42:05.2094] I [glusterd-handler.c:675:glusterd_handle_cli_get_volume] glusterd: Received get vol req
[2010-10-03 08:42:05.2240] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:42:05.2509] I [glusterd-handler.c:675:glusterd_handle_cli_get_volume] glusterd: Received get vol req
[2010-10-03 08:42:05.2561] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:47:48.614346] I [glusterd-handler.c:531:glusterd_handle_cli_probe] glusterd: Received CLI probe req 10.198.110.16 6969
[2010-10-03 08:47:48.619484] I [glusterd-handler.c:366:glusterd_friend_find] glusterd: Unable to find hostname: 10.198.110.16
[2010-10-03 08:47:48.619516] I [glusterd-handler.c:2438:glusterd_probe_begin] glusterd: Unable to find peerinfo for host: 10.198.110.16 (6969)
[2010-10-03 08:47:48.622682] W [rpc-transport.c:841:rpc_transport_load] rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2010-10-03 08:47:48.624994] I [glusterd-handler.c:2419:glusterd_friend_add] glusterd: connect returned 0
[2010-10-03 08:47:48.625823] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 11
[2010-10-03 08:47:48.625855] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 0
[2010-10-03 08:47:48.625905] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 0 to 0
[2010-10-03 08:47:48.630522] I [glusterd3_1-mops.c:82:glusterd3_1_probe_cbk] glusterd: Received probe resp from uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e, host: 10.198.110.16
[2010-10-03 08:47:48.630571] I [glusterd-handler.c:354:glusterd_friend_find] glusterd: Unable to find peer by uuid
[2010-10-03 08:47:48.630587] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 0
[2010-10-03 08:47:48.630605] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 2
[2010-10-03 08:47:48.630701] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 0 to 1
[2010-10-03 08:47:48.630837] I [glusterd3_1-mops.c:139:glusterd3_1_probe_cbk] glusterd: Received resp to probe req
[2010-10-03 08:47:48.648461] I [rpc-clnt.c:691:rpc_clnt_handle_cbk] rpc-clnt: recieved rpc message (XID: 0x2a, Ver: 2, Program: 52743234, ProgVers: 1, Proc: 1) from rpc-transport (management)
[2010-10-03 08:47:48.648991] I [glusterd3_1-mops.c:186:glusterd3_1_friend_add_cbk] glusterd: Received ACC from uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e, host: 10.198.110.16, port: 0
[2010-10-03 08:47:48.649024] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 1
[2010-10-03 08:47:48.649042] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:47:48.649094] I [glusterd-handler.c:2592:glusterd_xfer_cli_probe_resp] glusterd: Responded to CLI, ret: 0
[2010-10-03 08:47:48.649113] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 1 to 4
[2010-10-03 08:47:48.649983] I [glusterd-handler.c:2249:glusterd_handle_probe_query] glusterd: Received probe from uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e
[2010-10-03 08:47:48.650009] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 4
[2010-10-03 08:47:48.650044] I [glusterd-handler.c:2272:glusterd_handle_probe_query] glusterd: Responded to 10.192.141.187, op_ret: 0, op_errno: 0, ret: 0
[2010-10-03 08:47:48.650063] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:47:48.651662] I [glusterd-handler.c:2028:glusterd_handle_incoming_friend_req] glusterd: Received probe from uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e
[2010-10-03 08:47:48.651704] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 4
[2010-10-03 08:47:48.651743] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 7
[2010-10-03 08:47:48.651769] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 4
[2010-10-03 08:47:48.651799] I [glusterd-handler.c:2569:glusterd_xfer_friend_add_resp] glusterd: Responded to 10.192.141.187 (0), ret: 0
[2010-10-03 08:47:48.651840] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 4 to 4
[2010-10-03 08:47:48.651968] I [glusterd3_1-mops.c:964:glusterd3_1_friend_update] : Added uuid: bd364a85-e8b3-43d6-8a88-363df38be63c, host: 10.192.134.144
[2010-10-03 08:47:48.651989] I [glusterd3_1-mops.c:964:glusterd3_1_friend_update] : Added uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e, host: 10.198.110.16
[2010-10-03 08:47:48.652053] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 4 to 3
[2010-10-03 08:47:48.652136] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:47:48.659452] I [glusterd3_1-mops.c:371:glusterd3_1_friend_update_cbk] glusterd: Received ACC from uuid: 
[2010-10-03 08:47:48.669955] I [glusterd-handler.c:2146:glusterd_handle_friend_update] glusterd: Received friend update from uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e
[2010-10-03 08:47:48.670020] I [glusterd-handler.c:2191:glusterd_handle_friend_update] : Received uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe, hostname:10.192.141.187
[2010-10-03 08:47:48.670040] I [glusterd-handler.c:2194:glusterd_handle_friend_update] : Received my uuid as Friend
[2010-10-03 08:47:48.670057] I [glusterd-handler.c:2191:glusterd_handle_friend_update] : Received uuid: bd364a85-e8b3-43d6-8a88-363df38be63c, hostname:10.192.134.144
[2010-10-03 08:47:48.670072] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:47:48.670110] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:47:48.670906] I [glusterd3_1-mops.c:371:glusterd3_1_friend_update_cbk] glusterd: Received ACC from uuid: 
[2010-10-03 08:47:55.331443] I [glusterd-handler.c:531:glusterd_handle_cli_probe] glusterd: Received CLI probe req 10.214.231.112 6969
[2010-10-03 08:47:55.333106] I [glusterd-handler.c:366:glusterd_friend_find] glusterd: Unable to find hostname: 10.214.231.112
[2010-10-03 08:47:55.333126] I [glusterd-handler.c:2438:glusterd_probe_begin] glusterd: Unable to find peerinfo for host: 10.214.231.112 (6969)
[2010-10-03 08:47:55.335673] W [rpc-transport.c:841:rpc_transport_load] rpc-transport: missing 'option transport-type'. defaulting to "socket"
[2010-10-03 08:47:55.338047] I [glusterd-handler.c:2419:glusterd_friend_add] glusterd: connect returned 0
[2010-10-03 08:47:55.338491] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 11
[2010-10-03 08:47:55.338523] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.214.231.112 found.. state: 0
[2010-10-03 08:47:55.338565] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 0 to 0
[2010-10-03 08:47:55.341155] I [glusterd3_1-mops.c:82:glusterd3_1_probe_cbk] glusterd: Received probe resp from uuid: 71042154-2a52-4796-a246-1120731a2fa4, host: 10.214.231.112
[2010-10-03 08:47:55.341180] I [glusterd-handler.c:354:glusterd_friend_find] glusterd: Unable to find peer by uuid
[2010-10-03 08:47:55.341195] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.214.231.112 found.. state: 0
[2010-10-03 08:47:55.341212] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 2
[2010-10-03 08:47:55.341291] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 0 to 1
[2010-10-03 08:47:55.341409] I [glusterd3_1-mops.c:139:glusterd3_1_probe_cbk] glusterd: Received resp to probe req
[2010-10-03 08:47:55.348316] I [rpc-clnt.c:691:rpc_clnt_handle_cbk] rpc-clnt: recieved rpc message (XID: 0x2a, Ver: 2, Program: 52743234, ProgVers: 1, Proc: 1) from rpc-transport (management)
[2010-10-03 08:47:55.348380] I [glusterd3_1-mops.c:186:glusterd3_1_friend_add_cbk] glusterd: Received ACC from uuid: 71042154-2a52-4796-a246-1120731a2fa4, host: 10.214.231.112, port: 0
[2010-10-03 08:47:55.348421] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 1
[2010-10-03 08:47:55.348439] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:47:55.348491] I [glusterd-handler.c:2592:glusterd_xfer_cli_probe_resp] glusterd: Responded to CLI, ret: 0
[2010-10-03 08:47:55.348509] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 1 to 4
[2010-10-03 08:47:55.349018] I [glusterd-handler.c:2249:glusterd_handle_probe_query] glusterd: Received probe from uuid: 71042154-2a52-4796-a246-1120731a2fa4
[2010-10-03 08:47:55.349041] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 4
[2010-10-03 08:47:55.349075] I [glusterd-handler.c:2272:glusterd_handle_probe_query] glusterd: Responded to 10.192.141.187, op_ret: 0, op_errno: 0, ret: 0
[2010-10-03 08:47:55.349098] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:47:55.349626] I [glusterd-handler.c:2028:glusterd_handle_incoming_friend_req] glusterd: Received probe from uuid: 71042154-2a52-4796-a246-1120731a2fa4
[2010-10-03 08:47:55.349692] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 4
[2010-10-03 08:47:55.349714] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 7
[2010-10-03 08:47:55.349738] I [glusterd-sm.c:699:glusterd_friend_sm_inject_event] glusterd: Enqueuing event: 4
[2010-10-03 08:47:55.349770] I [glusterd-handler.c:2569:glusterd_xfer_friend_add_resp] glusterd: Responded to 10.192.141.187 (0), ret: 0
[2010-10-03 08:47:55.349790] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 4 to 4
[2010-10-03 08:47:55.349884] I [glusterd3_1-mops.c:964:glusterd3_1_friend_update] : Added uuid: bd364a85-e8b3-43d6-8a88-363df38be63c, host: 10.192.134.144
[2010-10-03 08:47:55.349905] I [glusterd3_1-mops.c:964:glusterd3_1_friend_update] : Added uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e, host: 10.198.110.16
[2010-10-03 08:47:55.349924] I [glusterd3_1-mops.c:964:glusterd3_1_friend_update] : Added uuid: 71042154-2a52-4796-a246-1120731a2fa4, host: 10.214.231.112
[2010-10-03 08:47:55.350006] I [glusterd-sm.c:526:glusterd_friend_sm_transition_state] : Transitioning from 4 to 3
[2010-10-03 08:47:55.350092] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:47:55.350305] I [glusterd-handler.c:2146:glusterd_handle_friend_update] glusterd: Received friend update from uuid: 71042154-2a52-4796-a246-1120731a2fa4
[2010-10-03 08:47:55.350333] I [glusterd-handler.c:2191:glusterd_handle_friend_update] : Received uuid: 7c9a6b04-3436-4e9c-812e-91be61c407fe, hostname:10.192.141.187
[2010-10-03 08:47:55.350354] I [glusterd-handler.c:2194:glusterd_handle_friend_update] : Received my uuid as Friend
[2010-10-03 08:47:55.350385] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:47:55.356573] I [glusterd3_1-mops.c:371:glusterd3_1_friend_update_cbk] glusterd: Received ACC from uuid: 
[2010-10-03 08:47:55.366917] I [glusterd3_1-mops.c:371:glusterd3_1_friend_update_cbk] glusterd: Received ACC from uuid: 
[2010-10-03 08:47:55.387203] I [glusterd3_1-mops.c:371:glusterd3_1_friend_update_cbk] glusterd: Received ACC from uuid: 
[2010-10-03 08:48:02.933690] I [glusterd-op-sm.c:4901:glusterd_op_set_cli_op] : Returning 0
[2010-10-03 08:48:02.933871] I [glusterd-handler.c:1043:glusterd_handle_add_brick] glusterd: Received add brick req
[2010-10-03 08:48:02.934014] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 3
[2010-10-03 08:48:02.934145] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.214.231.112 found.. state: 3
[2010-10-03 08:48:02.934269] I [glusterd-utils.c:222:glusterd_lock] glusterd: Cluster lock held by 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:02.934310] I [glusterd-handler.c:2636:glusterd_op_txn_begin] glusterd: Acquired local lock
[2010-10-03 08:48:02.934328] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 1
[2010-10-03 08:48:02.934343] I [glusterd-handler.c:2640:glusterd_op_txn_begin] glusterd: Returning 0
[2010-10-03 08:48:02.934696] I [glusterd3_1-mops.c:1054:glusterd3_1_cluster_lock] glusterd: Sent lock req to 3 peers
[2010-10-03 08:48:02.934715] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 0 to 1 due to event 1
[2010-10-03 08:48:02.934732] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:02.935421] I [glusterd3_1-mops.c:410:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:48:02.935440] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:02.935456] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:02.935482] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 1 due to event 3
[2010-10-03 08:48:02.935511] I [glusterd3_1-mops.c:410:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e
[2010-10-03 08:48:02.935530] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:02.935545] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:02.935577] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 1 due to event 3
[2010-10-03 08:48:02.936172] I [glusterd3_1-mops.c:410:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: 71042154-2a52-4796-a246-1120731a2fa4
[2010-10-03 08:48:02.936200] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:02.936215] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:02.936231] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 4
[2010-10-03 08:48:02.936245] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 1 due to event 3
[2010-10-03 08:48:02.936271] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.198.110.16:/mnt/oct3
[2010-10-03 08:48:02.936288] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 3
[2010-10-03 08:48:02.936308] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 3
[2010-10-03 08:48:02.936324] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.214.231.112:/mnt/oct3
[2010-10-03 08:48:02.936340] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.214.231.112 found.. state: 3
[2010-10-03 08:48:02.936359] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.214.231.112 found.. state: 3
[2010-10-03 08:48:02.936426] I [glusterd3_1-mops.c:1195:glusterd3_1_stage_op] glusterd: Sent op req to 3 peers
[2010-10-03 08:48:02.936443] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 3 due to event 4
[2010-10-03 08:48:02.937554] I [glusterd3_1-mops.c:533:glusterd3_1_stage_op_cbk] glusterd: Received RJT from uuid: 71042154-2a52-4796-a246-1120731a2fa4
[2010-10-03 08:48:02.937574] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:02.937590] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 7
[2010-10-03 08:48:02.937666] I [glusterd3_1-mops.c:1108:glusterd3_1_cluster_unlock] glusterd: Sent unlock req to 3 peers
[2010-10-03 08:48:02.937682] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 3 to 7 due to event 7
[2010-10-03 08:48:02.937715] I [glusterd3_1-mops.c:533:glusterd3_1_stage_op_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:48:02.937742] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:02.937762] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:02.937777] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 7 due to event 3
[2010-10-03 08:48:02.938747] I [glusterd3_1-mops.c:471:glusterd3_1_cluster_unlock_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:48:02.938775] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:02.938791] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:02.938806] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 7 due to event 3
[2010-10-03 08:48:02.938834] I [glusterd3_1-mops.c:471:glusterd3_1_cluster_unlock_cbk] glusterd: Received ACC from uuid: 71042154-2a52-4796-a246-1120731a2fa4
[2010-10-03 08:48:02.938850] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:02.938865] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:02.938880] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 4
[2010-10-03 08:48:02.938895] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 7 due to event 3
[2010-10-03 08:48:02.938912] I [glusterd-op-sm.c:4277:glusterd_op_txn_complete] glusterd: Cleared local lock
[2010-10-03 08:48:02.938947] I [glusterd-op-sm.c:4241:glusterd_op_send_cli_response] : Returning 0
[2010-10-03 08:48:02.938966] I [glusterd-op-sm.c:4320:glusterd_op_txn_complete] glusterd: Returning 0
[2010-10-03 08:48:02.938981] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 0 due to event 4
[2010-10-03 08:48:02.939360] I [glusterd3_1-mops.c:533:glusterd3_1_stage_op_cbk] glusterd: Received RJT from uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e
[2010-10-03 08:48:02.939379] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:02.939396] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 7
[2010-10-03 08:48:02.939446] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 0 to 0 due to event 7
[2010-10-03 08:48:02.941048] I [glusterd3_1-mops.c:471:glusterd3_1_cluster_unlock_cbk] glusterd: Received ACC from uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e
[2010-10-03 08:48:02.941069] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:02.941085] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:02.941112] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 0 to 0 due to event 3
[2010-10-03 08:48:04.391591] I [glusterd-op-sm.c:4901:glusterd_op_set_cli_op] : Returning 0
[2010-10-03 08:48:04.391670] I [glusterd-handler.c:1043:glusterd_handle_add_brick] glusterd: Received add brick req
[2010-10-03 08:48:04.391930] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 3
[2010-10-03 08:48:04.392068] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.214.231.112 found.. state: 3
[2010-10-03 08:48:04.392360] I [glusterd-utils.c:222:glusterd_lock] glusterd: Cluster lock held by 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:04.392380] I [glusterd-handler.c:2636:glusterd_op_txn_begin] glusterd: Acquired local lock
[2010-10-03 08:48:04.392438] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 1
[2010-10-03 08:48:04.392454] I [glusterd-handler.c:2640:glusterd_op_txn_begin] glusterd: Returning 0
[2010-10-03 08:48:04.392595] I [glusterd3_1-mops.c:1054:glusterd3_1_cluster_lock] glusterd: Sent lock req to 3 peers
[2010-10-03 08:48:04.392628] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 0 to 1 due to event 1
[2010-10-03 08:48:04.392666] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:04.393937] I [glusterd3_1-mops.c:410:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e
[2010-10-03 08:48:04.393960] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:04.393976] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:04.393992] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 1 due to event 3
[2010-10-03 08:48:04.394107] I [glusterd3_1-mops.c:410:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: 71042154-2a52-4796-a246-1120731a2fa4
[2010-10-03 08:48:04.394124] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:04.394140] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:04.394155] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 1 due to event 3
[2010-10-03 08:48:04.394182] I [glusterd3_1-mops.c:410:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:48:04.394293] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:04.394309] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:04.394324] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 4
[2010-10-03 08:48:04.394339] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 1 due to event 3
[2010-10-03 08:48:04.394365] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.198.110.16:/mnt/oct3
[2010-10-03 08:48:04.394392] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 3
[2010-10-03 08:48:04.394418] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 3
[2010-10-03 08:48:04.394434] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.214.231.112:/mnt/oct3
[2010-10-03 08:48:04.394450] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.214.231.112 found.. state: 3
[2010-10-03 08:48:04.394469] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.214.231.112 found.. state: 3
[2010-10-03 08:48:04.394536] I [glusterd3_1-mops.c:1195:glusterd3_1_stage_op] glusterd: Sent op req to 3 peers
[2010-10-03 08:48:04.394577] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 3 due to event 4
[2010-10-03 08:48:04.396090] I [glusterd3_1-mops.c:533:glusterd3_1_stage_op_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:48:04.396111] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:04.396129] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:04.396145] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 3 to 3 due to event 3
[2010-10-03 08:48:04.398039] I [glusterd3_1-mops.c:533:glusterd3_1_stage_op_cbk] glusterd: Received ACC from uuid: 71042154-2a52-4796-a246-1120731a2fa4
[2010-10-03 08:48:04.398061] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:04.398078] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:04.398094] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 3 to 3 due to event 3
[2010-10-03 08:48:04.398123] I [glusterd3_1-mops.c:533:glusterd3_1_stage_op_cbk] glusterd: Received ACC from uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e
[2010-10-03 08:48:04.398140] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:04.398168] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:04.398185] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 5
[2010-10-03 08:48:04.398200] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 3 to 3 due to event 3
[2010-10-03 08:48:04.398232] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 3
[2010-10-03 08:48:04.398259] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.214.231.112 found.. state: 3
[2010-10-03 08:48:04.399657] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.198.110.16:/mnt/oct3
[2010-10-03 08:48:04.399685] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 3
[2010-10-03 08:48:04.399702] I [glusterd-utils.c:696:glusterd_brickinfo_get] : Found brick
[2010-10-03 08:48:04.399717] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.214.231.112:/mnt/oct3
[2010-10-03 08:48:04.399733] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.214.231.112 found.. state: 3
[2010-10-03 08:48:04.399749] I [glusterd-utils.c:696:glusterd_brickinfo_get] : Found brick
[2010-10-03 08:48:04.399817] E [glusterd-store.c:215:glusterd_store_delete_brick] : Unlink failed on /etc/glusterd/vols/dhtqa39/bricks/10.198.110.16:-mnt-oct3
[2010-10-03 08:48:04.399841] E [glusterd-store.c:215:glusterd_store_delete_brick] : Unlink failed on /etc/glusterd/vols/dhtqa39/bricks/10.214.231.112:-mnt-oct3
[2010-10-03 08:48:04.400862] I [glusterd-utils.c:805:glusterd_service_stop] : Stopping gluster nfsd running in pid: 17528
[2010-10-03 08:48:05.522319] I [glusterd3_1-mops.c:1285:glusterd3_1_commit_op] glusterd: Sent op req to 3 peers
[2010-10-03 08:48:05.522419] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 3 to 5 due to event 5
[2010-10-03 08:48:05.523700] I [rpc-clnt.c:691:rpc_clnt_handle_cbk] rpc-clnt: recieved rpc message (XID: 0x2a, Ver: 2, Program: 52743234, ProgVers: 1, Proc: 1) from rpc-transport (management)
[2010-10-03 08:48:05.524361] I [rpc-clnt.c:691:rpc_clnt_handle_cbk] rpc-clnt: recieved rpc message (XID: 0x2a, Ver: 2, Program: 52743234, ProgVers: 1, Proc: 1) from rpc-transport (management)
[2010-10-03 08:48:05.524491] I [rpc-clnt.c:691:rpc_clnt_handle_cbk] rpc-clnt: recieved rpc message (XID: 0x2a, Ver: 2, Program: 52743234, ProgVers: 1, Proc: 1) from rpc-transport (management)
[2010-10-03 08:48:06.238008] I [glusterd3_1-mops.c:680:glusterd3_1_commit_op_cbk] glusterd: Received ACC from uuid: 71042154-2a52-4796-a246-1120731a2fa4
[2010-10-03 08:48:06.238050] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:06.238068] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:06.238086] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 5 to 5 due to event 3
[2010-10-03 08:48:06.257916] I [glusterd3_1-mops.c:680:glusterd3_1_commit_op_cbk] glusterd: Received ACC from uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e
[2010-10-03 08:48:06.257950] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:06.257966] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:06.257981] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 5 to 5 due to event 3
[2010-10-03 08:48:06.664684] I [glusterd3_1-mops.c:680:glusterd3_1_commit_op_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:48:06.664753] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:06.664770] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:06.664788] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 6
[2010-10-03 08:48:06.664823] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 5 to 5 due to event 3
[2010-10-03 08:48:06.664917] I [glusterd3_1-mops.c:1108:glusterd3_1_cluster_unlock] glusterd: Sent unlock req to 3 peers
[2010-10-03 08:48:06.664934] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 5 to 7 due to event 6
[2010-10-03 08:48:06.665269] I [glusterd3_1-mops.c:471:glusterd3_1_cluster_unlock_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:48:06.665293] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:06.665323] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:06.665339] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 7 due to event 3
[2010-10-03 08:48:06.665407] I [glusterd3_1-mops.c:471:glusterd3_1_cluster_unlock_cbk] glusterd: Received ACC from uuid: 71042154-2a52-4796-a246-1120731a2fa4
[2010-10-03 08:48:06.665429] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:06.665449] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:06.665465] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 7 due to event 3
[2010-10-03 08:48:06.665519] I [glusterd3_1-mops.c:471:glusterd3_1_cluster_unlock_cbk] glusterd: Received ACC from uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e
[2010-10-03 08:48:06.665540] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:06.665555] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:06.665571] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 4
[2010-10-03 08:48:06.665585] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 7 due to event 3
[2010-10-03 08:48:06.665604] I [glusterd-op-sm.c:4277:glusterd_op_txn_complete] glusterd: Cleared local lock
[2010-10-03 08:48:06.665662] I [glusterd-op-sm.c:4241:glusterd_op_send_cli_response] : Returning 0
[2010-10-03 08:48:06.665682] I [glusterd-op-sm.c:4320:glusterd_op_txn_complete] glusterd: Returning 0
[2010-10-03 08:48:06.665698] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 0 due to event 4
[2010-10-03 08:48:08.658121] I [glusterd-op-sm.c:4901:glusterd_op_set_cli_op] : Returning 0
[2010-10-03 08:48:08.658238] I [glusterd-handler.c:1043:glusterd_handle_add_brick] glusterd: Received add brick req
[2010-10-03 08:48:08.658393] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 3
[2010-10-03 08:48:08.658592] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.214.231.112 found.. state: 3
[2010-10-03 08:48:08.658786] I [glusterd-utils.c:222:glusterd_lock] glusterd: Cluster lock held by 7c9a6b04-3436-4e9c-812e-91be61c407fe
[2010-10-03 08:48:08.658802] I [glusterd-handler.c:2636:glusterd_op_txn_begin] glusterd: Acquired local lock
[2010-10-03 08:48:08.658818] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 1
[2010-10-03 08:48:08.658833] I [glusterd-handler.c:2640:glusterd_op_txn_begin] glusterd: Returning 0
[2010-10-03 08:48:08.658967] I [glusterd3_1-mops.c:1054:glusterd3_1_cluster_lock] glusterd: Sent lock req to 3 peers
[2010-10-03 08:48:08.658992] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 0 to 1 due to event 1
[2010-10-03 08:48:08.659026] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:08.659683] I [glusterd3_1-mops.c:410:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e
[2010-10-03 08:48:08.659701] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:08.659720] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:08.659801] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 1 due to event 3
[2010-10-03 08:48:08.659846] I [glusterd3_1-mops.c:410:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:48:08.659862] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:08.659877] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:08.659892] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 1 due to event 3
[2010-10-03 08:48:08.659920] I [glusterd3_1-mops.c:410:glusterd3_1_cluster_lock_cbk] glusterd: Received ACC from uuid: 71042154-2a52-4796-a246-1120731a2fa4
[2010-10-03 08:48:08.659936] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:08.659950] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:08.659965] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 4
[2010-10-03 08:48:08.659980] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 1 due to event 3
[2010-10-03 08:48:08.660007] I [glusterd-utils.c:660:glusterd_brickinfo_get] : brick: 10.198.110.16:/mnt/oct3
[2010-10-03 08:48:08.660025] I [glusterd-utils.c:1985:glusterd_friend_find_by_hostname] glusterd: Friend 10.198.110.16 found.. state: 3
[2010-10-03 08:48:08.660041] I [glusterd-utils.c:696:glusterd_brickinfo_get] : Found brick
[2010-10-03 08:48:08.660056] E [glusterd-op-sm.c:718:glusterd_op_stage_add_brick] : Adding duplicate brick: 10.198.110.16:/mnt/oct3
[2010-10-03 08:48:08.660073] E [glusterd3_1-mops.c:1164:glusterd3_1_stage_op] : Staging failed
[2010-10-03 08:48:08.660088] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 7
[2010-10-03 08:48:08.660103] E [glusterd-op-sm.c:4819:glusterd_op_sm] glusterd: handler returned: -1
[2010-10-03 08:48:08.660166] I [glusterd3_1-mops.c:1108:glusterd3_1_cluster_unlock] glusterd: Sent unlock req to 3 peers
[2010-10-03 08:48:08.660183] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 1 to 7 due to event 7
[2010-10-03 08:48:08.660542] I [glusterd3_1-mops.c:471:glusterd3_1_cluster_unlock_cbk] glusterd: Received ACC from uuid: bd364a85-e8b3-43d6-8a88-363df38be63c
[2010-10-03 08:48:08.660568] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:08.660589] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:08.660648] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 7 due to event 3
[2010-10-03 08:48:08.660679] I [glusterd3_1-mops.c:471:glusterd3_1_cluster_unlock_cbk] glusterd: Received ACC from uuid: 1d93452f-3be9-4bb4-b1f2-264e3d825b5e
[2010-10-03 08:48:08.660694] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:08.660710] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:08.660725] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 7 due to event 3
[2010-10-03 08:48:08.660752] I [glusterd3_1-mops.c:471:glusterd3_1_cluster_unlock_cbk] glusterd: Received ACC from uuid: 71042154-2a52-4796-a246-1120731a2fa4
[2010-10-03 08:48:08.660767] I [glusterd-handler.c:87:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: 3
[2010-10-03 08:48:08.660782] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 3
[2010-10-03 08:48:08.660797] I [glusterd-op-sm.c:4758:glusterd_op_sm_inject_event] glusterd: Enqueuing event: 4
[2010-10-03 08:48:08.660812] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 7 due to event 3
[2010-10-03 08:48:08.660830] I [glusterd-op-sm.c:4277:glusterd_op_txn_complete] glusterd: Cleared local lock
[2010-10-03 08:48:08.660891] I [glusterd-op-sm.c:4241:glusterd_op_send_cli_response] : Returning 0
[2010-10-03 08:48:08.660910] I [glusterd-op-sm.c:4320:glusterd_op_txn_complete] glusterd: Returning 0
[2010-10-03 08:48:08.660925] I [glusterd-op-sm.c:4439:glusterd_op_sm_transition_state] : Transitioning from 7 to 0 due to event 4
[2010-10-03 08:48:56.789130] I [glusterd-handler.c:675:glusterd_handle_cli_get_volume] glusterd: Received get vol req
[2010-10-03 08:48:56.789250] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
[2010-10-03 08:48:56.789422] I [glusterd-handler.c:675:glusterd_handle_cli_get_volume] glusterd: Received get vol req
[2010-10-03 08:48:56.789471] W [glusterd3_1-mops.c:1454:glusterd_handle_rpc_msg] : Returning 0
=============================
Comment 2 Amar Tumballi 2010-10-03 22:14:57 EDT
Shishir, Can you make out anything from the logs? Lets work on this today, as 'add-brick' is the most highlighted feature of 3.1 release.
Comment 3 Amar Tumballi 2010-10-03 23:59:49 EDT
Below log is the exact reason why its failed..

[glusterd-op-sm.c:718:glusterd_op_stage_add_brick] : Adding duplicate brick:
10.198.110.16:/mnt/oct3
[2010-10-03 08:48:02.720547] E [glusterd-op-sm.c:4370:glusterd_op_ac_stage_op]
: Validate failed: -1


Need to see why its saying a duplicate brick.

Amar
Comment 4 Pranith Kumar K 2010-10-04 23:19:08 EDT
fixed as the commit 2420fd1f0cc5c357776d3766884b86859f3c3781. Not sure why it is not updated in the bug.

Moving to resolved.

Note You need to log in before you can comment on or make changes to this bug.