Attachment 826106 Details for Bug 1032122 – last words from glusterd at debug

last words from glusterd at debug

notes (text/plain), 30.19 KB, created by Lukas Bezdicka on 2013-11-19 15:17:02 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Lukas Bezdicka

Created: 2013-11-19 15:17:02 UTC

Size: 30.19 KB

patch

obsolete

>node1:
>
>
>[2013-11-18 13:24:46.991280] E [glusterd-handler.c:497:glusterd_op_txn_begin] 0-management: Unable to acquire lock on localhost, ret: -1
>[2013-11-18 13:24:46.991289] D [glusterd-handler.c:522:glusterd_op_txn_begin] 0-management: Returning -1
>[2013-11-18 13:24:46.991352] D [glusterd-rpc-ops.c:183:glusterd_op_send_cli_response] 0-management: Returning 0
>[2013-11-18 13:24:46.995212] D [socket.c:486:__socket_rwv] 0-socket.management: EOF on socket
>[2013-11-18 13:24:46.995240] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now
>[2013-11-18 13:24:46.995296] D [socket.c:486:__socket_rwv] 0-socket.management: EOF on socket
>[2013-11-18 13:24:46.995309] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now
>[2013-11-18 13:24:47.092028] D [mem-pool.c:422:mem_get] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103) [0x7f6b97ec80d3] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x116) [0x7f6b97ec7e56] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0x79) [0x7f6b97ec7ab9]))) 0-mem-pool: Mem pool is full. Callocing mem
>[2013-11-18 13:24:47.093228] I [glusterd-volume-ops.c:473:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume kachna
>[2013-11-18 13:24:47.093247] D [glusterd-utils.c:1084:glusterd_volinfo_find] 0-: Volume kachna found
>[2013-11-18 13:24:47.093255] D [glusterd-utils.c:1091:glusterd_volinfo_find] 0-: Returning 0
>[2013-11-18 13:24:47.093280] E [glusterd-utils.c:329:glusterd_lock] 0-management: Unable to get lock for uuid: b0c590fd-0fa4-4100-9675-5028c2e135cd, lock held by: b0c590fd-0fa4-4100-9675-5028c2e135cd
>[2013-11-18 13:24:47.093291] E [glusterd-handler.c:497:glusterd_op_txn_begin] 0-management: Unable to acquire lock on localhost, ret: -1
>[2013-11-18 13:24:47.093300] D [glusterd-handler.c:522:glusterd_op_txn_begin] 0-management: Returning -1
>[2013-11-18 13:24:47.093435] D [glusterd-rpc-ops.c:183:glusterd_op_send_cli_response] 0-management: Returning 0
>[2013-11-18 13:24:47.116936] D [socket.c:486:__socket_rwv] 0-socket.management: EOF on socket
>[2013-11-18 13:24:47.116979] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now
>[2013-11-18 13:24:47.200106] D [mem-pool.c:422:mem_get] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103) [0x7f6b97ec80d3] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x116) [0x7f6b97ec7e56] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0x79) [0x7f6b97ec7ab9]))) 0-mem-pool: Mem pool is full. Callocing mem
>[2013-11-18 13:24:47.201234] I [glusterd-volume-ops.c:473:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume kachna
>[2013-11-18 13:24:47.201253] D [glusterd-utils.c:1084:glusterd_volinfo_find] 0-: Volume kachna found
>[2013-11-18 13:24:47.201261] D [glusterd-utils.c:1091:glusterd_volinfo_find] 0-: Returning 0
>[2013-11-18 13:24:47.201285] E [glusterd-utils.c:329:glusterd_lock] 0-management: Unable to get lock for uuid: b0c590fd-0fa4-4100-9675-5028c2e135cd, lock held by: b0c590fd-0fa4-4100-9675-5028c2e135cd
>[2013-11-18 13:24:47.201298] E [glusterd-handler.c:497:glusterd_op_txn_begin] 0-management: Unable to acquire lock on localhost, ret: -1
>[2013-11-18 13:24:47.201319] D [glusterd-handler.c:522:glusterd_op_txn_begin] 0-management: Returning -1
>[2013-11-18 13:24:47.201416] D [glusterd-rpc-ops.c:183:glusterd_op_send_cli_response] 0-management: Returning 0
>[2013-11-18 13:24:47.206284] D [socket.c:486:__socket_rwv] 0-socket.management: EOF on socket
>[2013-11-18 13:24:47.206318] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now
>[2013-11-18 13:24:47.354470] D [mem-pool.c:422:mem_get] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103) [0x7f6b97ec80d3] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x116) [0x7f6b97ec7e56] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0x79) [0x7f6b97ec7ab9]))) 0-mem-pool: Mem pool is full. Callocing mem
>[2013-11-18 13:24:47.355788] I [glusterd-volume-ops.c:473:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume kachna
>[2013-11-18 13:24:47.355821] D [glusterd-utils.c:1084:glusterd_volinfo_find] 0-: Volume kachna found
>[2013-11-18 13:24:47.355830] D [glusterd-utils.c:1091:glusterd_volinfo_find] 0-: Returning 0
>[2013-11-18 13:24:47.355857] E [glusterd-utils.c:329:glusterd_lock] 0-management: Unable to get lock for uuid: b0c590fd-0fa4-4100-9675-5028c2e135cd, lock held by: b0c590fd-0fa4-4100-9675-5028c2e135cd
>[2013-11-18 13:24:47.355870] E [glusterd-handler.c:497:glusterd_op_txn_begin] 0-management: Unable to acquire lock on localhost, ret: -1
>[2013-11-18 13:24:47.355878] D [glusterd-handler.c:522:glusterd_op_txn_begin] 0-management: Returning -1
>[2013-11-18 13:24:47.355998] D [glusterd-rpc-ops.c:183:glusterd_op_send_cli_response] 0-management: Returning 0
>[2013-11-18 13:24:47.363192] D [socket.c:486:__socket_rwv] 0-socket.management: EOF on socket
>[2013-11-18 13:24:47.363209] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now
>[2013-11-18 13:24:47.502265] D [mem-pool.c:422:mem_get] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103) [0x7f6b97ec80d3] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x116) [0x7f6b97ec7e56] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0x79) [0x7f6b97ec7ab9]))) 0-mem-pool: Mem pool is full. Callocing mem
>[2013-11-18 13:24:47.503396] I [glusterd-volume-ops.c:473:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume kachna
>[2013-11-18 13:24:47.503416] D [glusterd-utils.c:1084:glusterd_volinfo_find] 0-: Volume kachna found
>[2013-11-18 13:24:47.503425] D [glusterd-utils.c:1091:glusterd_volinfo_find] 0-: Returning 0
>[2013-11-18 13:24:47.503449] E [glusterd-utils.c:329:glusterd_lock] 0-management: Unable to get lock for uuid: b0c590fd-0fa4-4100-9675-5028c2e135cd, lock held by: b0c590fd-0fa4-4100-9675-5028c2e135cd
>[2013-11-18 13:24:47.503463] E [glusterd-handler.c:497:glusterd_op_txn_begin] 0-management: Unable to acquire lock on localhost, ret: -1
>[2013-11-18 13:24:47.503471] D [glusterd-handler.c:522:glusterd_op_txn_begin] 0-management: Returning -1
>[2013-11-18 13:24:47.503634] D [glusterd-rpc-ops.c:183:glusterd_op_send_cli_response] 0-management: Returning 0
>[2013-11-18 13:24:47.516392] D [socket.c:486:__socket_rwv] 0-socket.management: EOF on socket
>[2013-11-18 13:24:47.516419] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now
>[2013-11-18 13:24:47.550043] D [mem-pool.c:422:mem_get] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103) [0x7f6b97ec80d3] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x116) [0x7f6b97ec7e56] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0x79) [0x7f6b97ec7ab9]))) 0-mem-pool: Mem pool is full. Callocing mem
>[2013-11-18 13:24:47.551073] I [glusterd-volume-ops.c:473:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume kachna
>[2013-11-18 13:24:47.551091] D [glusterd-utils.c:1084:glusterd_volinfo_find] 0-: Volume kachna found
>[2013-11-18 13:24:47.551098] D [glusterd-utils.c:1091:glusterd_volinfo_find] 0-: Returning 0
>[2013-11-18 13:24:47.551120] E [glusterd-utils.c:329:glusterd_lock] 0-management: Unable to get lock for uuid: b0c590fd-0fa4-4100-9675-5028c2e135cd, lock held by: b0c590fd-0fa4-4100-9675-5028c2e135cd
>[2013-11-18 13:24:47.551132] E [glusterd-handler.c:497:glusterd_op_txn_begin] 0-management: Unable to acquire lock on localhost, ret: -1
>[2013-11-18 13:24:47.551140] D [glusterd-handler.c:522:glusterd_op_txn_begin] 0-management: Returning -1
>[2013-11-18 13:24:47.551225] D [glusterd-rpc-ops.c:183:glusterd_op_send_cli_response] 0-management: Returning 0
>[2013-11-18 13:24:47.556042] D [socket.c:486:__socket_rwv] 0-socket.management: EOF on socket
>[2013-11-18 13:24:47.556061] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now
>[2013-11-18 13:24:47.792304] D [mem-pool.c:422:mem_get] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103) [0x7f6b97ec80d3] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x116) [0x7f6b97ec7e56] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0x79) [0x7f6b97ec7ab9]))) 0-mem-pool: Mem pool is full. Callocing mem
>[2013-11-18 13:24:47.793217] I [glusterd-volume-ops.c:473:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume kachna
>[2013-11-18 13:24:47.793239] D [glusterd-utils.c:1084:glusterd_volinfo_find] 0-: Volume kachna found
>[2013-11-18 13:24:47.793248] D [glusterd-utils.c:1091:glusterd_volinfo_find] 0-: Returning 0
>[2013-11-18 13:24:47.793273] E [glusterd-utils.c:329:glusterd_lock] 0-management: Unable to get lock for uuid: b0c590fd-0fa4-4100-9675-5028c2e135cd, lock held by: b0c590fd-0fa4-4100-9675-5028c2e135cd
>[2013-11-18 13:24:47.793285] E [glusterd-handler.c:497:glusterd_op_txn_begin] 0-management: Unable to acquire lock on localhost, ret: -1
>[2013-11-18 13:24:47.793294] D [glusterd-handler.c:522:glusterd_op_txn_begin] 0-management: Returning -1
>[2013-11-18 13:24:47.793459] D [glusterd-rpc-ops.c:183:glusterd_op_send_cli_response] 0-management: Returning 0
>[2013-11-18 13:24:47.838011] D [socket.c:486:__socket_rwv] 0-socket.management: EOF on socket
>[2013-11-18 13:24:47.838080] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now
>[2013-11-18 13:24:51.790684] D [socket.c:486:__socket_rwv] 0-socket.management: EOF on socket
>[2013-11-18 13:24:51.790729] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now
>[2013-11-18 13:24:52.079580] D [mem-pool.c:422:mem_get] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103) [0x7f6b97ec80d3] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x116) [0x7f6b97ec7e56] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0x79) [0x7f6b97ec7ab9]))) 0-mem-pool: Mem pool is full. Callocing mem
>[2013-11-18 13:24:52.080552] I [glusterd-volume-ops.c:473:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume kachna
>[2013-11-18 13:24:52.080594] D [glusterd-utils.c:1084:glusterd_volinfo_find] 0-: Volume kachna found
>[2013-11-18 13:24:52.080603] D [glusterd-utils.c:1091:glusterd_volinfo_find] 0-: Returning 0
>[2013-11-18 13:24:52.080627] E [glusterd-utils.c:329:glusterd_lock] 0-management: Unable to get lock for uuid: b0c590fd-0fa4-4100-9675-5028c2e135cd, lock held by: b0c590fd-0fa4-4100-9675-5028c2e135cd
>[2013-11-18 13:24:52.080639] E [glusterd-handler.c:497:glusterd_op_txn_begin] 0-management: Unable to acquire lock on localhost, ret: -1
>[2013-11-18 13:24:52.080648] D [glusterd-handler.c:522:glusterd_op_txn_begin] 0-management: Returning -1
>[2013-11-18 13:24:52.080748] D [glusterd-rpc-ops.c:183:glusterd_op_send_cli_response] 0-management: Returning 0
>[2013-11-18 13:24:52.116874] D [socket.c:486:__socket_rwv] 0-socket.management: EOF on socket
>[2013-11-18 13:24:52.116912] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now
>[2013-11-18 13:24:52.406187] D [mem-pool.c:422:mem_get] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103) [0x7f6b97ec80d3] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x116) [0x7f6b97ec7e56] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0x79) [0x7f6b97ec7ab9]))) 0-mem-pool: Mem pool is full. Callocing mem
>[2013-11-18 13:24:52.413251] I [glusterd-volume-ops.c:473:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume kachna
>[2013-11-18 13:24:52.413280] D [glusterd-utils.c:1084:glusterd_volinfo_find] 0-: Volume kachna found
>[2013-11-18 13:24:52.413290] D [glusterd-utils.c:1091:glusterd_volinfo_find] 0-: Returning 0
>[2013-11-18 13:24:52.413325] E [glusterd-utils.c:329:glusterd_lock] 0-management: Unable to get lock for uuid: b0c590fd-0fa4-4100-9675-5028c2e135cd, lock held by: b0c590fd-0fa4-4100-9675-5028c2e135cd
>[2013-11-18 13:24:52.413337] E [glusterd-handler.c:497:glusterd_op_txn_begin] 0-management: Unable to acquire lock on localhost, ret: -1
>[2013-11-18 13:24:52.413346] D [glusterd-handler.c:522:glusterd_op_txn_begin] 0-management: Returning -1
>[2013-11-18 13:24:52.413421] D [glusterd-rpc-ops.c:183:glusterd_op_send_cli_response] 0-management: Returning 0
>[2013-11-18 13:24:52.428888] D [socket.c:486:__socket_rwv] 0-socket.management: EOF on socket
>[2013-11-18 13:24:52.428915] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now
>[2013-11-18 13:24:52.665222] D [mem-pool.c:422:mem_get] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103) [0x7f6b97ec80d3] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x116) [0x7f6b97ec7e56] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0x79) [0x7f6b97ec7ab9]))) 0-mem-pool: Mem pool is full. Callocing mem
>[2013-11-18 13:24:52.668372] I [glusterd-volume-ops.c:473:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume kachna
>[2013-11-18 13:24:52.668392] D [glusterd-utils.c:1084:glusterd_volinfo_find] 0-: Volume kachna found
>[2013-11-18 13:24:52.668400] D [glusterd-utils.c:1091:glusterd_volinfo_find] 0-: Returning 0
>[2013-11-18 13:24:52.668425] E [glusterd-utils.c:329:glusterd_lock] 0-management: Unable to get lock for uuid: b0c590fd-0fa4-4100-9675-5028c2e135cd, lock held by: b0c590fd-0fa4-4100-9675-5028c2e135cd
>[2013-11-18 13:24:52.668863] E [glusterd-handler.c:497:glusterd_op_txn_begin] 0-management: Unable to acquire lock on localhost, ret: -1
>[2013-11-18 13:24:52.668874] D [glusterd-handler.c:522:glusterd_op_txn_begin] 0-management: Returning -1
>[2013-11-18 13:24:52.668955] D [glusterd-rpc-ops.c:183:glusterd_op_send_cli_response] 0-management: Returning 0
>[2013-11-18 13:24:52.729289] D [socket.c:486:__socket_rwv] 0-socket.management: EOF on socket
>[2013-11-18 13:24:52.729324] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now
>[2013-11-18 13:24:52.774536] D [socket.c:486:__socket_rwv] 0-socket.management: EOF on socket
>[2013-11-18 13:24:52.774609] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now
>[2013-11-18 13:24:53.035040] D [mem-pool.c:422:mem_get] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103) [0x7f6b97ec80d3] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x116) [0x7f6b97ec7e56] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0x79) [0x7f6b97ec7ab9]))) 0-mem-pool: Mem pool is full. Callocing mem
>[2013-11-18 13:24:53.035995] I [glusterd-handler.c:2360:__glusterd_handle_cli_profile_volume] 0-management: Received volume profile req for volume kachna
>[2013-11-18 13:24:53.036021] E [glusterd-utils.c:329:glusterd_lock] 0-management: Unable to get lock for uuid: b0c590fd-0fa4-4100-9675-5028c2e135cd, lock held by: b0c590fd-0fa4-4100-9675-5028c2e135cd
>[2013-11-18 13:24:53.036032] E [glusterd-handler.c:497:glusterd_op_txn_begin] 0-management: Unable to acquire lock on localhost, ret: -1
>[2013-11-18 13:24:53.036042] D [glusterd-handler.c:522:glusterd_op_txn_begin] 0-management: Returning -1
>[2013-11-18 13:24:53.036143] D [glusterd-rpc-ops.c:183:glusterd_op_send_cli_response] 0-management: Returning 0
>[2013-11-18 13:24:53.036153] D [glusterd-handler.c:2384:__glusterd_handle_cli_profile_volume] 0-management: Returning 0
>[2013-11-18 13:24:53.036405] D [mem-pool.c:422:mem_get] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103) [0x7f6b97ec80d3] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x116) [0x7f6b97ec7e56] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0x79) [0x7f6b97ec7ab9]))) 0-mem-pool: Mem pool is full. Callocing mem
>[2013-11-18 13:24:53.037418] I [glusterd-volume-ops.c:473:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume kachna
>[2013-11-18 13:24:53.037443] D [glusterd-utils.c:1084:glusterd_volinfo_find] 0-: Volume kachna found
>[2013-11-18 13:24:53.037454] D [glusterd-utils.c:1091:glusterd_volinfo_find] 0-: Returning 0
>[2013-11-18 13:24:53.037473] E [glusterd-utils.c:329:glusterd_lock] 0-management: Unable to get lock for uuid: b0c590fd-0fa4-4100-9675-5028c2e135cd, lock held by: b0c590fd-0fa4-4100-9675-5028c2e135cd
>[2013-11-18 13:24:53.037483] E [glusterd-handler.c:497:glusterd_op_txn_begin] 0-management: Unable to acquire lock on localhost, ret: -1
>[2013-11-18 13:24:53.037493] D [glusterd-handler.c:522:glusterd_op_txn_begin] 0-management: Returning -1
>[2013-11-18 13:24:53.037578] D [glusterd-rpc-ops.c:183:glusterd_op_send_cli_response] 0-management: Returning 0
>[2013-11-18 13:24:53.093073] D [socket.c:486:__socket_rwv] 0-socket.management: EOF on socket
>[2013-11-18 13:24:53.093115] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now
>[2013-11-18 13:24:53.113516] D [socket.c:486:__socket_rwv] 0-socket.management: EOF on socket
>[2013-11-18 13:24:53.113551] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now
>[2013-11-18 13:24:53.231883] D [mem-pool.c:422:mem_get] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103) [0x7f6b97ec80d3] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x116) [0x7f6b97ec7e56] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0x79) [0x7f6b97ec7ab9]))) 0-mem-pool: Mem pool is full. Callocing mem
>[2013-11-18 13:24:53.232889] I [glusterd-handler.c:2360:__glusterd_handle_cli_profile_volume] 0-management: Received volume profile req for volume kachna
>[2013-11-18 13:24:53.232916] E [glusterd-utils.c:329:glusterd_lock] 0-management: Unable to get lock for uuid: b0c590fd-0fa4-4100-9675-5028c2e135cd, lock held by: b0c590fd-0fa4-4100-9675-5028c2e135cd
>[2013-11-18 13:24:53.232931] E [glusterd-handler.c:497:glusterd_op_txn_begin] 0-management: Unable to acquire lock on localhost, ret: -1
>[2013-11-18 13:24:53.232940] D [glusterd-handler.c:522:glusterd_op_txn_begin] 0-management: Returning -1
>[2013-11-18 13:24:53.233016] D [glusterd-rpc-ops.c:183:glusterd_op_send_cli_response] 0-management: Returning 0
>[2013-11-18 13:24:53.233026] D [glusterd-handler.c:2384:__glusterd_handle_cli_profile_volume] 0-management: Returning 0
>[2013-11-18 13:24:53.258466] D [socket.c:486:__socket_rwv] 0-socket.management: EOF on socket
>[2013-11-18 13:24:53.258500] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now
>[2013-11-18 13:24:53.475149] D [mem-pool.c:422:mem_get] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103) [0x7f6b97ec80d3] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x116) [0x7f6b97ec7e56] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0x79) [0x7f6b97ec7ab9]))) 0-mem-pool: Mem pool is full. Callocing mem
>[2013-11-18 13:24:53.476102] I [glusterd-handler.c:2360:__glusterd_handle_cli_profile_volume] 0-management: Received volume profile req for volume kachna
>[2013-11-18 13:24:53.476128] E [glusterd-utils.c:329:glusterd_lock] 0-management: Unable to get lock for uuid: b0c590fd-0fa4-4100-9675-5028c2e135cd, lock held by: b0c590fd-0fa4-4100-9675-5028c2e135cd
>[2013-11-18 13:24:53.476142] E [glusterd-handler.c:497:glusterd_op_txn_begin] 0-management: Unable to acquire lock on localhost, ret: -1
>[2013-11-18 13:24:53.476151] D [glusterd-handler.c:522:glusterd_op_txn_begin] 0-management: Returning -1
>[2013-11-18 13:24:53.476262] D [glusterd-rpc-ops.c:183:glusterd_op_send_cli_response] 0-management: Returning 0
>[2013-11-18 13:24:53.476273] D [glusterd-handler.c:2384:__glusterd_handle_cli_profile_volume] 0-management: Returning 0
>[2013-11-18 13:24:53.493859] D [socket.c:486:__socket_rwv] 0-socket.management: EOF on socket
>[2013-11-18 13:24:53.493898] D [socket.c:2236:socket_event_handler] 0-transport: disconnecting now
>[2013-11-18 13:24:53.729594] D [mem-pool.c:422:mem_get] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103) [0x7f6b97ec80d3] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x116) [0x7f6b97ec7e56] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0x79) [0x7f6b97ec7ab9]))) 0-mem-pool: Mem pool is full. Callocing mem
>[2013-11-18 13:24:53.730401] I [glusterd-handler.c:2360:__glusterd_handle_cli_profile_volume] 0-management: Received volume profile req for volume kachna
>[2013-11-18 13:24:53.730425] E [glusterd-utils.c:329:glusterd_lock] 0-management: Unable to get lock for uuid: b0c590fd-0fa4-4100-9675-5028c2e135cd, lock held by: b0c590fd-0fa4-4100-9675-5028c2e135cd
>[2013-11-18 13:24:53.730439] E [glusterd-handler.c:497:glusterd_op_txn_begin] 0-management: Unable to acquire lock on localhost, ret: -1
>
>
>
>node2:
>
>
>-4100-9675-5028c2e135cd
>[2013-11-18 13:24:47.019954] D [glusterd-op-sm.c:2442:glusterd_op_ac_lock] 0-management: Lock Returned -1
>[2013-11-18 13:24:47.019966] D [glusterd-handler.c:1776:glusterd_op_lock_send_resp] 0-management: Responded to lock, ret: 0
>[2013-11-18 13:24:47.019972] E [glusterd-op-sm.c:5378:glusterd_op_sm] 0-management: handler returned: -1
>[2013-11-18 13:24:47.020039] D [mem-pool.c:422:mem_get] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103) [0x7fd4dcac00d3] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x116) [0x7fd4dcabfe56] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0x79) [0x7fd4dcabfab9]))) 0-mem-pool: Mem pool is full. Callocing mem
>[2013-11-18 13:24:47.020747] D [glusterd-handler.c:549:__glusterd_handle_cluster_lock] 0-management: Received LOCK from uuid: b0c590fd-0fa4-4100-9675-5028c2e135cd
>[2013-11-18 13:24:47.020758] D [glusterd-utils.c:4796:glusterd_friend_find_by_uuid] 0-management: Friend found... state: Peer in Cluster
>[2013-11-18 13:24:47.020765] D [glusterd-op-sm.c:5288:glusterd_op_sm_inject_event] 0-management: Enqueue event: 'GD_OP_EVENT_LOCK'
>[2013-11-18 13:24:47.020771] D [glusterd-handler.c:572:__glusterd_handle_cluster_lock] 0-management: Returning 0
>[2013-11-18 13:24:47.020778] D [glusterd-op-sm.c:5365:glusterd_op_sm] 0-management: Dequeued event of type: 'GD_OP_EVENT_LOCK'
>[2013-11-18 13:24:47.020787] E [glusterd-utils.c:329:glusterd_lock] 0-management: Unable to get lock for uuid: b0c590fd-0fa4-4100-9675-5028c2e135cd, lock held by: b0c590fd-0fa4-4100-9675-5028c2e135cd
>[2013-11-18 13:24:47.020796] D [glusterd-op-sm.c:2442:glusterd_op_ac_lock] 0-management: Lock Returned -1
>[2013-11-18 13:24:47.020808] D [glusterd-handler.c:1776:glusterd_op_lock_send_resp] 0-management: Responded to lock, ret: 0
>[2013-11-18 13:24:47.020815] E [glusterd-op-sm.c:5378:glusterd_op_sm] 0-management: handler returned: -1
>[2013-11-18 13:24:47.020970] D [mem-pool.c:422:mem_get] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103) [0x7fd4dcac00d3] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x116) [0x7fd4dcabfe56] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0x79) [0x7fd4dcabfab9]))) 0-mem-pool: Mem pool is full. Callocing mem
>[2013-11-18 13:24:47.021695] D [glusterd-handler.c:549:__glusterd_handle_cluster_lock] 0-management: Received LOCK from uuid: b0c590fd-0fa4-4100-9675-5028c2e135cd
>[2013-11-18 13:24:47.021707] D [glusterd-utils.c:4796:glusterd_friend_find_by_uuid] 0-management: Friend found... state: Peer in Cluster
>[2013-11-18 13:24:47.021714] D [glusterd-op-sm.c:5288:glusterd_op_sm_inject_event] 0-management: Enqueue event: 'GD_OP_EVENT_LOCK'
>[2013-11-18 13:24:47.021720] D [glusterd-handler.c:572:__glusterd_handle_cluster_lock] 0-management: Returning 0
>[2013-11-18 13:24:47.021726] D [glusterd-op-sm.c:5365:glusterd_op_sm] 0-management: Dequeued event of type: 'GD_OP_EVENT_LOCK'
>[2013-11-18 13:24:47.021753] E [glusterd-utils.c:329:glusterd_lock] 0-management: Unable to get lock for uuid: b0c590fd-0fa4-4100-9675-5028c2e135cd, lock held by: b0c590fd-0fa4-4100-9675-5028c2e135cd
>[2013-11-18 13:24:47.021761] D [glusterd-op-sm.c:2442:glusterd_op_ac_lock] 0-management: Lock Returned -1
>[2013-11-18 13:24:47.021773] D [glusterd-handler.c:1776:glusterd_op_lock_send_resp] 0-management: Responded to lock, ret: 0
>[2013-11-18 13:24:47.021780] E [glusterd-op-sm.c:5378:glusterd_op_sm] 0-management: handler returned: -1
>[2013-11-18 13:24:47.021931] D [mem-pool.c:422:mem_get] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103) [0x7fd4dcac00d3] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x116) [0x7fd4dcabfe56] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0x79) [0x7fd4dcabfab9]))) 0-mem-pool: Mem pool is full. Callocing mem
>[2013-11-18 13:24:47.022668] D [glusterd-handler.c:549:__glusterd_handle_cluster_lock] 0-management: Received LOCK from uuid: b0c590fd-0fa4-4100-9675-5028c2e135cd
>[2013-11-18 13:24:47.022680] D [glusterd-utils.c:4796:glusterd_friend_find_by_uuid] 0-management: Friend found... state: Peer in Cluster
>[2013-11-18 13:24:47.022687] D [glusterd-op-sm.c:5288:glusterd_op_sm_inject_event] 0-management: Enqueue event: 'GD_OP_EVENT_LOCK'
>[2013-11-18 13:24:47.022693] D [glusterd-handler.c:572:__glusterd_handle_cluster_lock] 0-management: Returning 0
>[2013-11-18 13:24:47.022699] D [glusterd-op-sm.c:5365:glusterd_op_sm] 0-management: Dequeued event of type: 'GD_OP_EVENT_LOCK'
>[2013-11-18 13:24:47.022707] E [glusterd-utils.c:329:glusterd_lock] 0-management: Unable to get lock for uuid: b0c590fd-0fa4-4100-9675-5028c2e135cd, lock held by: b0c590fd-0fa4-4100-9675-5028c2e135cd
>[2013-11-18 13:24:47.022715] D [glusterd-op-sm.c:2442:glusterd_op_ac_lock] 0-management: Lock Returned -1
>[2013-11-18 13:24:47.022726] D [glusterd-handler.c:1776:glusterd_op_lock_send_resp] 0-management: Responded to lock, ret: 0
>[2013-11-18 13:24:47.022732] E [glusterd-op-sm.c:5378:glusterd_op_sm] 0-management: handler returned: -1
>[2013-11-18 13:24:47.022797] D [mem-pool.c:422:mem_get] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103) [0x7fd4dcac00d3] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x116) [0x7fd4dcabfe56] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0x79) [0x7fd4dcabfab9]))) 0-mem-pool: Mem pool is full. Callocing mem
>[2013-11-18 13:24:47.023537] D [glusterd-handler.c:549:__glusterd_handle_cluster_lock] 0-management: Received LOCK from uuid: b0c590fd-0fa4-4100-9675-5028c2e135cd
>[2013-11-18 13:24:47.023549] D [glusterd-utils.c:4796:glusterd_friend_find_by_uuid] 0-management: Friend found... state: Peer in Cluster
>[2013-11-18 13:24:47.023556] D [glusterd-op-sm.c:5288:glusterd_op_sm_inject_event] 0-management: Enqueue event: 'GD_OP_EVENT_LOCK'
>[2013-11-18 13:24:47.023562] D [glusterd-handler.c:572:__glusterd_handle_cluster_lock] 0-management: Returning 0
>[2013-11-18 13:24:47.023569] D [glusterd-op-sm.c:5365:glusterd_op_sm] 0-management: Dequeued event of type: 'GD_OP_EVENT_LOCK'
>[2013-11-18 13:24:47.023577] E [glusterd-utils.c:329:glusterd_lock] 0-management: Unable to get lock for uuid: b0c590fd-0fa4-4100-9675-5028c2e135cd, lock held by: b0c590fd-0fa4-4100-9675-5028c2e135cd
>[2013-11-18 13:24:47.023598] D [glusterd-op-sm.c:2442:glusterd_op_ac_lock] 0-management: Lock Returned -1
>[2013-11-18 13:24:47.023611] D [glusterd-handler.c:1776:glusterd_op_lock_send_resp] 0-management: Responded to lock, ret: 0
>[2013-11-18 13:24:47.023618] E [glusterd-op-sm.c:5378:glusterd_op_sm] 0-management: handler returned: -1
>[2013-11-18 13:24:47.023686] D [mem-pool.c:422:mem_get] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103) [0x7fd4dcac00d3] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x116) [0x7fd4dcabfe56] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0x79) [0x7fd4dcabfab9]))) 0-mem-pool: Mem pool is full. Callocing mem
>[2013-11-18 13:24:47.024738] D [glusterd-handler.c:549:__glusterd_handle_cluster_lock] 0-management: Received LOCK from uuid: b0c590fd-0fa4-4100-9675-5028c2e135cd
>[2013-11-18 13:24:47.024752] D [glusterd-utils.c:4796:glusterd_friend_find_by_uuid] 0-management: Friend found... state: Peer in Cluster
>[2013-11-18 13:24:47.024760] D [glusterd-op-sm.c:5288:glusterd_op_sm_inject_event] 0-management: Enqueue event: 'GD_OP_EVENT_LOCK'
>[2013-11-18 13:24:47.024766] D [glusterd-handler.c:572:__glusterd_handle_cluster_lock] 0-management: Returning 0
>[2013-11-18 13:24:47.024772] D [glusterd-op-sm.c:5365:glusterd_op_sm] 0-management: Dequeued event of type: 'GD_OP_EVENT_LOCK'
>[2013-11-18 13:24:47.024780] E [glusterd-utils.c:329:glusterd_lock] 0-management: Unable to get lock for uuid: b0c590fd-0fa4-4100-9675-5028c2e135cd, lock held by: b0c590fd-0fa4-4100-9675-5028c2e135cd
>[2013-11-18 13:24:47.024787] D [glusterd-op-sm.c:2442:glusterd_op_ac_lock] 0-management: Lock Returned -1
>[2013-11-18 13:24:47.024799] D [glusterd-handler.c:1776:glusterd_op_lock_send_resp] 0-management: Responded to lock, ret: 0
>[2013-11-18 13:24:47.024806] E [glusterd-op-sm.c:5378:glusterd_op_sm] 0-management: handler returned: -1
>[2013-11-18 13:24:47.024887] D [mem-pool.c:422:mem_get] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103) [0x7fd4dcac00d3] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x116) [0x7fd4dcabfe56] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0x79) [0x7fd4dcabfab9]))) 0-mem-pool: Mem pool is full. Callocing mem
>[2013-11-18 13:24:47.025622] D [glusterd-handler.c:549:__glusterd_handle_cluster_lock] 0-management: Received LOCK from uuid: b0c590fd-0fa4-4100-9675-5028c2e135cd
>[2013-11-18 13:24:47.025633] D [glusterd-utils.c:4796:glusterd_friend_find_by_uuid] 0-management: Friend found... state: Peer in Cluster
>[2013-11-18 13:24:47.025640] D [glusterd-op-sm.c:5288:glusterd_op_sm_inject_event] 0-management: Enqueue event: 'GD_OP_EVENT_LOCK'
>[2013-11-18 13:24:47.025646] D [glusterd-handler.c:572:__glusterd_handle_cluster_lock] 0-management: Returning 0
>[2013-11-18 13:24:47.025652] D [glusterd-op-sm.c:5365:glusterd_op_sm] 0-management: Dequeued event of type: 'GD_OP_EVENT_LOCK'
>[2013-11-18 13:24:47.025661] E [glusterd-utils.c:329:glusterd_lock] 0-management: Unable to get lock for uuid: b0c590fd-0fa4-4100-9675-5028c2e135cd, lock held by: b0c590fd-0fa4-4100-9675-5028c2e135cd
>[2013-11-18 13:24:47.025667] D [glusterd-op-sm.c:2442:glusterd_op_ac_lock] 0-management: Lock Returned -1
>[2013-11-18 13:24:47.025685] D [glusterd-handler.c:1776:glusterd_op_lock_send_resp] 0-management: Responded to lock, ret: 0
>[2013-11-18 13:24:47.025692] E [glusterd-op-sm.c:5378:glusterd_op_sm] 0-management: handler returned: -1
>[2013-11-18 13:24:47.025756] D [mem-pool.c:422:mem_get] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_notify+0x103) [0x7fd4dcac00d3] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_handle_rpc_call+0x116) [0x7fd4dcabfe56] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0x79) [0x7fd4dcabfab9]))) 0-mem-pool: Mem pool is full. Callocing mem
>gfrpc.so.0(rpcsvc_handle_rpc_call+0x116) [0x7fd4dcabfe56] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0x79) [0x7fd4dcabfab9]))) 0-mem-pool: Mem pool is full. Callocing mem
>[2013-11-18 13:24:47.039077] D [glusterd-utils.c:4796:glusterd_friend_find_bgfrpc.so.0(rpcsvc_handle_rpc_call+0x116) [0x7fd4dcabfe56] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0x79) [0x7fd4dcabfab9]))) 0-mem-pool: Mem pool is full. Callocing mem
>[2013-11-18 13:24:47.058154] D [glusterd-utils.c:4796:glusterd_friend_find_bgfrpc.so.0(rpcsvc_handle_rpc_call+0x116) [0x7fd4dcabfe56] (-->/usr/lib64/libgfrpc.so.0(rpcsvc_request_create+0x79) [0x7fd4dcabfab9]))) 0-mem-pool: Mem pool is full. Callocing mem
>[2013-11-18 13:24:47.062051] D [glusterd-utils.c:4796:glusterd_friend_find_b[2013-11-18 13:24:47.151554] D [glusterd-handler.c:1776:glusterd_op_lock_send_resp] 0-management: Responded to lock, ret: 0
>[2013-11-18 13:24:47.151605] E [glusterd-op-sm.c:5378:glusterd_op_sm] 0-management: handler returned: -1
>[2013-11-18 13:24:47.1520
>

Actions: View

Attachments on bug 1032122: 826106 | 833514