Attachment 1446228 Details for Bug 1584644 – glusterd.log

glusterd.log

glusterd.log (text/plain), 32.88 KB, created by Saravanakumar on 2018-05-31 11:27:03 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Saravanakumar

Created: 2018-05-31 11:27:03 UTC

Size: 32.88 KB

patch

obsolete

>sh-4.2# tail -300 /var/log/glusterfs/glusterd.log
>[2018-05-29 08:18:23.239354] I [MSGID: 106062] [glusterd-volume-ops.c:2625:glusterd_op_start_volume] 0-management: Global dict not present.
>[2018-05-29 08:18:23.239426] W [MSGID: 106603] [glusterd-utils.c:5509:find_compat_brick_in_vol] 0-management: cluster.max-bricks-per-process options isn't set. Continuing with no limit set for brick multiplexing.
>[2018-05-29 08:18:23.239466] I [glusterd-utils.c:5340:attach_brick] 0-management: add brick /var/lib/heketi/mounts/vg_a629f4e6dbb4933ac2d8f1c23c9bf4bb/brick_3781f861dcfc2e71f201387e4dd8d273/brick to existing process for /var/lib/heketi/mounts/vg_06dcc4216ea19fbbb5761eaf54d6f5dc/brick_ab5bbe8b6bb8e236b135ebfaab987f71/brick
>[2018-05-29 08:18:24.404218] I [rpc-clnt.c:1060:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
>[2018-05-29 08:18:24.404358] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped
>[2018-05-29 08:18:24.404375] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped
>[2018-05-29 08:18:24.405036] I [MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management: Stopping glustershd daemon running in pid: 421
>[2018-05-29 08:18:25.405428] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped
>[2018-05-29 08:18:25.405591] I [MSGID: 106567] [glusterd-svc-mgmt.c:197:glusterd_svc_start] 0-management: Starting glustershd service
>[2018-05-29 08:18:25.410951] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
>[2018-05-29 08:18:25.410977] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped
>[2018-05-29 08:18:25.411006] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
>[2018-05-29 08:18:25.411016] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped
>[2018-05-29 08:18:27.705385] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd879a) [0x7f8384b6579a] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd8255) [0x7f8384b65255] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f8389b2c1c5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh --volname=vol_2aaf562cfd128cc6a174022f18b683b0 --first=no --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd
>[2018-05-29 08:18:27.721464] E [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd879a) [0x7f8384b6579a] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xd81b6) [0x7f8384b651b6] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f8389b2c1c5] ) 0-management: Failed to execute script: /var/lib/glusterd/hooks/1/start/post/S30samba-start.sh --volname=vol_2aaf562cfd128cc6a174022f18b683b0 --first=no --version=1 --volume-op=start --gd-workdir=/var/lib/glusterd
>[2018-05-30 09:29:59.352801] I [MSGID: 100030] [glusterfsd.c:2441:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.8.4 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
>[2018-05-30 09:29:59.361631] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536
>[2018-05-30 09:29:59.361665] I [MSGID: 106479] [glusterd.c:1509:init] 0-management: Using /var/lib/glusterd as working directory
>[2018-05-30 09:29:59.361672] I [MSGID: 106479] [glusterd.c:1514:init] 0-management: Using /var/run/gluster as pid file working directory
>[2018-05-30 09:29:59.365975] E [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.8.4/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
>[2018-05-30 09:29:59.365998] W [rpc-transport.c:287:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine
>[2018-05-30 09:29:59.366004] W [rpcsvc.c:1646:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed
>[2018-05-30 09:29:59.366012] E [MSGID: 106243] [glusterd.c:1797:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
>[2018-05-30 09:30:03.580438] I [MSGID: 106513] [glusterd-store.c:2135:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 31101
>[2018-05-30 09:30:03.886093] I [MSGID: 106498] [glusterd-handler.c:3670:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
>[2018-05-30 09:30:03.886260] I [rpc-clnt.c:1060:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2018-05-30 09:30:03.890824] I [rpc-clnt.c:1060:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2018-05-30 09:30:03.886214] I [MSGID: 106498] [glusterd-handler.c:3670:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
>[2018-05-30 09:30:03.896014] I [MSGID: 106544] [glusterd.c:157:glusterd_uuid_init] 0-management: retrieved UUID: c2ef4540-2864-4a9e-bc43-f9a1f2e3db21
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume management
>  2:     type mgmt/glusterd
>  3:     option rpc-auth.auth-glusterfs on
>  4:     option rpc-auth.auth-unix on
>  5:     option rpc-auth.auth-null on
>  6:     option rpc-auth-allow-insecure on
>  7:     option transport.listen-backlog 10
>  8:     option event-threads 1
>  9:     option ping-timeout 0
> 10:     option transport.socket.read-fail-log off
> 11:     option transport.socket.keepalive-interval 2
> 12:     option transport.socket.keepalive-time 10
> 13:     option transport-type rdma
> 14:     option working-directory /var/lib/glusterd
> 15: end-volume
> 16:  
>+------------------------------------------------------------------------------+
>[2018-05-30 09:30:03.909129] I [MSGID: 101190] [event-epoll.c:602:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
>[2018-05-30 09:30:03.910018] I [MSGID: 106163] [glusterd-handshake.c:1284:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31101
>[2018-05-30 09:30:04.145378] I [MSGID: 106163] [glusterd-handshake.c:1284:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31101
>[2018-05-30 09:30:04.262341] I [MSGID: 106493] [glusterd-rpc-ops.c:478:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: df78611e-886a-439f-b2b0-6a1dd404311e, host: dhcp41-180.lab.eng.blr.redhat.com, port: 0
>[2018-05-30 09:30:04.499418] I [MSGID: 106490] [glusterd-handler.c:2611:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: e2da7fca-5fdf-4502-b160-5c109b58d6a1
>[2018-05-30 09:30:04.574518] I [MSGID: 106493] [glusterd-handler.c:3873:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.70.43.135 (0), ret: 0, op_ret: 0
>[2018-05-30 09:30:04.950120] I [MSGID: 106492] [glusterd-handler.c:2789:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: df78611e-886a-439f-b2b0-6a1dd404311e
>[2018-05-30 09:30:05.174969] I [MSGID: 106502] [glusterd-handler.c:2834:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
>[2018-05-30 09:30:05.175094] I [MSGID: 106493] [glusterd-rpc-ops.c:693:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: df78611e-886a-439f-b2b0-6a1dd404311e
>[2018-05-30 09:30:05.175120] I [MSGID: 106493] [glusterd-rpc-ops.c:478:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: e2da7fca-5fdf-4502-b160-5c109b58d6a1, host: 10.70.43.135, port: 0
>[2018-05-30 09:30:05.266553] I [MSGID: 106490] [glusterd-handler.c:2611:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: df78611e-886a-439f-b2b0-6a1dd404311e
>[2018-05-30 09:30:05.542933] I [MSGID: 106493] [glusterd-handler.c:3873:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to dhcp41-180.lab.eng.blr.redhat.com (0), ret: 0, op_ret: 0
>[2018-05-30 09:30:05.856075] I [MSGID: 106492] [glusterd-handler.c:2789:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: e2da7fca-5fdf-4502-b160-5c109b58d6a1
>[2018-05-30 09:30:05.897651] I [MSGID: 106502] [glusterd-handler.c:2834:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
>[2018-05-30 09:30:05.897768] I [MSGID: 106493] [glusterd-rpc-ops.c:693:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: e2da7fca-5fdf-4502-b160-5c109b58d6a1
>[2018-05-30 09:30:05.897797] I [MSGID: 106492] [glusterd-handler.c:2789:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: df78611e-886a-439f-b2b0-6a1dd404311e
>[2018-05-30 09:30:05.931064] I [MSGID: 106502] [glusterd-handler.c:2834:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
>[2018-05-30 09:30:05.931217] I [MSGID: 106493] [glusterd-rpc-ops.c:693:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: df78611e-886a-439f-b2b0-6a1dd404311e
>[2018-05-30 09:30:05.931246] I [MSGID: 106492] [glusterd-handler.c:2789:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: e2da7fca-5fdf-4502-b160-5c109b58d6a1
>[2018-05-30 09:30:05.964436] I [MSGID: 106502] [glusterd-handler.c:2834:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
>[2018-05-30 09:30:05.964611] I [MSGID: 106493] [glusterd-rpc-ops.c:693:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: e2da7fca-5fdf-4502-b160-5c109b58d6a1
>[2018-05-30 09:30:05.964820] I [rpc-clnt.c:1060:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600
>[2018-05-30 09:30:05.964920] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped
>[2018-05-30 09:30:05.964938] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped
>[2018-05-30 09:30:05.965202] I [rpc-clnt.c:1060:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600
>[2018-05-30 09:30:05.965764] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped
>[2018-05-30 09:30:05.965780] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped
>[2018-05-30 09:30:05.965796] I [MSGID: 106567] [glusterd-svc-mgmt.c:197:glusterd_svc_start] 0-management: Starting glustershd service
>[2018-05-30 09:30:06.983727] I [rpc-clnt.c:1060:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600
>[2018-05-30 09:30:06.983865] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped
>[2018-05-30 09:30:06.983881] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: quotad service is stopped
>[2018-05-30 09:30:06.983902] I [rpc-clnt.c:1060:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600
>[2018-05-30 09:30:06.983977] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
>[2018-05-30 09:30:06.983988] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped
>[2018-05-30 09:30:06.984008] I [rpc-clnt.c:1060:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600
>[2018-05-30 09:30:06.984065] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
>[2018-05-30 09:30:06.984074] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped
>[2018-05-30 09:30:06.984168] I [glusterd-utils.c:5876:glusterd_brick_start] 0-management: starting a fresh brick process for brick /var/lib/heketi/mounts/vg_06dcc4216ea19fbbb5761eaf54d6f5dc/brick_ab5bbe8b6bb8e236b135ebfaab987f71/brick
>[2018-05-30 09:30:06.988603] I [rpc-clnt.c:1060:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
>[2018-05-30 09:30:07.252915] W [MSGID: 106603] [glusterd-utils.c:5509:find_compat_brick_in_vol] 0-management: cluster.max-bricks-per-process options isn't set. Continuing with no limit set for brick multiplexing.
>[2018-05-30 09:30:07.255990] I [socket.c:2465:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
>[2018-05-30 09:30:07.258105] I [MSGID: 106005] [glusterd-handler.c:6120:__glusterd_brick_rpc_notify] 0-management: Brick 10.70.42.122:/var/lib/heketi/mounts/vg_06dcc4216ea19fbbb5761eaf54d6f5dc/brick_ab5bbe8b6bb8e236b135ebfaab987f71/brick has disconnected from glusterd.
>[2018-05-30 09:30:07.259235] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped
>[2018-05-30 09:30:07.259294] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped
>[2018-05-30 09:30:07.260035] I [MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management: Stopping glustershd daemon running in pid: 478
>[2018-05-30 09:30:07.261052] W [socket.c:595:__socket_rwv] 0-glustershd: readv on /var/run/gluster/619715b97601ca5701db5c3573427f9a.socket failed (No data available)
>[2018-05-30 09:30:08.260283] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped
>[2018-05-30 09:30:08.260353] I [MSGID: 106567] [glusterd-svc-mgmt.c:197:glusterd_svc_start] 0-management: Starting glustershd service
>[2018-05-30 09:30:09.264308] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped
>[2018-05-30 09:30:09.264341] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: quotad service is stopped
>[2018-05-30 09:30:09.264387] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
>[2018-05-30 09:30:09.264400] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped
>[2018-05-30 09:30:09.264429] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
>[2018-05-30 09:30:09.264437] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped
>[2018-05-30 09:30:09.264491] I [glusterd-utils.c:5784:glusterd_brick_start] 0-management: discovered already-running brick /var/lib/heketi/mounts/vg_06dcc4216ea19fbbb5761eaf54d6f5dc/brick_ab5bbe8b6bb8e236b135ebfaab987f71/brick
>[2018-05-30 09:30:09.264504] I [MSGID: 106143] [glusterd-pmap.c:278:pmap_registry_bind] 0-pmap: adding brick /var/lib/heketi/mounts/vg_06dcc4216ea19fbbb5761eaf54d6f5dc/brick_ab5bbe8b6bb8e236b135ebfaab987f71/brick on port 49152
>[2018-05-30 09:30:09.840018] I [rpc-clnt.c:1060:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
>[2018-05-30 09:30:09.840247] I [rpc-clnt.c:1060:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
>[2018-05-30 09:30:09.840438] I [MSGID: 106143] [glusterd-pmap.c:278:pmap_registry_bind] 0-pmap: adding brick /var/lib/heketi/mounts/vg_06dcc4216ea19fbbb5761eaf54d6f5dc/brick_ab5bbe8b6bb8e236b135ebfaab987f71/brick on port 49152
>[2018-05-30 09:30:09.840913] I [MSGID: 106488] [glusterd-handler.c:1540:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
>[2018-05-30 09:30:11.840580] I [glusterd-utils.c:5340:attach_brick] 0-management: add brick /var/lib/heketi/mounts/vg_a629f4e6dbb4933ac2d8f1c23c9bf4bb/brick_3781f861dcfc2e71f201387e4dd8d273/brick to existing process for /var/lib/heketi/mounts/vg_06dcc4216ea19fbbb5761eaf54d6f5dc/brick_ab5bbe8b6bb8e236b135ebfaab987f71/brick
>The message "I [MSGID: 106488] [glusterd-handler.c:1540:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 2 times between [2018-05-30 09:30:09.840913] and [2018-05-30 09:30:09.843250]
>[2018-05-30 11:25:23.467760] W [socket.c:595:__socket_rwv] 0-management: readv on 10.70.41.180:24007 failed (No data available)
>[2018-05-30 11:25:23.467809] I [MSGID: 106004] [glusterd-handler.c:6366:__glusterd_peer_rpc_notify] 0-management: Peer <dhcp41-180.lab.eng.blr.redhat.com> (<df78611e-886a-439f-b2b0-6a1dd404311e>), in state <Peer in Cluster>, has disconnected from glusterd.
>[2018-05-30 11:25:23.467898] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x22c4a) [0x7f6c3beadc4a] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2c9f8) [0x7f6c3beb79f8] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xddd45) [0x7f6c3bf68d45] ) 0-management: Lock for vol heketidbstorage not held
>[2018-05-30 11:25:23.467908] W [MSGID: 106118] [glusterd-handler.c:6391:__glusterd_peer_rpc_notify] 0-management: Lock not released for heketidbstorage
>[2018-05-30 11:25:23.467944] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x22c4a) [0x7f6c3beadc4a] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2c9f8) [0x7f6c3beb79f8] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xddd45) [0x7f6c3bf68d45] ) 0-management: Lock for vol vol_2aaf562cfd128cc6a174022f18b683b0 not held
>[2018-05-30 11:25:23.467952] W [MSGID: 106118] [glusterd-handler.c:6391:__glusterd_peer_rpc_notify] 0-management: Lock not released for vol_2aaf562cfd128cc6a174022f18b683b0
>[2018-05-30 11:25:37.240674] E [socket.c:2360:socket_connect_finish] 0-management: connection to 10.70.41.180:24007 failed (No route to host); disconnecting socket
>[2018-05-30 11:27:09.231427] I [MSGID: 106163] [glusterd-handshake.c:1284:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31101
>[2018-05-30 11:27:09.345542] I [MSGID: 106490] [glusterd-handler.c:2611:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: df78611e-886a-439f-b2b0-6a1dd404311e
>[2018-05-30 11:27:09.578232] I [MSGID: 106493] [glusterd-handler.c:3873:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to dhcp41-180.lab.eng.blr.redhat.com (0), ret: 0, op_ret: 0
>[2018-05-30 11:27:09.788967] I [MSGID: 106492] [glusterd-handler.c:2789:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: df78611e-886a-439f-b2b0-6a1dd404311e
>[2018-05-30 11:27:09.961585] I [MSGID: 106502] [glusterd-handler.c:2834:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
>[2018-05-30 11:27:09.963594] I [MSGID: 106493] [glusterd-rpc-ops.c:693:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: df78611e-886a-439f-b2b0-6a1dd404311e
>[2018-05-30 11:27:11.887642] I [MSGID: 106493] [glusterd-rpc-ops.c:478:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: df78611e-886a-439f-b2b0-6a1dd404311e, host: dhcp41-180.lab.eng.blr.redhat.com, port: 0
>[2018-05-30 11:27:11.936670] I [MSGID: 106492] [glusterd-handler.c:2789:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: df78611e-886a-439f-b2b0-6a1dd404311e
>[2018-05-30 11:27:12.011652] I [MSGID: 106502] [glusterd-handler.c:2834:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
>[2018-05-30 11:27:12.087772] I [MSGID: 106493] [glusterd-rpc-ops.c:693:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: df78611e-886a-439f-b2b0-6a1dd404311e
>[2018-05-30 11:28:40.226569] I [MSGID: 106004] [glusterd-handler.c:6366:__glusterd_peer_rpc_notify] 0-management: Peer <dhcp41-180.lab.eng.blr.redhat.com> (<df78611e-886a-439f-b2b0-6a1dd404311e>), in state <Peer in Cluster>, has disconnected from glusterd.
>[2018-05-30 11:28:40.226794] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x22c4a) [0x7f6c3beadc4a] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2c9f8) [0x7f6c3beb79f8] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xddd45) [0x7f6c3bf68d45] ) 0-management: Lock for vol heketidbstorage not held
>[2018-05-30 11:28:40.226807] W [MSGID: 106118] [glusterd-handler.c:6391:__glusterd_peer_rpc_notify] 0-management: Lock not released for heketidbstorage
>[2018-05-30 11:28:40.226821] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x22c4a) [0x7f6c3beadc4a] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2c9f8) [0x7f6c3beb79f8] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xddd45) [0x7f6c3bf68d45] ) 0-management: Lock for vol vol_2aaf562cfd128cc6a174022f18b683b0 not held
>[2018-05-30 11:28:40.226827] W [MSGID: 106118] [glusterd-handler.c:6391:__glusterd_peer_rpc_notify] 0-management: Lock not released for vol_2aaf562cfd128cc6a174022f18b683b0
>[2018-05-30 11:29:08.430612] E [socket.c:2360:socket_connect_finish] 0-management: connection to 10.70.41.180:24007 failed (No route to host); disconnecting socket
>[2018-05-30 11:33:02.481585] I [MSGID: 106163] [glusterd-handshake.c:1284:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31101
>[2018-05-30 11:33:03.037090] I [MSGID: 106490] [glusterd-handler.c:2611:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: df78611e-886a-439f-b2b0-6a1dd404311e
>[2018-05-30 11:33:03.320641] I [MSGID: 106493] [glusterd-handler.c:3873:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to dhcp41-180.lab.eng.blr.redhat.com (0), ret: 0, op_ret: 0
>[2018-05-30 11:33:03.568008] I [MSGID: 106492] [glusterd-handler.c:2789:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: df78611e-886a-439f-b2b0-6a1dd404311e
>[2018-05-30 11:33:03.609649] I [MSGID: 106502] [glusterd-handler.c:2834:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
>[2018-05-30 11:33:03.638565] I [MSGID: 106493] [glusterd-rpc-ops.c:693:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: df78611e-886a-439f-b2b0-6a1dd404311e
>[2018-05-30 11:33:03.664173] I [MSGID: 106493] [glusterd-rpc-ops.c:478:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: df78611e-886a-439f-b2b0-6a1dd404311e, host: dhcp41-180.lab.eng.blr.redhat.com, port: 0
>[2018-05-30 11:33:03.764352] I [MSGID: 106492] [glusterd-handler.c:2789:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: df78611e-886a-439f-b2b0-6a1dd404311e
>[2018-05-30 11:33:03.818044] I [MSGID: 106502] [glusterd-handler.c:2834:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
>[2018-05-30 11:33:03.972670] I [MSGID: 106493] [glusterd-rpc-ops.c:693:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: df78611e-886a-439f-b2b0-6a1dd404311e
>[2018-05-30 11:51:04.341440] I [MSGID: 106004] [glusterd-handler.c:6366:__glusterd_peer_rpc_notify] 0-management: Peer <dhcp41-180.lab.eng.blr.redhat.com> (<df78611e-886a-439f-b2b0-6a1dd404311e>), in state <Peer in Cluster>, has disconnected from glusterd.
>[2018-05-30 11:51:04.341614] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x22c4a) [0x7f6c3beadc4a] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2c9f8) [0x7f6c3beb79f8] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xddd45) [0x7f6c3bf68d45] ) 0-management: Lock for vol heketidbstorage not held
>[2018-05-30 11:51:04.341629] W [MSGID: 106118] [glusterd-handler.c:6391:__glusterd_peer_rpc_notify] 0-management: Lock not released for heketidbstorage
>[2018-05-30 11:51:04.341651] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x22c4a) [0x7f6c3beadc4a] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2c9f8) [0x7f6c3beb79f8] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xddd45) [0x7f6c3bf68d45] ) 0-management: Lock for vol vol_2aaf562cfd128cc6a174022f18b683b0 not held
>[2018-05-30 11:51:04.341659] W [MSGID: 106118] [glusterd-handler.c:6391:__glusterd_peer_rpc_notify] 0-management: Lock not released for vol_2aaf562cfd128cc6a174022f18b683b0
>[2018-05-30 11:51:14.867080] E [socket.c:2360:socket_connect_finish] 0-management: connection to 10.70.41.180:24007 failed (Connection refused); disconnecting socket
>[2018-05-30 11:52:02.628160] I [MSGID: 106163] [glusterd-handshake.c:1284:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31101
>[2018-05-30 11:52:03.303798] I [MSGID: 106490] [glusterd-handler.c:2611:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: df78611e-886a-439f-b2b0-6a1dd404311e
>[2018-05-30 11:52:03.355668] I [MSGID: 106493] [glusterd-handler.c:3873:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to dhcp41-180.lab.eng.blr.redhat.com (0), ret: 0, op_ret: 0
>[2018-05-30 11:52:05.070703] I [MSGID: 106492] [glusterd-handler.c:2789:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: df78611e-886a-439f-b2b0-6a1dd404311e
>[2018-05-30 11:52:05.155944] I [MSGID: 106502] [glusterd-handler.c:2834:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
>[2018-05-30 11:52:05.245486] I [MSGID: 106493] [glusterd-rpc-ops.c:693:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: df78611e-886a-439f-b2b0-6a1dd404311e
>[2018-05-30 11:52:07.133080] I [MSGID: 106493] [glusterd-rpc-ops.c:478:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: df78611e-886a-439f-b2b0-6a1dd404311e, host: dhcp41-180.lab.eng.blr.redhat.com, port: 0
>[2018-05-30 11:52:07.189379] I [MSGID: 106492] [glusterd-handler.c:2789:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: df78611e-886a-439f-b2b0-6a1dd404311e
>[2018-05-30 11:52:07.247900] I [MSGID: 106502] [glusterd-handler.c:2834:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
>[2018-05-30 11:52:07.277972] I [MSGID: 106493] [glusterd-rpc-ops.c:693:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: df78611e-886a-439f-b2b0-6a1dd404311e
>[2018-05-30 11:54:45.522462] I [MSGID: 106004] [glusterd-handler.c:6366:__glusterd_peer_rpc_notify] 0-management: Peer <dhcp41-180.lab.eng.blr.redhat.com> (<df78611e-886a-439f-b2b0-6a1dd404311e>), in state <Peer in Cluster>, has disconnected from glusterd.
>[2018-05-30 11:54:45.522626] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x22c4a) [0x7f6c3beadc4a] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2c9f8) [0x7f6c3beb79f8] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xddd45) [0x7f6c3bf68d45] ) 0-management: Lock for vol heketidbstorage not held
>[2018-05-30 11:54:45.522640] W [MSGID: 106118] [glusterd-handler.c:6391:__glusterd_peer_rpc_notify] 0-management: Lock not released for heketidbstorage
>[2018-05-30 11:54:45.522654] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x22c4a) [0x7f6c3beadc4a] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2c9f8) [0x7f6c3beb79f8] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xddd45) [0x7f6c3bf68d45] ) 0-management: Lock for vol vol_2aaf562cfd128cc6a174022f18b683b0 not held
>[2018-05-30 11:54:45.522660] W [MSGID: 106118] [glusterd-handler.c:6391:__glusterd_peer_rpc_notify] 0-management: Lock not released for vol_2aaf562cfd128cc6a174022f18b683b0
>[2018-05-30 11:54:58.912689] E [socket.c:2360:socket_connect_finish] 0-management: connection to 10.70.41.180:24007 failed (No route to host); disconnecting socket
>[2018-05-30 12:11:02.742636] I [MSGID: 106163] [glusterd-handshake.c:1284:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31101
>[2018-05-30 12:11:02.856621] I [MSGID: 106490] [glusterd-handler.c:2611:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: df78611e-886a-439f-b2b0-6a1dd404311e
>[2018-05-30 12:11:02.948422] I [MSGID: 106493] [glusterd-handler.c:3873:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to dhcp41-180.lab.eng.blr.redhat.com (0), ret: 0, op_ret: 0
>[2018-05-30 12:11:03.123772] I [MSGID: 106492] [glusterd-handler.c:2789:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: df78611e-886a-439f-b2b0-6a1dd404311e
>[2018-05-30 12:11:03.240136] I [MSGID: 106502] [glusterd-handler.c:2834:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
>[2018-05-30 12:11:04.563232] I [MSGID: 106493] [glusterd-rpc-ops.c:693:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: df78611e-886a-439f-b2b0-6a1dd404311e
>[2018-05-30 12:11:05.009405] I [MSGID: 106493] [glusterd-rpc-ops.c:478:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: df78611e-886a-439f-b2b0-6a1dd404311e, host: dhcp41-180.lab.eng.blr.redhat.com, port: 0
>[2018-05-30 12:11:05.181936] I [MSGID: 106492] [glusterd-handler.c:2789:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: df78611e-886a-439f-b2b0-6a1dd404311e
>[2018-05-30 12:11:05.240319] I [MSGID: 106502] [glusterd-handler.c:2834:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
>[2018-05-30 12:11:05.273702] I [MSGID: 106493] [glusterd-rpc-ops.c:693:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: df78611e-886a-439f-b2b0-6a1dd404311e
>[2018-05-30 12:11:29.129383] I [MSGID: 106004] [glusterd-handler.c:6366:__glusterd_peer_rpc_notify] 0-management: Peer <dhcp41-180.lab.eng.blr.redhat.com> (<df78611e-886a-439f-b2b0-6a1dd404311e>), in state <Peer in Cluster>, has disconnected from glusterd.
>[2018-05-30 12:11:29.129563] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x22c4a) [0x7f6c3beadc4a] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2c9f8) [0x7f6c3beb79f8] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xddd45) [0x7f6c3bf68d45] ) 0-management: Lock for vol heketidbstorage not held
>[2018-05-30 12:11:29.129575] W [MSGID: 106118] [glusterd-handler.c:6391:__glusterd_peer_rpc_notify] 0-management: Lock not released for heketidbstorage
>[2018-05-30 12:11:29.129588] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x22c4a) [0x7f6c3beadc4a] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2c9f8) [0x7f6c3beb79f8] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xddd45) [0x7f6c3bf68d45] ) 0-management: Lock for vol vol_2aaf562cfd128cc6a174022f18b683b0 not held
>[2018-05-30 12:11:29.129594] W [MSGID: 106118] [glusterd-handler.c:6391:__glusterd_peer_rpc_notify] 0-management: Lock not released for vol_2aaf562cfd128cc6a174022f18b683b0
>[2018-05-30 12:11:39.941751] E [socket.c:2360:socket_connect_finish] 0-management: connection to 10.70.41.180:24007 failed (Connection refused); disconnecting socket
>[2018-05-30 12:12:27.484299] I [MSGID: 106163] [glusterd-handshake.c:1284:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31101
>[2018-05-30 12:12:27.539180] I [MSGID: 106490] [glusterd-handler.c:2611:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: df78611e-886a-439f-b2b0-6a1dd404311e
>[2018-05-30 12:12:28.135734] I [MSGID: 106493] [glusterd-handler.c:3873:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to dhcp41-180.lab.eng.blr.redhat.com (0), ret: 0, op_ret: 0
>[2018-05-30 12:12:29.067877] I [MSGID: 106492] [glusterd-handler.c:2789:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: df78611e-886a-439f-b2b0-6a1dd404311e
>[2018-05-30 12:12:29.105827] I [MSGID: 106502] [glusterd-handler.c:2834:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
>[2018-05-30 12:12:29.201178] I [MSGID: 106493] [glusterd-rpc-ops.c:693:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: df78611e-886a-439f-b2b0-6a1dd404311e
>[2018-05-30 12:12:30.547600] I [MSGID: 106493] [glusterd-rpc-ops.c:478:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: df78611e-886a-439f-b2b0-6a1dd404311e, host: dhcp41-180.lab.eng.blr.redhat.com, port: 0
>[2018-05-30 12:12:30.749465] I [MSGID: 106492] [glusterd-handler.c:2789:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: df78611e-886a-439f-b2b0-6a1dd404311e
>[2018-05-30 12:12:30.897968] I [MSGID: 106502] [glusterd-handler.c:2834:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
>[2018-05-30 12:12:30.898076] I [MSGID: 106493] [glusterd-rpc-ops.c:693:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: df78611e-886a-439f-b2b0-6a1dd404311e
>[2018-05-31 10:28:16.898518] I [MSGID: 106004] [glusterd-handler.c:6366:__glusterd_peer_rpc_notify] 0-management: Peer <dhcp41-180.lab.eng.blr.redhat.com> (<df78611e-886a-439f-b2b0-6a1dd404311e>), in state <Peer in Cluster>, has disconnected from glusterd.
>[2018-05-31 10:28:16.898768] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x22c4a) [0x7f6c3beadc4a] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2c9f8) [0x7f6c3beb79f8] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xddd45) [0x7f6c3bf68d45] ) 0-management: Lock for vol heketidbstorage not held
>[2018-05-31 10:28:16.898778] W [MSGID: 106118] [glusterd-handler.c:6391:__glusterd_peer_rpc_notify] 0-management: Lock not released for heketidbstorage
>[2018-05-31 10:28:16.898804] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x22c4a) [0x7f6c3beadc4a] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0x2c9f8) [0x7f6c3beb79f8] -->/usr/lib64/glusterfs/3.8.4/xlator/mgmt/glusterd.so(+0xddd45) [0x7f6c3bf68d45] ) 0-management: Lock for vol vol_2aaf562cfd128cc6a174022f18b683b0 not held
>[2018-05-31 10:28:16.898810] W [MSGID: 106118] [glusterd-handler.c:6391:__glusterd_peer_rpc_notify] 0-management: Lock not released for vol_2aaf562cfd128cc6a174022f18b683b0
>[2018-05-31 10:28:30.562615] E [socket.c:2360:socket_connect_finish] 0-management: connection to 10.70.41.180:24007 failed (No route to host); disconnecting socket
>

Actions: View

Attachments on bug 1584644: 1446228 | 1446229