Attachment 926218 Details for Bug 1129486 – glustershd.log from server gluster1

glustershd.log from server gluster1

gluster1-glustershd.log (text/plain), 14.97 KB, created by Scott Merrill on 2014-08-12 21:17:40 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Scott Merrill

Created: 2014-08-12 21:17:40 UTC

Size: 14.97 KB

patch

obsolete

>[2014-08-12 20:59:45.374475] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.5.2 (/usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/3fcda4bc5b4dbcfd512a4d62a3864ba3.socket --xlator-option *replicate*.node-uuid=bb61111b-b048-4dc9-b54d-12a0cc2dd8a9)
>[2014-08-12 20:59:45.380680] I [socket.c:3561:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-08-12 20:59:45.380741] I [socket.c:3576:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-08-12 20:59:45.380937] I [socket.c:3561:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-08-12 20:59:45.380962] I [socket.c:3576:socket_init] 0-glusterfs: using system polling thread
>[2014-08-12 20:59:45.395383] I [graph.c:254:gf_add_cmdline_options] 0-test-replicate-0: adding option 'node-uuid' for volume 'test-replicate-0' with value 'bb61111b-b048-4dc9-b54d-12a0cc2dd8a9'
>[2014-08-12 20:59:45.399812] I [socket.c:3561:socket_init] 0-test-client-1: SSL support is NOT enabled
>[2014-08-12 20:59:45.399840] I [socket.c:3576:socket_init] 0-test-client-1: using system polling thread
>[2014-08-12 20:59:45.400326] I [socket.c:3561:socket_init] 0-test-client-0: SSL support is NOT enabled
>[2014-08-12 20:59:45.400348] I [socket.c:3576:socket_init] 0-test-client-0: using system polling thread
>[2014-08-12 20:59:45.400381] I [client.c:2294:notify] 0-test-client-0: parent translators are ready, attempting connect on transport
>[2014-08-12 20:59:45.407034] I [client.c:2294:notify] 0-test-client-1: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume test-client-0
>  2:     type protocol/client
>  3:     option remote-host gluster1
>  4:     option remote-subvolume /bricks/brick1/brick
>  5:     option transport-type socket
>  6:     option username 0a75ffa8-0a6f-4722-b8a9-3fe5a9853fb4
>  7:     option password d4b3c8a7-b210-47c7-8d97-9db70ba8d5bc
>  8: end-volume
>  9: 
> 10: volume test-client-1
> 11:     type protocol/client
> 12:     option remote-host gluster2
> 13:     option remote-subvolume /bricks/brick1/brick
> 14:     option transport-type socket
> 15:     option username 0a75ffa8-0a6f-4722-b8a9-3fe5a9853fb4
> 16:     option password d4b3c8a7-b210-47c7-8d97-9db70ba8d5bc
> 17: end-volume
> 18: 
> 19: volume test-replicate-0
> 20:     type cluster/replicate
> 21:     option node-uuid bb61111b-b048-4dc9-b54d-12a0cc2dd8a9
> 22:     option background-self-heal-count 0
> 23:     option metadata-self-heal on
> 24:     option data-self-heal on
> 25:     option entry-self-heal on
> 26:     option self-heal-daemon on
> 27:     option iam-self-heal-daemon yes
> 28:     subvolumes test-client-0 test-client-1
> 29: end-volume
> 30: 
> 31: volume glustershd
> 32:     type debug/io-stats
> 33:     subvolumes test-replicate-0
> 34: end-volume
> 35: 
>+------------------------------------------------------------------------------+
>[2014-08-12 20:59:45.413828] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-test-client-0: changing port to 49152 (from 0)
>[2014-08-12 20:59:45.420550] I [client-handshake.c:1677:select_server_supported_programs] 0-test-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-08-12 20:59:45.425827] I [client-handshake.c:1462:client_setvolume_cbk] 0-test-client-0: Connected to 192.168.30.107:49152, attached to remote volume '/bricks/brick1/brick'.
>[2014-08-12 20:59:45.425868] I [client-handshake.c:1474:client_setvolume_cbk] 0-test-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-08-12 20:59:45.425998] I [afr-common.c:4131:afr_notify] 0-test-replicate-0: Subvolume 'test-client-0' came back up; going online.
>[2014-08-12 20:59:45.426100] I [client-handshake.c:450:client_set_lk_version_cbk] 0-test-client-0: Server lk version = 1
>[2014-08-12 20:59:45.427422] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-test-replicate-0: Another crawl is in progress for test-client-0
>[2014-08-12 20:59:45.427502] E [afr-self-heald.c:1479:afr_find_child_position] 0-test-replicate-0: getxattr failed on test-client-1 - (Transport endpoint is not connected)
>[2014-08-12 20:59:45.705481] E [client-handshake.c:1760:client_query_portmap_cbk] 0-test-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
>[2014-08-12 20:59:45.705584] I [client.c:2229:client_rpc_notify] 0-test-client-1: disconnected from 192.168.30.108:24007. Client process will keep trying to connect to glusterd until brick's port is available
>[2014-08-12 20:59:49.400435] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-test-client-1: changing port to 49152 (from 0)
>[2014-08-12 20:59:49.407295] I [client-handshake.c:1677:select_server_supported_programs] 0-test-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-08-12 20:59:49.407659] I [client-handshake.c:1462:client_setvolume_cbk] 0-test-client-1: Connected to 192.168.30.108:49152, attached to remote volume '/bricks/brick1/brick'.
>[2014-08-12 20:59:49.407683] I [client-handshake.c:1474:client_setvolume_cbk] 0-test-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-08-12 20:59:49.407888] I [client-handshake.c:450:client_set_lk_version_cbk] 0-test-client-1: Server lk version = 1
>[2014-08-12 21:04:23.227339] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
>[2014-08-12 21:04:23.230830] I [glusterfsd-mgmt.c:1307:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
>[2014-08-12 21:04:23.810295] W [glusterfsd.c:1095:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x7f89c7ee53dd] (-->/lib64/libpthread.so.0(+0x7df3) [0x7f89c859cdf3] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f89c9b6a1b5]))) 0-: received signum (15), shutting down
>[2014-08-12 21:04:24.823959] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.5.2 (/usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/3fcda4bc5b4dbcfd512a4d62a3864ba3.socket --xlator-option *replicate*.node-uuid=bb61111b-b048-4dc9-b54d-12a0cc2dd8a9)
>[2014-08-12 21:04:24.829948] I [socket.c:3561:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
>[2014-08-12 21:04:24.830011] I [socket.c:3576:socket_init] 0-socket.glusterfsd: using system polling thread
>[2014-08-12 21:04:24.830202] I [socket.c:3561:socket_init] 0-glusterfs: SSL support is NOT enabled
>[2014-08-12 21:04:24.830220] I [socket.c:3576:socket_init] 0-glusterfs: using system polling thread
>[2014-08-12 21:04:24.844810] I [graph.c:254:gf_add_cmdline_options] 0-test-replicate-1: adding option 'node-uuid' for volume 'test-replicate-1' with value 'bb61111b-b048-4dc9-b54d-12a0cc2dd8a9'
>[2014-08-12 21:04:24.844839] I [graph.c:254:gf_add_cmdline_options] 0-test-replicate-0: adding option 'node-uuid' for volume 'test-replicate-0' with value 'bb61111b-b048-4dc9-b54d-12a0cc2dd8a9'
>[2014-08-12 21:04:24.852812] I [socket.c:3561:socket_init] 0-test-client-3: SSL support is NOT enabled
>[2014-08-12 21:04:24.852850] I [socket.c:3576:socket_init] 0-test-client-3: using system polling thread
>[2014-08-12 21:04:24.853317] I [socket.c:3561:socket_init] 0-test-client-2: SSL support is NOT enabled
>[2014-08-12 21:04:24.853338] I [socket.c:3576:socket_init] 0-test-client-2: using system polling thread
>[2014-08-12 21:04:24.853785] I [socket.c:3561:socket_init] 0-test-client-1: SSL support is NOT enabled
>[2014-08-12 21:04:24.853800] I [socket.c:3576:socket_init] 0-test-client-1: using system polling thread
>[2014-08-12 21:04:24.854249] I [socket.c:3561:socket_init] 0-test-client-0: SSL support is NOT enabled
>[2014-08-12 21:04:24.854270] I [socket.c:3576:socket_init] 0-test-client-0: using system polling thread
>[2014-08-12 21:04:24.854316] I [client.c:2294:notify] 0-test-client-0: parent translators are ready, attempting connect on transport
>[2014-08-12 21:04:24.861671] I [client.c:2294:notify] 0-test-client-1: parent translators are ready, attempting connect on transport
>[2014-08-12 21:04:24.867969] I [client.c:2294:notify] 0-test-client-2: parent translators are ready, attempting connect on transport
>[2014-08-12 21:04:24.874257] I [client.c:2294:notify] 0-test-client-3: parent translators are ready, attempting connect on transport
>Final graph:
>+------------------------------------------------------------------------------+
>  1: volume test-client-0
>  2:     type protocol/client
>  3:     option remote-host gluster1
>  4:     option remote-subvolume /bricks/brick1/brick
>  5:     option transport-type socket
>  6:     option username 0a75ffa8-0a6f-4722-b8a9-3fe5a9853fb4
>  7:     option password d4b3c8a7-b210-47c7-8d97-9db70ba8d5bc
>  8: end-volume
>  9: 
> 10: volume test-client-1
> 11:     type protocol/client
> 12:     option remote-host gluster2
> 13:     option remote-subvolume /bricks/brick1/brick
> 14:     option transport-type socket
> 15:     option username 0a75ffa8-0a6f-4722-b8a9-3fe5a9853fb4
> 16:     option password d4b3c8a7-b210-47c7-8d97-9db70ba8d5bc
> 17: end-volume
> 18: 
> 19: volume test-replicate-0
> 20:     type cluster/replicate
> 21:     option node-uuid bb61111b-b048-4dc9-b54d-12a0cc2dd8a9
> 22:     option background-self-heal-count 0
> 23:     option metadata-self-heal on
> 24:     option data-self-heal on
> 25:     option entry-self-heal on
> 26:     option self-heal-daemon on
> 27:     option iam-self-heal-daemon yes
> 28:     subvolumes test-client-0 test-client-1
> 29: end-volume
> 30: 
> 31: volume test-client-2
> 32:     type protocol/client
> 33:     option remote-host gluster1
> 34:     option remote-subvolume /bricks/brick2/brick
> 35:     option transport-type socket
> 36:     option username 0a75ffa8-0a6f-4722-b8a9-3fe5a9853fb4
> 37:     option password d4b3c8a7-b210-47c7-8d97-9db70ba8d5bc
> 38: end-volume
> 39: 
> 40: volume test-client-3
> 41:     type protocol/client
> 42:     option remote-host gluster2
> 43:     option remote-subvolume /bricks/brick2/brick
> 44:     option transport-type socket
> 45:     option username 0a75ffa8-0a6f-4722-b8a9-3fe5a9853fb4
> 46:     option password d4b3c8a7-b210-47c7-8d97-9db70ba8d5bc
> 47: end-volume
> 48: 
> 49: volume test-replicate-1
> 50:     type cluster/replicate
> 51:     option node-uuid bb61111b-b048-4dc9-b54d-12a0cc2dd8a9
> 52:     option background-self-heal-count 0
> 53:     option metadata-self-heal on
> 54:     option data-self-heal on
> 55:     option entry-self-heal on
> 56:     option self-heal-daemon on
> 57:     option iam-self-heal-daemon yes
> 58:     subvolumes test-client-2 test-client-3
> 59: end-volume
> 60: 
> 61: volume glustershd
> 62:     type debug/io-stats
> 63:     subvolumes test-replicate-0 test-replicate-1
> 64: end-volume
> 65: 
>+------------------------------------------------------------------------------+
>[2014-08-12 21:04:24.881340] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-test-client-0: changing port to 49152 (from 0)
>[2014-08-12 21:04:24.887746] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-test-client-2: changing port to 49153 (from 0)
>[2014-08-12 21:04:24.894397] I [client-handshake.c:1677:select_server_supported_programs] 0-test-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-08-12 21:04:24.894513] I [client-handshake.c:1677:select_server_supported_programs] 0-test-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-08-12 21:04:24.894724] I [client-handshake.c:1462:client_setvolume_cbk] 0-test-client-0: Connected to 192.168.30.107:49152, attached to remote volume '/bricks/brick1/brick'.
>[2014-08-12 21:04:24.894744] I [client-handshake.c:1474:client_setvolume_cbk] 0-test-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2014-08-12 21:04:24.894831] I [afr-common.c:4131:afr_notify] 0-test-replicate-0: Subvolume 'test-client-0' came back up; going online.
>[2014-08-12 21:04:24.894981] I [client-handshake.c:450:client_set_lk_version_cbk] 0-test-client-0: Server lk version = 1
>[2014-08-12 21:04:24.895973] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-test-replicate-0: Another crawl is in progress for test-client-0
>[2014-08-12 21:04:24.895992] E [afr-self-heald.c:1479:afr_find_child_position] 0-test-replicate-0: getxattr failed on test-client-1 - (Transport endpoint is not connected)
>[2014-08-12 21:04:24.897809] E [afr-self-heal-data.c:1611:afr_sh_data_open_cbk] 0-test-replicate-0: open of <gfid:e5e936cb-4ad9-4e56-8ace-980b74ce19af> failed on child test-client-1 (Transport endpoint is not connected)
>[2014-08-12 21:04:24.899709] I [client-handshake.c:1462:client_setvolume_cbk] 0-test-client-2: Connected to 192.168.30.107:49153, attached to remote volume '/bricks/brick2/brick'.
>[2014-08-12 21:04:24.899748] I [client-handshake.c:1474:client_setvolume_cbk] 0-test-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2014-08-12 21:04:24.899837] I [afr-common.c:4131:afr_notify] 0-test-replicate-1: Subvolume 'test-client-2' came back up; going online.
>[2014-08-12 21:04:24.899983] I [client-handshake.c:450:client_set_lk_version_cbk] 0-test-client-2: Server lk version = 1
>[2014-08-12 21:04:24.900982] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-test-replicate-1: Another crawl is in progress for test-client-2
>[2014-08-12 21:04:24.901003] E [afr-self-heald.c:1479:afr_find_child_position] 0-test-replicate-1: getxattr failed on test-client-3 - (Transport endpoint is not connected)
>[2014-08-12 21:04:24.901122] E [afr-self-heal-data.c:1611:afr_sh_data_open_cbk] 0-test-replicate-0: open of <gfid:e5e936cb-4ad9-4e56-8ace-980b74ce19af> failed on child test-client-1 (Transport endpoint is not connected)
>[2014-08-12 21:04:26.441960] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-test-client-1: changing port to 49152 (from 0)
>[2014-08-12 21:04:26.442046] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-test-client-3: changing port to 49153 (from 0)
>[2014-08-12 21:04:26.455350] I [client-handshake.c:1677:select_server_supported_programs] 0-test-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-08-12 21:04:26.455510] I [client-handshake.c:1677:select_server_supported_programs] 0-test-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
>[2014-08-12 21:04:26.455725] I [client-handshake.c:1462:client_setvolume_cbk] 0-test-client-1: Connected to 192.168.30.108:49152, attached to remote volume '/bricks/brick1/brick'.
>[2014-08-12 21:04:26.455747] I [client-handshake.c:1474:client_setvolume_cbk] 0-test-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2014-08-12 21:04:26.458302] I [client-handshake.c:450:client_set_lk_version_cbk] 0-test-client-1: Server lk version = 1
>[2014-08-12 21:04:26.460787] I [client-handshake.c:1462:client_setvolume_cbk] 0-test-client-3: Connected to 192.168.30.108:49153, attached to remote volume '/bricks/brick2/brick'.
>[2014-08-12 21:04:26.460828] I [client-handshake.c:1474:client_setvolume_cbk] 0-test-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2014-08-12 21:04:26.461133] I [client-handshake.c:450:client_set_lk_version_cbk] 0-test-client-3: Server lk version = 1

Actions: View

Attachments on bug 1129486: 926217 | 926218 | 926219