Attachment 587189 Details for Bug 825562 – mysql1-clearlocks-mnt.log

mysql1-clearlocks-mnt.log

file_825562.txt (text/plain), 59.36 KB, created by Joe Julian on 2012-05-28 09:37:31 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Joe Julian

Created: 2012-05-28 09:37:31 UTC

Size: 59.36 KB

patch

obsolete

>[2012-05-27 12:45:25.852883] I [glusterfsd.c:1651:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.0qa43
>[2012-05-27 12:45:26.025771] E [glusterfsd.c:430:get_volfp] 0-glusterfsd: /var/lib/glusterd/vols/mysql1/trusted-mysql1-fuse.vol: No such file or directory
>[2012-05-27 12:45:26.025959] E [glusterfsd.c:1602:glusterfs_volumes_init] 0-glusterfsd: Cannot reach volume specification file
>[2012-05-27 12:52:04.039369] I [glusterfsd.c:1651:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.0qa43
>[2012-05-27 12:52:04.069981] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-3: adding option 'entry-self-heal' for volume 'mysql1-replicate-3' with value 'off'
>[2012-05-27 12:52:04.070051] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-3: adding option 'metadata-self-heal' for volume 'mysql1-replicate-3' with value 'off'
>[2012-05-27 12:52:04.070080] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-3: adding option 'data-self-heal' for volume 'mysql1-replicate-3' with value 'off'
>[2012-05-27 12:52:04.070108] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-2: adding option 'entry-self-heal' for volume 'mysql1-replicate-2' with value 'off'
>[2012-05-27 12:52:04.070135] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-2: adding option 'metadata-self-heal' for volume 'mysql1-replicate-2' with value 'off'
>[2012-05-27 12:52:04.070177] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-2: adding option 'data-self-heal' for volume 'mysql1-replicate-2' with value 'off'
>[2012-05-27 12:52:04.070204] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-1: adding option 'entry-self-heal' for volume 'mysql1-replicate-1' with value 'off'
>[2012-05-27 12:52:04.070231] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-1: adding option 'metadata-self-heal' for volume 'mysql1-replicate-1' with value 'off'
>[2012-05-27 12:52:04.070257] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-1: adding option 'data-self-heal' for volume 'mysql1-replicate-1' with value 'off'
>[2012-05-27 12:52:04.070284] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-0: adding option 'entry-self-heal' for volume 'mysql1-replicate-0' with value 'off'
>[2012-05-27 12:52:04.070311] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-0: adding option 'metadata-self-heal' for volume 'mysql1-replicate-0' with value 'off'
>[2012-05-27 12:52:04.070337] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-0: adding option 'data-self-heal' for volume 'mysql1-replicate-0' with value 'off'
>[2012-05-27 12:52:04.070370] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-client-10: adding option 'remote-port' for volume 'mysql1-client-10' with value '24016'
>[2012-05-27 12:52:04.070400] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-client-7: adding option 'remote-port' for volume 'mysql1-client-7' with value '24015'
>[2012-05-27 12:52:04.070430] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-client-4: adding option 'remote-port' for volume 'mysql1-client-4' with value '24014'
>[2012-05-27 12:52:04.070461] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-client-1: adding option 'remote-port' for volume 'mysql1-client-1' with value '24013'
>[2012-05-27 12:52:04.110596] I [client.c:2142:notify] 0-mysql1-client-0: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:04.116784] I [client.c:2142:notify] 0-mysql1-client-1: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:04.122225] I [client.c:2142:notify] 0-mysql1-client-2: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:04.127242] I [client.c:2142:notify] 0-mysql1-client-3: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:04.132102] I [client.c:2142:notify] 0-mysql1-client-4: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:04.137420] I [client.c:2142:notify] 0-mysql1-client-5: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:04.142521] I [client.c:2142:notify] 0-mysql1-client-6: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:04.147545] I [client.c:2142:notify] 0-mysql1-client-7: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:04.152574] I [client.c:2142:notify] 0-mysql1-client-8: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:04.157629] I [client.c:2142:notify] 0-mysql1-client-9: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:04.162650] I [client.c:2142:notify] 0-mysql1-client-10: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:04.167607] I [client.c:2142:notify] 0-mysql1-client-11: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume mysql1-client-0
>  2:     type protocol/client
>  3:     option remote-host ewcs2
>  4:     option remote-subvolume /var/spool/glusterfs/a_mysql1
>  5:     option transport-type tcp
>  6: end-volume
>  7: 
>  8: volume mysql1-client-1
>  9:     type protocol/client
> 10:     option remote-host ewcs4
> 11:     option remote-subvolume /var/spool/glusterfs/a_mysql1
> 12:     option transport-type tcp
> 13: end-volume
> 14: 
> 15: volume mysql1-client-2
> 16:     type protocol/client
> 17:     option remote-host ewcs7
> 18:     option remote-subvolume /var/spool/glusterfs/a_mysql1
> 19:     option transport-type tcp
> 20: end-volume
> 21: 
> 22: volume mysql1-client-3
> 23:     type protocol/client
> 24:     option remote-host ewcs2
> 25:     option remote-subvolume /var/spool/glusterfs/b_mysql1
> 26:     option transport-type tcp
> 27: end-volume
> 28: 
> 29: volume mysql1-client-4
> 30:     type protocol/client
> 31:     option remote-host ewcs4
> 32:     option remote-subvolume /var/spool/glusterfs/b_mysql1
> 33:     option transport-type tcp
> 34: end-volume
> 35: 
> 36: volume mysql1-client-5
> 37:     type protocol/client
> 38:     option remote-host ewcs7
> 39:     option remote-subvolume /var/spool/glusterfs/b_mysql1
> 40:     option transport-type tcp
> 41: end-volume
> 42: 
> 43: volume mysql1-client-6
> 44:     type protocol/client
> 45:     option remote-host ewcs2
> 46:     option remote-subvolume /var/spool/glusterfs/c_mysql1
> 47:     option transport-type tcp
> 48: end-volume
> 49: 
> 50: volume mysql1-client-7
> 51:     type protocol/client
> 52:     option remote-host ewcs4
> 53:     option remote-subvolume /var/spool/glusterfs/c_mysql1
> 54:     option transport-type tcp
> 55: end-volume
> 56: 
> 57: volume mysql1-client-8
> 58:     type protocol/client
> 59:     option remote-host ewcs7
> 60:     option remote-subvolume /var/spool/glusterfs/c_mysql1
> 61:     option transport-type tcp
> 62: end-volume
> 63: 
> 64: volume mysql1-client-9
> 65:     type protocol/client
> 66:     option remote-host ewcs2
> 67:     option remote-subvolume /var/spool/glusterfs/d_mysql1
> 68:     option transport-type tcp
> 69: end-volume
> 70: 
> 71: volume mysql1-client-10
> 72:     type protocol/client
> 73:     option remote-host ewcs4
> 74:     option remote-subvolume /var/spool/glusterfs/d_mysql1
> 75:     option transport-type tcp
> 76: end-volume
> 77: 
> 78: volume mysql1-client-11
> 79:     type protocol/client
> 80:     option remote-host ewcs7
> 81:     option remote-subvolume /var/spool/glusterfs/d_mysql1
> 82:     option transport-type tcp
> 83: end-volume
> 84: 
> 85: volume mysql1-replicate-0
> 86:     type cluster/replicate
> 87:     subvolumes mysql1-client-0 mysql1-client-1 mysql1-client-2
> 88: end-volume
> 89: 
> 90: volume mysql1-replicate-1
> 91:     type cluster/replicate
> 92:     subvolumes mysql1-client-3 mysql1-client-4 mysql1-client-5
> 93: end-volume
> 94: 
> 95: volume mysql1-replicate-2
> 96:     type cluster/replicate
> 97:     subvolumes mysql1-client-6 mysql1-client-7 mysql1-client-8
> 98: end-volume
> 99: 
>100: volume mysql1-replicate-3
>101:     type cluster/replicate
>102:     subvolumes mysql1-client-9 mysql1-client-10 mysql1-client-11
>103: end-volume
>104: 
>105: volume mysql1-dht
>106:     type cluster/distribute
>107:     subvolumes mysql1-replicate-0 mysql1-replicate-1 mysql1-replicate-2 mysql1-replicate-3
>108: end-volume
>109: 
>110: volume mysql1-md-cache
>111:     type performance/md-cache
>112:     subvolumes mysql1-dht
>113: end-volume
>114: 
>115: volume mysql1
>116:     type debug/io-stats
>117:     option latency-measurement off
>118:     option count-fop-hits off
>119:     subvolumes mysql1-md-cache
>120: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-05-27 12:52:04.173997] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-4: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:04.174287] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-7: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:04.174513] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-10: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:04.174753] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-mysql1-client-9: changing port to 24060 (from 0)
>[2012-05-27 12:52:04.174838] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-mysql1-client-8: changing port to 24143 (from 0)
>[2012-05-27 12:52:04.174902] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-mysql1-client-6: changing port to 24059 (from 0)
>[2012-05-27 12:52:04.174959] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-mysql1-client-5: changing port to 24142 (from 0)
>[2012-05-27 12:52:04.175046] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-4: Connected to 10.0.0.4:24014, attached to remote volume '/var/spool/glusterfs/b_mysql1'.
>[2012-05-27 12:52:04.175078] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:04.175211] I [afr-common.c:3628:afr_notify] 0-mysql1-replicate-1: Subvolume 'mysql1-client-4' came back up; going online.
>[2012-05-27 12:52:04.175281] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-mysql1-client-3: changing port to 24058 (from 0)
>[2012-05-27 12:52:04.175338] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-mysql1-client-2: changing port to 24141 (from 0)
>[2012-05-27 12:52:04.175396] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-1: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:04.175569] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-10: Connected to 10.0.0.4:24016, attached to remote volume '/var/spool/glusterfs/d_mysql1'.
>[2012-05-27 12:52:04.175636] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-10: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:04.175826] I [afr-common.c:3628:afr_notify] 0-mysql1-replicate-3: Subvolume 'mysql1-client-10' came back up; going online.
>[2012-05-27 12:52:04.175914] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-mysql1-client-11: changing port to 24144 (from 0)
>[2012-05-27 12:52:04.175995] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-7: Connected to 10.0.0.4:24015, attached to remote volume '/var/spool/glusterfs/c_mysql1'.
>[2012-05-27 12:52:04.176023] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:04.176119] I [afr-common.c:3628:afr_notify] 0-mysql1-replicate-2: Subvolume 'mysql1-client-7' came back up; going online.
>[2012-05-27 12:52:04.176261] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-4: Server lk version = 1
>[2012-05-27 12:52:04.176379] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-mysql1-client-0: changing port to 24057 (from 0)
>[2012-05-27 12:52:04.176457] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-1: Connected to 10.0.0.4:24013, attached to remote volume '/var/spool/glusterfs/a_mysql1'.
>[2012-05-27 12:52:04.176486] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:04.176615] I [afr-common.c:3628:afr_notify] 0-mysql1-replicate-0: Subvolume 'mysql1-client-1' came back up; going online.
>[2012-05-27 12:52:04.176825] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-7: Server lk version = 1
>[2012-05-27 12:52:04.176918] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-10: Server lk version = 1
>[2012-05-27 12:52:04.177012] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-1: Server lk version = 1
>[2012-05-27 12:52:08.127171] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-9: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:08.127941] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-9: Connected to 10.0.0.12:24060, attached to remote volume '/var/spool/glusterfs/d_mysql1'.
>[2012-05-27 12:52:08.128006] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:08.128206] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-9: Server lk version = 1
>[2012-05-27 12:52:08.132263] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-8: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:08.132834] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-8: Connected to 10.0.0.120:24143, attached to remote volume '/var/spool/glusterfs/c_mysql1'.
>[2012-05-27 12:52:08.132886] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:08.133046] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-8: Server lk version = 1
>[2012-05-27 12:52:08.137549] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-6: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:08.138190] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-6: Connected to 10.0.0.12:24059, attached to remote volume '/var/spool/glusterfs/c_mysql1'.
>[2012-05-27 12:52:08.138264] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:08.138671] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-6: Server lk version = 1
>[2012-05-27 12:52:08.142696] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-5: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:08.143173] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-5: Connected to 10.0.0.120:24142, attached to remote volume '/var/spool/glusterfs/b_mysql1'.
>[2012-05-27 12:52:08.143248] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:08.143419] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-5: Server lk version = 1
>[2012-05-27 12:52:08.148064] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-3: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:08.148830] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-3: Connected to 10.0.0.12:24058, attached to remote volume '/var/spool/glusterfs/b_mysql1'.
>[2012-05-27 12:52:08.148905] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:08.149231] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-3: Server lk version = 1
>[2012-05-27 12:52:08.153415] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-2: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:08.153896] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-2: Connected to 10.0.0.120:24141, attached to remote volume '/var/spool/glusterfs/a_mysql1'.
>[2012-05-27 12:52:08.153976] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:08.154167] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-2: Server lk version = 1
>[2012-05-27 12:52:08.158687] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-11: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:08.159152] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-11: Connected to 10.0.0.120:24144, attached to remote volume '/var/spool/glusterfs/d_mysql1'.
>[2012-05-27 12:52:08.159222] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-11: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:08.159582] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-11: Server lk version = 1
>[2012-05-27 12:52:08.164398] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-0: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:08.164858] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-0: Connected to 10.0.0.12:24057, attached to remote volume '/var/spool/glusterfs/a_mysql1'.
>[2012-05-27 12:52:08.164895] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:08.174401] I [fuse-bridge.c:4138:fuse_graph_setup] 0-fuse: switched to graph 0
>[2012-05-27 12:52:08.174695] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-0: Server lk version = 1
>[2012-05-27 12:52:08.174878] I [fuse-bridge.c:3376:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.10
>[2012-05-27 12:52:08.175746] I [afr-common.c:1965:afr_set_root_inode_on_first_lookup] 0-mysql1-replicate-0: added root inode
>[2012-05-27 12:52:08.178182] I [afr-common.c:1965:afr_set_root_inode_on_first_lookup] 0-mysql1-replicate-2: added root inode
>[2012-05-27 12:52:08.178560] I [afr-common.c:1965:afr_set_root_inode_on_first_lookup] 0-mysql1-replicate-1: added root inode
>[2012-05-27 12:52:08.178962] I [afr-common.c:1965:afr_set_root_inode_on_first_lookup] 0-mysql1-replicate-3: added root inode
>[2012-05-27 12:52:08.192998] W [afr-self-heal-data.c:826:afr_lookup_select_read_child_by_txn_type] 0-mysql1-replicate-0: /mariadb/mysql/tables_priv.MYI: Possible split-brain
>[2012-05-27 12:52:08.193087] W [afr-common.c:1227:afr_detect_self_heal_by_lookup_status] 0-mysql1-replicate-0: split brain detected during lookup of /mariadb/mysql/tables_priv.MYI.
>[2012-05-27 12:52:08.193118] I [afr-common.c:1189:afr_detect_self_heal_by_iatt] 0-mysql1-replicate-0: size differs for /mariadb/mysql/tables_priv.MYI 
>[2012-05-27 12:52:08.193160] I [afr-common.c:1341:afr_launch_self_heal] 0-mysql1-replicate-0: background  data missing-entry gfid self-heal triggered. path: /mariadb/mysql/tables_priv.MYI, reason: lookup detected pending operations
>[2012-05-27 12:52:08.194974] I [afr-self-heal-common.c:1318:afr_sh_missing_entries_lookup_done] 0-mysql1-replicate-0: No sources for dir of /mariadb/mysql/tables_priv.MYI, in missing entry self-heal, continuing with the rest of the self-heals
>[2012-05-27 12:52:08.195442] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-mysql1-replicate-0: background  data missing-entry gfid self-heal completed on /mariadb/mysql/tables_priv.MYI
>[2012-05-27 12:52:08.380818] I [fuse-bridge.c:4037:fuse_thread_proc] 0-fuse: unmounting /tmp/mysql1.3LoOl0
>[2012-05-27 12:52:08.382741] W [glusterfsd.c:816:cleanup_and_exit] (-->/lib/libc.so.6(clone+0x5e) [0xdd40ae] (-->/lib/libpthread.so.0 [0x9e6852] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0x1ff) [0x804c33f]))) 0-: received signum (15), shutting down
>[2012-05-27 12:52:15.587346] I [glusterfsd.c:1651:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.0qa43
>[2012-05-27 12:52:15.621049] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-3: adding option 'entry-self-heal' for volume 'mysql1-replicate-3' with value 'off'
>[2012-05-27 12:52:15.621139] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-3: adding option 'metadata-self-heal' for volume 'mysql1-replicate-3' with value 'off'
>[2012-05-27 12:52:15.621168] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-3: adding option 'data-self-heal' for volume 'mysql1-replicate-3' with value 'off'
>[2012-05-27 12:52:15.621200] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-2: adding option 'entry-self-heal' for volume 'mysql1-replicate-2' with value 'off'
>[2012-05-27 12:52:15.621227] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-2: adding option 'metadata-self-heal' for volume 'mysql1-replicate-2' with value 'off'
>[2012-05-27 12:52:15.621253] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-2: adding option 'data-self-heal' for volume 'mysql1-replicate-2' with value 'off'
>[2012-05-27 12:52:15.621280] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-1: adding option 'entry-self-heal' for volume 'mysql1-replicate-1' with value 'off'
>[2012-05-27 12:52:15.621307] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-1: adding option 'metadata-self-heal' for volume 'mysql1-replicate-1' with value 'off'
>[2012-05-27 12:52:15.621333] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-1: adding option 'data-self-heal' for volume 'mysql1-replicate-1' with value 'off'
>[2012-05-27 12:52:15.621359] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-0: adding option 'entry-self-heal' for volume 'mysql1-replicate-0' with value 'off'
>[2012-05-27 12:52:15.621386] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-0: adding option 'metadata-self-heal' for volume 'mysql1-replicate-0' with value 'off'
>[2012-05-27 12:52:15.621412] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-0: adding option 'data-self-heal' for volume 'mysql1-replicate-0' with value 'off'
>[2012-05-27 12:52:15.621445] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-client-10: adding option 'remote-port' for volume 'mysql1-client-10' with value '24016'
>[2012-05-27 12:52:15.621476] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-client-7: adding option 'remote-port' for volume 'mysql1-client-7' with value '24015'
>[2012-05-27 12:52:15.621507] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-client-4: adding option 'remote-port' for volume 'mysql1-client-4' with value '24014'
>[2012-05-27 12:52:15.621536] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-client-1: adding option 'remote-port' for volume 'mysql1-client-1' with value '24013'
>[2012-05-27 12:52:15.660615] I [client.c:2142:notify] 0-mysql1-client-0: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:15.666604] I [client.c:2142:notify] 0-mysql1-client-1: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:15.671864] I [client.c:2142:notify] 0-mysql1-client-2: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:15.676739] I [client.c:2142:notify] 0-mysql1-client-3: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:15.681722] I [client.c:2142:notify] 0-mysql1-client-4: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:15.686620] I [client.c:2142:notify] 0-mysql1-client-5: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:15.691745] I [client.c:2142:notify] 0-mysql1-client-6: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:15.696640] I [client.c:2142:notify] 0-mysql1-client-7: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:15.701920] I [client.c:2142:notify] 0-mysql1-client-8: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:15.706825] I [client.c:2142:notify] 0-mysql1-client-9: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:15.711848] I [client.c:2142:notify] 0-mysql1-client-10: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:15.717128] I [client.c:2142:notify] 0-mysql1-client-11: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume mysql1-client-0
>  2:     type protocol/client
>  3:     option remote-host ewcs2
>  4:     option remote-subvolume /var/spool/glusterfs/a_mysql1
>  5:     option transport-type tcp
>  6: end-volume
>  7: 
>  8: volume mysql1-client-1
>  9:     type protocol/client
> 10:     option remote-host ewcs4
> 11:     option remote-subvolume /var/spool/glusterfs/a_mysql1
> 12:     option transport-type tcp
> 13: end-volume
> 14: 
> 15: volume mysql1-client-2
> 16:     type protocol/client
> 17:     option remote-host ewcs7
> 18:     option remote-subvolume /var/spool/glusterfs/a_mysql1
> 19:     option transport-type tcp
> 20: end-volume
> 21: 
> 22: volume mysql1-client-3
> 23:     type protocol/client
> 24:     option remote-host ewcs2
> 25:     option remote-subvolume /var/spool/glusterfs/b_mysql1
> 26:     option transport-type tcp
> 27: end-volume
> 28: 
> 29: volume mysql1-client-4
> 30:     type protocol/client
> 31:     option remote-host ewcs4
> 32:     option remote-subvolume /var/spool/glusterfs/b_mysql1
> 33:     option transport-type tcp
> 34: end-volume
> 35: 
> 36: volume mysql1-client-5
> 37:     type protocol/client
> 38:     option remote-host ewcs7
> 39:     option remote-subvolume /var/spool/glusterfs/b_mysql1
> 40:     option transport-type tcp
> 41: end-volume
> 42: 
> 43: volume mysql1-client-6
> 44:     type protocol/client
> 45:     option remote-host ewcs2
> 46:     option remote-subvolume /var/spool/glusterfs/c_mysql1
> 47:     option transport-type tcp
> 48: end-volume
> 49: 
> 50: volume mysql1-client-7
> 51:     type protocol/client
> 52:     option remote-host ewcs4
> 53:     option remote-subvolume /var/spool/glusterfs/c_mysql1
> 54:     option transport-type tcp
> 55: end-volume
> 56: 
> 57: volume mysql1-client-8
> 58:     type protocol/client
> 59:     option remote-host ewcs7
> 60:     option remote-subvolume /var/spool/glusterfs/c_mysql1
> 61:     option transport-type tcp
> 62: end-volume
> 63: 
> 64: volume mysql1-client-9
> 65:     type protocol/client
> 66:     option remote-host ewcs2
> 67:     option remote-subvolume /var/spool/glusterfs/d_mysql1
> 68:     option transport-type tcp
> 69: end-volume
> 70: 
> 71: volume mysql1-client-10
> 72:     type protocol/client
> 73:     option remote-host ewcs4
> 74:     option remote-subvolume /var/spool/glusterfs/d_mysql1
> 75:     option transport-type tcp
> 76: end-volume
> 77: 
> 78: volume mysql1-client-11
> 79:     type protocol/client
> 80:     option remote-host ewcs7
> 81:     option remote-subvolume /var/spool/glusterfs/d_mysql1
> 82:     option transport-type tcp
> 83: end-volume
> 84: 
> 85: volume mysql1-replicate-0
> 86:     type cluster/replicate
> 87:     subvolumes mysql1-client-0 mysql1-client-1 mysql1-client-2
> 88: end-volume
> 89: 
> 90: volume mysql1-replicate-1
> 91:     type cluster/replicate
> 92:     subvolumes mysql1-client-3 mysql1-client-4 mysql1-client-5
> 93: end-volume
> 94: 
> 95: volume mysql1-replicate-2
> 96:     type cluster/replicate
> 97:     subvolumes mysql1-client-6 mysql1-client-7 mysql1-client-8
> 98: end-volume
> 99: 
>100: volume mysql1-replicate-3
>101:     type cluster/replicate
>102:     subvolumes mysql1-client-9 mysql1-client-10 mysql1-client-11
>103: end-volume
>104: 
>105: volume mysql1-dht
>106:     type cluster/distribute
>107:     subvolumes mysql1-replicate-0 mysql1-replicate-1 mysql1-replicate-2 mysql1-replicate-3
>108: end-volume
>109: 
>110: volume mysql1-md-cache
>111:     type performance/md-cache
>112:     subvolumes mysql1-dht
>113: end-volume
>114: 
>115: volume mysql1
>116:     type debug/io-stats
>117:     option latency-measurement off
>118:     option count-fop-hits off
>119:     subvolumes mysql1-md-cache
>120: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-05-27 12:52:15.724079] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-4: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:15.724385] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-7: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:15.724609] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-10: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:15.724861] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-1: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:15.725005] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-mysql1-client-9: changing port to 24060 (from 0)
>[2012-05-27 12:52:15.725084] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-mysql1-client-8: changing port to 24143 (from 0)
>[2012-05-27 12:52:15.725138] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-mysql1-client-6: changing port to 24059 (from 0)
>[2012-05-27 12:52:15.725199] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-mysql1-client-5: changing port to 24142 (from 0)
>[2012-05-27 12:52:15.725291] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-4: Connected to 10.0.0.4:24014, attached to remote volume '/var/spool/glusterfs/b_mysql1'.
>[2012-05-27 12:52:15.725325] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:15.725434] I [afr-common.c:3628:afr_notify] 0-mysql1-replicate-1: Subvolume 'mysql1-client-4' came back up; going online.
>[2012-05-27 12:52:15.725506] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-mysql1-client-3: changing port to 24058 (from 0)
>[2012-05-27 12:52:15.725566] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-mysql1-client-2: changing port to 24141 (from 0)
>[2012-05-27 12:52:15.725644] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-10: Connected to 10.0.0.4:24016, attached to remote volume '/var/spool/glusterfs/d_mysql1'.
>[2012-05-27 12:52:15.725675] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-10: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:15.725790] I [afr-common.c:3628:afr_notify] 0-mysql1-replicate-3: Subvolume 'mysql1-client-10' came back up; going online.
>[2012-05-27 12:52:15.725850] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-mysql1-client-11: changing port to 24144 (from 0)
>[2012-05-27 12:52:15.725919] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-7: Connected to 10.0.0.4:24015, attached to remote volume '/var/spool/glusterfs/c_mysql1'.
>[2012-05-27 12:52:15.725948] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:15.726036] I [afr-common.c:3628:afr_notify] 0-mysql1-replicate-2: Subvolume 'mysql1-client-7' came back up; going online.
>[2012-05-27 12:52:15.726155] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-4: Server lk version = 1
>[2012-05-27 12:52:15.726289] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-1: Connected to 10.0.0.4:24013, attached to remote volume '/var/spool/glusterfs/a_mysql1'.
>[2012-05-27 12:52:15.726320] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:15.726417] I [afr-common.c:3628:afr_notify] 0-mysql1-replicate-0: Subvolume 'mysql1-client-1' came back up; going online.
>[2012-05-27 12:52:15.726474] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-mysql1-client-0: changing port to 24057 (from 0)
>[2012-05-27 12:52:15.726557] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-1: Server lk version = 1
>[2012-05-27 12:52:15.726635] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-7: Server lk version = 1
>[2012-05-27 12:52:15.726715] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-10: Server lk version = 1
>[2012-05-27 12:52:19.678505] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-9: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:19.679177] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-9: Connected to 10.0.0.12:24060, attached to remote volume '/var/spool/glusterfs/d_mysql1'.
>[2012-05-27 12:52:19.679252] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:19.679528] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-9: Server lk version = 1
>[2012-05-27 12:52:19.683815] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-8: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:19.684652] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-8: Connected to 10.0.0.120:24143, attached to remote volume '/var/spool/glusterfs/c_mysql1'.
>[2012-05-27 12:52:19.684711] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:19.684922] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-8: Server lk version = 1
>[2012-05-27 12:52:19.688932] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-6: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:19.689419] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-6: Connected to 10.0.0.12:24059, attached to remote volume '/var/spool/glusterfs/c_mysql1'.
>[2012-05-27 12:52:19.689491] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:19.690936] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-6: Server lk version = 1
>[2012-05-27 12:52:19.694754] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-5: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:19.695218] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-5: Connected to 10.0.0.120:24142, attached to remote volume '/var/spool/glusterfs/b_mysql1'.
>[2012-05-27 12:52:19.695259] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:19.695551] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-5: Server lk version = 1
>[2012-05-27 12:52:19.699455] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-3: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:19.700016] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-3: Connected to 10.0.0.12:24058, attached to remote volume '/var/spool/glusterfs/b_mysql1'.
>[2012-05-27 12:52:19.700090] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:19.700436] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-3: Server lk version = 1
>[2012-05-27 12:52:19.704946] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-2: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:19.705716] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-2: Connected to 10.0.0.120:24141, attached to remote volume '/var/spool/glusterfs/a_mysql1'.
>[2012-05-27 12:52:19.705778] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:19.706037] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-2: Server lk version = 1
>[2012-05-27 12:52:19.710182] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-11: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:19.710853] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-11: Connected to 10.0.0.120:24144, attached to remote volume '/var/spool/glusterfs/d_mysql1'.
>[2012-05-27 12:52:19.710955] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-11: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:19.711274] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-11: Server lk version = 1
>[2012-05-27 12:52:19.716114] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-0: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:19.716579] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-0: Connected to 10.0.0.12:24057, attached to remote volume '/var/spool/glusterfs/a_mysql1'.
>[2012-05-27 12:52:19.716619] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:19.726158] I [fuse-bridge.c:4138:fuse_graph_setup] 0-fuse: switched to graph 0
>[2012-05-27 12:52:19.726454] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-0: Server lk version = 1
>[2012-05-27 12:52:19.726633] I [fuse-bridge.c:3376:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.10
>[2012-05-27 12:52:19.727517] I [afr-common.c:1965:afr_set_root_inode_on_first_lookup] 0-mysql1-replicate-0: added root inode
>[2012-05-27 12:52:19.729764] I [afr-common.c:1965:afr_set_root_inode_on_first_lookup] 0-mysql1-replicate-2: added root inode
>[2012-05-27 12:52:19.730076] I [afr-common.c:1965:afr_set_root_inode_on_first_lookup] 0-mysql1-replicate-1: added root inode
>[2012-05-27 12:52:19.730504] I [afr-common.c:1965:afr_set_root_inode_on_first_lookup] 0-mysql1-replicate-3: added root inode
>[2012-05-27 12:52:19.743684] W [afr-self-heal-data.c:826:afr_lookup_select_read_child_by_txn_type] 0-mysql1-replicate-0: /mariadb/mysql/tables_priv.MYI: Possible split-brain
>[2012-05-27 12:52:19.743770] W [afr-common.c:1227:afr_detect_self_heal_by_lookup_status] 0-mysql1-replicate-0: split brain detected during lookup of /mariadb/mysql/tables_priv.MYI.
>[2012-05-27 12:52:19.743800] I [afr-common.c:1189:afr_detect_self_heal_by_iatt] 0-mysql1-replicate-0: size differs for /mariadb/mysql/tables_priv.MYI 
>[2012-05-27 12:52:19.743827] I [afr-common.c:1189:afr_detect_self_heal_by_iatt] 0-mysql1-replicate-0: size differs for /mariadb/mysql/tables_priv.MYI 
>[2012-05-27 12:52:19.743867] I [afr-common.c:1341:afr_launch_self_heal] 0-mysql1-replicate-0: background  data missing-entry gfid self-heal triggered. path: /mariadb/mysql/tables_priv.MYI, reason: lookup detected pending operations
>[2012-05-27 12:52:19.745579] I [afr-self-heal-common.c:1318:afr_sh_missing_entries_lookup_done] 0-mysql1-replicate-0: No sources for dir of /mariadb/mysql/tables_priv.MYI, in missing entry self-heal, continuing with the rest of the self-heals
>[2012-05-27 12:52:19.746045] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-mysql1-replicate-0: background  data missing-entry gfid self-heal completed on /mariadb/mysql/tables_priv.MYI
>[2012-05-27 12:52:19.752412] I [fuse-bridge.c:4037:fuse_thread_proc] 0-fuse: unmounting /tmp/mysql1.TI4o1D
>[2012-05-27 12:52:19.753120] W [glusterfsd.c:816:cleanup_and_exit] (-->/lib/libc.so.6(clone+0x5e) [0x5dd0ae] (-->/lib/libpthread.so.0 [0x115852] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0x1ff) [0x804c33f]))) 0-: received signum (15), shutting down
>[2012-05-27 12:52:24.124073] I [glusterfsd.c:1651:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.0qa43
>[2012-05-27 12:52:24.152573] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-3: adding option 'entry-self-heal' for volume 'mysql1-replicate-3' with value 'off'
>[2012-05-27 12:52:24.152626] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-3: adding option 'metadata-self-heal' for volume 'mysql1-replicate-3' with value 'off'
>[2012-05-27 12:52:24.152654] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-3: adding option 'data-self-heal' for volume 'mysql1-replicate-3' with value 'off'
>[2012-05-27 12:52:24.152680] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-2: adding option 'entry-self-heal' for volume 'mysql1-replicate-2' with value 'off'
>[2012-05-27 12:52:24.152735] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-2: adding option 'metadata-self-heal' for volume 'mysql1-replicate-2' with value 'off'
>[2012-05-27 12:52:24.152762] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-2: adding option 'data-self-heal' for volume 'mysql1-replicate-2' with value 'off'
>[2012-05-27 12:52:24.152788] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-1: adding option 'entry-self-heal' for volume 'mysql1-replicate-1' with value 'off'
>[2012-05-27 12:52:24.152814] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-1: adding option 'metadata-self-heal' for volume 'mysql1-replicate-1' with value 'off'
>[2012-05-27 12:52:24.152839] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-1: adding option 'data-self-heal' for volume 'mysql1-replicate-1' with value 'off'
>[2012-05-27 12:52:24.152865] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-0: adding option 'entry-self-heal' for volume 'mysql1-replicate-0' with value 'off'
>[2012-05-27 12:52:24.152891] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-0: adding option 'metadata-self-heal' for volume 'mysql1-replicate-0' with value 'off'
>[2012-05-27 12:52:24.152922] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-replicate-0: adding option 'data-self-heal' for volume 'mysql1-replicate-0' with value 'off'
>[2012-05-27 12:52:24.152953] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-client-10: adding option 'remote-port' for volume 'mysql1-client-10' with value '24016'
>[2012-05-27 12:52:24.152983] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-client-7: adding option 'remote-port' for volume 'mysql1-client-7' with value '24015'
>[2012-05-27 12:52:24.153012] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-client-4: adding option 'remote-port' for volume 'mysql1-client-4' with value '24014'
>[2012-05-27 12:52:24.153041] I [graph.c:241:gf_add_cmdline_options] 0-mysql1-client-1: adding option 'remote-port' for volume 'mysql1-client-1' with value '24013'
>[2012-05-27 12:52:24.192303] I [client.c:2142:notify] 0-mysql1-client-0: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:24.198520] I [client.c:2142:notify] 0-mysql1-client-1: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:24.203942] I [client.c:2142:notify] 0-mysql1-client-2: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:24.208945] I [client.c:2142:notify] 0-mysql1-client-3: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:24.214206] I [client.c:2142:notify] 0-mysql1-client-4: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:24.219229] I [client.c:2142:notify] 0-mysql1-client-5: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:24.224389] I [client.c:2142:notify] 0-mysql1-client-6: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:24.229357] I [client.c:2142:notify] 0-mysql1-client-7: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:24.234611] I [client.c:2142:notify] 0-mysql1-client-8: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:24.239563] I [client.c:2142:notify] 0-mysql1-client-9: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:24.244643] I [client.c:2142:notify] 0-mysql1-client-10: parent translators are ready, attempting connect on transport
>[2012-05-27 12:52:24.249739] I [client.c:2142:notify] 0-mysql1-client-11: parent translators are ready, attempting connect on transport
>Given volfile:
>+------------------------------------------------------------------------------+
>  1: volume mysql1-client-0
>  2:     type protocol/client
>  3:     option remote-host ewcs2
>  4:     option remote-subvolume /var/spool/glusterfs/a_mysql1
>  5:     option transport-type tcp
>  6: end-volume
>  7: 
>  8: volume mysql1-client-1
>  9:     type protocol/client
> 10:     option remote-host ewcs4
> 11:     option remote-subvolume /var/spool/glusterfs/a_mysql1
> 12:     option transport-type tcp
> 13: end-volume
> 14: 
> 15: volume mysql1-client-2
> 16:     type protocol/client
> 17:     option remote-host ewcs7
> 18:     option remote-subvolume /var/spool/glusterfs/a_mysql1
> 19:     option transport-type tcp
> 20: end-volume
> 21: 
> 22: volume mysql1-client-3
> 23:     type protocol/client
> 24:     option remote-host ewcs2
> 25:     option remote-subvolume /var/spool/glusterfs/b_mysql1
> 26:     option transport-type tcp
> 27: end-volume
> 28: 
> 29: volume mysql1-client-4
> 30:     type protocol/client
> 31:     option remote-host ewcs4
> 32:     option remote-subvolume /var/spool/glusterfs/b_mysql1
> 33:     option transport-type tcp
> 34: end-volume
> 35: 
> 36: volume mysql1-client-5
> 37:     type protocol/client
> 38:     option remote-host ewcs7
> 39:     option remote-subvolume /var/spool/glusterfs/b_mysql1
> 40:     option transport-type tcp
> 41: end-volume
> 42: 
> 43: volume mysql1-client-6
> 44:     type protocol/client
> 45:     option remote-host ewcs2
> 46:     option remote-subvolume /var/spool/glusterfs/c_mysql1
> 47:     option transport-type tcp
> 48: end-volume
> 49: 
> 50: volume mysql1-client-7
> 51:     type protocol/client
> 52:     option remote-host ewcs4
> 53:     option remote-subvolume /var/spool/glusterfs/c_mysql1
> 54:     option transport-type tcp
> 55: end-volume
> 56: 
> 57: volume mysql1-client-8
> 58:     type protocol/client
> 59:     option remote-host ewcs7
> 60:     option remote-subvolume /var/spool/glusterfs/c_mysql1
> 61:     option transport-type tcp
> 62: end-volume
> 63: 
> 64: volume mysql1-client-9
> 65:     type protocol/client
> 66:     option remote-host ewcs2
> 67:     option remote-subvolume /var/spool/glusterfs/d_mysql1
> 68:     option transport-type tcp
> 69: end-volume
> 70: 
> 71: volume mysql1-client-10
> 72:     type protocol/client
> 73:     option remote-host ewcs4
> 74:     option remote-subvolume /var/spool/glusterfs/d_mysql1
> 75:     option transport-type tcp
> 76: end-volume
> 77: 
> 78: volume mysql1-client-11
> 79:     type protocol/client
> 80:     option remote-host ewcs7
> 81:     option remote-subvolume /var/spool/glusterfs/d_mysql1
> 82:     option transport-type tcp
> 83: end-volume
> 84: 
> 85: volume mysql1-replicate-0
> 86:     type cluster/replicate
> 87:     subvolumes mysql1-client-0 mysql1-client-1 mysql1-client-2
> 88: end-volume
> 89: 
> 90: volume mysql1-replicate-1
> 91:     type cluster/replicate
> 92:     subvolumes mysql1-client-3 mysql1-client-4 mysql1-client-5
> 93: end-volume
> 94: 
> 95: volume mysql1-replicate-2
> 96:     type cluster/replicate
> 97:     subvolumes mysql1-client-6 mysql1-client-7 mysql1-client-8
> 98: end-volume
> 99: 
>100: volume mysql1-replicate-3
>101:     type cluster/replicate
>102:     subvolumes mysql1-client-9 mysql1-client-10 mysql1-client-11
>103: end-volume
>104: 
>105: volume mysql1-dht
>106:     type cluster/distribute
>107:     subvolumes mysql1-replicate-0 mysql1-replicate-1 mysql1-replicate-2 mysql1-replicate-3
>108: end-volume
>109: 
>110: volume mysql1-md-cache
>111:     type performance/md-cache
>112:     subvolumes mysql1-dht
>113: end-volume
>114: 
>115: volume mysql1
>116:     type debug/io-stats
>117:     option latency-measurement off
>118:     option count-fop-hits off
>119:     subvolumes mysql1-md-cache
>120: end-volume
>
>+------------------------------------------------------------------------------+
>[2012-05-27 12:52:24.256129] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-4: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:24.256393] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-7: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:24.256609] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-10: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:24.256836] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-1: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:24.257025] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-mysql1-client-9: changing port to 24060 (from 0)
>[2012-05-27 12:52:24.257125] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-mysql1-client-8: changing port to 24143 (from 0)
>[2012-05-27 12:52:24.257184] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-mysql1-client-6: changing port to 24059 (from 0)
>[2012-05-27 12:52:24.257245] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-mysql1-client-5: changing port to 24142 (from 0)
>[2012-05-27 12:52:24.257331] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-4: Connected to 10.0.0.4:24014, attached to remote volume '/var/spool/glusterfs/b_mysql1'.
>[2012-05-27 12:52:24.257363] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-4: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:24.257469] I [afr-common.c:3628:afr_notify] 0-mysql1-replicate-1: Subvolume 'mysql1-client-4' came back up; going online.
>[2012-05-27 12:52:24.257536] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-mysql1-client-3: changing port to 24058 (from 0)
>[2012-05-27 12:52:24.257594] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-mysql1-client-2: changing port to 24141 (from 0)
>[2012-05-27 12:52:24.257674] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-10: Connected to 10.0.0.4:24016, attached to remote volume '/var/spool/glusterfs/d_mysql1'.
>[2012-05-27 12:52:24.257703] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-10: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:24.257794] I [afr-common.c:3628:afr_notify] 0-mysql1-replicate-3: Subvolume 'mysql1-client-10' came back up; going online.
>[2012-05-27 12:52:24.257846] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-mysql1-client-11: changing port to 24144 (from 0)
>[2012-05-27 12:52:24.257918] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-7: Connected to 10.0.0.4:24015, attached to remote volume '/var/spool/glusterfs/c_mysql1'.
>[2012-05-27 12:52:24.257946] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-7: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:24.258037] I [afr-common.c:3628:afr_notify] 0-mysql1-replicate-2: Subvolume 'mysql1-client-7' came back up; going online.
>[2012-05-27 12:52:24.258172] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-4: Server lk version = 1
>[2012-05-27 12:52:24.258296] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-1: Connected to 10.0.0.4:24013, attached to remote volume '/var/spool/glusterfs/a_mysql1'.
>[2012-05-27 12:52:24.258326] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-1: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:24.258423] I [afr-common.c:3628:afr_notify] 0-mysql1-replicate-0: Subvolume 'mysql1-client-1' came back up; going online.
>[2012-05-27 12:52:24.258477] I [rpc-clnt.c:1660:rpc_clnt_reconfig] 0-mysql1-client-0: changing port to 24057 (from 0)
>[2012-05-27 12:52:24.258555] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-1: Server lk version = 1
>[2012-05-27 12:52:24.258633] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-7: Server lk version = 1
>[2012-05-27 12:52:24.258695] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-10: Server lk version = 1
>[2012-05-27 12:52:28.211235] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-9: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:28.212123] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-9: Connected to 10.0.0.12:24060, attached to remote volume '/var/spool/glusterfs/d_mysql1'.
>[2012-05-27 12:52:28.212185] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-9: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:28.212442] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-9: Server lk version = 1
>[2012-05-27 12:52:28.216419] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-8: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:28.216912] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-8: Connected to 10.0.0.120:24143, attached to remote volume '/var/spool/glusterfs/c_mysql1'.
>[2012-05-27 12:52:28.217012] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-8: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:28.217244] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-8: Server lk version = 1
>[2012-05-27 12:52:28.221859] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-6: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:28.222655] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-6: Connected to 10.0.0.12:24059, attached to remote volume '/var/spool/glusterfs/c_mysql1'.
>[2012-05-27 12:52:28.222722] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-6: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:28.223084] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-6: Server lk version = 1
>[2012-05-27 12:52:28.227001] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-5: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:28.227429] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-5: Connected to 10.0.0.120:24142, attached to remote volume '/var/spool/glusterfs/b_mysql1'.
>[2012-05-27 12:52:28.227520] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-5: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:28.227789] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-5: Server lk version = 1
>[2012-05-27 12:52:28.232539] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-3: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:28.233377] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-3: Connected to 10.0.0.12:24058, attached to remote volume '/var/spool/glusterfs/b_mysql1'.
>[2012-05-27 12:52:28.233449] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-3: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:28.233795] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-3: Server lk version = 1
>[2012-05-27 12:52:28.237724] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-2: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:28.238167] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-2: Connected to 10.0.0.120:24141, attached to remote volume '/var/spool/glusterfs/a_mysql1'.
>[2012-05-27 12:52:28.238257] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-2: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:28.238559] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-2: Server lk version = 1
>[2012-05-27 12:52:28.243397] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-11: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:28.244121] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-11: Connected to 10.0.0.120:24144, attached to remote volume '/var/spool/glusterfs/d_mysql1'.
>[2012-05-27 12:52:28.244182] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-11: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:28.244509] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-11: Server lk version = 1
>[2012-05-27 12:52:28.249282] I [client-handshake.c:1628:select_server_supported_programs] 0-mysql1-client-0: Using Program GlusterFS 3.3.0qa43, Num (1298437), Version (330)
>[2012-05-27 12:52:28.249789] I [client-handshake.c:1425:client_setvolume_cbk] 0-mysql1-client-0: Connected to 10.0.0.12:24057, attached to remote volume '/var/spool/glusterfs/a_mysql1'.
>[2012-05-27 12:52:28.249852] I [client-handshake.c:1437:client_setvolume_cbk] 0-mysql1-client-0: Server and Client lk-version numbers are not same, reopening the fds
>[2012-05-27 12:52:28.259450] I [fuse-bridge.c:4138:fuse_graph_setup] 0-fuse: switched to graph 0
>[2012-05-27 12:52:28.259740] I [client-handshake.c:453:client_set_lk_version_cbk] 0-mysql1-client-0: Server lk version = 1
>[2012-05-27 12:52:28.259917] I [fuse-bridge.c:3376:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.10
>[2012-05-27 12:52:28.260787] I [afr-common.c:1965:afr_set_root_inode_on_first_lookup] 0-mysql1-replicate-0: added root inode
>[2012-05-27 12:52:28.263021] I [afr-common.c:1965:afr_set_root_inode_on_first_lookup] 0-mysql1-replicate-2: added root inode
>[2012-05-27 12:52:28.263355] I [afr-common.c:1965:afr_set_root_inode_on_first_lookup] 0-mysql1-replicate-1: added root inode
>[2012-05-27 12:52:28.263909] I [afr-common.c:1965:afr_set_root_inode_on_first_lookup] 0-mysql1-replicate-3: added root inode
>[2012-05-27 12:52:28.291047] W [afr-self-heal-data.c:826:afr_lookup_select_read_child_by_txn_type] 0-mysql1-replicate-0: /mariadb/mysql/tables_priv.MYI: Possible split-brain
>[2012-05-27 12:52:28.291172] W [afr-common.c:1227:afr_detect_self_heal_by_lookup_status] 0-mysql1-replicate-0: split brain detected during lookup of /mariadb/mysql/tables_priv.MYI.
>[2012-05-27 12:52:28.291231] I [afr-common.c:1189:afr_detect_self_heal_by_iatt] 0-mysql1-replicate-0: size differs for /mariadb/mysql/tables_priv.MYI 
>[2012-05-27 12:52:28.291327] I [afr-common.c:1341:afr_launch_self_heal] 0-mysql1-replicate-0: background  data missing-entry gfid self-heal triggered. path: /mariadb/mysql/tables_priv.MYI, reason: lookup detected pending operations
>[2012-05-27 12:52:28.294588] I [afr-self-heal-common.c:1318:afr_sh_missing_entries_lookup_done] 0-mysql1-replicate-0: No sources for dir of /mariadb/mysql/tables_priv.MYI, in missing entry self-heal, continuing with the rest of the self-heals
>[2012-05-27 12:52:28.295434] I [afr-self-heal-common.c:2159:afr_self_heal_completion_cbk] 0-mysql1-replicate-0: background  data missing-entry gfid self-heal completed on /mariadb/mysql/tables_priv.MYI
>[2012-05-27 12:52:28.305664] I [fuse-bridge.c:4037:fuse_thread_proc] 0-fuse: unmounting /tmp/mysql1.15wOhK
>[2012-05-27 12:52:28.306871] W [glusterfsd.c:816:cleanup_and_exit] (-->/lib/libc.so.6(clone+0x5e) [0x3220ae] (-->/lib/libpthread.so.0 [0x6a1852] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0x1ff) [0x804c33f]))) 0-: received signum (15), shutting down

Actions: View

Attachments on bug 825562: 587189 | 587198 | 587200 | 587203