Description of problem: I met a problem in one of my recent small project. I tried to build a gluster file system over several pogoplug devices(arm cpu). I have 3 pogoplug and all of them were installed Arch Linux, which have gluster 3.4.2 installed. Two of the pogoplug were configured as gluster server, and two 1T usb hard drivers attached to it separately. The third pogoplug were configured as a gluster client, and I can mount the glusterfs successfully. I also have a laptop(intel cpu) installed with Ubuntu 13 64bit, and I installed the gluster 3.4.2 by the instructions from http://download.gluster.org/pub/gluster/glusterfs/3.4/3.4.2/Ubuntu/Ubuntu.README and I can also mount the gluster filesytem successfully. But, I found that there's some files in a random directories lost in my laptop mount point, while in the pogoplug I can browse them. And I restart one of the gluster server, and remount the file system in my laptop, the files were back, but some other files lost when I changed to another directory. Version-Release number of selected component (if applicable): gluster 3.4.2 in all of platforms (arm, intel, Arch Linux, Ubuntu 13 64bit) How reproducible: 1. Hardwares 1) pogoplug1, gluster server, ip 192.168.2.7 2) pogoplug2, gluster server, ip 192.168.2.8 3) pogoplug3, gluster client, ip 192.168.2.5 4) laptop, gluster client, ip 192.168.2.91 2. connection +-----+ +------+ |pogo3| |laptop| +--+--+ +--+---+ | | ====== local network ======= | | +--+--+ +--+--+ |pogo1| |pogo2| +-----+ +-----+ Steps to Reproduce: 1. start gluster servers in pogoplug1 and pogoplug2 2. start gluster client in pogoplug3(mount) 3. start gluster client in laptop(mount) 4. browser files Actual results: some files in a random directories lost in my laptop mount point. Expected results: the file should be there. Additional info: =========================================================== There're a lot of error message in pogoplug1's log file /var/log/glusterfs/etc-glusterfs-glusterd.vol.log [2014-03-14 16:15:09.631981] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) [2014-03-14 16:15:10.632728] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) [2014-03-14 16:15:11.827808] I [glusterd-handler.c:952:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2014-03-14 16:15:12.633559] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) [2014-03-14 16:15:13.634621] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) =========================================================== pogoplug2's log file /var/log/glusterfs/etc-glusterfs-glusterd.vol.log [2014-03-14 16:55:45.251856] I [glusterfsd.c:1910:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.4.2 (/usr/sbin/glusterd -p /run/glusterd.pid) [2014-03-14 16:55:45.353211] I [glusterd.c:961:init] 0-management: Using /var/lib/glusterd as working directory [2014-03-14 16:55:45.367758] I [socket.c:3480:socket_init] 0-socket.management: SSL support is NOT enabled [2014-03-14 16:55:45.367958] I [socket.c:3495:socket_init] 0-socket.management: using system polling thread [2014-03-14 16:55:45.369898] E [rpc-transport.c:253:rpc_transport_load] 0-rpc-transport: /usr/lib/glusterfs/3.4.2/rpc-transport/rdma.so: cannot open shared object file: No such file or directory [2014-03-14 16:55:45.370041] W [rpc-transport.c:257:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine [2014-03-14 16:55:45.370144] W [rpcsvc.c:1389:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed [2014-03-14 16:56:03.022247] I [glusterd-store.c:1339:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2 [2014-03-14 16:56:03.063619] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0 [2014-03-14 16:56:03.063886] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1 [2014-03-14 16:56:03.263168] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0 [2014-03-14 16:56:03.263436] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1 [2014-03-14 16:56:03.463129] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0 [2014-03-14 16:56:03.463401] E [glusterd-store.c:1858:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1 [2014-03-14 16:56:03.662476] I [glusterd-handler.c:2818:glusterd_friend_add] 0-management: connect returned 0 [2014-03-14 16:56:03.663191] I [rpc-clnt.c:962:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2014-03-14 16:56:03.663714] I [socket.c:3480:socket_init] 0-management: SSL support is NOT enabled [2014-03-14 16:56:03.663831] I [socket.c:3495:socket_init] 0-management: using system polling thread [2014-03-14 16:56:03.703083] I [glusterd.c:125:glusterd_uuid_init] 0-management: retrieved UUID: e1b96b84-49bf-4ead-b044-03ba141af3be Given volfile: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option working-directory /var/lib/glusterd 4: option transport-type socket,rdma 5: option transport.socket.keepalive-time 10 6: option transport.socket.keepalive-interval 2 7: option transport.socket.read-fail-log off 8: # option base-port 49152 9: end-volume +------------------------------------------------------------------------------+ [2014-03-14 16:56:05.712944] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /media/yhfu3tmirror/glusterfs-brick1 on port 49152 [2014-03-14 16:56:05.722766] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /media/yhfu1t2vol/glusterfs-brick1 on port 49154 [2014-03-14 16:56:05.732657] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /media/yhfu1t2vol/glusterfs-test on port 49153 [2014-03-14 16:56:05.738813] I [glusterd-handshake.c:557:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 2 [2014-03-14 16:56:06.915986] I [glusterd-handler.c:1956:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 101bf18d-2bc9-4a83-a06b-5c946a1f1fc2 [2014-03-14 16:56:06.917327] I [glusterd-handler.c:2987:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to home-pogoplug-v4-2 (0), ret: 0 [2014-03-14 16:56:07.026131] I [glusterd-sm.c:494:glusterd_ac_send_friend_update] 0-: Added uuid: 101bf18d-2bc9-4a83-a06b-5c946a1f1fc2, host: home-pogoplug-v4-1 [2014-03-14 16:56:07.137486] I [rpc-clnt.c:962:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2014-03-14 16:56:07.138012] I [socket.c:3480:socket_init] 0-management: SSL support is NOT enabled [2014-03-14 16:56:07.138185] I [socket.c:3495:socket_init] 0-management: using system polling thread [2014-03-14 16:56:07.139587] I [rpc-clnt.c:962:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2014-03-14 16:56:07.140003] I [socket.c:3480:socket_init] 0-management: SSL support is NOT enabled [2014-03-14 16:56:07.140110] I [socket.c:3495:socket_init] 0-management: using system polling thread [2014-03-14 16:56:07.141530] I [rpc-clnt.c:962:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2014-03-14 16:56:07.141965] I [socket.c:3480:socket_init] 0-management: SSL support is NOT enabled [2014-03-14 16:56:07.142071] I [socket.c:3495:socket_init] 0-management: using system polling thread [2014-03-14 16:56:07.143761] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick [2014-03-14 16:56:07.145274] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick [2014-03-14 16:56:07.147654] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick [2014-03-14 16:56:07.149072] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick [2014-03-14 16:56:07.151474] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick [2014-03-14 16:56:07.152915] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick [2014-03-14 16:56:07.175917] I [rpc-clnt.c:962:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2014-03-14 16:56:07.176501] I [socket.c:3480:socket_init] 0-management: SSL support is NOT enabled [2014-03-14 16:56:07.176622] I [socket.c:3495:socket_init] 0-management: using system polling thread [2014-03-14 16:56:07.176922] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) [2014-03-14 16:56:07.218274] I [rpc-clnt.c:962:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2014-03-14 16:56:07.218867] I [socket.c:3480:socket_init] 0-management: SSL support is NOT enabled [2014-03-14 16:56:07.218986] I [socket.c:3495:socket_init] 0-management: using system polling thread [2014-03-14 16:56:07.219265] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) [2014-03-14 16:56:07.233472] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick [2014-03-14 16:56:07.235302] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick [2014-03-14 16:56:07.236874] I [glusterd-utils.c:1079:glusterd_volume_brickinfo_get] 0-management: Found brick [2014-03-14 16:56:07.346729] I [glusterd-rpc-ops.c:345:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 101bf18d-2bc9-4a83-a06b-5c946a1f1fc2, host: home-pogoplug-v4-1, port: 0 [2014-03-14 16:56:08.786310] I [glusterd-rpc-ops.c:542:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 101bf18d-2bc9-4a83-a06b-5c946a1f1fc2 [2014-03-14 16:56:08.786816] I [glusterd-handler.c:2118:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 101bf18d-2bc9-4a83-a06b-5c946a1f1fc2 [2014-03-14 16:56:08.787067] I [glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received uuid: e1b96b84-49bf-4ead-b044-03ba141af3be, hostname:home-pogoplug-v4-2 [2014-03-14 16:56:08.787186] I [glusterd-handler.c:2172:__glusterd_handle_friend_update] 0-: Received my uuid as Friend [2014-03-14 16:56:11.642627] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) [2014-03-14 16:56:12.200137] W [socket.c:514:__socket_rwv] 0-management: readv failed (Connection reset by peer) [2014-03-14 16:56:12.227157] W [socket.c:1962:__socket_proto_state_machine] 0-management: reading from socket failed. Error (Connection reset by peer), peer (/var/run/ff922de1ba43b735c09dcb06b7aa17cb.socket) [2014-03-14 16:56:14.704416] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) [2014-03-14 16:56:14.704865] E [socket.c:2788:socket_connect] 0-management: connection attempt failed (Connection refused) =========================================================== pogoplug1's log file /var/log/glusterfs/cli.log [2014-03-14 16:15:08.824370] W [rpc-transport.c:175:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket" [2014-03-14 16:15:08.883164] I [socket.c:3480:socket_init] 0-glusterfs: SSL support is NOT enabled [2014-03-14 16:15:08.883392] I [socket.c:3495:socket_init] 0-glusterfs: using system polling thread [2014-03-14 16:15:11.862868] I [cli-rpc-ops.c:332:gf_cli_list_friends_cbk] 0-cli: Received resp to list: 0 [2014-03-14 16:15:11.894708] I [input.c:36:cli_batch] 0-: Exiting with: 0 =========================================================== pogoplug2's log file /var/log/glusterfs/cli.log [2014-03-05 13:14:47.061967] W [rpc-transport.c:175:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket" [2014-03-05 13:14:47.077431] I [socket.c:3480:socket_init] 0-glusterfs: SSL support is NOT enabled [2014-03-05 13:14:47.077663] I [socket.c:3495:socket_init] 0-glusterfs: using system polling thread [2014-03-05 13:15:21.507624] I [cli-rpc-ops.c:153:gf_cli_probe_cbk] 0-cli: Received resp to probe [2014-03-05 13:15:21.508204] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2014-03-05 13:15:54.915285] W [rpc-transport.c:175:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket" [2014-03-05 13:15:54.924739] I [socket.c:3480:socket_init] 0-glusterfs: SSL support is NOT enabled [2014-03-05 13:15:54.924962] I [socket.c:3495:socket_init] 0-glusterfs: using system polling thread [2014-03-05 13:15:55.951333] I [cli-rpc-ops.c:332:gf_cli_list_friends_cbk] 0-cli: Received resp to list: 0 [2014-03-05 13:15:55.952295] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2014-03-05 13:16:05.177906] W [rpc-transport.c:175:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket" [2014-03-05 13:16:05.183610] I [socket.c:3480:socket_init] 0-glusterfs: SSL support is NOT enabled [2014-03-05 13:16:05.183826] I [socket.c:3495:socket_init] 0-glusterfs: using system polling thread [2014-03-05 13:16:06.112691] I [cli-rpc-ops.c:545:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0 [2014-03-05 13:16:06.113012] I [cli-rpc-ops.c:778:gf_cli_get_volume_cbk] 0-cli: Returning: 0 [2014-03-05 13:16:06.113503] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2014-03-05 13:16:25.335633] W [rpc-transport.c:175:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket" [2014-03-05 13:16:25.341258] I [socket.c:3480:socket_init] 0-glusterfs: SSL support is NOT enabled [2014-03-05 13:16:25.341462] I [socket.c:3495:socket_init] 0-glusterfs: using system polling thread [2014-03-05 13:17:19.035421] I [input.c:36:cli_batch] 0-: Exiting with: 0 [2014-03-05 13:17:22.365940] W [rpc-transport.c:175:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket" [2014-03-05 13:17:22.371571] I [socket.c:3480:socket_init] 0-glusterfs: SSL support is NOT enabled [2014-03-05 13:17:22.371776] I [socket.c:3495:socket_init] 0-glusterfs: using system polling thread [2014-03-05 13:17:23.320204] I [cli-rpc-ops.c:545:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0 [2014-03-05 13:17:23.321559] I [cli-rpc-ops.c:778:gf_cli_get_volume_cbk] 0-cli: Returning: 0 [2014-03-05 13:17:23.323734] I [cli-rpc-ops.c:545:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0 [2014-03-05 13:17:23.324582] I [cli-rpc-ops.c:778:gf_cli_get_volume_cbk] 0-cli: Returning: 0 [2014-03-05 13:17:23.326514] I [cli-rpc-ops.c:545:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0 [2014-03-05 13:17:23.327285] I [cli-rpc-ops.c:778:gf_cli_get_volume_cbk] 0-cli: Returning: 0 [2014-03-05 13:17:23.328729] I [cli-rpc-ops.c:545:gf_cli_get_volume_cbk] 0-cli: Received resp to get vol: 0 [2014-03-05 13:17:23.328905] I [cli-rpc-ops.c:778:gf_cli_get_volume_cbk] 0-cli: Returning: 0 [2014-03-05 13:17:23.329119] I [input.c:36:cli_batch] 0-: Exiting with: 0 [1970-01-01 00:03:47.959501] W [rpc-transport.c:175:rpc_transport_load] 0-rpc-transport: missing 'option transport-type'. defaulting to "socket" [1970-01-01 00:03:47.967588] I [socket.c:3480:socket_init] 0-glusterfs: SSL support is NOT enabled [1970-01-01 00:03:47.967809] I [socket.c:3495:socket_init] 0-glusterfs: using system polling thread [1970-01-01 00:03:48.882772] E [socket.c:2157:socket_connect_finish] 0-glusterfs: connection to 127.0.0.1:24007 failed (Connection refused) [1970-01-01 00:03:48.883186] W [socket.c:514:__socket_rwv] 0-glusterfs: readv failed (No data available) =========================================================== laptop's log file /var/log/glusterfs/media-filecache1t-.log [2014-03-14 16:56:10.761632] I [fuse-bridge.c:5260:fini] 0-fuse: Unmounting '/media/filecache1t/'. [2014-03-14 16:56:15.898323] I [glusterfsd.c:1910:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.2 (/usr/sbin/glusterfs --fuse-mountopts=allow_other,default_permissions,max_read=131072 --volfile=/etc/glusterfs/filecache1t-fuse.vol --fuse-mountopts=allow_other,default_permissions,max_read=131072 /media/filecache1t/) [2014-03-14 16:56:15.904693] I [socket.c:3480:socket_init] 0-filecache1t-client-1: SSL support is NOT enabled [2014-03-14 16:56:15.904727] I [socket.c:3495:socket_init] 0-filecache1t-client-1: using system polling thread [2014-03-14 16:56:15.905185] I [socket.c:3480:socket_init] 0-filecache1t-client-0: SSL support is NOT enabled [2014-03-14 16:56:15.905206] I [socket.c:3495:socket_init] 0-filecache1t-client-0: using system polling thread [2014-03-14 16:56:15.905236] I [client.c:2154:notify] 0-filecache1t-client-0: parent translators are ready, attempting connect on transport [2014-03-14 16:56:15.905869] I [client.c:2154:notify] 0-filecache1t-client-1: parent translators are ready, attempting connect on transport Given volfile: +------------------------------------------------------------------------------+ 1: volume filecache1t-client-0 2: type protocol/client 3: option password 3263d4ed-db4e-4c98-24a1-901980a4ae37 4: option username 4f71cd59-0c59-05cf-97c3-ddca00a570c1 5: option transport-type tcp 6: option remote-subvolume /media/yhfu1t1vol/glusterfs-brick1 7: option remote-host 192.168.2.7 8: end-volume 9: 10: volume filecache1t-client-1 11: type protocol/client 12: option password 3263d4ed-db4e-4c98-24a1-901980a4ae37 13: option username 4f71cd59-0c59-05cf-97c3-ddca00a570c1 14: option transport-type tcp 15: option remote-subvolume /media/yhfu1t2vol/glusterfs-brick1 16: option remote-host 192.168.2.8 17: end-volume 18: 19: volume filecache1t-replicate-0 20: type cluster/replicate 21: subvolumes filecache1t-client-0 filecache1t-client-1 22: end-volume 23: 24: volume filecache1t-dht 25: type cluster/distribute 26: subvolumes filecache1t-replicate-0 27: end-volume 28: 29: volume filecache1t-write-behind 30: type performance/write-behind 31: subvolumes filecache1t-dht 32: end-volume 33: 34: volume filecache1t-read-ahead 35: type performance/read-ahead 36: subvolumes filecache1t-write-behind 37: end-volume 38: 39: volume filecache1t-io-cache 40: type performance/io-cache 41: option cache-size 100MB 42: subvolumes filecache1t-read-ahead 43: end-volume 44: 45: volume filecache1t-quick-read 46: type performance/quick-read 47: option cache-size 100MB 48: subvolumes filecache1t-io-cache 49: end-volume 50: 51: volume filecache1t-open-behind 52: type performance/open-behind 53: subvolumes filecache1t-quick-read 54: end-volume 55: 56: volume filecache1t-md-cache 57: type performance/md-cache 58: subvolumes filecache1t-open-behind 59: end-volume 60: 61: volume filecache1t 62: type debug/io-stats 63: option count-fop-hits off 64: option latency-measurement off 65: subvolumes filecache1t-md-cache 66: end-volume 67: +------------------------------------------------------------------------------+ [2014-03-14 16:56:15.913541] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-filecache1t-client-0: changing port to 49153 (from 0) [2014-03-14 16:56:15.913599] W [socket.c:514:__socket_rwv] 0-filecache1t-client-0: readv failed (No data available) [2014-03-14 16:56:15.924134] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-filecache1t-client-1: changing port to 49154 (from 0) [2014-03-14 16:56:15.924210] W [socket.c:514:__socket_rwv] 0-filecache1t-client-1: readv failed (No data available) [2014-03-14 16:56:15.924504] I [client-handshake.c:1659:select_server_supported_programs] 0-filecache1t-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2014-03-14 16:56:15.931587] I [client-handshake.c:1456:client_setvolume_cbk] 0-filecache1t-client-0: Connected to 192.168.2.7:49153, attached to remote volume '/media/yhfu1t1vol/glusterfs-brick1'. [2014-03-14 16:56:15.931611] I [client-handshake.c:1468:client_setvolume_cbk] 0-filecache1t-client-0: Server and Client lk-version numbers are not same, reopening the fds [2014-03-14 16:56:15.931679] I [afr-common.c:3698:afr_notify] 0-filecache1t-replicate-0: Subvolume 'filecache1t-client-0' came back up; going online. [2014-03-14 16:56:15.932743] I [client-handshake.c:450:client_set_lk_version_cbk] 0-filecache1t-client-0: Server lk version = 1 [2014-03-14 16:56:15.946848] I [client-handshake.c:1659:select_server_supported_programs] 0-filecache1t-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2014-03-14 16:56:15.965319] I [client-handshake.c:1456:client_setvolume_cbk] 0-filecache1t-client-1: Connected to 192.168.2.8:49154, attached to remote volume '/media/yhfu1t2vol/glusterfs-brick1'. [2014-03-14 16:56:15.965358] I [client-handshake.c:1468:client_setvolume_cbk] 0-filecache1t-client-1: Server and Client lk-version numbers are not same, reopening the fds [2014-03-14 16:56:15.970470] I [fuse-bridge.c:4769:fuse_graph_setup] 0-fuse: switched to graph 0 [2014-03-14 16:56:15.970673] I [client-handshake.c:450:client_set_lk_version_cbk] 0-filecache1t-client-1: Server lk version = 1 [2014-03-14 16:56:15.970927] I [fuse-bridge.c:3724:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.22 [2014-03-14 16:56:15.988044] I [afr-common.c:2057:afr_set_root_inode_on_first_lookup] 0-filecache1t-replicate-0: added root inode
ARM can run in Big-Engian mode or Little-Endian mode, what Endianness do your Pogoplugs have? Mixing Little-Endian (Intel) and Big-Endian will very likely have some issues (see bug 951903). There is also a bug that involves 64-bit clients and 32-bit servers. Mounting over glusterfs-fuse will return errors for some directories (more than 20 entries). Mounting over NFS does not have any issues (bug 1074023). If you access the mounted volume from a terminal window, do you get any error messages?
All of systems are Little-Endian I run the command command to detect the endian(0-big, 1-little) echo -n I | hexdump -o | awk '{ print substr($2,6,1); exit}' and output were 1s.
An easier check is the 'lscpu' command, 'uname -m', or /proc/cpuinfo ... Also, please let us know this: - If you access the mounted volume from a terminal window, do you get any error messages? - Do you have the same problem when mounting over NFS? - Are the missing files available on any of the raw bricks? At the moment, I expect that this is a duplicate of bug 1074023.
I changed a little bit of my network recently, and I changed the OS system for the x86 arch to a i686 Arch Linux (32bit), so that the systems involved is all Arch Linux 32bit; the difference is the gluster server running on ARM cpu, and the client is on x86 cpu. Now I also found there's I/O error for some directories when I check the files from the gluster client. here's the cpu info I gathered: 1. gluster client: a VM cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 2 model name : QEMU Virtual CPU version 1.5.0 stepping : 3 microcode : 0x1 cpu MHz : 2526.862 cache size : 4096 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fdiv_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 4 wp : yes flags : fpu de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm pni vmx cx16 hypervisor lahf_lm bogomips : 5055.29 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: lscpu Architecture: i686 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 1 On-line CPU(s) list: 0 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 2 Model name: QEMU Virtual CPU version 1.5.0 Stepping: 3 CPU MHz: 2526.862 BogoMIPS: 5055.29 Virtualization: VT-x Hypervisor vendor: KVM Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 4096K uname -m i686 uname -a Linux home-fileserver 3.13.7-1-ARCH #1 SMP PREEMPT Mon Mar 24 19:50:04 CET 2014 i686 GNU/Linux 2. gluster server(s): ARM cat /proc/cpuinfo Processor : Feroceon 88FR131 rev 1 (v5l) BogoMIPS : 795.44 Features : swp half thumb fastmult edsp CPU implementer : 0x56 CPU architecture: 5TE CPU variant : 0x2 CPU part : 0x131 CPU revision : 1 Hardware : Marvell OpenRD Ultimate Board Revision : 0000 Serial : 0000000000000000 lscpu Architecture: armv5tel Byte Order: Little Endian CPU(s): 1 On-line CPU(s) list: 0 uname -m armv5tel uname -a Linux home-pogoplug-v4-1 3.1.10-32-ARCH #1 PREEMPT Tue Feb 11 06:26:34 MST 2014 armv5tel GNU/Linux
And other questions: - If you access the mounted volume from a terminal window, do you get any error messages? Not quiet sure about the question. I enter a directory from the mount point, and got ls -l ls: reading directory .: Input/output error total 0 - Do you have the same problem when mounting over NFS? I failed to mount the file system by NFS, mount -vv -o mountproto=udp,vers=3 -t nfs 192.168.2.7:filecache1t /media/test1/ mount.nfs: timeout set for Wed Apr 9 23:49:56 2014 mount.nfs: trying text-based options 'mountproto=udp,vers=3,addr=192.168.2.7,mountaddr=192.168.2.7' mount.nfs: prog 100003, trying vers=3, prot=6 mount.nfs: trying 192.168.2.7 prog 100003 vers 3 prot TCP port 2049 mount.nfs: prog 100005, trying vers=3, prot=17 mount.nfs: portmap query failed: RPC: Program not registered mount.nfs: requested NFS version or transport protocol is not supported - Are the missing files available on any of the raw bricks? Yes, all of the files are exist in the raw bricks. I mounted in the server, use the same configure file(eg. /etc/glusterfs/datastore.vol), and I can access all of the files too.
GlusterFS 3.7.0 has been released (http://www.gluster.org/pipermail/gluster-users/2015-May/021901.html), and the Gluster project maintains N-2 supported releases. The last two releases before 3.7 are still maintained, at the moment these are 3.6 and 3.5. This bug has been filed against the 3,4 release, and will not get fixed in a 3.4 version any more. Please verify if newer versions are affected with the reported problem. If that is the case, update the bug with a note, and update the version if you can. In case updating the version is not possible, leave a comment in this bug report with the version you tested, and set the "Need additional information the selected bugs from" below the comment box to "bugs". If there is no response by the end of the month, this bug will get automatically closed.
GlusterFS 3.4.x has reached end-of-life.\ \ If this bug still exists in a later release please reopen this and change the version or open a new bug.