Description of problem: If we expand a new brick to a WORM volume and create some new files, the files located in the new brick can be renamed. Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. create a dht volume and enable WORM feature 2. add one brick to the volume 3. fix layout of the volume 4. create some files in the client directory 5. rename files which are located in the new added brick Actual results: Mv operation succeeds without any error reported. But the new name and the original name both exists. Expected results: Mv operation is not allowed. Additional info:
Can you provide brick and client logs
This is what I did: ----------- $ command line; # comments I added bdfs-node1 is the hostname Two bricks were used. One used the system storage mounted on "/". The other used /dev/sdc1 mounted on "/data/br1". ----------- $ gluster v create worm-test bdfs-node1:/data/br0 force $ gluster v set worm-test worm on $ gluster v start worm-test $ mkdir client $ mount -t glusterfs bdfs-node1:worm-test ./client/ $ export LANG=en_US $ cd client/ $ for i in `seq 0 9`; do dd if=/dev/zero of=./dummy_$i bs=1M count=1; done $ gluster v add-brick worm-test bdfs-node1:/data/br1/br1 force $ gluster v rebalance worm-test fix-layout start $ gluster v rebalance worm-test start force $ gluster v rebalance worm-test status # To make sure the progress completed. $ for i in `seq 10 99`; do dd if=/dev/zero of=./dummy_$i bs=1M count=1; done $ ls /data/br1/br1/ root@bdfs-node1:~# ls /data/br1/br1/ -l total 5124 ---------T 2 root root 0 Nov 29 12:09 dummy_5 -rw-r--r-- 3 root root 1048576 Nov 29 12:15 dummy_61 -rw-r--r-- 2 root root 1048576 Nov 29 12:15 dummy_64 -rw-r--r-- 2 root root 1048576 Nov 29 12:15 dummy_65 -rw-r--r-- 2 root root 1048576 Nov 29 12:15 dummy_68 $ mv dummy_5 dummy_bk mv: cannot move 'dummy_5' to 'dummy_bk': Read-only file system # This is OK. $ mv dummy_61 dummy_bk # Operation succeeded without any ERROR reported. This should not be permitted. $ ls -l dummy_61 dummy_bk -rw-r--r-- 2 root root 1048576 Nov 29 12:15 dummy_61 -rw-r--r-- 2 root root 1048576 Nov 29 12:15 dummy_bk # ls /data/br1/br1/ -l total 5124 ---------T 2 root root 0 Nov 29 12:09 dummy_5 -rw-r--r-- 3 root root 1048576 Nov 29 12:15 dummy_61 -rw-r--r-- 2 root root 1048576 Nov 29 12:15 dummy_64 -rw-r--r-- 2 root root 1048576 Nov 29 12:15 dummy_65 -rw-r--r-- 2 root root 1048576 Nov 29 12:15 dummy_68 -rw-r--r-- 3 root root 1048576 Nov 29 12:15 dummy_bk # Both files exist.
Client logs: cat /var/log/glusterfs/.-client-.log [2015-11-29 04:04:48.792173] I [MSGID: 100030] [glusterfsd.c:2318:main] 0-/usr/local/sbin/glusterfs: Started running /usr/local/sbin/glusterfs version 3.7.6 (args: /usr/local/sbin/glusterfs --volfile-server=bdfs-node1 --volfile-id=worm-test ./client/) [2015-11-29 04:04:48.806833] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2015-11-29 04:04:48.813542] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2015-11-29 04:04:48.814130] I [MSGID: 114020] [client.c:2118:notify] 0-worm-test-client-0: parent translators are ready, attempting connect on transport Final graph: +------------------------------------------------------------------------------+ 1: volume worm-test-client-0 2: type protocol/client 3: option ping-timeout 42 4: option remote-host bdfs-node1 5: option remote-subvolume /data/br0 6: option transport-type socket 7: option username f5c71cc1-e15e-450d-8a0f-9bb0b65bf9c2 8: option password 6df9bdfe-b597-4cbb-a640-34e8915f1f66 9: option send-gids true 10: end-volume 11: 12: volume worm-test-dht 13: type cluster/distribute 14: subvolumes worm-test-client-0 15: end-volume 16: 17: volume worm-test-write-behind 18: type performance/write-behind 19: subvolumes worm-test-dht 20: end-volume 21: 22: volume worm-test-read-ahead 23: type performance/read-ahead 24: subvolumes worm-test-write-behind 25: end-volume 26: 27: volume worm-test-readdir-ahead 28: type performance/readdir-ahead 29: subvolumes worm-test-read-ahead 30: end-volume 31: 32: volume worm-test-io-cache 33: type performance/io-cache 34: subvolumes worm-test-readdir-ahead 35: end-volume 36: 37: volume worm-test-quick-read 38: type performance/quick-read 39: subvolumes worm-test-io-cache 40: end-volume 41: 42: volume worm-test-open-behind 43: type performance/open-behind 44: subvolumes worm-test-quick-read 45: end-volume 46: 47: volume worm-test-md-cache 48: type performance/md-cache 49: subvolumes worm-test-open-behind 50: end-volume 51: 52: volume worm-test 53: type debug/io-stats 54: option latency-measurement off 55: option count-fop-hits off 56: subvolumes worm-test-md-cache 57: end-volume 58: 59: volume meta-autoload 60: type meta 61: subvolumes worm-test 62: end-volume 63: +------------------------------------------------------------------------------+ [2015-11-29 04:04:48.815578] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-worm-test-client-0: changing port to 49153 (from 0) [2015-11-29 04:04:48.816906] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-worm-test-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-11-29 04:04:48.822359] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-worm-test-client-0: Connected to worm-test-client-0, attached to remote volume '/data/br0'. [2015-11-29 04:04:48.822437] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-worm-test-client-0: Server and Client lk-version numbers are not same, reopening the fds [2015-11-29 04:04:48.830526] I [fuse-bridge.c:5137:fuse_graph_setup] 0-fuse: switched to graph 0 [2015-11-29 04:04:48.830702] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-worm-test-client-0: Server lk version = 1 [2015-11-29 04:04:48.830863] I [fuse-bridge.c:4030:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22 kernel 7.22 [2015-11-29 04:04:48.832030] I [MSGID: 109063] [dht-layout.c:702:dht_layout_normalize] 0-worm-test-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0 [2015-11-29 04:04:48.832982] I [MSGID: 109036] [dht-common.c:7869:dht_log_new_layout_for_dir_selfheal] 0-worm-test-dht: Setting layout of / with [Subvol_name: worm-test-client-0, Err: -1 , Start: 0 , Stop: 4294967295 , Hash: 1 ], [2015-11-29 04:07:05.064172] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed [2015-11-29 04:07:05.074662] I [MSGID: 114020] [client.c:2118:notify] 2-worm-test-client-0: parent translators are ready, attempting connect on transport [2015-11-29 04:07:05.076871] I [MSGID: 114020] [client.c:2118:notify] 2-worm-test-client-1: parent translators are ready, attempting connect on transport [2015-11-29 04:07:05.077366] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 2-worm-test-client-0: changing port to 49153 (from 0) Final graph: +------------------------------------------------------------------------------+ 1: volume worm-test-client-0 2: type protocol/client 3: option ping-timeout 42 4: option remote-host bdfs-node1 5: option remote-subvolume /data/br0 6: option transport-type socket 7: option username f5c71cc1-e15e-450d-8a0f-9bb0b65bf9c2 8: option password 6df9bdfe-b597-4cbb-a640-34e8915f1f66 9: option send-gids true 10: end-volume 11: 12: volume worm-test-client-1 13: type protocol/client 14: option ping-timeout 42 15: option remote-host bdfs-node1 16: option remote-subvolume /data/br1/br1 17: option transport-type socket 18: option username f5c71cc1-e15e-450d-8a0f-9bb0b65bf9c2 19: option password 6df9bdfe-b597-4cbb-a640-34e8915f1f66 20: option send-gids true 21: end-volume 22: 23: volume worm-test-dht 24: type cluster/distribute 25: subvolumes worm-test-client-0 worm-test-client-1 26: end-volume 27: 28: volume worm-test-write-behind 29: type performance/write-behind 30: subvolumes worm-test-dht 31: end-volume 32: 33: volume worm-test-read-ahead 34: type performance/read-ahead 35: subvolumes worm-test-write-behind 36: end-volume 37: 38: volume worm-test-readdir-ahead 39: type performance/readdir-ahead 40: subvolumes worm-test-read-ahead 41: end-volume 42: 43: volume worm-test-io-cache 44: type performance/io-cache 45: subvolumes worm-test-readdir-ahead 46: end-volume 47: 48: volume worm-test-quick-read 49: type performance/quick-read 50: subvolumes worm-test-io-cache 51: end-volume 52: 53: volume worm-test-open-behind 54: type performance/open-behind 55: subvolumes worm-test-quick-read 56: end-volume 57: 58: volume worm-test-md-cache 59: type performance/md-cache 60: subvolumes worm-test-open-behind 61: end-volume 62: 63: volume worm-test 64: type debug/io-stats 65: option latency-measurement off 66: option count-fop-hits off 67: subvolumes worm-test-md-cache 68: end-volume 69: 70: volume meta-autoload 71: type meta 72: subvolumes worm-test 73: end-volume 74: +------------------------------------------------------------------------------+ [2015-11-29 04:07:05.078032] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 2-worm-test-client-1: changing port to 49154 (from 0) [2015-11-29 04:07:05.078376] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 2-worm-test-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-11-29 04:07:05.078525] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 2-worm-test-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2015-11-29 04:07:05.078670] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 2-worm-test-client-0: Connected to worm-test-client-0, attached to remote volume '/data/br0'. [2015-11-29 04:07:05.078695] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 2-worm-test-client-0: Server and Client lk-version numbers are not same, reopening the fds [2015-11-29 04:07:05.078842] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 2-worm-test-client-0: Server lk version = 1 [2015-11-29 04:07:05.081482] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 2-worm-test-client-1: Connected to worm-test-client-1, attached to remote volume '/data/br1/br1'. [2015-11-29 04:07:05.081549] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 2-worm-test-client-1: Server and Client lk-version numbers are not same, reopening the fds [2015-11-29 04:07:05.090749] I [fuse-bridge.c:5137:fuse_graph_setup] 0-fuse: switched to graph 2 [2015-11-29 04:07:05.090884] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 2-worm-test-client-1: Server lk version = 1 [2015-11-29 04:07:05.221094] I [MSGID: 114021] [client.c:2127:notify] 0-worm-test-client-0: current graph is no longer active, destroying rpc_client [2015-11-29 04:07:05.221224] I [MSGID: 114018] [client.c:2042:client_rpc_notify] 0-worm-test-client-0: disconnected from worm-test-client-0. Client process will keep trying to connect to glusterd until brick's port is available [2015-11-29 04:07:39.845022] I [MSGID: 109064] [dht-layout.c:808:dht_layout_dir_mismatch] 2-worm-test-dht: subvol: worm-test-client-0; inode layout - 0 - 4294967295 - 1; disk layout - 216892300 - 4294967295 - 1 [2015-11-29 04:07:39.845095] I [MSGID: 109018] [dht-common.c:811:dht_revalidate_cbk] 2-worm-test-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001 [2015-11-29 04:07:39.845128] I [MSGID: 109064] [dht-layout.c:808:dht_layout_dir_mismatch] 2-worm-test-dht: subvol: worm-test-client-1; inode layout - 0 - 0 - 0; disk layout - 0 - 216892299 - 1 [2015-11-29 04:07:39.845164] I [MSGID: 109018] [dht-common.c:811:dht_revalidate_cbk] 2-worm-test-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001 [2015-11-29 04:15:08.834496] I [MSGID: 109066] [dht-rename.c:1411:dht_rename] 2-worm-test-dht: renaming /dummy_0 (hash=worm-test-client-0/cache=worm-test-client-0) => /a.out (hash=worm-test-client-0/cache=<nul>) [2015-11-29 04:15:08.835599] W [MSGID: 114031] [client-rpc-fops.c:2738:client3_3_rename_cbk] 2-worm-test-client-0: remote operation failed [Read-only file system] [2015-11-29 04:15:08.835642] W [MSGID: 109030] [dht-rename.c:715:dht_rename_cbk] 2-worm-test-dht: /dummy_0: Rename on worm-test-client-0 failed, (gfid = 54ba3d42-3db9-4cb5-91b7-aa104de11593) [Read-only file system] [2015-11-29 04:15:08.835964] W [fuse-bridge.c:1777:fuse_rename_cbk] 0-glusterfs-fuse: 2794: /dummy_0 -> /a.out => -1 (Read-only file system) [2015-11-29 04:15:24.536094] I [MSGID: 109066] [dht-rename.c:1411:dht_rename] 2-worm-test-dht: renaming /dummy_5 (hash=worm-test-client-1/cache=worm-test-client-0) => /dummy_bk (hash=worm-test-client-0/cache=<nul>) [2015-11-29 04:15:24.537169] W [MSGID: 109030] [dht-rename.c:715:dht_rename_cbk] 2-worm-test-dht: /dummy_5: Rename on worm-test-client-0 failed, (gfid = 443047b0-83cb-40ef-ad8c-c8120de0043a) [Read-only file system] [2015-11-29 04:15:24.537419] W [fuse-bridge.c:1777:fuse_rename_cbk] 0-glusterfs-fuse: 2919: /dummy_5 -> /dummy_bk => -1 (Read-only file system) [2015-11-29 04:16:08.247116] I [MSGID: 109066] [dht-rename.c:1411:dht_rename] 2-worm-test-dht: renaming /dummy_61 (hash=worm-test-client-1/cache=worm-test-client-1) => /dummy_bk (hash=worm-test-client-0/cache=<nul>) The message "W [MSGID: 114031] [client-rpc-fops.c:2738:client3_3_rename_cbk] 2-worm-test-client-0: remote operation failed [Read-only file system]" repeated 2 times between [2015-11-29 04:15:08.835599] and [2015-11-29 04:16:08.248904] [2015-11-29 04:16:08.248913] I [MSGID: 109030] [dht-rename.c:729:dht_rename_cbk] 2-worm-test-dht: /dummy_61: Rename (linkto file) on worm-test-client-0 failed, (gfid = bed7d688-a8ef-4cf8-86a3-d9a133ecca15) [Read-only file system] [2015-11-29 04:16:08.249222] W [MSGID: 114031] [client-rpc-fops.c:664:client3_3_unlink_cbk] 2-worm-test-client-1: remote operation failed [Read-only file system] [2015-11-29 04:16:08.249252] W [MSGID: 109034] [dht-rename.c:510:dht_rename_unlink_cbk] 2-worm-test-dht: /dummy_61: Rename: unlink on worm-test-client-1 failed [Read-only file system]
brick /data/br0 log: cat /var/log/glusterfs/bricks/data-br0.log [2015-11-29 04:04:08.021460] I [MSGID: 100030] [glusterfsd.c:2318:main] 0-/usr/local/sbin/glusterfsd: Started running /usr/local/sbin/glusterfsd version 3.7.6 (args: /usr/local/sbin/glusterfsd -s bdfs-node1 --volfile-id worm-test.bdfs-node1.data-br0 -p /var/lib/glusterd/vols/worm-test/run/bdfs-node1-data-br0.pid -S /var/run/gluster/c75a848207b61e8e7a7a7f92accda3ca.socket --brick-name /data/br0 -l /var/log/glusterfs/bricks/data-br0.log --xlator-option *-posix.glusterd-uuid=615f2a8a-d501-4ee2-a2b6-4113933e472a --brick-port 49153 --xlator-option worm-test-server.listen-port=49153) [2015-11-29 04:04:08.026274] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2015-11-29 04:04:08.033794] I [graph.c:269:gf_add_cmdline_options] 0-worm-test-server: adding option 'listen-port' for volume 'worm-test-server' with value '49153' [2015-11-29 04:04:08.033853] I [graph.c:269:gf_add_cmdline_options] 0-worm-test-posix: adding option 'glusterd-uuid' for volume 'worm-test-posix' with value '615f2a8a-d501-4ee2-a2b6-4113933e472a' [2015-11-29 04:04:08.034205] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/data/br0: skip format check for non-addr auth option auth.login./data/br0.allow [2015-11-29 04:04:08.034229] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/data/br0: skip format check for non-addr auth option auth.login.f5c71cc1-e15e-450d-8a0f-9bb0b65bf9c2.password [2015-11-29 04:04:08.034388] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2015-11-29 04:04:08.035748] I [rpcsvc.c:2215:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64 [2015-11-29 04:04:08.035894] W [MSGID: 101002] [options.c:957:xl_opt_validate] 0-worm-test-server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction [2015-11-29 04:04:08.037941] I [MSGID: 121050] [ctr-helper.c:256:extract_ctr_options] 0-gfdbdatastore: CTR Xlator is disabled. [2015-11-29 04:04:08.037983] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-worm-test-changetimerecorder: Failed to retrieve sql-db-pagesize from params.Assigning default value: 4096 [2015-11-29 04:04:08.038007] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-worm-test-changetimerecorder: Failed to retrieve sql-db-cachesize from params.Assigning default value: 1000 [2015-11-29 04:04:08.038029] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-worm-test-changetimerecorder: Failed to retrieve sql-db-journalmode from params.Assigning default value: wal [2015-11-29 04:04:08.038050] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-worm-test-changetimerecorder: Failed to retrieve sql-db-wal-autocheckpoint from params.Assigning default value: 1000 [2015-11-29 04:04:08.038071] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-worm-test-changetimerecorder: Failed to retrieve sql-db-sync from params.Assigning default value: normal [2015-11-29 04:04:08.038091] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-worm-test-changetimerecorder: Failed to retrieve sql-db-autovacuum from params.Assigning default value: none [2015-11-29 04:04:08.045369] I [trash.c:2366:init] 0-worm-test-trash: no option specified for 'eliminate', using NULL [2015-11-29 04:04:08.046135] W [graph.c:357:_log_if_unknown_option] 0-worm-test-server: option 'rpc-auth.auth-glusterfs' is not recognized [2015-11-29 04:04:08.046192] W [graph.c:357:_log_if_unknown_option] 0-worm-test-server: option 'rpc-auth.auth-unix' is not recognized [2015-11-29 04:04:08.046239] W [graph.c:357:_log_if_unknown_option] 0-worm-test-server: option 'rpc-auth.auth-null' is not recognized [2015-11-29 04:04:08.046316] W [graph.c:357:_log_if_unknown_option] 0-worm-test-quota: option 'timeout' is not recognized [2015-11-29 04:04:08.046343] W [graph.c:357:_log_if_unknown_option] 0-worm-test-marker: option 'quota-version' is not recognized [2015-11-29 04:04:08.046409] W [graph.c:357:_log_if_unknown_option] 0-worm-test-trash: option 'brick-path' is not recognized Final graph: +------------------------------------------------------------------------------+ 1: volume worm-test-posix 2: type storage/posix 3: option glusterd-uuid 615f2a8a-d501-4ee2-a2b6-4113933e472a 4: option directory /data/br0 5: option volume-id efd32766-94b2-4284-ae65-acdaf4b899b9 6: end-volume 7: 8: volume worm-test-trash 9: type features/trash 10: option trash-dir .trashcan 11: option brick-path /data/br0 12: option trash-internal-op off 13: subvolumes worm-test-posix 14: end-volume 15: 16: volume worm-test-changetimerecorder 17: type features/changetimerecorder 18: option db-type sqlite3 19: option hot-brick off 20: option db-name br0.db 21: option db-path /data/br0/.glusterfs/ 22: option record-exit off 23: option ctr_link_consistency off 24: option ctr_hardlink_heal_expire_period 300 25: option ctr_inode_heal_expire_period 300 26: option record-entry on 27: option ctr-enabled off 28: option record-counters off 29: subvolumes worm-test-trash 30: end-volume 31: 32: volume worm-test-changelog 33: type features/changelog 34: option changelog-brick /data/br0 35: option changelog-dir /data/br0/.glusterfs/changelogs 36: option changelog-barrier-timeout 120 37: subvolumes worm-test-changetimerecorder 38: end-volume 39: 40: volume worm-test-bitrot-stub 41: type features/bitrot-stub 42: option export /data/br0 43: subvolumes worm-test-changelog 44: end-volume 45: 46: volume worm-test-access-control 47: type features/access-control 48: subvolumes worm-test-bitrot-stub 49: end-volume 50: 51: volume worm-test-locks 52: type features/locks 53: subvolumes worm-test-access-control 54: end-volume 55: 56: volume worm-test-upcall 57: type features/upcall 58: option cache-invalidation off 59: subvolumes worm-test-locks 60: end-volume 61: 62: volume worm-test-io-threads 63: type performance/io-threads 64: subvolumes worm-test-upcall 65: end-volume 66: 67: volume worm-test-marker 68: type features/marker 69: option volume-uuid efd32766-94b2-4284-ae65-acdaf4b899b9 70: option timestamp-file /var/lib/glusterd/vols/worm-test/marker.tstamp 71: option quota-version 0 72: option xtime off 73: option gsync-force-xtime off 74: option quota off 75: option inode-quota off 76: subvolumes worm-test-io-threads 77: end-volume 78: 79: volume worm-test-barrier 80: type features/barrier 81: option barrier disable 82: option barrier-timeout 120 83: subvolumes worm-test-marker 84: end-volume 85: 86: volume worm-test-index 87: type features/index 88: option index-base /data/br0/.glusterfs/indices 89: subvolumes worm-test-barrier 90: end-volume 91: 92: volume worm-test-quota 93: type features/quota 94: option volume-uuid worm-test 95: option server-quota off 96: option timeout 0 97: option deem-statfs off 98: subvolumes worm-test-index 99: end-volume 100: 101: volume worm-test-worm 102: type features/worm 103: option worm on 104: subvolumes worm-test-quota 105: end-volume 106: 107: volume worm-test-read-only 108: type features/read-only 109: option read-only off 110: subvolumes worm-test-worm 111: end-volume 112: 113: volume /data/br0 114: type debug/io-stats 115: option latency-measurement off 116: option count-fop-hits off 117: subvolumes worm-test-read-only 118: end-volume 119: 120: volume worm-test-server 121: type protocol/server 122: option transport.socket.listen-port 49153 123: option rpc-auth.auth-glusterfs on 124: option rpc-auth.auth-unix on 125: option rpc-auth.auth-null on 126: option rpc-auth-allow-insecure on 127: option transport-type tcp 128: option auth.login./data/br0.allow f5c71cc1-e15e-450d-8a0f-9bb0b65bf9c2 129: option auth.login.f5c71cc1-e15e-450d-8a0f-9bb0b65bf9c2.password 6df9bdfe-b597-4cbb-a640-34e8915f1f66 130: option auth.addr./data/br0.allow * 131: subvolumes /data/br0 132: end-volume 133: +------------------------------------------------------------------------------+ [2015-11-29 04:04:48.817212] I [login.c:81:gf_auth] 0-auth/login: allowed user names: f5c71cc1-e15e-450d-8a0f-9bb0b65bf9c2 [2015-11-29 04:04:48.817277] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-worm-test-server: accepted client from bdfs-node1-13919-2015/11/29-04:04:48:784141-worm-test-client-0-0-0 (version: 3.7.6) [2015-11-29 04:07:05.064299] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed [2015-11-29 04:07:05.069809] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2015-11-29 04:07:05.078577] I [login.c:81:gf_auth] 0-auth/login: allowed user names: f5c71cc1-e15e-450d-8a0f-9bb0b65bf9c2 [2015-11-29 04:07:05.078599] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-worm-test-server: accepted client from bdfs-node1-13919-2015/11/29-04:04:48:784141-worm-test-client-0-2-0 (version: 3.7.6) [2015-11-29 04:07:05.221231] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-worm-test-server: disconnecting connection from bdfs-node1-13919-2015/11/29-04:04:48:784141-worm-test-client-0-0-0 [2015-11-29 04:07:05.221301] I [MSGID: 101055] [client_t.c:419:gf_client_unref] 0-worm-test-server: Shutting down connection bdfs-node1-13919-2015/11/29-04:04:48:784141-worm-test-client-0-0-0 [2015-11-29 04:07:39.569252] I [login.c:81:gf_auth] 0-auth/login: allowed user names: f5c71cc1-e15e-450d-8a0f-9bb0b65bf9c2 [2015-11-29 04:07:39.569318] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-worm-test-server: accepted client from bdfs-node1-14545-2015/11/29-04:07:34:551062-worm-test-client-0-0-0 (version: 3.7.6) [2015-11-29 04:07:39.596869] E [MSGID: 115070] [server-rpc-fops.c:1555:server_open_cbk] 0-worm-test-server: 43: OPEN /dummy_5 (443047b0-83cb-40ef-ad8c-c8120de0043a) ==> (Read-only file system) [Read-only file system] [2015-11-29 04:07:39.620850] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-worm-test-server: disconnecting connection from bdfs-node1-14545-2015/11/29-04:07:34:551062-worm-test-client-0-0-0 [2015-11-29 04:07:39.620933] I [MSGID: 101055] [client_t.c:419:gf_client_unref] 0-worm-test-server: Shutting down connection bdfs-node1-14545-2015/11/29-04:07:34:551062-worm-test-client-0-0-0 [2015-11-29 04:09:12.329897] I [login.c:81:gf_auth] 0-auth/login: allowed user names: f5c71cc1-e15e-450d-8a0f-9bb0b65bf9c2 [2015-11-29 04:09:12.329957] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-worm-test-server: accepted client from bdfs-node1-14832-2015/11/29-04:09:07:314727-worm-test-client-0-0-0 (version: 3.7.6) [2015-11-29 04:09:12.362482] E [MSGID: 115070] [server-rpc-fops.c:1555:server_open_cbk] 0-worm-test-server: 55: OPEN /dummy_5 (443047b0-83cb-40ef-ad8c-c8120de0043a) ==> (Read-only file system) [Read-only file system] [2015-11-29 04:09:12.390323] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-worm-test-server: disconnecting connection from bdfs-node1-14832-2015/11/29-04:09:07:314727-worm-test-client-0-0-0 [2015-11-29 04:09:12.390403] I [MSGID: 101055] [client_t.c:419:gf_client_unref] 0-worm-test-server: Shutting down connection bdfs-node1-14832-2015/11/29-04:09:07:314727-worm-test-client-0-0-0 [2015-11-29 04:13:01.109471] I [login.c:81:gf_auth] 0-auth/login: allowed user names: f5c71cc1-e15e-450d-8a0f-9bb0b65bf9c2 [2015-11-29 04:13:01.109519] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-worm-test-server: accepted client from bdfs-node1-15539-2015/11/29-04:12:56:94646-worm-test-client-0-0-0 (version: 3.7.6) [2015-11-29 04:13:01.139488] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-worm-test-server: disconnecting connection from bdfs-node1-15539-2015/11/29-04:12:56:94646-worm-test-client-0-0-0 [2015-11-29 04:13:01.139567] I [MSGID: 101055] [client_t.c:419:gf_client_unref] 0-worm-test-server: Shutting down connection bdfs-node1-15539-2015/11/29-04:12:56:94646-worm-test-client-0-0-0 [2015-11-29 04:15:08.835453] I [MSGID: 115061] [server-rpc-fops.c:1024:server_rename_cbk] 0-worm-test-server: 2057: RENAME /dummy_0 (00000000-0000-0000-0000-000000000000/dummy_0) -> /a.out (00000000-0000-0000-0000-000000000000/a.out) ==> (Read-only file system) [Read-only file system] [2015-11-29 04:15:24.537004] I [MSGID: 115061] [server-rpc-fops.c:1024:server_rename_cbk] 0-worm-test-server: 2196: RENAME /dummy_5 (00000000-0000-0000-0000-000000000000/dummy_5) -> /dummy_bk (00000000-0000-0000-0000-000000000000/dummy_bk) ==> (Read-only file system) [Read-only file system] [2015-11-29 04:16:08.248810] I [MSGID: 115061] [server-rpc-fops.c:1024:server_rename_cbk] 0-worm-test-server: 3687: RENAME /dummy_61 (00000000-0000-0000-0000-000000000000/dummy_61) -> /dummy_bk (00000000-0000-0000-0000-000000000000/dummy_bk) ==> (Read-only file system) [Read-only file system]
brick1 /data/br1/br1 logs: $ cat /var/log/glusterfs/bricks/data-br1-br1.log [2015-11-29 04:07:04.894778] I [MSGID: 100030] [glusterfsd.c:2318:main] 0-/usr/local/sbin/glusterfsd: Started running /usr/local/sbin/glusterfsd version 3.7.6 (args: /usr/local/sbin/glusterfsd -s bdfs-node1 --volfile-id worm-test.bdfs-node1.data-br1-br1 -p /var/lib/glusterd/vols/worm-test/run/bdfs-node1-data-br1-br1.pid -S /var/run/gluster/e918773766a09570089866c58002ef52.socket --brick-name /data/br1/br1 -l /var/log/glusterfs/bricks/data-br1-br1.log --xlator-option *-posix.glusterd-uuid=615f2a8a-d501-4ee2-a2b6-4113933e472a --brick-port 49154 --xlator-option worm-test-server.listen-port=49154) [2015-11-29 04:07:04.903836] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2015-11-29 04:07:04.911202] I [graph.c:269:gf_add_cmdline_options] 0-worm-test-server: adding option 'listen-port' for volume 'worm-test-server' with value '49154' [2015-11-29 04:07:04.911260] I [graph.c:269:gf_add_cmdline_options] 0-worm-test-posix: adding option 'glusterd-uuid' for volume 'worm-test-posix' with value '615f2a8a-d501-4ee2-a2b6-4113933e472a' [2015-11-29 04:07:04.911607] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/data/br1/br1: skip format check for non-addr auth option auth.login./data/br1/br1.allow [2015-11-29 04:07:04.911631] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/data/br1/br1: skip format check for non-addr auth option auth.login.f5c71cc1-e15e-450d-8a0f-9bb0b65bf9c2.password [2015-11-29 04:07:04.911826] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2015-11-29 04:07:04.913111] I [rpcsvc.c:2215:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64 [2015-11-29 04:07:04.913274] W [MSGID: 101002] [options.c:957:xl_opt_validate] 0-worm-test-server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction [2015-11-29 04:07:04.915284] I [MSGID: 121050] [ctr-helper.c:256:extract_ctr_options] 0-gfdbdatastore: CTR Xlator is disabled. [2015-11-29 04:07:04.915322] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-worm-test-changetimerecorder: Failed to retrieve sql-db-pagesize from params.Assigning default value: 4096 [2015-11-29 04:07:04.915345] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-worm-test-changetimerecorder: Failed to retrieve sql-db-cachesize from params.Assigning default value: 1000 [2015-11-29 04:07:04.915366] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-worm-test-changetimerecorder: Failed to retrieve sql-db-journalmode from params.Assigning default value: wal [2015-11-29 04:07:04.915388] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-worm-test-changetimerecorder: Failed to retrieve sql-db-wal-autocheckpoint from params.Assigning default value: 1000 [2015-11-29 04:07:04.915409] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-worm-test-changetimerecorder: Failed to retrieve sql-db-sync from params.Assigning default value: normal [2015-11-29 04:07:04.915428] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-worm-test-changetimerecorder: Failed to retrieve sql-db-autovacuum from params.Assigning default value: none [2015-11-29 04:07:05.020367] I [trash.c:2366:init] 0-worm-test-trash: no option specified for 'eliminate', using NULL [2015-11-29 04:07:05.021300] W [graph.c:357:_log_if_unknown_option] 0-worm-test-server: option 'rpc-auth.auth-glusterfs' is not recognized [2015-11-29 04:07:05.021369] W [graph.c:357:_log_if_unknown_option] 0-worm-test-server: option 'rpc-auth.auth-unix' is not recognized [2015-11-29 04:07:05.021417] W [graph.c:357:_log_if_unknown_option] 0-worm-test-server: option 'rpc-auth.auth-null' is not recognized [2015-11-29 04:07:05.021493] W [graph.c:357:_log_if_unknown_option] 0-worm-test-quota: option 'timeout' is not recognized [2015-11-29 04:07:05.021518] W [graph.c:357:_log_if_unknown_option] 0-worm-test-marker: option 'quota-version' is not recognized [2015-11-29 04:07:05.021583] W [graph.c:357:_log_if_unknown_option] 0-worm-test-trash: option 'brick-path' is not recognized Final graph: +------------------------------------------------------------------------------+ 1: volume worm-test-posix 2: type storage/posix 3: option glusterd-uuid 615f2a8a-d501-4ee2-a2b6-4113933e472a 4: option directory /data/br1/br1 5: option volume-id efd32766-94b2-4284-ae65-acdaf4b899b9 6: end-volume 7: 8: volume worm-test-trash 9: type features/trash 10: option trash-dir .trashcan 11: option brick-path /data/br1/br1 12: option trash-internal-op off 13: subvolumes worm-test-posix 14: end-volume 15: 16: volume worm-test-changetimerecorder 17: type features/changetimerecorder 18: option db-type sqlite3 19: option hot-brick off 20: option db-name br1.db 21: option db-path /data/br1/br1/.glusterfs/ 22: option record-exit off 23: option ctr_link_consistency off 24: option ctr_hardlink_heal_expire_period 300 25: option ctr_inode_heal_expire_period 300 26: option record-entry on 27: option ctr-enabled off 28: option record-counters off 29: subvolumes worm-test-trash 30: end-volume 31: 32: volume worm-test-changelog 33: type features/changelog 34: option changelog-brick /data/br1/br1 35: option changelog-dir /data/br1/br1/.glusterfs/changelogs 36: option changelog-barrier-timeout 120 37: subvolumes worm-test-changetimerecorder 38: end-volume 39: 40: volume worm-test-bitrot-stub 41: type features/bitrot-stub 42: option export /data/br1/br1 43: subvolumes worm-test-changelog 44: end-volume 45: 46: volume worm-test-access-control 47: type features/access-control 48: subvolumes worm-test-bitrot-stub 49: end-volume 50: 51: volume worm-test-locks 52: type features/locks 53: subvolumes worm-test-access-control 54: end-volume 55: 56: volume worm-test-upcall 57: type features/upcall 58: option cache-invalidation off 59: subvolumes worm-test-locks 60: end-volume 61: 62: volume worm-test-io-threads 63: type performance/io-threads 64: subvolumes worm-test-upcall 65: end-volume 66: 67: volume worm-test-marker 68: type features/marker 69: option volume-uuid efd32766-94b2-4284-ae65-acdaf4b899b9 70: option timestamp-file /var/lib/glusterd/vols/worm-test/marker.tstamp 71: option quota-version 0 72: option xtime off 73: option gsync-force-xtime off 74: option quota off 75: option inode-quota off 76: subvolumes worm-test-io-threads 77: end-volume 78: 79: volume worm-test-barrier 80: type features/barrier 81: option barrier disable 82: option barrier-timeout 120 83: subvolumes worm-test-marker 84: end-volume 85: 86: volume worm-test-index 87: type features/index 88: option index-base /data/br1/br1/.glusterfs/indices 89: subvolumes worm-test-barrier 90: end-volume 91: 92: volume worm-test-quota 93: type features/quota 94: option volume-uuid worm-test 95: option server-quota off 96: option timeout 0 97: option deem-statfs off 98: subvolumes worm-test-index 99: end-volume 100: 101: volume worm-test-worm 102: type features/worm 103: option worm on 104: subvolumes worm-test-quota 105: end-volume 106: 107: volume worm-test-read-only 108: type features/read-only 109: option read-only off 110: subvolumes worm-test-worm 111: end-volume 112: 113: volume /data/br1/br1 114: type debug/io-stats 115: option latency-measurement off 116: option count-fop-hits off 117: subvolumes worm-test-read-only 118: end-volume 119: 120: volume worm-test-server 121: type protocol/server 122: option transport.socket.listen-port 49154 123: option rpc-auth.auth-glusterfs on 124: option rpc-auth.auth-unix on 125: option rpc-auth.auth-null on 126: option rpc-auth-allow-insecure on 127: option transport-type tcp 128: option auth.login./data/br1/br1.allow f5c71cc1-e15e-450d-8a0f-9bb0b65bf9c2 129: option auth.login.f5c71cc1-e15e-450d-8a0f-9bb0b65bf9c2.password 6df9bdfe-b597-4cbb-a640-34e8915f1f66 130: option auth.addr./data/br1/br1.allow * 131: subvolumes /data/br1/br1 132: end-volume 133: +------------------------------------------------------------------------------+ [2015-11-29 04:07:05.064317] I [glusterfsd-mgmt.c:57:mgmt_cbk_spec] 0-mgmt: Volume file changed [2015-11-29 04:07:05.069588] I [glusterfsd-mgmt.c:1513:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing [2015-11-29 04:07:05.078713] I [login.c:81:gf_auth] 0-auth/login: allowed user names: f5c71cc1-e15e-450d-8a0f-9bb0b65bf9c2 [2015-11-29 04:07:05.078739] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-worm-test-server: accepted client from bdfs-node1-13919-2015/11/29-04:04:48:784141-worm-test-client-1-2-0 (version: 3.7.6) [2015-11-29 04:07:39.566024] I [login.c:81:gf_auth] 0-auth/login: allowed user names: f5c71cc1-e15e-450d-8a0f-9bb0b65bf9c2 [2015-11-29 04:07:39.566081] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-worm-test-server: accepted client from bdfs-node1-14545-2015/11/29-04:07:34:551062-worm-test-client-1-0-0 (version: 3.7.6) [2015-11-29 04:07:39.621012] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-worm-test-server: disconnecting connection from bdfs-node1-14545-2015/11/29-04:07:34:551062-worm-test-client-1-0-0 [2015-11-29 04:07:39.621081] I [MSGID: 101055] [client_t.c:419:gf_client_unref] 0-worm-test-server: Shutting down connection bdfs-node1-14545-2015/11/29-04:07:34:551062-worm-test-client-1-0-0 [2015-11-29 04:09:12.329992] I [login.c:81:gf_auth] 0-auth/login: allowed user names: f5c71cc1-e15e-450d-8a0f-9bb0b65bf9c2 [2015-11-29 04:09:12.330035] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-worm-test-server: accepted client from bdfs-node1-14832-2015/11/29-04:09:07:314727-worm-test-client-1-0-0 (version: 3.7.6) [2015-11-29 04:09:12.390453] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-worm-test-server: disconnecting connection from bdfs-node1-14832-2015/11/29-04:09:07:314727-worm-test-client-1-0-0 [2015-11-29 04:09:12.390525] I [MSGID: 101055] [client_t.c:419:gf_client_unref] 0-worm-test-server: Shutting down connection bdfs-node1-14832-2015/11/29-04:09:07:314727-worm-test-client-1-0-0 [2015-11-29 04:13:01.109668] I [login.c:81:gf_auth] 0-auth/login: allowed user names: f5c71cc1-e15e-450d-8a0f-9bb0b65bf9c2 [2015-11-29 04:13:01.109710] I [MSGID: 115029] [server-handshake.c:612:server_setvolume] 0-worm-test-server: accepted client from bdfs-node1-15539-2015/11/29-04:12:56:94646-worm-test-client-1-0-0 (version: 3.7.6) [2015-11-29 04:13:01.139875] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-worm-test-server: disconnecting connection from bdfs-node1-15539-2015/11/29-04:12:56:94646-worm-test-client-1-0-0 [2015-11-29 04:13:01.139926] I [MSGID: 101055] [client_t.c:419:gf_client_unref] 0-worm-test-server: Shutting down connection bdfs-node1-15539-2015/11/29-04:12:56:94646-worm-test-client-1-0-0 [2015-11-29 04:16:08.249134] E [MSGID: 115062] [server-rpc-fops.c:1097:server_unlink_cbk] 0-worm-test-server: 2430: UNLINK /dummy_61 (00000000-0000-0000-0000-000000000001/dummy_61) ==> (Read-only file system) [Read-only file system]
RCA: Here, a new brick is added and then, rebalance process is carried out in the volume. Now, the files present in the new brick may have different hash/cache. When rename is carried out(for the files present in new brick), it is combination of LINK(with new name) and UNLINK(original file). (Note, this is applicable only if it is a Special file). In current worm translator: 1. LINK operation is NOT implemented in worm, so it ends up with default LINK operation. so, LINK file will be created. 2. UNLINK will fail as it is implemented as ro_unlink. So, it ends up with both original file and new(renamed) file.
REVIEW: http://review.gluster.org/14020 (worm: Avoid link fop, if it is internal operation) posted (#1) for review on master by Saravanakumar Arumugam (sarumuga)
This update is done in bulk based on the state of the patch and the time since last activity. If the issue is still seen, please reopen the bug.