Hide Forgot
Description of problem: cat show different file content from different glusterfs client. Version-Release number of selected component (if applicable): gluster7 How reproducible: Steps to Reproduce: 1. on sn0 node, change time +30 days 2. touch a file in one volume 3. echo sth to that file 4. change time +30 day on sn0 again 5. restore time back (should -60 day), and check that the data is back to normal now 6. append sth to the file on sn0 7. log to sn1 append sth to the file from sn1 client 8. cat the file from sn0 and sn1 , the content is different on two nodes. Actual results: file content should are not the same Expected results: file content should be the same ///////////////////operation steps//////////////////////// Log on sn0 node(mn-0 On my env) [root@mn-0:/mnt/log/master/fsaudit] # date -s "+30 day" Thu Apr 9 08:59:47 EEST 2020 [root@mn-0:/mnt/log/master/fsaudit] # ls alarms auth.log internal-ssh.log [root@mn-0:/mnt/log/master/fsaudit] # touch testfile [root@mn-0:/mnt/log/master/fsaudit] # echo "from mn-0">testfile [root@mn-0:/mnt/log/master/fsaudit] # cat testfile from mn-0 [root@mn-0:/mnt/log/master/fsaudit] # stat testfile File: testfile Size: 10 Blocks: 1 IO Block: 131072 regular file Device: 2ah/42d Inode: 10674483048284813578 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2020-04-09 08:59:51.729843706 +0300 Modify: 2020-04-09 08:59:59.259739585 +0300 Change: 2020-04-09 08:59:59.259739585 +0300 Birth: - [root@mn-0:/mnt/log/master/fsaudit] # date -s "+30 day" Sat May 9 09:00:29 EEST 2020 [root@mn-0:/mnt/log/master/fsaudit] # date -s "-30 day" Thu Apr 9 09:00:34 EEST 2020 [root@mn-0:/mnt/log/master/fsaudit] # date -s "-30 day" Tue Mar 10 08:00:37 EET 2020 [root@mn-0:/mnt/log/master/fsaudit] # date Tue Mar 10 08:00:38 EET 2020 [root@mn-0:/mnt/log/master/fsaudit] # ls alarms auth.log internal-ssh.log testfile [root@mn-0:/mnt/log/master/fsaudit] # stat testfile File: testfile Size: 10 Blocks: 1 IO Block: 131072 regular file Device: 2ah/42d Inode: 10674483048284813578 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2020-04-09 08:59:51.729843706 +0300 Modify: 2020-04-09 08:59:59.259739585 +0300 Change: 2020-04-09 08:59:59.259739585 +0300 Birth: - [root@mn-0:/mnt/log/master/fsaudit] # echo "append after change time back">>testfile [root@mn-0:/mnt/log/master/fsaudit] # stat testfile File: testfile Size: 40 Blocks: 1 IO Block: 131072 regular file Device: 2ah/42d Inode: 10674483048284813578 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2020-04-09 08:59:51.729843706 +0300 Modify: 2020-04-09 08:59:59.259739585 +0300 Change: 2020-04-09 08:59:59.259739585 +0300 Birth: - [root@mn-0:/mnt/log/master/fsaudit] # cat testfile from mn-0 append after change time back Log in sn1 (mn-1 in my env): [root@mn-1:/mnt/log/master/fsaudit] # date Tue Mar 10 07:59:34 EET 2020 [root@mn-1:/mnt/log/master/fsaudit] # cat testfile from mn-0 append after change time back [root@mn-1:/mnt/log/master/fsaudit] # echo "append from mn-1">>testfile [root@mn-1:/mnt/log/master/fsaudit] # stat testfile File: testfile Size: 57 Blocks: 1 IO Block: 131072 regular file Device: 29h/41d Inode: 10674483048284813578 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2020-04-09 08:59:51.729843706 +0300 Modify: 2020-04-09 08:59:59.259739585 +0300 Change: 2020-04-09 08:59:59.259739585 +0300 Birth: - [root@mn-1:/mnt/log/master/fsaudit] # cat testfile from mn-0 append after change time back append from mn-1 [root@mn-0:/mnt/log/master/fsaudit] # cat testfile from mn-0 append after change time back Additional info:
However, after I disable utime, everything is back to normal [root@mn-0:/mnt/log/master/fsaudit] # gluster v set log ctime.noatime off volume set: success [root@mn-0:/mnt/log/master/fsaudit] # gluster v set log features.ctime off volume set: success [root@mn-0:/mnt/log/master/fsaudit] # cat tt from mn-1 after date +30 from mn-1 after date normal from mn0 [root@mn-0:/mnt/log/master/fsaudit] # ssh mn-1.local USAGE OF THE ROOT ACCOUNT AND THE FULL BASH IS RECOMMENDED ONLY FOR LIMITED USE. PLEASE USE A NON-ROOT ACCOUNT AND THE SCLI SHELL (fsclish) AND/OR LIMITED BASH SHELL. Read /opt/nokia/share/security/readme_root.txt for more details. [root@mn-1:/root] # cat /mnt/log/master/fsaudit/tt from mn-1 after date +30 from mn-1 after date normal from mn0 [root@mn-1:/root]
However, if i disable feature.utime. there will be following error [root@cp-0:/mnt/export] # rm -rf /mnt/export/testbak;cp -rf /mnt/export/test /mnt/export/testbak;tar -cf /mnt/export/test-tar.tar -C /mnt/export/ testbak tar: testbak/test.tar: file changed as we read it
[root@mn-0:/home/robot] # echo "fromm mn-0">>/mnt/export/testfile [root@mn-0:/home/robot] # stat /mnt/export/testfile File: /mnt/export/testfile Size: 30 Blocks: 1 IO Block: 131072 regular file Device: 28h/40d Inode: 9855109080001305442 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 615/_nokfsuifileshare) Access: 2020-05-10 09:33:59.713840197 +0300 Modify: 2020-05-10 09:33:59.713840197 +0300 Change: 2020-05-10 09:33:59.714413772 +0300 Birth: - [root@mn-0:/home/robot] # cat /mnt/export/testfil cat: /mnt/export/testfil: No such file or directory [root@mn-0:/home/robot] # cat /mnt/export/testfile from mn0 from mn-1 fromm mn-0 [root@mn-0:/home/robot] # date Wed 11 Mar 2020 09:05:58 AM EET [root@mn-0:/home/robot] it seems that after change system time to future, and restore, even when you write sth to the file the file's modify/access time keeps to be the future, i think this is wrong.
I figured out what's happening. The issue is that the file has 'c|a|m' time set to future (The file is created after the date is set to +30 days). This is done from client-1. On client-2 with correct date, when data is appended, it doesn't update the mtime and ctime because of both mtime and ctime is less than already set time on the file. This protection is required to keep the latest time when two clients are writing to the same file. We update c|m|a time only if it's greater than existing time. As a result, the perf xlators on client1 which relies on mtime doesn't send read to server as it thinks nothing is changed as in this case the times haven't changed. Workarounds: 1. Disabling quick-read solved the issue for me. I don't know how real this kind of workload is? Is this a normal scenario ? The other thing to do is to remove that protection of updating time only if it's greater but that would open up the race when two clients are updating the same file. This would result in keeping the older time than the latest. This requires code change and I don't think that should be done.
This bug is moved to https://github.com/gluster/glusterfs/issues/887, and will be tracked there from now on. Visit GitHub issues URL for further details
REVIEW: https://review.gluster.org/24229 (enable update mdata after date change, if system data change to future, then change back, mdata should not stay in future.) posted (#1) for review on master by None
This bug is moved to https://github.com/gluster/glusterfs/issues/1123, and will be tracked there from now on. Visit GitHub issues URL for further details