Bug 1811907

Summary: different glusterfs client see different file content but when check brick, the data are same.
Product: [Community] GlusterFS Reporter: zhou lin <zz.sh.cynthia>
Component: fuseAssignee: bugs <bugs>
Status: CLOSED UPSTREAM QA Contact:
Severity: high Docs Contact:
Priority: unspecified    
Version: 7CC: bugs, khiremat
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-03-18 06:10:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description zhou lin 2020-03-10 07:17:39 UTC 
Description of problem:
cat show different file content from different glusterfs client.

Version-Release number of selected component (if applicable):

gluster7
How reproducible:


Steps to Reproduce:
1.	on sn0 node, change time +30 days
2.	touch a file in one volume
3.	echo sth to that file
4.	change time +30 day on sn0 again
5.	restore time back (should -60 day), and check that the data is back to normal now
6.	append sth to the file on sn0
7.	log to sn1 append sth to the file from sn1 client
8.	cat the file from sn0 and sn1 , the content is different on two nodes.


Actual results:
file content should are not the same

Expected results:
file content should be the same

///////////////////operation steps////////////////////////

Log on sn0 node(mn-0 On my env)
 
 
[root@mn-0:/mnt/log/master/fsaudit]
# date -s "+30 day"
Thu Apr  9 08:59:47 EEST 2020
[root@mn-0:/mnt/log/master/fsaudit]
# ls
alarms  auth.log  internal-ssh.log
[root@mn-0:/mnt/log/master/fsaudit]
# touch testfile
[root@mn-0:/mnt/log/master/fsaudit]
# echo "from mn-0">testfile
[root@mn-0:/mnt/log/master/fsaudit]
# cat testfile
from mn-0
[root@mn-0:/mnt/log/master/fsaudit]
# stat testfile
  File: testfile
  Size: 10              Blocks: 1          IO Block: 131072 regular file
Device: 2ah/42d Inode: 10674483048284813578  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2020-04-09 08:59:51.729843706 +0300
Modify: 2020-04-09 08:59:59.259739585 +0300
Change: 2020-04-09 08:59:59.259739585 +0300
Birth: -
[root@mn-0:/mnt/log/master/fsaudit]
# date -s "+30 day"
Sat May  9 09:00:29 EEST 2020
[root@mn-0:/mnt/log/master/fsaudit]
# date -s "-30 day"
Thu Apr  9 09:00:34 EEST 2020
[root@mn-0:/mnt/log/master/fsaudit]
# date -s "-30 day"
Tue Mar 10 08:00:37 EET 2020
[root@mn-0:/mnt/log/master/fsaudit]
# date
Tue Mar 10 08:00:38 EET 2020
[root@mn-0:/mnt/log/master/fsaudit]
# ls
alarms  auth.log  internal-ssh.log  testfile
[root@mn-0:/mnt/log/master/fsaudit]
# stat testfile
  File: testfile
  Size: 10              Blocks: 1          IO Block: 131072 regular file
Device: 2ah/42d Inode: 10674483048284813578  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2020-04-09 08:59:51.729843706 +0300
Modify: 2020-04-09 08:59:59.259739585 +0300
Change: 2020-04-09 08:59:59.259739585 +0300
Birth: -
[root@mn-0:/mnt/log/master/fsaudit]
# echo "append after change time back">>testfile
[root@mn-0:/mnt/log/master/fsaudit]
# stat testfile
  File: testfile
  Size: 40              Blocks: 1          IO Block: 131072 regular file
Device: 2ah/42d Inode: 10674483048284813578  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2020-04-09 08:59:51.729843706 +0300
Modify: 2020-04-09 08:59:59.259739585 +0300
Change: 2020-04-09 08:59:59.259739585 +0300
Birth: -
[root@mn-0:/mnt/log/master/fsaudit]
# cat testfile
from mn-0
append after change time back
 
 
Log in sn1 (mn-1 in my env):
 
[root@mn-1:/mnt/log/master/fsaudit]
# date
Tue Mar 10 07:59:34 EET 2020
[root@mn-1:/mnt/log/master/fsaudit]
# cat testfile
from mn-0
append after change time back
[root@mn-1:/mnt/log/master/fsaudit]
# echo "append from mn-1">>testfile
[root@mn-1:/mnt/log/master/fsaudit]
# stat testfile
  File: testfile
  Size: 57              Blocks: 1          IO Block: 131072 regular file
Device: 29h/41d Inode: 10674483048284813578  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2020-04-09 08:59:51.729843706 +0300
Modify: 2020-04-09 08:59:59.259739585 +0300
Change: 2020-04-09 08:59:59.259739585 +0300
Birth: -
[root@mn-1:/mnt/log/master/fsaudit]
# cat testfile
from mn-0
append after change time back
append from mn-1
 
[root@mn-0:/mnt/log/master/fsaudit]
# cat testfile
from mn-0
append after change time back



Additional info:
Comment 1 zhou lin 2020-03-10 07:20:24 UTC 
However, after I disable utime, everything is back to normal
[root@mn-0:/mnt/log/master/fsaudit]
# gluster v set  log ctime.noatime  off
volume set: success
[root@mn-0:/mnt/log/master/fsaudit]
# gluster v set  log features.ctime  off
volume set: success
[root@mn-0:/mnt/log/master/fsaudit]
# cat tt
from mn-1 after date +30
from mn-1 after date normal
from mn0
[root@mn-0:/mnt/log/master/fsaudit]
# ssh mn-1.local
 
USAGE OF THE ROOT ACCOUNT AND THE FULL BASH IS RECOMMENDED ONLY FOR LIMITED USE. PLEASE USE A NON-ROOT ACCOUNT AND THE SCLI SHELL (fsclish) AND/OR LIMITED BASH SHELL.
 
Read /opt/nokia/share/security/readme_root.txt for more details.
 
[root@mn-1:/root]
# cat /mnt/log/master/fsaudit/tt
from mn-1 after date +30
from mn-1 after date normal
from mn0
[root@mn-1:/root]
Comment 2 zhou lin 2020-03-11 06:02:52 UTC 
However, if i disable feature.utime. there will be following error

[root@cp-0:/mnt/export]
# rm -rf /mnt/export/testbak;cp -rf /mnt/export/test /mnt/export/testbak;tar -cf /mnt/export/test-tar.tar -C /mnt/export/ testbak
tar: testbak/test.tar: file changed as we read it
Comment 3 zhou lin 2020-03-11 07:07:55 UTC 
[root@mn-0:/home/robot]
# echo "fromm mn-0">>/mnt/export/testfile
[root@mn-0:/home/robot]
# stat /mnt/export/testfile
  File: /mnt/export/testfile
  Size: 30              Blocks: 1          IO Block: 131072 regular file
Device: 28h/40d Inode: 9855109080001305442  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (  615/_nokfsuifileshare)
Access: 2020-05-10 09:33:59.713840197 +0300
Modify: 2020-05-10 09:33:59.713840197 +0300
Change: 2020-05-10 09:33:59.714413772 +0300
 Birth: -
[root@mn-0:/home/robot]
# cat /mnt/export/testfil
cat: /mnt/export/testfil: No such file or directory
[root@mn-0:/home/robot]
# cat /mnt/export/testfile
from mn0
from mn-1
fromm mn-0
[root@mn-0:/home/robot]
# date
Wed 11 Mar 2020 09:05:58 AM EET
[root@mn-0:/home/robot]

it seems that after change system time to future, and restore, even when you write sth to the file the file's modify/access time keeps to be the future, i think this is wrong.
Comment 4 Kotresh HR 2020-03-11 14:05:08 UTC 
I figured out what's happening. The issue is that the file has 'c|a|m' time set to future (The file is created after the date is set to +30 days). This
is done from client-1. On client-2 with correct date, when data is appended, it doesn't update the mtime and ctime because of both mtime and ctime is less than
already set time on the file. This protection is required to keep the latest time when two clients are writing to the same file. We update c|m|a time only if it's greater than
existing time. As a result, the perf xlators on client1 which relies on mtime doesn't send read to server as it thinks nothing is changed as in this case the times haven't
changed.

Workarounds:
1. Disabling quick-read solved the issue for me.

I don't know how real this kind of workload is? Is this a normal scenario ?
The other thing to do is to remove that protection of updating time only if it's greater but that would open up the race when two clients are updating the same file.
This would result in keeping the older time than the latest. This requires code change and I don't think that should be done.
Comment 5 Worker Ant 2020-03-12 12:22:48 UTC 
This bug is moved to https://github.com/gluster/glusterfs/issues/887, and will be tracked there from now on. Visit GitHub issues URL for further details
Comment 6 Worker Ant 2020-03-17 04:47:24 UTC 
REVIEW: https://review.gluster.org/24229 (enable update mdata after date change, if system data change to future, then change back, mdata should not stay in future.) posted (#1) for review on master by None
Comment 7 Worker Ant 2020-03-18 06:10:19 UTC 
This bug is moved to https://github.com/gluster/glusterfs/issues/1123, and will be tracked there from now on. Visit GitHub issues URL for further details