Description of problem: When we try to create a new microsoft word file over a mounted tier volume in windows client the file does get created but it is not getting listed on the share, even after repeated refresh action over the share the new file created is not listed. Then when we try to create another new microsoft word file it gets renamed and created and only after that when we refresh the first file is listed. In an another instance when we create a microsoft word file in windows client-1 it gets created but the same file is not listed in same volume share from another windows client-2 even after continuous refresh of the share. (around a 6minutes) and then again when we try to create another new microsoft word file it gets renamed and created and only after that when we refresh the first file is listed in windows client-2 Version-Release number of selected component (if applicable): glusterfs-3.7.9-3.el7rhgs.x86_64 samba-client-libs-4.4.2-1.el7rhgs.x86_64 Windows Client 8.1 How reproducible: 6/10 times Steps to Reproduce: 1.Tier volume mount in windows client1 & windows client2 2.Create a new microsoft word file by right click in the share 3.Refresh from both the windows client Actual results: File is not getting listed Expected results: File should get listed immediately Additional info:
Update : --------- The issue is not always hit on the fresh / first attempt. But gradually when we work on the share intensely like running multiple IOs and keeping the share busy we get the above issue. When testing with tiered volume this issue was seen and hence reported. Now when i am trying with other volumes, i am able to reproduce the same. Few information below: ------------------------ 1.The issue is also observed in Distribute & Disp-Rep volume. 2.The issue exists not only with Microsoft Windows files but also text files. 3.Whether files and directories are created or deleted it is not getting reflected in the share mount even after a gap of 15-20minutes.
The notify sub-system has changed in Samba 4.3. The notify watch into the file system can be provided by different callbacks depending on the configuration and capabilities of the (build)system. The default is inotify. But in our case of the glusterfs vfs module, this can obviously not work since our paths are not paths in a posix file system that inotify can work with. Hence we need to disable inotify by setting "notify:inotify = no" in smb.conf (global section) (on all nodes). ==> Please test this next. If proper functionality is confirmed, we should probably make sure by a code change in the glustefs vfs module, that this is enforced. Ultimately, we might extend Samba to have notify-watch as a VFS operation that can then be provided by the glusterfs vfs module...
Update: Instead of doing notify:inotify = no, the documented setting 'kernel change notify = no' should achieve exactly the same thing. Cheers - Michael
I tested with the change suggested. i.e with kernel change notify = no and verified that it works exactly as expected. I will remove the needinfo on vdas and would suggest that we make a build with the patch as provided by Michael in the mail thread. Also, changing the bug to post state to indicate that we have a patch ready. Would request that we get the build as soon as possible.
My observation with new build: 1. If both the windows clients have the same dir open then a new file create/delete etc is reflected immedietely in the other windows client. (note that this is happening only after we brought the "Kernel change notify = no" option, earlier this wasn't working either). 2. Navigate to a dir in windows client1 say /a/b/c. Navigate to the same dir in windows client2. Now, create a file file.txt in client1, you should be able to see it in client2 immediately. Now, in client2, move back to / dir of the share. On client1, create another file say file2.txt. If you navigate to /a/b/c in client2 now, you won't find file2.txt. This does not change even after multiple refreshes are performed on the GUI. Performing a DIR in powershell at the same location does not show new file too. This is the bug. Few more subtle points a. If /a/b/c was never accessed in client2 and it was now accessed for the first time then all files are seen. So it has to do something with notification. b. The same test as mentioned above in 2. was performed on a xfs share with no gluster involved and the same bug was observed. I am attaching a packet capture showing packets on client2 when a refresh was performed but no new packets were seen.
Created attachment 1162820 [details] packet capture
(In reply to Raghavendra Talur from comment #15) > My observation with new build: > > 1. If both the windows clients have the same dir open then a new file > create/delete etc is reflected immedietely in the other windows client. > (note that this is happening only after we brought the "Kernel change notify > = no" option, earlier this wasn't working either). > > 2. Navigate to a dir in windows client1 say /a/b/c. Navigate to the same dir > in windows client2. Now, create a file file.txt in client1, you should be > able to see it in client2 immediately. Now, in client2, move back to / dir > of the share. On client1, create another file say file2.txt. If you navigate > to /a/b/c in client2 now, you won't find file2.txt. This does not change > even after multiple refreshes are performed on the GUI. Performing a DIR in > powershell at the same location does not show new file too. This is the bug. > > > Few more subtle points > a. If /a/b/c was never accessed in client2 and it was now accessed for the > first time then all files are seen. So it has to do something with > notification. > b. The same test as mentioned above in 2. was performed on a xfs share with > no gluster involved and the same bug was observed. > > > I am attaching a packet capture showing packets on client2 when a refresh > was performed but no new packets were seen. Thanks for the updates. I will look into this more deeply. But we need to very specific, in a cluster, where who connects, since it may make a difference. We have at least three scenarios, assuming two different clients connecting: case1: client1 connects to IP1, client2 connects to IP2, which is on the same node as IP1 case2: client1 connects to IP1, client2 connects to IP2, which is on a different node than IP1 case3: client1 connects to IP1 client2 connects to IP1 (the *same* IP)
(In reply to Raghavendra Talur from comment #15) > My observation with new build: Comments on these observations since I only commented on the scenarios in my last comment: > 1. If both the windows clients have the same dir open then a new file > create/delete etc is reflected immedietely in the other windows client. > (note that this is happening only after we brought the "Kernel change notify > = no" option, earlier this wasn't working either). Good! So at least this hase changed (improved) something. > 2. Navigate to a dir in windows client1 say /a/b/c. Navigate to the same dir > in windows client2. Now, create a file file.txt in client1, you should be > able to see it in client2 immediately. Now, in client2, move back to / dir > of the share. On client1, create another file say file2.txt. If you navigate > to /a/b/c in client2 now, you won't find file2.txt. This does not change > even after multiple refreshes are performed on the GUI. Performing a DIR in > powershell at the same location does not show new file too. This is the bug. Very interesting, indeed. I'll dig deeper here... > Few more subtle points > a. If /a/b/c was never accessed in client2 and it was now accessed for the > first time then all files are seen. So it has to do something with > notification. Right. > b. The same test as mentioned above in 2. was performed on a xfs share with > no gluster involved and the same bug was observed. Good point. It is always good to have as minimal a reproducer as possible. Will try to put this into a samba selftest. > I am attaching a packet capture showing packets on client2 when a refresh > was performed but no new packets were seen.
Summary of the findings so far: The remaining issue seems to be a client issue: Specific versions of windows 7 and windows 8 seem to be affected under certain configuration scenarios. And it does not seem to be a blocker. Windows 7 (not fully patched) test result: - with SMB1: no problem - with SMB2 and oplocks disabled: no problem - with SMB2 and smb2 leases enabled: no problem - only with SMB2 and oplocks but not leases: problem - this problem also exists with Samba 4.2 ==> not a regression. Windows 8: - with this machine the issue was first discovered - still re-testing the full matrix with that one Windows 8.1: - no issue seen with any config / version. Based on this, the change of 'kernel change notify = no' seems to have fixed the original issue, and the remaining issues seem to be client-bugs in specific Windows-versions regarding notify in combination with oplocks. Moving back to ON_QA for verification.
I confirmed that the bug happens with windows 8 against samba 4.2 (and 4.3) as well. So no regression for Windows 8, either.
According to the original bug posted (refer comment #4) this bug is not reproducible with the fix provided. So marking this as Verified. Howerever a new bug will be opened for the bug mentioned in comment #12.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1245