Description of problem:
Reading data from files over Samba share causes them to
temporarily appear as modified when same files are examined via
NFS share. Does not happen if cached 'ls' data is present.
Results in spurious rebuilding of output files when 'make' is
run concurrently from Samba and NFS shares. Note that in our
environment objects for different platforms are kept in separate
directories in a manner similar to the way 'gcc' builds work.
The fictious source file timestamp modifications are what
cause the problem.
Version-Release number of selected component (if applicable):
Samba+NFS server, x86_64
Windows 2003 R2 SP2 X64, current patches
NFS client, i686
1) place a bunch of files in a directory
(verified this with 'gzip' source)
2) issue 'touch time_mark'
3) create Samba share of directory
4) create NFSv3 share of directory
5) issue 'find . -type f -print | xargs cat >/dev/null'
in directory via Samba client
(alternately 'make -n' could be run)
6) issue 'find . -type f -newer time_mark -ls'
in directory via NFS client
7) observe that all files are reported as modified!!!
8) repeat (6) after 'acregmax' interval and observe
that files are no longer reported as modified
In practice this has been causing 'make' to rebuild things
that it should not. Putting it nicely, it's been driving
us nuts for many months.
Obviously, the files were never modified and should not
be reported as such.
If you run 'ls -l' in the directory via the NFS share, the
problem is cured. You may have to unmount and remount the share
to bring back the bug.
Before we figured this out, tried NFSv4 for awhile. Seemed like
it fixed the above issue but it's not certain. NFSv4 + PoPToP
started hosing our system so it's gone and I have no further
patience or time for more experiments.
A closer looks shows that the 'cat' command is not
require. Simply 'find'ing the files on the Windows
find . -type f -print >/dev/null
produces the behavior. Also determined the problem
appears for both 'i686' and 'x86_64' servers, so it's
a portable bug.
Also it seems that more than a handful of files
is required to produce the problem.
does not happen at top-level of NFSv3 mount, must be in subdirectory
if any directories are touched by the 'find' on Samba share, problem is suppressed
does not happen when files reside on 'ext2' file system
does happen when files reside on 'ext3' file system
have 'noreservation' option active
too lazy to try it with 'noreservation' turned off
tried mounting Samba share remotely with CIFS
problem does not happen with 'find' run on CIFS mount
files accessed via Windows UNC path, not mapped drive
obviously CYGWIN is in use on the Windows system
can also produce the problem using CMD shell and 'type \\server\...\file' command
CMD 'dir' doesn't work, probably because directory file is examined
does happen with even tiny number of files
does happen with WXP SP2 32-bit Samba client
can produce problem with MKS, but a 'cat' is also required:
find . -type f -print | xargs cat >/dev/null
Created attachment 267991 [details]
TAR archive of traces files showing problem
Traces showing problem. Debug level 10 Samba
trace plus Wireshark capture.
# Windows 'type' access of file
SMB access of 'djl0' at packet 21
Last Write: Nov 24, 2007 02:20:44.000000000
# first 'find' invocation
NFS access of 'djl0' at packet 97
mtime: Nov 24, 2007 04:00:45.000000000
NFS access of 'djl0' at packet 107
mtime: Nov 24, 2007 04:00:45.000000000
# second 'find' invocation
NFS access of 'djl0' at packet 176
mtime: Nov 24, 2007 02:20:44.000000000
Created attachment 268101 [details]
file lease test program
Examined Samba traces and discerned that the cause of the problem:
fcntl(fd, F_SETLEASE, F_WRLCK);
on files that are opened for read access. The Linux kernel sets
the modify time as seen via NFS as current when the activity
trace is put in place and then restores the modify time once
it's released. Reproduced this with the attached test case. The
altered modify time appears *only* via NFS. On the local system
it's not visible. Definitely a NFS bug.
Checked NFSv4 and issue is not present there. Appears only with
NVSv3 and is in the server logic, not in the client side.
The Plot Thickens
It turns out the observed behavior is by design; full details at
I don't fully understand all the interactions, so I'm pasting
comments from Bruce Fields, a 'nfsd' maintainer. Appended the
same information to the Samba bug report.
>------- Comment #5 From email@example.com 2007-11-26 08:57:56 [reply] -------
>This is by design--see fs/locks.c:lease_get_mtime().
>The argument is: if somebody has a write lease on the file (are
>you exporting this via Samba? That'd be the typical user), then
>they're caching writes--they're explicitly telling us that we do
>not know whether the file is still the same, because we may have
>modified it on a remote client and not told us about it. So
>lease_get_mtime() reports the current time as the mtime,
>prompting you to actually try opening the file, at which point
>the write lease gets broken and any cached writes get flushed
>It's a terrible kludge, I agree, and maybe we should remove it.
>But I'd first like to understand what circumstances prompted
>somebody to add it originally, and talk to the Samba people
>about how they're using these write leases.
>(In NFSv4, by the way, there's a callback to the client to allow
>the server to find out the attributes of a file that the client
>holding a write lease is caching, which solves this problem. We
>haven't implemented that yet; it seems likely to be an enormous
>pain. I wonder if Samba would want something similar?)
>------- Comment #6 From firstname.lastname@example.org 2007-11-26 10:17:08 [reply] -------
>Yes, using a Samba mount along with NFS mounts to 'make' build
>an application on multiple platforms simultaneously. When
>re-building existing trees the result is spurious target
>building, a real problem. Took months of frustration to
>figure out what was going on.
>Actually reported this as a bug to Samba along with RH, and a
>request for info from the Samba team provoked the analysis that
>isolated the problem. Closed the Samba bug but it's all still
>My impression from looking at the Samba code is that the leases
>are put in place so that files can be monitored for changes,
>but that's just a guess. Perhaps Samba can use a read lease
>instead of a write lease.
>------- Comment #7 From email@example.com 2007-11-26 10:55:17 [reply] -------
>Thanks for the pointer to the samba bug. I understand the
>My feeling is that: yes, it's suboptimal (but perhaps not a bug)
>for samba to be requesting a write lease when a read lease
>(sufficient to alert it to modifications of the file) would be
>It seems to me that the real bug is the incomplete lease
>implementation--if the purpose of a lease is to allow a remote
>client to cache writes to the file, then there's no way for us
>to give a sensible answer to a stat call, unless we break the
>Perhaps we could find out (in the samba case) what the
>consequences would be of not updating mtime in this case? I
>suppose the worst case would be that a modification to a file
>made on a samba client could be indefinitely delayed from being
>flushed to the server.
>If we agree that that would be less of a problem than these
>spurious bumps in the mtime, then the best solution for now may
>just be to rip out lease_get_mtime(). I'll cook up a straw-man
>------- Comment #8 From firstname.lastname@example.org 2007-11-26 10:56:16 [reply] -------
>Created an attachment (id=13763) [details]
>Not sure if this is what we want to do, but it's at least
>something you could test to confirm the source of the problem.
>------- Comment #9 From email@example.com 2007-11-26
>11:13:01 [reply] -------
>Thanks! Happen to have a kernel source tree set up for the
>RH images running on the server, so I can try this out in
>the next week or so. Seems certain to work.
>In the interim perhaps you could start a dialog with Samba about
>the reasoning behind the application of the F_WRLCK? Or if you
>wish I could do so using the Samaba bug report cited above.
Created attachment 270671 [details]
Adapted kernel.org authored patch for RHEL 4.5 2.6.9-55 kernel
(attached). Works as expected and eliminates spurious target
rebuilding when a concurrent Samaba and NFS 'make' is run on
same source tree. Test case also shows expected behavior.
Obviously did not perform any regression testing with other
Samba file scenarios. In our build trees the same files and
directories are never written by more than one server.
Concise description of patch rationale from original:
>From: J. Bruce Fields <>
>Date: Mon, 26 Nov 2007 13:48:34 -0500
>Subject: [PATCH] nfsd: stop incrementing mtime on presence of write lease
>The lease_get_mtime() function has the effect of setting the mtime of a
>file (as far as any nfs client is concerned) to the current time,
>whenever there is a write lease held on the file.
>The presence of a write lease may mean that some client (almost
>certainly a samba client) is caching writes to that file. Thus
>increasing the mtime has the effect of making the nfs client read data
>from that file, thus opening the file for read, thus breaking the write
>lease, thus forcing the samba client to flush any cached writes.
>However Samba seems to be requesting write leases even when a read lease
>would do. And unfortunately the consequences of the spurious mtime
>updates--make unnecessarily rebuilding, for example--may be worse than
>the consequences of no mtime update--modifications from a samba client
>taking longer to be noticed on the nfs client.
>So perhaps we should rip out lease_get_mtime().
Hmmm...at first glance this sounds tricky and I'm not clear whether it's
something we'd want to take for RHEL4. I need to do a more thorough review of
the problem and patch though, so let me do that before I NAK anything...
This definitely needs to go upstream first and will need some soak time before
it can go into any RHEL product. Given that RHEL4 is approaching maintenance
phase, it's probably unlikely that this will ever make it into there. RHEL5
might be a possibility if there's resolution upstream.
Most likely, samba is getting leases in order to emulate opportunistic locks
(aka oplocks) on the share. One possible workaround might be to disable kernel
oplocks in smb.conf:
kernel oplocks = no
...or maybe by judicious use of the "veto oplock files" parameter. Of course,
that means that oplocks/leases won't be broken by actvity on the files outside
of samba, but that's probably similar to the behavior you'll see by removing
An even better option might be to remove NFS from the picture altogether and
just use CIFS with unix extensions on the Linux clients.
In any case, I don't see this happening for RHEL4 so I'm going to NAK this for
now. Feel free to open a similar case for RHEL5 if you like, though until this
patch goes upstream it's not an option there either...
Development Management has reviewed and declined this request. You may appeal
this decision by reopening this request.
Tried "kernel oplocks = no" in the Samba config
and it does work around the issue.