Bug 623803

Have you confirmed that this is indeed a problem on RHEL6?

If so, what kernels are the client and server running? What sort of filesystem is being exported?

Jeff,

I tested this in loopback on RHEL 6 using kernel-2.6.32-59.1.el6.x86_64.  I tried exporting several filesystem types including ext3, ext4, and btrfs.

I just reran the test using kernel-2.6.32-71.el6.x86_64 on both NFS client and server.  I used btrfs as the underlying filesystem, but I believe that the type of filesystem is irrelevant.

Needless to say, this reproduced right away for me between 2 rhel 6 boxes (the client is a VM, the server bare metal).

I'm afraid I've not been able to do so:

...created an 8G file on an export ext4 filesystem:

server$ dd if=/dev/zero of=/export/filler bs=1M count=8000
server$ df -k /export
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/vg_data-export
                      20642428   9479488  10114364  49% /export

...mounted up the filesystem on the client using NFSv3. Here's the fstab entry:

dantu.rdu.redhat.com:/export	/mnt/dantu	nfs	noauto	0 0
client# mount /mnt/dantu -o vers=3

...then acquired the lock:

client# ./locktest /mnt/dantu/filler
Press <Enter> to try to get lock: 
waiting...Locked.
Press <Enter> to release lock: 

...removed the file on server -- free space is unchanged since lockd holds it open for now:

server$ rm /export/filler
server$ df -k /export
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/vg_data-export
                      20642428   9479488  10114364  49% /export

...hit ^c on the client to kill locktest. df on the server shows the space as immediately freed:

server$ df -k /export
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/vg_data-export
                      20642428   1287484  18306368   7% /export

...so apparently you and I are doing something different. Perhaps you can clarify what that is?

Jeff,

I just retested...  In RHEL 6, the problem is with NFS V4.  NFS V2 and NFS V3 report that the disk space is freed promptly after killing locktest.

-Marc

Ok, that helps. I can reproduce it with that too. Interestingly, when I shut down nfs serving from the server free space still isn't shown. I had to shut it down, unmount and remount before the free space was visible.

I'll have to do some analysis to figure out what's happening.

Some more info. Locking isn't necessary here. This happens when you just hold the file open and then delete it. Josef said it sounds like the inode is ending up on the unused list instead of being cleaned up properly when nfsd's filp is closed.

That seems likely -- when I do this:

    # echo 2 > /proc/sys/vm/drop_caches

...the space magically reappears.

See also 636926, describing the problem RHEL6/NFSv4 problem.  I cann't find any leak of NFSv4 data structures--as far as NFSv4 is concerned I think it's done with the inode in question--but the inode isn't freed.  I think we must be misusing the vfs reference counting somehow.  I don't know whether you're seeing a symptom of the same problem.

Thanks, Bruce...

Yeah, looks like the same problem. I'll close this one in favor of that one as it seems to have more info.

*** This bug has been marked as a duplicate of bug 636926 ***