Bug 601800 - NFS-over-GFS out-of-order GETATTR Reply causes corruption
Summary: NFS-over-GFS out-of-order GETATTR Reply causes corruption
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kernel
Version: 5.6
Hardware: All
OS: Linux
Target Milestone: rc
: ---
Assignee: Jeff Layton
QA Contact: Cluster QE
: 438676 (view as bug list)
Depends On:
TreeView+ depends on / blocked
Reported: 2010-06-08 16:28 UTC by Abhijith Das
Modified: 2018-11-14 18:10 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2011-01-13 21:36:22 UTC

Attachments (Terms of Use)
relevant portion of tcpdump on the server when the crash occurs (536.48 KB, application/octet-stream)
2010-06-08 16:35 UTC, Abhijith Das
no flags Details
nfs log snippet on client when crash occured (3.92 KB, text/plain)
2010-06-08 16:37 UTC, Abhijith Das
no flags Details
patchset1 -- 4 patches from upstream that should correct problem (11.00 KB, patch)
2010-06-09 17:06 UTC, Jeff Layton
no flags Details | Diff

System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2011:0017 normal SHIPPED_LIVE Important: Red Hat Enterprise Linux 5.6 kernel security and bug fix update 2011-01-13 10:37:42 UTC

Description Abhijith Das 2010-06-08 16:28:16 UTC
Description of problem:
While testing a patch for bz 245024, the qe test program genesis crashed with corrupted data.
The setup consists of:
- one GFS server (running 2.6.18-201.el5) exported via NFS.
- one client (also running 2.6.18-201.el5) mounting the NFS filesystem

Running genesis (modified by me to only do append operations) on the client in the nfs mount with the following command line:
/root/sts-rhel5/src/genesis/genesis -i 300s -p5 -n5 -S 5555 -d 1 -k -L fcntl 2> /tmp/genlog 
does the following:

genesis creates 5 files in 1 directory and forks 5 processes to do the following continuously for 300s. 
1. open any one of the 5 files at random
2. lock the file using locking specified (fcntl here)
3. seek to the end of the file
4. write a random number of bytes (upto 1024)
5. seek back to offset where the write started
6. read the number of bytes written in step 4
7. verify that the bytes written and bytes read back are identical
8. unlock file
9. close file

One of the processes will fail to verify the write (reads nulls in step 6) and crashes the test.

Comment 1 Abhijith Das 2010-06-08 16:35:01 UTC
Created attachment 422268 [details]
relevant portion of tcpdump on the server when the crash occurs

When viewed with wireshark, this tcpdump shows that an out-of-order GETATTR Reply packet (for inode 79) was sent after a WRITE operation. The WRITE operation wrote 1024@694092 making isize=695116... but the out-of-order GETATTR reply sends the isize as 694092 and the client is unable to read to the actual end of the file because of this.

Comment 2 Abhijith Das 2010-06-08 16:37:49 UTC
Created attachment 422269 [details]
nfs log snippet on client when crash occured

Comment 4 Jeff Layton 2010-06-08 18:09:48 UTC
It looks like 47c287183705bdbe3c10bf1d57636589d1f336b3 applies cleanly to RHEL5. I think it's the right fix for this. I'm adding this to my test kernels and will plan to do some regression testing with it.

I think it's reasonable that we can make 5.6 with this too (PM willing), so I'll go ahead and propose it for that.

Comment 5 Jeff Layton 2010-06-09 17:06:41 UTC
Created attachment 422625 [details]
patchset1 -- 4 patches from upstream that should correct problem

The only problem with this patch is that it might mean that we occasionally lose attribute updates that are valid. There are a few other patches that went in after this one that help recover some of that. This set reflects the backport of those. I think we'll want all 4.

A nice bonus to this is that it seems to reduce some of the false cache invalidations we see without this patch. Whenever these calls get reordered like this, we not only get the metadata wrong, but we have to invalidate the cache for the inode. By discarding stale updates like this, we minimize this effect to a large degree. That helps performance.

Comment 7 Jeff Layton 2010-06-28 12:44:59 UTC
*** Bug 438676 has been marked as a duplicate of this bug. ***

Comment 8 RHEL Product and Program Management 2010-06-28 12:48:44 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update

Comment 9 Abhijith Das 2010-06-29 15:01:20 UTC
*** Bug 245024 has been marked as a duplicate of this bug. ***

Comment 20 Jarod Wilson 2010-07-23 15:28:52 UTC
in kernel-2.6.18-208.el5
You can download this test kernel from http://people.redhat.com/jwilson/el5

Detailed testing feedback is always welcomed.

Comment 23 errata-xmlrpc 2011-01-13 21:36:22 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.


Note You need to log in before you can comment on or make changes to this bug.