Bug 200974 - NFS (client) file corruption with 2.6.17-1.2488.fc6xen
NFS (client) file corruption with 2.6.17-1.2488.fc6xen
Product: Fedora
Classification: Fedora
Component: kernel (Show other bugs)
x86_64 Linux
medium Severity medium
: ---
: ---
Assigned To: Steve Dickson
Brian Brock
Depends On:
  Show dependency treegraph
Reported: 2006-08-01 17:52 EDT by Mike Gahagan
Modified: 2007-11-30 17:11 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2007-03-09 15:43:00 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Mike Gahagan 2006-08-01 17:52:44 EDT
Description of problem:

There is some appearant client side nfs file corruption with
2.6.17-1.2488.fc6xen. I found this while trying to install rpms over nfs. A
quick run of rpm -K shows that files larger than about 3MB or so are showing
checksum failures where smaller files are apperantly ok. I have not noticed any
corruption with other protocols as of yet. I can reproduce both on dom-0 as well
as a guest system on an AMD x86_64 box.

Version-Release number of selected component (if applicable):
8-1-2006 rawhide, 2.6.17-1.2488.fc6xen

How reproducible:

Steps to Reproduce:
1.Mount an NFS repository
2.WAtch rpm install attempts blow up due to bad checksums.
Actual results:

rpm files > than 3MB or so can't be used, it likely affects anything else, but I
have not done very extensive testing as of yet.

Expected results:

rpms installable.

Additional info:
Comment 2 Steve Dickson 2006-08-01 22:18:33 EDT
Does this only happen when a XEN kernel?
Comment 3 Mike Gahagan 2006-08-02 10:32:04 EDT
This is happening on the non-Xen kernel as well. What's interesting is the file
size is always correct. I did a diff on the files and did not get very much back
so it's not like the file is getting truncated or filled with zeros. 

-bash-3.1# ll kernel-2.6.17-1.2505.fc6.src.rpm
-rw-r--r-- 14 root root 49890117 Aug  1 21:17 kernel-2.6.17-1.2505.fc6.src.rpm
-bash-3.1# rpm -K kernel-2.6.17-1.2505.fc6.src.rpm
kernel-2.6.17-1.2505.fc6.src.rpm: sha1 MD5 NOT OK
-bash-3.1# md5sum kernel-2.6.17-1.2505.fc6.src.rpm
da35d19179b034e487802ed9c2e2f2b2  kernel-2.6.17-1.2505.fc6.src.rpm
-bash-3.1# uname -a
Linux et-4.test.redhat.com 2.6.17-1.2488.fc6 #1 SMP Mon Jul 31 21:09:02 EDT 2006
x86_64 x86_64 x86_64 GNU/Linux
Comment 4 Mike Gahagan 2006-08-02 10:34:42 EDT
Lowering the r/wsize to 8192 did not help, also transfers over http or ssh do
not seem to be affected.
Comment 5 Mike Gahagan 2006-08-02 11:10:12 EDT
No change in behavior in 2.6.17-1.2505.fc6
Comment 6 Steve Dickson 2006-08-02 11:28:32 EDT
hmm... I'm not seeing this at all.... I using a 2.6.17-1.2488.fc6 kernel
and installing kernel-2.6.17-1.2505.fc6.i686.rpm and both kernel-debuginfo
packages from curly with no problem at all... 

So I guess I'm going to need some bzip2 binary tethereal trace
to try and figure out whats going on here... from the 
client please use 'tethereal -w /tmp/data.pcap host <server> ; bzip2 /tmp/data.pcap'

Comment 7 David Howells 2006-08-02 11:34:16 EDT
This sounds like one of the problems IBM has seen, where the installer kernel 
sometimes mucks up NFS retrieval somehow, somewhere.

See bug 168981.
Comment 9 Mike Gahagan 2006-08-02 12:52:42 EDT
This is on an amd x86_64 box in case that makes any difference. I also just
tried to do a yum update from a guest system on the same box and the first
package it tried to download and install was corrupted. This was the first time
I've seen it with HTTP.
Comment 16 Mike Gahagan 2006-08-25 11:04:28 EDT
I have also seen this behavior with FC5 GA using NFS over TCP, using NFS over
UDP works around the problem.
Comment 20 Neil Horman 2007-03-05 11:06:03 EST
From a protocol standpoint, I think this is all running properly.  What I see is
the following:

1) Lost segments (frames 323, 333, 337, 343, 420, 4201, 11581)
     These are all flowing from the server to the client, and most likely
represent simple network congestion.  I'd be interested to know the results of
the dropped frame counters on the NFS client to see if these drops are occuring
on the client or elsewhere on the network.  My guess would be the latter, but it
would be interesting to know just the same.

2) Duplicate ACKS (even frames 324-410, odd frames 4103-4197, even frames
      These are expected behavior.  After every lost segment from (1), we enter
an out of order delivery mode until such time as the lost segment is
retransmitted.  during this time, we respond to each frame that does not satisfy
the missing segment bytes with an ACK indicating the last in-order segment we
recevied.  These are recorded as "duplicate acks" by wireshark (since they are),
but are specifcified by RFC 2581 (tcp congestion control) as correct.  Three
duplicate ACKs received by the other peer should trigger a Fast retransmit of
the missing data starting at the sequence number provided by the duplicate ACK.
 Unfortunately, the NFS server (I assume our Netapp) is ignoring the RFC
prescribed method of fast retransmit detection, in that fast retransmit should
begin after 3 duplicate ACKs are received and clearly it is waiting for dozens
of duplicate ACKs.  Not sure if that is caused by frames being dropped on the
network, or just being ignored by the Netapp, but we should definately find out,
since this looks to me like the most likely prospect in terms of what might be
causing a file corruption.

3) Out of Order packets (frames 414, 416, 418, 4200, 11695, 11697, 11699)
      These are also expected behavior.  The first of each of these out of order
sequences are the start of the Netapps fast retransmit algorithm, in which it
attempts to fill in the missing segment bytes that the duplicate acks of (2)
were indicating.  Each set of frames seems to correctly fill in the missing
sequence properly, although I would be interested to correlate the corrupt
segments of the transmitted rpm file to the retransmitted frames.  If there is a
correlation, it would suggest that, while the netapp is sending the retransmits
properly, it isn't providing the proper data in those frames.

So Moving forward, I think we should do two things.

A) See if we can figure out if the Netapp NFS server is just never seeing these
duplicate ACKs, or if it is choosing to ignore the recommendation of RFC 2581. 
If it is doing the latter, lets find out why, and what the impact of that is on
our client.

B) Do a binary diff of the corrupted RPM file with the good RPM file (bdiff and
vbindiff can do this), and see if we can correlate the corrupted segments with
the retransmitted data on the tcpdump.
Comment 21 Steve Dickson 2007-03-09 09:22:35 EST
Is this happening with more recent kernels?
Comment 22 Mike Gahagan 2007-03-09 15:43:00 EST
I haven't seen this problem in quite some time on the particular test box I
first saw the problem on. I suspect this might have been hardware all along. I
just did some nfs copies from curly along with a few runs of 'rpm -K' and can't
see any problems. I'm just going to close this assuming this was a hardware fault.

Note You need to log in before you can comment on or make changes to this bug.