Bug 3758 - nfs client write bug
nfs client write bug
Status: CLOSED NOTABUG
Product: Red Hat Linux
Classification: Retired
Component: kernel (Show other bugs)
6.0
i386 Linux
high Severity high
: ---
: ---
Assigned To: David Lawrence
:
: 3807 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 1999-06-27 02:35 EDT by brm
Modified: 2008-05-01 11:37 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 1999-06-28 09:24:34 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description brm 1999-06-27 02:35:20 EDT
We have an intermittent NFS write bug, on Redhat 6.0 with
all the latest updates, both single-processor (with de4x5
ethernet) and 4-proc SMP i686 (with 3c59x).  During CVS
access and sometimes during assembly/linking, sections of
files are written out shifted up 1-3 characters with 0's
inserted on one end.  We're using a Sun as a file-server
(SunOS 5.5), and haven't observed the problem with an SGI
Irix fileserver (Irix 5.3), but I've used tcpdump to watch
the packets and find that the data is sent already
corrupted, so I'm pretty sure it's on the Linux end, and the
server is just affecting the packet ordering in a way that
produces the bug.  An example which often exhibits the
problem is running
    /usr/bin/as -V -Qy -o tmp.o formatted-local.s
with the input file which I've put in
    http://suif.stanford.edu/~brm/formatted-local.s
for your perusal.  The output of od -x on the output tmp.o
when correct, has the line
> 0110000 642e 5f2e 765f 5f74 3174 7332 6d69 6c70
and, when incorrect, the line
< 0110000 2e00 2e64 5f5f 7476 745f 3231 6973 706d
(corruption begins at byte 36865, with a 0 byte inserted and
211 bytes shifted down 1; the 212th byte is missing).  A
2.2.10 kernel still has the problem.
  I spent as much time as I can afford on this, using
tcpdump, strace, and feeble debugging of a 2.2.10 kernel
(which also has the problem, perhaps more consistently).
It appears that the problem results from the following
write: [from strace]
  _llseek(0x3, 0, 0x8ca1, 0xbfffef58, 0)  = 0
  write(3, "\0.symtab\0.strtab\0.shstrtab\0."..., 1075)=1075
which makes it down to the kernel intact, to the routine
generic_file_write [in /usr/src/linux/mm/filemap.c]
which writes the first part on one page (863 bytes) and
then copies the rest (212 bytes) and schedules it as an
asynchronous write.  Immediately following is a 52-byte
write to a different block
  _llseek(0x3, 0, 0, 0xbfffeef0, 0)       = 0
  write(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\1"..., 52) = 52
which gets handled as a synchronous write and either goes
before or after the 212-byte scheduled write.  Apparently if
that write gets sent before the async write, then the 212
byte block is corrupted.  Why, I can't figure out.

I have also experienced very similar symptoms in some CVS
repository file corruption recently, with 3 0 bytes inserted
and a section of the file shifted 3 bytes over.

I've been looking around, but can't find a mailing list or a
contact for the NFS driver.  Where are they located?
Thanks.

I feel a bit presumptuous giving this a "high" priority,
since apparently noone else has noticed it, but we have $40K
to spend on new computers and unless I fix this the pro-Sun
camp will get to spend it on Sparcs... :-)


------- Additional Comments From   06/27/99 16:40 -------
Our NFS filesystem is mounted (from a Sun Ultra) like:
  failaka:/sow/1/1 on /sow/1/1 type nfs
	(rw,nosuid,nodev,bg,addr=171.64.73.157)
and I have experimented today and have not been able to reproduce
the problem with a Linux file server.  So I suspect this may be hard
for anyone else to reproduce. :-(  Perhaps there's some regression
test of the NFS filesystem that someone could run that might evidence
the problem. :-)
Comment 1 Alan Cox 1999-06-27 18:35:59 EDT
This is a known bug in Solaris. You need to get the Solaris Errata or
move to Solaris 2.6 or higher. The sun patches should be on sunsolve.
The report you have is the classic offset by 1-3 bytes problem that is
the clear evidence of the bug.
Comment 2 Jay Turner 1999-06-28 09:24:59 EDT
Discarding bug as it is actually known problem with Solaris and not
problem with Red Hat Linux.
Comment 3 Jay Turner 1999-06-29 12:56:59 EDT
*** Bug 3807 has been marked as a duplicate of this bug. ***

When using Red Hat Linux 6.0 as an NFS client to a host
running Solaris 2.5.1, I've seen file corruption.  The
specific trigger is running ld to link executables -- the
output file is messed up with some regularity.  I do not
know which side of the NFS connection is to blame.  It could
be a bug in the Solaris server.  I have not yet seen the
problem with servers running Solaris 2.6.  When I look at
the bytes that are wrong it looks like some byte or word
swapping has occurred.

Here is some output of "cmp -l" showing some of the
differences between a good file and a corrupted one:

 36525  33   0
 36527   0  33
 36529   1   0
 36531   0   1
 36533   6   0
 36535   0   6
 36541 100   0
 36543   0 100
 36545 140   0
 36546  42   0
 36547   0 140
 36548   0  42
 36557  20   0
 36559   0  20
 36565  41   0
 36567   0  41
 36569  11   0
 36571   0  11
 36581 314   0
 36582 224   0
 36583   0 314
 36584   0 224
 36585 260   0
 36586  23   0
 36587   0 260
 36588   0  23
 36589  14   0
 36591   0  14
 36593   1   0
 36595   0   1
 36597   4   0
 36599   0   4
 36601  10   0
 36603   0  10
 36605  53   0
 36607   0  53


------- Additional Comments From   06/29/99 12:36 -------
We have a large amount of experience with Red Hat 5.1 and the bug was
definitely not present in that release.  It only appeared after the
6.0 upgrade.
Comment 4 Bernard DAUTREVAUX 2000-03-27 08:04:59 EST
I'm using RedHat-6.1 kernel and I have exactly the same kind of NFS bug, with
file systems mounted from a Solaris-2.6 PC; I've have it occasionnaly on gcc
compiles (as I usually compile to the local disk) but have it also on some of
my programs that write through NFS.

After some head scratching I discover it was linked with using fseek on an NFS
mounted file, where when seeking at say offset 8000 from the beginning of the
file and writing 2000 bytes (in a signed fwrite), I get one zero (but may be up
to three zeroes) inserted at 1000, then the first 191 bytes I wrote (see the
pattern: up to byte 8191) then the 192nd byte is omitted and the rest of my
write gets out correctly.

Suspecting a possible problem with stdio, I added a flush, then an fseek in 0
followed by a small fread there, then the correct fseek in 8000. if I then
wrote less than 193 bytes everything is OK; but if I write 2000 bytes, the byte
that should be written at offset 8192 is missing (IIRC) and the others are
shifted :-<

To work around this I finally have to fclose() the file and the fopen() it
again, always in binary mode but for update... then all seems to work OK for
more than one month now (although gcc compiles still fail occasionally).

The main point here is that I get these problem with several kernels (stock
6.0, stock 6.1, stock 6.1smp and 2.2.14-8smp) mounting an NFS file system from
a Solaris-2.6 PC, that work perfectly with RedHat-4.0, RedHat-5.2, hpUX-9.07,
hpUX-10.20, Solaris-2.5 (Sparc) and AIX-4.2... Even if its a problem with
Solaris, it's a pity that all the proprietary OSes are able to work OK but not
Linux!...

Note You need to log in before you can comment on or make changes to this bug.