Description of problem:

Our application is an RDF store running on a loose cluster of x86-64 FC5
machines. It uses mmap() for some potentially very large files. Until recently
everything worked as we expected, but upon upgrading to kernel-2.6.18-1.2200.fc5
we experienced a very frustrating data loss problem. Our investigation is
ongoing, but I think it's time to write this up as a bug.

We have found a way to reproduce this problem on our development system, but
unfortunately it requires importing several gigabytes of RDF (which takes an
hour or two) into our custom software.

The noticeable data loss (in such a large volume of data it is hard to be sure
if this is the only problem) is that our "header" structure in the first block
of an mmap'd file sometimes reverts to an earlier version after we munmap() and
close() the file. We added a timestamp to the header which is updated by our
software, and in our debug logs we see that the timestamp has some value (e.g.
1162918168) and then later when the same file is re-opened its timestamp has
rewound somehow (e.g. to 1162918165) and the other metadata in this header
structure has also reverted to some earlier no longer appropriate values.
Therefore we believe that a dirty block is somehow lost by the kernel, instead
of being written to disk.

Our application does a consistency check when re-opening the file so we notice
this error immediately.

If we revert the kernel on the cluster to kernel-2.6.17-1.2187_FC5 the previous
update, we are unable to reproduce this problem after many hours and many many
gigabytes of data imported. Indeed our production system is currently using
kernel-2.6.17-1.2187_FC5 to store well over a billion RDF triples.


Version-Release number of selected component (if applicable):
kernel-2.6.18-1.2200.fc5

How reproducible:
We can reliably reproduce this, but unfortunately we have failed to narrow it
down to a test case that can be uploaded to your Bugzilla.

Steps to Reproduce:
1. Reboot to kernel-2.6.18-1.2200.fc5
2. Run our test import of the US TIGER geographic database in RDF form
  
Actual results:
Somewhere during the import, usually by half way the store complains that
consistency checks have failed. The exact point where the data loss starts to
occur seems to vary somewhat, perhaps depending on memory pressure? Afterwards
logs show that data written to mmap'd memory and then msync'd and munmap'd seems
to have mysteriously vanished.

Expected results:
Import should run to completion, data written to mmap'd memory should appear
identical on disk.

Additional info:
Are there any known problems or suspected problems in the upstream Linux kernel
that could be related? It will probably be very hard to arrange for you to
reproduce this on a machine under Red Hat's control, for two reasons: Firstly
you'd need our software and a set of identical x86-64 machines to run it on, and
secondly you'd need the gigabytes of test data (or we would have to find some
way to generate fictitious data from a pseudo-random seed, verify that it
triggers the bug and then send you the generator and seed).

Since we don't yet have a support contract we can't expect you to go to such
lengths. I will update this bug if we discover anything further, if a future FC5
kernel release does or does not fix our symptoms, or if we are able to reduce
our testcase to something that can be uploaded to Bugzilla.

Maybe there is an existing Linux test suite for this sort of bug? If so please
point me to it.