OS: Redhat Advanced Server 3/Update2 with kernel 2.4.21-20.ELsmp System: 4 Opteron CPUs with 32GB RAM Our Developers wrote a short program to test asynchron and synchron msync calls The Result: the calls behave equal and return immediately We expected: async msync call return immediately and synchron calls after successfull sync. With the plain Linux-Kernels we see not this behavior, so we think this is some extension from redhat.
What are the details of the test program? In upstream 2.4.21 kernels, MS_ASYNC would wait for existing IO to complete if it found any IO in flight. The implication is that a single MS_ASYNC call will be asynchronous --- it will submit background IO but not wait for it --- but a subsequent call will be pseudo-synchronous, as it waits not for its own IO to complete, but for the *previous* call's IO to complete. So if you time MS_ASYNC in a loop, yes, you'll see it costing the same as MS_SYNC. This was upstream behaviour. It has changed for 2.6 kernels, but RHEL3 inherited that upstream behaviour for 2.4.21 and it is too late to change it now. As a Linux-specific workaround, calling msync() with flags == 0 (ie. no MS_SYNC or MS_ASYNC at all) will have a fully async effect even on 2.4.21-era kernels.
Incidentally, see also bug #117993 for a closely-related issue which is actually complaining about earlier kernels *not* submitting IO on MS_ASYNC.