Bug 1424805

Description of problem:

I have a program that does, in this order, `open()`, `sendfile()`, `close()` syscalls to copy a file.

When I have both `storage.linux-aio on` and `cluster.use-compound-fops on`, there is a race condition that makes the `close()` return `errno = EINVAL`.

This is especially suspicious because `close()` cannot result in `EINVAL` according to `man 2 close`.

`EINVAL` is also returned for other operations besides `close()`, e.g. for `fsync()`.

Turning off either of `storage.linux-aio` or `cluster.use-compound-fops` fixes the problem, so I assume that the bug only exists if both are enabled.

Version-Release number of selected component (if applicable):

3.9.1

How reproducible:

Always

Steps to Reproduce:
1. Over a list of local files (you need multiple because it's a nondeterministic race)
2. Copy each file to a gluster volume by using `open + sendfile + close` (properly checking return code and errno  of close()).

Actual results:

After a few files, close will return -1 and errno will be set to 22 (EINVAL).

Expected results:

All files copy without problem.

Additional info:

I found this commit https://github.com/gluster/glusterfs/commit/88d772c05c45c467bfccebfc51f6a0e0ea9ca287
which seems to deal with a similar issue; skimming over it, it seems to suggest that somehow the close is issued before the write operation is done.