Created attachment 1298720 [details]
simple script trying to discover non-atomic move

Description of problem:
it is possible to see multiple intermediate state while moving a file over one that already exists even if both the source and the destination are on the same volume.

normally, on ext4, moving a file is an atomic operation where either the src file is present with uncorrupted readable content or the destination is present with uncorrupted readable content.

Version-Release number of selected component (if applicable):
3.10
mainline

How reproducible: 
100%

Steps to Reproduce:
1. run the attached script twice in parallel on the same folder of the same gluster volume

Actual results:
multiple different results are currently possible

for a volume with a single brick, I get this error
md5sum: ./test: Operation not permitted
non atomic move!!! actual md5:  vs expected 0f343b0931126a20f133d67c2b018a3b

for a volume with multiple bricks using tcp transport, I get another error
md5sum: ./test: No such file or directory
non atomic move!!! actual md5:  vs expected 0f343b0931126a20f133d67c2b018a3b

Expected results:
same as ext4, no errors should ever happen, once created, the file should always exist since its only overwritten... and its content should either be the old content, or the new content, but not some intermediate state.