Created attachment 937092 [details]
Script that executes racy mv on hardlinks

Description of problem:
Assuming a and b are files that are hard linked. Issuing an 'mv a b' ends up in sending an unlink on a. This per se is fine, but when the same races with another 'mv b a' it ends up sending unlink on a and unlink on b causing the file to be lost as both copies are unlinked.

Version-Release number of selected component (if applicable):
coreutils-8.4-31.el6_5.2.x86_64 (On RHEL 6.5)
coreutils-8.21-21.fc20.x86_64 (on Fedora 20)

How reproducible:
Using gdb always, as the race an be recreated (see below)

Using the attached script, 1-8 files every 2nd or 3rd run of the script.

Otherwise we hit this with GlusterFS that is mounted as a FUSE FS where it can be observed frequently, if say 1000 such files are being renamed from 2 shells on the same mount point.

Steps to Reproduce:
1. With to attached script (please run under a directory that is empty)

2. Using gdb as follows, (gdb output attached) (see ===> for comments)
NOTE: This is on Fedora 20

[GDB instance 1, doing a mv file file1]
Reading symbols from /bin/mv...Reading symbols from /usr/lib/debug/usr/bin/mv.debug...done.
done.
Breakpoint 1 at 0x4059a0: file src/copy.c, line 1619.
Starting program: /usr/bin/mv file file1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Breakpoint 1, copy_internal (src_name=src_name@entry=0x7fffffffe738 "file", dst_name=0x7fffffffe73d "file1", new_dst=new_dst@entry=false, device=device@entry=0, ancestors=ancestors@entry=0x0, x=x@entry=0x7fffffffe2d0, 
    command_line_arg=command_line_arg@entry=true, first_dir_created_per_command_line_arg=first_dir_created_per_command_line_arg@entry=0x7fffffffe1f7, copy_into_self=copy_into_self@entry=0x7fffffffe21e, 
    rename_succeeded=rename_succeeded@entry=0x7fffffffe21f) at src/copy.c:1619
1619	{
Missing separate debuginfos, use: debuginfo-install libacl-2.2.52-4.fc20.x86_64 libattr-2.4.47-3.fc20.x86_64 libselinux-2.2.1-6.fc20.x86_64 pcre-8.33-4.fc20.x86_64 xz-libs-5.1.2-9alpha.fc20.x86_64
1635	  if (x->move_mode && rename_succeeded)
1636	    *rename_succeeded = false;
1638	  *copy_into_self = false;
1640	  if (XSTAT (x, src_name, &src_sb) != 0)
1638	  *copy_into_self = false;
1640	  if (XSTAT (x, src_name, &src_sb) != 0)
1646	  src_mode = src_sb.st_mode;
1648	  if (S_ISDIR (src_mode) && !x->recursive)
1646	  src_mode = src_sb.st_mode;
1648	  if (S_ISDIR (src_mode) && !x->recursive)
1658	  if (command_line_arg)
1661	           && x->backup_type == no_backups
1662	           && seen_file (x->src_info, src_name, &src_sb))
1669	      record_file (x->src_info, src_name, &src_sb);
1672	  if (!new_dst)
1684	         && ! (x->move_mode || x->symbolic_link || x->hard_link
1689	           : lstat (dst_name, &dst_sb))
1690	          != 0)
1687	      if ((use_stat
1709	          if (! same_file_ok (src_name, &src_sb, dst_name, &dst_sb,
1708	          have_dst_lstat = !use_stat;
1709	          if (! same_file_ok (src_name, &src_sb, dst_name, &dst_sb,
1717	          if (!S_ISDIR (src_mode) && x->update)
1767	          if (x->move_mode)
1614	               const struct cp_options *x,
1769	              if (abandon_move (x, dst_name, &dst_sb)
1770	                  || (unlink_src && unlink (src_name) == 0))
===> stop here and let the other instance continue till it finishes
Continuing.
[Inferior 1 (process 23465) exited normally]

[GDB instance 2, doing a mv file1 file] (Actually this need not be run from gdb anyway)
Reading symbols from /bin/mv...Reading symbols from /usr/lib/debug/usr/bin/mv.debug...done.
done.
Breakpoint 1 at 0x4059a0: file src/copy.c, line 1619.
Starting program: /usr/bin/mv file1 file
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Breakpoint 1, copy_internal (src_name=src_name@entry=0x7fffffffe738 "file1", dst_name=0x7fffffffe73e "file", new_dst=new_dst@entry=false, device=device@entry=0, ancestors=ancestors@entry=0x0, x=x@entry=0x7fffffffe2d0, 
    command_line_arg=command_line_arg@entry=true, first_dir_created_per_command_line_arg=first_dir_created_per_command_line_arg@entry=0x7fffffffe1f7, copy_into_self=copy_into_self@entry=0x7fffffffe21e, 
    rename_succeeded=rename_succeeded@entry=0x7fffffffe21f) at src/copy.c:1619
1619	{
Missing separate debuginfos, use: debuginfo-install libacl-2.2.52-4.fc20.x86_64 libattr-2.4.47-3.fc20.x86_64 libselinux-2.2.1-6.fc20.x86_64 pcre-8.33-4.fc20.x86_64 xz-libs-5.1.2-9alpha.fc20.x86_64
Continuing.
[Inferior 1 (process 23475) exited normally]

3. With GlusterFS, but that is not advised :-)

Actual results:
Both instances of the file are unlinked

Expected results:
One instance should survive, if a rename was sent to the FS, one of them would have failed to find the source and hence failed that rename, while the other should have suceeded.

Maybe expected behaviour from a gluster point of view, is that we should recieve a rename request in this case and not an unlink as there is no way for the FS to determine this is actually a rename otherwise.

Additional info:
From coreutils code from Fedora 20 debug info repository (coreutils-debuginfo-8.21-21.fc20.x86_64) on reading the code, here is what I think happens,

#-> calling
do_move (mv.c) -> copy(copy.c) -> copy_internal(copy.c) -> same_file_ok(copy.c)

In small_file_ok we decide to set unlink_src to true, based on::
x->move_mode, same_link && 1 < dst_sb_link->st_nlink && ! same_name (src_name, dst_name)

Which in copy_internal decides to unlink the source file.

I am not commenting on the fix as I am not aware of other corner cases, but the cause is from the decisions as mentioned above.

The fix, if deemed the right thing to do, would probably need to make it to other relase branches as well, as from a Gluster perspective, any of the other releases could act as a client mounting the gluster volume.