*** Bug 2143997 has been marked as a duplicate of this bug. ***
From conversation with Steven - adding our observation during morning sync with client this morning. Based on the symptoms (hanging up on an inode, not finding the file by inode number, debug logging indicating it's unlinked) it appears an index is built at the beginning of mirroring, then when it reaches that file in the index in memory, it doesn't check whether the inode is still linked any more, and hangs up trying to delete that unlinked-but-possibly-held-open file? The log entries are complaining about an unlinked file at $HexInode and when we convert that hex to dec and find $/volume -inum $inode no file is returned (find hangs, never finding the file), but to ensure that the MDS is functioning correctly and serving requests we do a find on other files by inode (determined by ls -li) and it returns the result promptly.
on rsync as a workaround - called from the script or normally from the shell prompt, it stalls. David decided to strace to find where it was stalling... and lo and behold it is now working. Strace slows down the process, and by virtue of that we suspect it's not hammering the MDS as hard, thereby allowing it to proceed through.