I got the following error while running the dbench with 20 clients for 1000 secs. I first created a 2 node dht set-up, to which I added a one more brick using add brick. The after rebalance i replaced a brick using replace-brick. Then I got following error.

  20     28858     5.17 MB/sec  execute 342 sec  latency 149.024 ms
  20     28873     5.16 MB/sec  execute 343 sec  latency 152.643 ms
  20     28901     5.16 MB/sec  execute 344 sec  latency 185.746 ms
  20     28928     5.15 MB/sec  execute 345 sec  latency 182.520 ms
[28718] unlink ./clients/client15/~dmtmp/PM/PMC185.TMP failed (No such file or directory) - expected NT_STATUS_OK
ERROR: child 15 failed at line 28718
Child failed with status 1
root@tortuga:/tmp/mnt# [28744] open ./clients/client17/~dmtmp/PM/BDES1.PRN failed for handle 11729 (No such file or directory)
[28849] unlink ./clients/client4/~dmtmp/PM/PMD393.TMP failed (No such file or directory) - expected NT_STATUS_OK
ERROR: child 4 failed at line 28849
[28726] unlink ./clients/client3/~dmtmp/PM/PMC186.TMP failed (No such file or directory) - expected NT_STATUS_OK
ERROR: child 3 failed at line 28726

I observed above error in both distributed and distributed-replicated set-up. It also appears both in fuse and nfs mount.

I think you can simplify the test by using a distribute setup to run dbench and while it is ongoing, just use replace brick. The issue is with replace brick really. This might be a duplicate of 1812.

Moving severity to enhancement, as it is a known issue in rebalance.

*** This bug has been marked as a duplicate of bug 1812 ***