Bug 802233

Description of problem: When add-brick was done to a distribute volume when dbench was running on the fuse client, the dbench process exited with ENOENT.


Version-Release number of selected component (if applicable): Upstream.


How reproducible: Consistently


Steps to Reproduce:
1. dbench -s 10
2. gluster volume add-brick test2 shortwing:/falcon/d3 shortwing:/falcon/d4
3.
  
Actual results:

dbench version 4.00 - Copyright Andrew Tridgell 1999-2004

Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs
7 of 10 processes prepared for launch   0 sec
10 of 10 processes prepared for launch   0 sec
releasing clients
  10        33    11.53 MB/sec  warmup   1 sec  latency 693.376 ms
  10        41     7.86 MB/sec  warmup   2 sec  latency 964.348 ms
  10        56     8.04 MB/sec  warmup   3 sec  latency 1354.727 ms
  10        65     7.08 MB/sec  warmup   4 sec  latency 1383.304 ms
  10        79     7.13 MB/sec  warmup   5 sec  latency 1486.787 ms
  10        90     6.99 MB/sec  warmup   6 sec  latency 1283.962 ms
  10        99     6.59 MB/sec  warmup   7 sec  latency 1214.622 ms
  10       108     6.32 MB/sec  warmup   8 sec  latency 1103.166 ms
  10       119     6.08 MB/sec  warmup   9 sec  latency 945.197 ms
  10       133     5.92 MB/sec  warmup  10 sec  latency 557.283 ms
  10       143     5.64 MB/sec  warmup  11 sec  latency 623.668 ms
  10       158     5.54 MB/sec  warmup  12 sec  latency 356.948 ms
  10       169     5.35 MB/sec  warmup  13 sec  latency 390.450 ms
  10       182     5.29 MB/sec  warmup  14 sec  latency 411.173 ms
[189] open ./clients/client2/~dmtmp/PWRPNT/TRIDOTS.POT failed for handle 9955 (No such file or directory)
(190) ERROR: handle 9955 was not found
Child failed with status 1
root@shortwing:/falcon/mnt# [197] open ./clients/client5/~dmtmp/COREL failed for handle 9956 (No such file or directory)
[199] open ./clients/client0/~dmtmp/COREL/CDRROLS.CFG failed for handle 9957 (No such file or directory)
[189] open ./clients/client6/~dmtmp/PWRPNT/TRIDOTS.POT failed for handle 9955 (No such file or directory)
[189] open ./clients/client8/~dmtmp/PWRPNT/TRIDOTS.POT failed for handle 9955 (No such file or directory)
[197] open ./clients/client7/~dmtmp/COREL failed for handle 9956 (No such file or directory)
[197] open ./clients/client4/~dmtmp/COREL failed for handle 9956 (No such file or directory)
[189] open ./clients/client9/~dmtmp/PWRPNT/TRIDOTS.POT failed for handle 9955 (No such file or directory)
[189] open ./clients/client3/~dmtmp/PWRPNT/TRIDOTS.POT failed for handle 9955 (No such file or directory


Expected results: The dbench process shouldn't exit


Additional info:

Fuse client log-

[2012-03-12 12:55:48.504279] E [fuse-bridge.c:330:fuse_lookup_resume] 0-fuse: failed to resolve path (null)
[2012-03-12 12:55:48.504718] D [dht-common.c:269:dht_discover_cbk] 1-test2-dht: lookup of <gfid:00000000-0000-0000-0000-000000000000> on test2-client-2 returned error (No such file or directory)
[2012-03-12 12:55:48.504830] D [dht-common.c:269:dht_discover_cbk] 1-test2-dht: lookup of <gfid:00000000-0000-0000-0000-000000000000> on test2-client-1 returned error (No such file or directory)
[2012-03-12 12:55:48.504854] D [dht-layout.c:596:dht_layout_normalize] 1-test2-dht: directory <gfid:00000000-0000-0000-0000-000000000000> looked up first time
[2012-03-12 12:55:48.504875] D [dht-layout.c:614:dht_layout_normalize] 1-test2-dht: path=<gfid:00000000-0000-0000-0000-000000000000> err=No such file or directory on subvol=test2-client-2
[2012-03-12 12:55:48.504910] D [dht-layout.c:614:dht_layout_normalize] 1-test2-dht: path=<gfid:00000000-0000-0000-0000-000000000000> err=No such file or directory on subvol=test2-client-1
[2012-03-12 12:55:48.504930] D [dht-common.c:197:dht_discover_complete] 1-test2-dht: normalizing failed on <gfid:00000000-0000-0000-0000-000000000000>
[2012-03-12 12:55:48.504958] W [fuse-resolve.c:150:fuse_resolve_gfid_cbk] 0-fuse: ac0dce4b-38c4-4585-a32f-e0b202938a27: failed to resolve (Invalid argument)