Description of problem: When add-brick was done to a distribute volume when dbench was running on the fuse client, the dbench process exited with ENOENT. Version-Release number of selected component (if applicable): Upstream. How reproducible: Consistently Steps to Reproduce: 1. dbench -s 10 2. gluster volume add-brick test2 shortwing:/falcon/d3 shortwing:/falcon/d4 3. Actual results: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 7 of 10 processes prepared for launch 0 sec 10 of 10 processes prepared for launch 0 sec releasing clients 10 33 11.53 MB/sec warmup 1 sec latency 693.376 ms 10 41 7.86 MB/sec warmup 2 sec latency 964.348 ms 10 56 8.04 MB/sec warmup 3 sec latency 1354.727 ms 10 65 7.08 MB/sec warmup 4 sec latency 1383.304 ms 10 79 7.13 MB/sec warmup 5 sec latency 1486.787 ms 10 90 6.99 MB/sec warmup 6 sec latency 1283.962 ms 10 99 6.59 MB/sec warmup 7 sec latency 1214.622 ms 10 108 6.32 MB/sec warmup 8 sec latency 1103.166 ms 10 119 6.08 MB/sec warmup 9 sec latency 945.197 ms 10 133 5.92 MB/sec warmup 10 sec latency 557.283 ms 10 143 5.64 MB/sec warmup 11 sec latency 623.668 ms 10 158 5.54 MB/sec warmup 12 sec latency 356.948 ms 10 169 5.35 MB/sec warmup 13 sec latency 390.450 ms 10 182 5.29 MB/sec warmup 14 sec latency 411.173 ms [189] open ./clients/client2/~dmtmp/PWRPNT/TRIDOTS.POT failed for handle 9955 (No such file or directory) (190) ERROR: handle 9955 was not found Child failed with status 1 root@shortwing:/falcon/mnt# [197] open ./clients/client5/~dmtmp/COREL failed for handle 9956 (No such file or directory) [199] open ./clients/client0/~dmtmp/COREL/CDRROLS.CFG failed for handle 9957 (No such file or directory) [189] open ./clients/client6/~dmtmp/PWRPNT/TRIDOTS.POT failed for handle 9955 (No such file or directory) [189] open ./clients/client8/~dmtmp/PWRPNT/TRIDOTS.POT failed for handle 9955 (No such file or directory) [197] open ./clients/client7/~dmtmp/COREL failed for handle 9956 (No such file or directory) [197] open ./clients/client4/~dmtmp/COREL failed for handle 9956 (No such file or directory) [189] open ./clients/client9/~dmtmp/PWRPNT/TRIDOTS.POT failed for handle 9955 (No such file or directory) [189] open ./clients/client3/~dmtmp/PWRPNT/TRIDOTS.POT failed for handle 9955 (No such file or directory Expected results: The dbench process shouldn't exit Additional info: Fuse client log- [2012-03-12 12:55:48.504279] E [fuse-bridge.c:330:fuse_lookup_resume] 0-fuse: failed to resolve path (null) [2012-03-12 12:55:48.504718] D [dht-common.c:269:dht_discover_cbk] 1-test2-dht: lookup of <gfid:00000000-0000-0000-0000-000000000000> on test2-client-2 returned error (No such file or directory) [2012-03-12 12:55:48.504830] D [dht-common.c:269:dht_discover_cbk] 1-test2-dht: lookup of <gfid:00000000-0000-0000-0000-000000000000> on test2-client-1 returned error (No such file or directory) [2012-03-12 12:55:48.504854] D [dht-layout.c:596:dht_layout_normalize] 1-test2-dht: directory <gfid:00000000-0000-0000-0000-000000000000> looked up first time [2012-03-12 12:55:48.504875] D [dht-layout.c:614:dht_layout_normalize] 1-test2-dht: path=<gfid:00000000-0000-0000-0000-000000000000> err=No such file or directory on subvol=test2-client-2 [2012-03-12 12:55:48.504910] D [dht-layout.c:614:dht_layout_normalize] 1-test2-dht: path=<gfid:00000000-0000-0000-0000-000000000000> err=No such file or directory on subvol=test2-client-1 [2012-03-12 12:55:48.504930] D [dht-common.c:197:dht_discover_complete] 1-test2-dht: normalizing failed on <gfid:00000000-0000-0000-0000-000000000000> [2012-03-12 12:55:48.504958] W [fuse-resolve.c:150:fuse_resolve_gfid_cbk] 0-fuse: ac0dce4b-38c4-4585-a32f-e0b202938a27: failed to resolve (Invalid argument)
Check if its already fixed.
*** This bug has been marked as a duplicate of bug 803201 ***
bug #803201, is for a replicate volume. But, this issue is on a distribute volume, hence moving back to assigned state.
*** Bug 808054 has been marked as a duplicate of this bug. ***
*** Bug 803328 has been marked as a duplicate of this bug. ***
Patch http://review.gluster.com/#change,3069 isn't fixing the issue.
CHANGE: http://review.gluster.com/3069 (cluster/distribute: handle dht_layout_normalize errors differently in dht_lookup_dir_cbk and dht_discover_complete.) merged in master by Anand Avati (avati)
Anush, Is this bug fixed or do we still have it open?
*** Bug 808056 has been marked as a duplicate of this bug. ***
*** Bug 814661 has been marked as a duplicate of this bug. ***