+++ This bug was initially created as a clone of Bug #1409206 +++
Description of problem:
When a brick is killed in a replica and `dd` is run, we see a lot of fstats being sent over the network, with a small (but very real) reduction in write throughput.
Throughput and profile info on a random brick when all bricks are up:
==================================================================
root@tuxpad fuse_mnt$ dd if=/dev/zero of=FILE bs=1024 count=10240
10240+0 records in
10240+0 records out
10485760 bytes (10 MB) copied, 2.9847 s, 3.5 MB/s
Brick: 127.0.0.2:/home/ravi/bricks/brick1
-----------------------------------------
Cumulative Stats:
Block Size: 1024b+
No. of Reads: 0
No. of Writes: 10240
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 2 RELEASE
0.00 0.00 us 0.00 us 0.00 us 6 RELEASEDIR
0.01 30.50 us 27.00 us 34.00 us 2 INODELK
0.01 70.00 us 70.00 us 70.00 us 1 OPEN
0.01 35.50 us 19.00 us 52.00 us 2 FLUSH
0.03 96.50 us 83.00 us 110.00 us 2 GETXATTR
0.04 253.00 us 253.00 us 253.00 us 1 TRUNCATE
0.07 225.50 us 202.00 us 249.00 us 2 FXATTROP
0.15 153.17 us 47.00 us 656.00 us 6 STATFS
0.17 537.00 us 207.00 us 867.00 us 2 XATTROP
0.22 685.00 us 22.00 us 1348.00 us 2 FINODELK
0.62 255.67 us 104.00 us 928.00 us 15 LOOKUP
98.66 59.72 us 35.00 us 4772.00 us 10240 WRITE
Duration: 673 seconds
Data Read: 0 bytes
Data Written: 10485760 bytes
Throughput and profile info of one of the 'up' bricks when one brick is down
===========================================================================
0:root@tuxpad fuse_mnt$ dd if=/dev/zero of=FILE bs=1024 count=10240
10240+0 records in
10240+0 records out
10485760 bytes (10 MB) copied, 4.24494 s, 2.5 MB/s
Brick: 127.0.0.2:/home/ravi/bricks/brick1
-----------------------------------------
Cumulative Stats:
Block Size: 1024b+
No. of Reads: 0
No. of Writes: 10240
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 2 RELEASE
0.00 0.00 us 0.00 us 0.00 us 5 RELEASEDIR
0.01 98.00 us 98.00 us 98.00 us 1 OPEN
0.01 57.50 us 43.00 us 72.00 us 2 INODELK
0.01 126.00 us 126.00 us 126.00 us 1 GETXATTR
0.02 184.00 us 184.00 us 184.00 us 1 TRUNCATE
0.02 113.00 us 109.00 us 117.00 us 2 FXATTROP
0.02 122.00 us 16.00 us 228.00 us 2 FLUSH
0.02 132.00 us 38.00 us 226.00 us 2 FINODELK
0.08 418.00 us 283.00 us 553.00 us 2 XATTROP
0.21 763.00 us 122.00 us 1630.00 us 3 LOOKUP
41.23 44.83 us 36.00 us 490.00 us 10240 WRITE
58.38 63.47 us 46.00 us 888.00 us 10240 FSTAT
Duration: 75 seconds
Data Read: 0 bytes
Data Written: 10485760 bytes
--- Additional comment from Worker Ant on 2016-12-30 04:58:52 EST ---
REVIEW: http://review.gluster.org/16309 (afr: Avoid resetting event_gen when brick is always down) posted (#1) for review on master by Ravishankar N (ravishankar)
Hi Karan,
As discussed, please try out the test case (`dd`) in a replicate volume. We would need to see if there is significant performance drop in I/O throughput when one brick of a replica is down for the bug to be considered for 3.2.0.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://rhn.redhat.com/errata/RHSA-2017-0486.html