When the replication process creates an entry as part of self-heal, it uses the iatt buffer to send the create operation. Previously, entries were created regardless of iatt buffer validity. This resulted in an empty file with GFID xattr of all zeroes being created when an invalid iatt buffer was sent. Checks are now included to ensure that iatt buffer content is valid before creating the entry in this situation.
Description of problem:
Application is seeing very high load while accessing gluster volume.The load is mainly based on the response time from gluster for the application's requests.The moment load crosses a threshold level (70-75% in their calculation), the application is not able to serve the incoming requests faster as it has to wait longer for gluster's response.
Volume Name: dv_repo
Type: Distributed-Replicate
Volume ID: 5662a158-6768-45bf-8dda-263c80d5a674
Status: Started
Number of Bricks: 3 x 3 = 9
Transport-type: tcp
Bricks:
Brick1: Server1:/export/dv_repo
Brick2: Server2:/export/dv_repo
Brick3: Server3:/export/dv_repo
Brick4: Server4:/export/dv_repo
Brick5: Server5:/export/dv_repo
Brick6: Server6:/export/dv_repo
Brick7: Server7:/export/dv_repo
Brick8: Server8:/export/dv_repo
Brick9: Server9/export/dv_repo
Options Reconfigured:
cluster.self-heal-daemon: disable
changelog.changelog: on
geo-replication.ignore-pid-check: on
geo-replication.indexing: on
performance.readdir-ahead: on
storage.owner-uid: 8080
Profile Info
---------------
gluster volume profile dv_repo info incremental
Brick: Server3:/export/dv_repo
----------------------------------------
Interval 4 Stats:
Block Size: 128b+ 256b+ 512b+
No. of Reads: 0 2 6
No. of Writes: 1 1 5
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 5 6 23
No. of Writes: 10 1001 418
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 28 76 120
No. of Writes: 2593 1976 643
Block Size: 65536b+ 131072b+
No. of Reads: 312 7105
No. of Writes: 5320 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 1293 FORGET
0.00 0.00 us 0.00 us 0.00 us 669 RELEASE
0.00 0.00 us 0.00 us 0.00 us 9 RELEASEDIR
0.00 51.44 us 38.00 us 94.00 us 9 OPENDIR
0.00 120.62 us 55.00 us 361.00 us 16 SETXATTR
0.00 43.00 us 18.00 us 84.00 us 52 STAT
0.00 658.50 us 421.00 us 1036.00 us 4 MKDIR
0.00 62.90 us 22.00 us 248.00 us 48 INODELK
0.00 215.34 us 34.00 us 798.00 us 41 GETXATTR
0.01 24.72 us 9.00 us 113.00 us 669 FLUSH
0.02 42.17 us 17.00 us 208.00 us 1292 STATFS
0.07 48.19 us 14.00 us 653.00 us 4519 FSTAT
0.12 504.29 us 166.00 us 1729.00 us 669 CREATE
0.14 526.19 us 136.00 us 2471.00 us 762 UNLINK
0.14 165.72 us 25.00 us 19349.00 us 2532 XATTROP
0.18 34.13 us 10.00 us 735.00 us 15164 FINODELK
0.24 90.53 us 21.00 us 1046.00 us 7690 READ
0.32 93.14 us 20.00 us 31061.00 us 10012 FXATTROP
0.55 134.86 us 34.00 us 18012.00 us 11968 WRITE
0.87 3181.46 us 104.00 us 20044.00 us 798 FSYNC
21.51 150.90 us 33.00 us 34797.00 us 417326 LOOKUP
75.84 1687.11 us 8.00 us 41182533.00 us 131608 ENTRYLK
Duration: 229 seconds
Data Read: 970995797 bytes
Data Written: 751416802 bytes
Brick: Server2:/export/dv_repo
----------------------------------------
Interval 4 Stats:
Block Size: 128b+ 512b+ 1024b+
No. of Reads: 0 0 0
No. of Writes: 1 3 8
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 2 2 8
No. of Writes: 526 402 2559
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 16 27 55
No. of Writes: 1898 363 1661
Block Size: 131072b+
No. of Reads: 1815
No. of Writes: 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 425 FORGET
0.00 0.00 us 0.00 us 0.00 us 237 RELEASE
0.00 0.00 us 0.00 us 0.00 us 9 RELEASEDIR
0.00 60.50 us 30.00 us 103.00 us 4 GETXATTR
0.00 44.83 us 33.00 us 58.00 us 6 STAT
0.00 62.11 us 40.00 us 91.00 us 9 OPENDIR
0.00 125.25 us 46.00 us 176.00 us 12 SETXATTR
0.00 163.72 us 21.00 us 370.00 us 18 READDIRP
0.00 98.73 us 50.00 us 295.00 us 33 XATTROP
0.00 28.43 us 10.00 us 88.00 us 237 FLUSH
0.01 38.64 us 14.00 us 155.00 us 474 FINODELK
0.01 204.55 us 43.00 us 552.00 us 128 MKNOD
0.02 206.38 us 45.00 us 439.00 us 252 CREATE
0.02 41.05 us 14.00 us 214.00 us 1292 STATFS
0.02 49.75 us 14.00 us 348.00 us 1128 FSTAT
0.03 87.43 us 24.00 us 700.00 us 918 FXATTROP
0.05 461.27 us 63.00 us 1959.00 us 252 UNLINK
0.08 91.03 us 22.00 us 385.00 us 1931 READ
0.08 6398.33 us 21.00 us 189665.00 us 30 INODELK
0.29 92.59 us 34.00 us 587.00 us 7421 WRITE
1.13 98.11 us 20.00 us 3159.00 us 26806 MKDIR
16.03 86.20 us 22.00 us 4126.00 us 433826 LOOKUP
82.22 2909.57 us 8.00 us 38563527.00 us 65936 ENTRYLK
Duration: 229 seconds
Data Read: 245544394 bytes
Data Written: 302268158 bytes
Brick: Server8:/export/dv_repo
----------------------------------------
Interval 4 Stats:
Block Size: 128b+ 256b+ 512b+
No. of Reads: 0 0 0
No. of Writes: 2 3 7
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 3 5 10
No. of Writes: 21 1539 1923
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 16 18 46
No. of Writes: 11861 8706 140
Block Size: 65536b+ 131072b+
No. of Reads: 192 3220
No. of Writes: 27 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 760 FORGET
0.00 0.00 us 0.00 us 0.00 us 328 RELEASE
0.00 0.00 us 0.00 us 0.00 us 9 RELEASEDIR
0.00 180.50 us 175.00 us 186.00 us 2 OPEN
0.00 39.18 us 18.00 us 59.00 us 22 STAT
0.00 98.33 us 31.00 us 132.00 us 9 OPENDIR
0.00 258.25 us 220.00 us 310.00 us 4 MKDIR
0.01 114.38 us 60.00 us 183.00 us 13 SETXATTR
0.01 62.46 us 27.00 us 125.00 us 26 INODELK
0.01 195.88 us 25.00 us 694.00 us 16 READDIRP
0.03 131.35 us 53.00 us 388.00 us 62 XATTROP
0.03 169.16 us 45.00 us 382.00 us 51 GETXATTR
0.04 28.41 us 10.00 us 84.00 us 328 FLUSH
0.13 47.58 us 19.00 us 192.00 us 668 FINODELK
0.26 49.78 us 16.00 us 136.00 us 1292 STATFS
0.31 117.72 us 34.00 us 260.00 us 666 FXATTROP
0.37 284.81 us 150.00 us 12933.00 us 328 CREATE
0.47 58.31 us 15.00 us 1513.00 us 2046 FSTAT
0.65 475.47 us 148.00 us 9187.00 us 343 UNLINK
0.94 50.19 us 11.00 us 368.00 us 4710 ENTRYLK
1.59 113.93 us 23.00 us 1890.00 us 3513 READ
7.14 74.13 us 30.00 us 550.00 us 24228 WRITE
88.00 165.02 us 33.00 us 32511.00 us 134156 LOOKUP
Duration: 229 seconds
Data Read: 443973556 bytes
Data Written: 371097376 bytes
Brick: Server6:/export/dv_repo
----------------------------------------
Interval 4 Stats:
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 0 0
No. of Writes: 1 2 2
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 0 0
No. of Writes: 2 1 13
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 0 1 7
No. of Writes: 17 39 3514
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 8 24 40
No. of Writes: 5045 26970 18290
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 109 211 5474
No. of Writes: 198 37 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 1290 FORGET
0.00 0.00 us 0.00 us 0.00 us 708 RELEASE
0.00 0.00 us 0.00 us 0.00 us 9 RELEASEDIR
0.00 84.00 us 70.00 us 98.00 us 2 OPEN
0.00 61.75 us 27.00 us 85.00 us 4 GETXATTR
0.00 69.89 us 43.00 us 101.00 us 9 OPENDIR
0.01 95.54 us 51.00 us 200.00 us 13 SETXATTR
0.01 44.52 us 14.00 us 87.00 us 31 STAT
0.01 48.50 us 23.00 us 99.00 us 32 INODELK
0.01 768.00 us 638.00 us 901.00 us 4 MKDIR
0.08 25.62 us 9.00 us 104.00 us 708 FLUSH
0.26 46.12 us 16.00 us 156.00 us 1292 STATFS
0.26 42.50 us 17.00 us 136.00 us 1412 FINODELK
0.54 44.83 us 12.00 us 181.00 us 2802 ENTRYLK
0.63 101.65 us 24.00 us 339.00 us 1428 FXATTROP
0.76 51.68 us 12.00 us 275.00 us 3379 FSTAT
1.42 473.12 us 128.00 us 2420.00 us 691 UNLINK
1.48 485.07 us 172.00 us 1078.00 us 706 CREATE
2.55 100.28 us 21.00 us 2085.00 us 5876 READ
16.61 70.79 us 27.00 us 2839.00 us 54131 WRITE
75.38 134.80 us 34.00 us 17308.00 us 129034 LOOKUP
Duration: 229 seconds
Data Read: 746041663 bytes
Data Written: 807564460 bytes
Brick: Server5/export/dv_repo
----------------------------------------
Interval 4 Stats:
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 0 0
No. of Writes: 1 2 2
Block Size: 64b+ 128b+ 256b+
No. of Reads: 1 0 1
No. of Writes: 2 1 13
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 2 4 4
No. of Writes: 17 39 3514
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 8 32 68
No. of Writes: 5045 26970 18290
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 86 229 5951
No. of Writes: 198 37 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 1290 FORGET
0.00 0.00 us 0.00 us 0.00 us 708 RELEASE
0.00 0.00 us 0.00 us 0.00 us 9 RELEASEDIR
0.00 112.50 us 109.00 us 116.00 us 2 OPEN
0.00 58.25 us 46.00 us 67.00 us 4 GETXATTR
0.00 87.22 us 41.00 us 130.00 us 9 OPENDIR
0.00 264.75 us 205.00 us 337.00 us 4 MKDIR
0.00 44.43 us 14.00 us 115.00 us 28 STAT
0.01 111.38 us 65.00 us 174.00 us 13 SETXATTR
0.01 54.50 us 25.00 us 139.00 us 32 INODELK
0.02 199.23 us 26.00 us 1125.00 us 22 READDIRP
0.07 26.31 us 9.00 us 122.00 us 708 FLUSH
0.24 47.20 us 16.00 us 138.00 us 1292 STATFS
0.25 45.51 us 18.00 us 206.00 us 1412 FINODELK
0.52 47.01 us 16.00 us 320.00 us 2802 ENTRYLK
0.61 109.87 us 26.00 us 229.00 us 1428 FXATTROP
0.68 246.35 us 132.00 us 3819.00 us 706 CREATE
0.77 53.48 us 11.00 us 198.00 us 3672 FSTAT
1.33 491.62 us 137.00 us 5182.00 us 691 UNLINK
2.58 103.18 us 20.00 us 2865.00 us 6388 READ
15.45 72.84 us 26.00 us 1442.00 us 54131 WRITE
77.45 153.20 us 34.00 us 104853.00 us 129033 LOOKUP
Duration: 229 seconds
Data Read: 809567845 bytes
Data Written: 807564460 bytes
Brick: Server9:/export/dv_repo
----------------------------------------
Interval 1 Stats:
Block Size: 128b+ 256b+ 512b+
No. of Reads: 0 1 1
No. of Writes: 2 3 7
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 0 2 8
No. of Writes: 21 1540 1923
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 7 26 35
No. of Writes: 11861 8707 140
Block Size: 65536b+ 131072b+
No. of Reads: 132 2331
No. of Writes: 27 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 728 FORGET
0.00 0.00 us 0.00 us 0.00 us 328 RELEASE
0.00 0.00 us 0.00 us 0.00 us 9 RELEASEDIR
0.00 124.00 us 115.00 us 133.00 us 2 OPEN
0.00 69.75 us 51.00 us 93.00 us 4 GETXATTR
0.00 88.00 us 47.00 us 145.00 us 9 OPENDIR
0.00 45.57 us 21.00 us 90.00 us 21 STAT
0.00 91.82 us 50.00 us 162.00 us 11 XATTROP
0.01 106.15 us 52.00 us 136.00 us 13 SETXATTR
0.01 58.54 us 26.00 us 124.00 us 26 INODELK
0.03 26.11 us 9.00 us 332.00 us 328 FLUSH
0.05 2926.75 us 378.00 us 5261.00 us 4 MKDIR
0.13 48.64 us 14.00 us 277.00 us 670 FINODELK
0.25 48.96 us 19.00 us 1462.00 us 1292 STATFS
0.32 1585.08 us 207.00 us 4095.00 us 51 MKNOD
0.33 123.49 us 55.00 us 839.00 us 666 FXATTROP
0.34 55.83 us 15.00 us 329.00 us 1511 FSTAT
0.77 565.10 us 123.00 us 9516.00 us 343 UNLINK
0.96 51.25 us 10.00 us 3426.00 us 4710 ENTRYLK
1.11 109.68 us 24.00 us 1135.00 us 2554 READ
2.01 1541.15 us 165.00 us 4827.00 us 328 CREATE
7.11 73.78 us 29.00 us 4718.00 us 24231 WRITE
86.57 162.33 us 32.00 us 22330.00 us 134159 LOOKUP
Duration: 229 seconds
Data Read: 321019450 bytes
Data Written: 371119591 bytes
Brick: Server1:/export/dv_repo
----------------------------------------
Interval 4 Stats:
Block Size: 128b+ 256b+ 512b+
No. of Reads: 1 4 10
No. of Writes: 1 1 5
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 3 16 25
No. of Writes: 10 1001 418
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 52 117 193
No. of Writes: 2593 1976 643
Block Size: 65536b+ 131072b+
No. of Reads: 531 11145
No. of Writes: 5320 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 1233 FORGET
0.00 0.00 us 0.00 us 0.00 us 669 RELEASE
0.00 0.00 us 0.00 us 0.00 us 9 RELEASEDIR
0.00 354.00 us 290.00 us 527.00 us 4 MKDIR
0.00 266.60 us 16.00 us 7646.00 us 105 STAT
0.00 3322.67 us 62.00 us 11074.00 us 9 OPENDIR
0.00 214.21 us 9.00 us 27224.00 us 669 FLUSH
0.00 74751.00 us 41.00 us 445195.00 us 6 READDIRP
0.00 482.80 us 22.00 us 74947.00 us 1292 STATFS
0.00 1240.61 us 198.00 us 29810.00 us 669 CREATE
0.00 1342.66 us 168.00 us 41128.00 us 762 UNLINK
0.00 292.44 us 12.00 us 44091.00 us 6578 FSTAT
0.00 200.80 us 10.00 us 42864.00 us 18898 FINODELK
0.00 362.70 us 22.00 us 36913.00 us 12102 READ
0.00 6783.40 us 110.00 us 61766.00 us 798 FSYNC
0.00 625.49 us 46.00 us 42693.00 us 11968 WRITE
0.00 965.13 us 27.00 us 64134.00 us 10012 FXATTROP
0.01 4200339.38 us 77.00 us 41078178.00 us 16 SETXATTR
0.02 1998934.33 us 28.00 us 41365012.00 us 85 GETXATTR
0.07 14500594.56 us 30.00 us 235894105.00 us 48 INODELK
0.07 274987.36 us 39.00 us 41596014.00 us 2532 XATTROP
46.33 1079715.51 us 52.00 us 41770605.00 us 417268 LOOKUP
53.50 3980262.68 us 11.00 us 472355943.00 us 130686 ENTRYLK
Duration: 229 seconds
Data Read: 1527648224 bytes
Data Written: 751416802 bytes
Brick: Server4:/export/dv_repo
----------------------------------------
Interval 4 Stats:
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 0 0
No. of Writes: 1 2 2
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 1 0
No. of Writes: 2 1 13
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 5 4 6
No. of Writes: 17 39 3514
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 11 17 54
No. of Writes: 5045 26970 18290
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 97 281 6181
No. of Writes: 198 37 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 1295 FORGET
0.00 0.00 us 0.00 us 0.00 us 708 RELEASE
0.00 0.00 us 0.00 us 0.00 us 9 RELEASEDIR
0.00 120.50 us 88.00 us 153.00 us 2 OPEN
0.00 90.89 us 60.00 us 141.00 us 9 OPENDIR
0.00 482.00 us 53.00 us 911.00 us 2 READDIRP
0.00 286.25 us 237.00 us 334.00 us 4 MKDIR
0.00 96.15 us 53.00 us 172.00 us 13 SETXATTR
0.01 45.62 us 18.00 us 101.00 us 29 STAT
0.01 55.53 us 26.00 us 111.00 us 32 INODELK
0.06 22.24 us 8.00 us 83.00 us 708 FLUSH
0.08 178.77 us 30.00 us 409.00 us 115 GETXATTR
0.22 39.85 us 15.00 us 178.00 us 1412 FINODELK
0.24 46.76 us 16.00 us 139.00 us 1292 STATFS
0.46 41.61 us 15.00 us 171.00 us 2802 ENTRYLK
0.57 101.40 us 26.00 us 217.00 us 1428 FXATTROP
0.70 252.43 us 160.00 us 513.00 us 706 CREATE
0.79 52.28 us 12.00 us 200.00 us 3843 FSTAT
1.27 467.65 us 136.00 us 5732.00 us 691 UNLINK
2.71 103.23 us 21.00 us 9713.00 us 6660 READ
13.90 65.20 us 28.00 us 3065.00 us 54131 WRITE
78.96 155.36 us 36.00 us 87299.00 us 129041 LOOKUP
Duration: 229 seconds
Data Read: 845204064 bytes
Data Written: 807564460 bytes
Brick: Server7:/export/dv_repo
----------------------------------------
Interval 4 Stats:
Block Size: 128b+ 256b+ 512b+
No. of Reads: 0 2 0
No. of Writes: 2 3 7
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 5 4 4
No. of Writes: 21 1539 1923
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 17 35 66
No. of Writes: 11863 8708 140
Block Size: 65536b+ 131072b+
No. of Reads: 199 3942
No. of Writes: 25 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 774 FORGET
0.00 0.00 us 0.00 us 0.00 us 327 RELEASE
0.00 0.00 us 0.00 us 0.00 us 9 RELEASEDIR
0.00 99.50 us 62.00 us 137.00 us 2 OPEN
0.00 236.00 us 29.00 us 443.00 us 2 READDIRP
0.00 42.18 us 21.00 us 71.00 us 22 STAT
0.00 281.00 us 255.00 us 311.00 us 4 MKDIR
0.01 100.62 us 67.00 us 180.00 us 13 SETXATTR
0.01 60.27 us 29.00 us 126.00 us 26 INODELK
0.01 207.33 us 35.00 us 1171.00 us 9 OPENDIR
0.02 107.02 us 65.00 us 269.00 us 51 XATTROP
0.04 26.83 us 8.00 us 105.00 us 327 FLUSH
0.05 174.41 us 35.00 us 447.00 us 75 GETXATTR
0.12 44.34 us 18.00 us 123.00 us 652 FINODELK
0.25 47.97 us 16.00 us 229.00 us 1292 STATFS
0.32 117.19 us 39.00 us 270.00 us 658 FXATTROP
0.36 269.22 us 166.00 us 1256.00 us 327 CREATE
0.53 45.34 us 11.00 us 480.00 us 2836 ENTRYLK
0.55 54.27 us 12.00 us 783.00 us 2482 FSTAT
0.77 553.87 us 149.00 us 12336.00 us 339 UNLINK
1.90 108.45 us 22.00 us 1621.00 us 4278 READ
7.03 70.74 us 28.00 us 1578.00 us 24231 WRITE
88.02 159.88 us 30.00 us 57080.00 us 134187 LOOKUP
Duration: 229 seconds
Data Read: 541364650 bytes
Data Written: 370965933 bytes
storage.owner-gid: 8080
diagnostics.latency-measurement: on
diagnostics.count-fop-hits: on
performance.low-prio-threads: 64
performance.normal-prio-threads: 64
performance.high-prio-threads: 64
performance.io-thread-count: 64
server.outstanding-rpc-limit: 128
network.inode-lru-limit: 32000
server.event-threads: 8
cluster.entry-self-heal: off
cluster.metadata-self-heal: off
cluster.data-self-heal: off
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2017:2730
Description of problem: Application is seeing very high load while accessing gluster volume.The load is mainly based on the response time from gluster for the application's requests.The moment load crosses a threshold level (70-75% in their calculation), the application is not able to serve the incoming requests faster as it has to wait longer for gluster's response. Volume Name: dv_repo Type: Distributed-Replicate Volume ID: 5662a158-6768-45bf-8dda-263c80d5a674 Status: Started Number of Bricks: 3 x 3 = 9 Transport-type: tcp Bricks: Brick1: Server1:/export/dv_repo Brick2: Server2:/export/dv_repo Brick3: Server3:/export/dv_repo Brick4: Server4:/export/dv_repo Brick5: Server5:/export/dv_repo Brick6: Server6:/export/dv_repo Brick7: Server7:/export/dv_repo Brick8: Server8:/export/dv_repo Brick9: Server9/export/dv_repo Options Reconfigured: cluster.self-heal-daemon: disable changelog.changelog: on geo-replication.ignore-pid-check: on geo-replication.indexing: on performance.readdir-ahead: on storage.owner-uid: 8080 Profile Info --------------- gluster volume profile dv_repo info incremental Brick: Server3:/export/dv_repo ---------------------------------------- Interval 4 Stats: Block Size: 128b+ 256b+ 512b+ No. of Reads: 0 2 6 No. of Writes: 1 1 5 Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 5 6 23 No. of Writes: 10 1001 418 Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 28 76 120 No. of Writes: 2593 1976 643 Block Size: 65536b+ 131072b+ No. of Reads: 312 7105 No. of Writes: 5320 0 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 1293 FORGET 0.00 0.00 us 0.00 us 0.00 us 669 RELEASE 0.00 0.00 us 0.00 us 0.00 us 9 RELEASEDIR 0.00 51.44 us 38.00 us 94.00 us 9 OPENDIR 0.00 120.62 us 55.00 us 361.00 us 16 SETXATTR 0.00 43.00 us 18.00 us 84.00 us 52 STAT 0.00 658.50 us 421.00 us 1036.00 us 4 MKDIR 0.00 62.90 us 22.00 us 248.00 us 48 INODELK 0.00 215.34 us 34.00 us 798.00 us 41 GETXATTR 0.01 24.72 us 9.00 us 113.00 us 669 FLUSH 0.02 42.17 us 17.00 us 208.00 us 1292 STATFS 0.07 48.19 us 14.00 us 653.00 us 4519 FSTAT 0.12 504.29 us 166.00 us 1729.00 us 669 CREATE 0.14 526.19 us 136.00 us 2471.00 us 762 UNLINK 0.14 165.72 us 25.00 us 19349.00 us 2532 XATTROP 0.18 34.13 us 10.00 us 735.00 us 15164 FINODELK 0.24 90.53 us 21.00 us 1046.00 us 7690 READ 0.32 93.14 us 20.00 us 31061.00 us 10012 FXATTROP 0.55 134.86 us 34.00 us 18012.00 us 11968 WRITE 0.87 3181.46 us 104.00 us 20044.00 us 798 FSYNC 21.51 150.90 us 33.00 us 34797.00 us 417326 LOOKUP 75.84 1687.11 us 8.00 us 41182533.00 us 131608 ENTRYLK Duration: 229 seconds Data Read: 970995797 bytes Data Written: 751416802 bytes Brick: Server2:/export/dv_repo ---------------------------------------- Interval 4 Stats: Block Size: 128b+ 512b+ 1024b+ No. of Reads: 0 0 0 No. of Writes: 1 3 8 Block Size: 2048b+ 4096b+ 8192b+ No. of Reads: 2 2 8 No. of Writes: 526 402 2559 Block Size: 16384b+ 32768b+ 65536b+ No. of Reads: 16 27 55 No. of Writes: 1898 363 1661 Block Size: 131072b+ No. of Reads: 1815 No. of Writes: 0 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 425 FORGET 0.00 0.00 us 0.00 us 0.00 us 237 RELEASE 0.00 0.00 us 0.00 us 0.00 us 9 RELEASEDIR 0.00 60.50 us 30.00 us 103.00 us 4 GETXATTR 0.00 44.83 us 33.00 us 58.00 us 6 STAT 0.00 62.11 us 40.00 us 91.00 us 9 OPENDIR 0.00 125.25 us 46.00 us 176.00 us 12 SETXATTR 0.00 163.72 us 21.00 us 370.00 us 18 READDIRP 0.00 98.73 us 50.00 us 295.00 us 33 XATTROP 0.00 28.43 us 10.00 us 88.00 us 237 FLUSH 0.01 38.64 us 14.00 us 155.00 us 474 FINODELK 0.01 204.55 us 43.00 us 552.00 us 128 MKNOD 0.02 206.38 us 45.00 us 439.00 us 252 CREATE 0.02 41.05 us 14.00 us 214.00 us 1292 STATFS 0.02 49.75 us 14.00 us 348.00 us 1128 FSTAT 0.03 87.43 us 24.00 us 700.00 us 918 FXATTROP 0.05 461.27 us 63.00 us 1959.00 us 252 UNLINK 0.08 91.03 us 22.00 us 385.00 us 1931 READ 0.08 6398.33 us 21.00 us 189665.00 us 30 INODELK 0.29 92.59 us 34.00 us 587.00 us 7421 WRITE 1.13 98.11 us 20.00 us 3159.00 us 26806 MKDIR 16.03 86.20 us 22.00 us 4126.00 us 433826 LOOKUP 82.22 2909.57 us 8.00 us 38563527.00 us 65936 ENTRYLK Duration: 229 seconds Data Read: 245544394 bytes Data Written: 302268158 bytes Brick: Server8:/export/dv_repo ---------------------------------------- Interval 4 Stats: Block Size: 128b+ 256b+ 512b+ No. of Reads: 0 0 0 No. of Writes: 2 3 7 Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 3 5 10 No. of Writes: 21 1539 1923 Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 16 18 46 No. of Writes: 11861 8706 140 Block Size: 65536b+ 131072b+ No. of Reads: 192 3220 No. of Writes: 27 0 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 760 FORGET 0.00 0.00 us 0.00 us 0.00 us 328 RELEASE 0.00 0.00 us 0.00 us 0.00 us 9 RELEASEDIR 0.00 180.50 us 175.00 us 186.00 us 2 OPEN 0.00 39.18 us 18.00 us 59.00 us 22 STAT 0.00 98.33 us 31.00 us 132.00 us 9 OPENDIR 0.00 258.25 us 220.00 us 310.00 us 4 MKDIR 0.01 114.38 us 60.00 us 183.00 us 13 SETXATTR 0.01 62.46 us 27.00 us 125.00 us 26 INODELK 0.01 195.88 us 25.00 us 694.00 us 16 READDIRP 0.03 131.35 us 53.00 us 388.00 us 62 XATTROP 0.03 169.16 us 45.00 us 382.00 us 51 GETXATTR 0.04 28.41 us 10.00 us 84.00 us 328 FLUSH 0.13 47.58 us 19.00 us 192.00 us 668 FINODELK 0.26 49.78 us 16.00 us 136.00 us 1292 STATFS 0.31 117.72 us 34.00 us 260.00 us 666 FXATTROP 0.37 284.81 us 150.00 us 12933.00 us 328 CREATE 0.47 58.31 us 15.00 us 1513.00 us 2046 FSTAT 0.65 475.47 us 148.00 us 9187.00 us 343 UNLINK 0.94 50.19 us 11.00 us 368.00 us 4710 ENTRYLK 1.59 113.93 us 23.00 us 1890.00 us 3513 READ 7.14 74.13 us 30.00 us 550.00 us 24228 WRITE 88.00 165.02 us 33.00 us 32511.00 us 134156 LOOKUP Duration: 229 seconds Data Read: 443973556 bytes Data Written: 371097376 bytes Brick: Server6:/export/dv_repo ---------------------------------------- Interval 4 Stats: Block Size: 8b+ 16b+ 32b+ No. of Reads: 0 0 0 No. of Writes: 1 2 2 Block Size: 64b+ 128b+ 256b+ No. of Reads: 0 0 0 No. of Writes: 2 1 13 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 0 1 7 No. of Writes: 17 39 3514 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 8 24 40 No. of Writes: 5045 26970 18290 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 109 211 5474 No. of Writes: 198 37 0 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 1290 FORGET 0.00 0.00 us 0.00 us 0.00 us 708 RELEASE 0.00 0.00 us 0.00 us 0.00 us 9 RELEASEDIR 0.00 84.00 us 70.00 us 98.00 us 2 OPEN 0.00 61.75 us 27.00 us 85.00 us 4 GETXATTR 0.00 69.89 us 43.00 us 101.00 us 9 OPENDIR 0.01 95.54 us 51.00 us 200.00 us 13 SETXATTR 0.01 44.52 us 14.00 us 87.00 us 31 STAT 0.01 48.50 us 23.00 us 99.00 us 32 INODELK 0.01 768.00 us 638.00 us 901.00 us 4 MKDIR 0.08 25.62 us 9.00 us 104.00 us 708 FLUSH 0.26 46.12 us 16.00 us 156.00 us 1292 STATFS 0.26 42.50 us 17.00 us 136.00 us 1412 FINODELK 0.54 44.83 us 12.00 us 181.00 us 2802 ENTRYLK 0.63 101.65 us 24.00 us 339.00 us 1428 FXATTROP 0.76 51.68 us 12.00 us 275.00 us 3379 FSTAT 1.42 473.12 us 128.00 us 2420.00 us 691 UNLINK 1.48 485.07 us 172.00 us 1078.00 us 706 CREATE 2.55 100.28 us 21.00 us 2085.00 us 5876 READ 16.61 70.79 us 27.00 us 2839.00 us 54131 WRITE 75.38 134.80 us 34.00 us 17308.00 us 129034 LOOKUP Duration: 229 seconds Data Read: 746041663 bytes Data Written: 807564460 bytes Brick: Server5/export/dv_repo ---------------------------------------- Interval 4 Stats: Block Size: 8b+ 16b+ 32b+ No. of Reads: 0 0 0 No. of Writes: 1 2 2 Block Size: 64b+ 128b+ 256b+ No. of Reads: 1 0 1 No. of Writes: 2 1 13 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 2 4 4 No. of Writes: 17 39 3514 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 8 32 68 No. of Writes: 5045 26970 18290 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 86 229 5951 No. of Writes: 198 37 0 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 1290 FORGET 0.00 0.00 us 0.00 us 0.00 us 708 RELEASE 0.00 0.00 us 0.00 us 0.00 us 9 RELEASEDIR 0.00 112.50 us 109.00 us 116.00 us 2 OPEN 0.00 58.25 us 46.00 us 67.00 us 4 GETXATTR 0.00 87.22 us 41.00 us 130.00 us 9 OPENDIR 0.00 264.75 us 205.00 us 337.00 us 4 MKDIR 0.00 44.43 us 14.00 us 115.00 us 28 STAT 0.01 111.38 us 65.00 us 174.00 us 13 SETXATTR 0.01 54.50 us 25.00 us 139.00 us 32 INODELK 0.02 199.23 us 26.00 us 1125.00 us 22 READDIRP 0.07 26.31 us 9.00 us 122.00 us 708 FLUSH 0.24 47.20 us 16.00 us 138.00 us 1292 STATFS 0.25 45.51 us 18.00 us 206.00 us 1412 FINODELK 0.52 47.01 us 16.00 us 320.00 us 2802 ENTRYLK 0.61 109.87 us 26.00 us 229.00 us 1428 FXATTROP 0.68 246.35 us 132.00 us 3819.00 us 706 CREATE 0.77 53.48 us 11.00 us 198.00 us 3672 FSTAT 1.33 491.62 us 137.00 us 5182.00 us 691 UNLINK 2.58 103.18 us 20.00 us 2865.00 us 6388 READ 15.45 72.84 us 26.00 us 1442.00 us 54131 WRITE 77.45 153.20 us 34.00 us 104853.00 us 129033 LOOKUP Duration: 229 seconds Data Read: 809567845 bytes Data Written: 807564460 bytes Brick: Server9:/export/dv_repo ---------------------------------------- Interval 1 Stats: Block Size: 128b+ 256b+ 512b+ No. of Reads: 0 1 1 No. of Writes: 2 3 7 Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 0 2 8 No. of Writes: 21 1540 1923 Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 7 26 35 No. of Writes: 11861 8707 140 Block Size: 65536b+ 131072b+ No. of Reads: 132 2331 No. of Writes: 27 0 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 728 FORGET 0.00 0.00 us 0.00 us 0.00 us 328 RELEASE 0.00 0.00 us 0.00 us 0.00 us 9 RELEASEDIR 0.00 124.00 us 115.00 us 133.00 us 2 OPEN 0.00 69.75 us 51.00 us 93.00 us 4 GETXATTR 0.00 88.00 us 47.00 us 145.00 us 9 OPENDIR 0.00 45.57 us 21.00 us 90.00 us 21 STAT 0.00 91.82 us 50.00 us 162.00 us 11 XATTROP 0.01 106.15 us 52.00 us 136.00 us 13 SETXATTR 0.01 58.54 us 26.00 us 124.00 us 26 INODELK 0.03 26.11 us 9.00 us 332.00 us 328 FLUSH 0.05 2926.75 us 378.00 us 5261.00 us 4 MKDIR 0.13 48.64 us 14.00 us 277.00 us 670 FINODELK 0.25 48.96 us 19.00 us 1462.00 us 1292 STATFS 0.32 1585.08 us 207.00 us 4095.00 us 51 MKNOD 0.33 123.49 us 55.00 us 839.00 us 666 FXATTROP 0.34 55.83 us 15.00 us 329.00 us 1511 FSTAT 0.77 565.10 us 123.00 us 9516.00 us 343 UNLINK 0.96 51.25 us 10.00 us 3426.00 us 4710 ENTRYLK 1.11 109.68 us 24.00 us 1135.00 us 2554 READ 2.01 1541.15 us 165.00 us 4827.00 us 328 CREATE 7.11 73.78 us 29.00 us 4718.00 us 24231 WRITE 86.57 162.33 us 32.00 us 22330.00 us 134159 LOOKUP Duration: 229 seconds Data Read: 321019450 bytes Data Written: 371119591 bytes Brick: Server1:/export/dv_repo ---------------------------------------- Interval 4 Stats: Block Size: 128b+ 256b+ 512b+ No. of Reads: 1 4 10 No. of Writes: 1 1 5 Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 3 16 25 No. of Writes: 10 1001 418 Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 52 117 193 No. of Writes: 2593 1976 643 Block Size: 65536b+ 131072b+ No. of Reads: 531 11145 No. of Writes: 5320 0 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 1233 FORGET 0.00 0.00 us 0.00 us 0.00 us 669 RELEASE 0.00 0.00 us 0.00 us 0.00 us 9 RELEASEDIR 0.00 354.00 us 290.00 us 527.00 us 4 MKDIR 0.00 266.60 us 16.00 us 7646.00 us 105 STAT 0.00 3322.67 us 62.00 us 11074.00 us 9 OPENDIR 0.00 214.21 us 9.00 us 27224.00 us 669 FLUSH 0.00 74751.00 us 41.00 us 445195.00 us 6 READDIRP 0.00 482.80 us 22.00 us 74947.00 us 1292 STATFS 0.00 1240.61 us 198.00 us 29810.00 us 669 CREATE 0.00 1342.66 us 168.00 us 41128.00 us 762 UNLINK 0.00 292.44 us 12.00 us 44091.00 us 6578 FSTAT 0.00 200.80 us 10.00 us 42864.00 us 18898 FINODELK 0.00 362.70 us 22.00 us 36913.00 us 12102 READ 0.00 6783.40 us 110.00 us 61766.00 us 798 FSYNC 0.00 625.49 us 46.00 us 42693.00 us 11968 WRITE 0.00 965.13 us 27.00 us 64134.00 us 10012 FXATTROP 0.01 4200339.38 us 77.00 us 41078178.00 us 16 SETXATTR 0.02 1998934.33 us 28.00 us 41365012.00 us 85 GETXATTR 0.07 14500594.56 us 30.00 us 235894105.00 us 48 INODELK 0.07 274987.36 us 39.00 us 41596014.00 us 2532 XATTROP 46.33 1079715.51 us 52.00 us 41770605.00 us 417268 LOOKUP 53.50 3980262.68 us 11.00 us 472355943.00 us 130686 ENTRYLK Duration: 229 seconds Data Read: 1527648224 bytes Data Written: 751416802 bytes Brick: Server4:/export/dv_repo ---------------------------------------- Interval 4 Stats: Block Size: 8b+ 16b+ 32b+ No. of Reads: 0 0 0 No. of Writes: 1 2 2 Block Size: 64b+ 128b+ 256b+ No. of Reads: 0 1 0 No. of Writes: 2 1 13 Block Size: 512b+ 1024b+ 2048b+ No. of Reads: 5 4 6 No. of Writes: 17 39 3514 Block Size: 4096b+ 8192b+ 16384b+ No. of Reads: 11 17 54 No. of Writes: 5045 26970 18290 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 97 281 6181 No. of Writes: 198 37 0 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 1295 FORGET 0.00 0.00 us 0.00 us 0.00 us 708 RELEASE 0.00 0.00 us 0.00 us 0.00 us 9 RELEASEDIR 0.00 120.50 us 88.00 us 153.00 us 2 OPEN 0.00 90.89 us 60.00 us 141.00 us 9 OPENDIR 0.00 482.00 us 53.00 us 911.00 us 2 READDIRP 0.00 286.25 us 237.00 us 334.00 us 4 MKDIR 0.00 96.15 us 53.00 us 172.00 us 13 SETXATTR 0.01 45.62 us 18.00 us 101.00 us 29 STAT 0.01 55.53 us 26.00 us 111.00 us 32 INODELK 0.06 22.24 us 8.00 us 83.00 us 708 FLUSH 0.08 178.77 us 30.00 us 409.00 us 115 GETXATTR 0.22 39.85 us 15.00 us 178.00 us 1412 FINODELK 0.24 46.76 us 16.00 us 139.00 us 1292 STATFS 0.46 41.61 us 15.00 us 171.00 us 2802 ENTRYLK 0.57 101.40 us 26.00 us 217.00 us 1428 FXATTROP 0.70 252.43 us 160.00 us 513.00 us 706 CREATE 0.79 52.28 us 12.00 us 200.00 us 3843 FSTAT 1.27 467.65 us 136.00 us 5732.00 us 691 UNLINK 2.71 103.23 us 21.00 us 9713.00 us 6660 READ 13.90 65.20 us 28.00 us 3065.00 us 54131 WRITE 78.96 155.36 us 36.00 us 87299.00 us 129041 LOOKUP Duration: 229 seconds Data Read: 845204064 bytes Data Written: 807564460 bytes Brick: Server7:/export/dv_repo ---------------------------------------- Interval 4 Stats: Block Size: 128b+ 256b+ 512b+ No. of Reads: 0 2 0 No. of Writes: 2 3 7 Block Size: 1024b+ 2048b+ 4096b+ No. of Reads: 5 4 4 No. of Writes: 21 1539 1923 Block Size: 8192b+ 16384b+ 32768b+ No. of Reads: 17 35 66 No. of Writes: 11863 8708 140 Block Size: 65536b+ 131072b+ No. of Reads: 199 3942 No. of Writes: 25 0 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 774 FORGET 0.00 0.00 us 0.00 us 0.00 us 327 RELEASE 0.00 0.00 us 0.00 us 0.00 us 9 RELEASEDIR 0.00 99.50 us 62.00 us 137.00 us 2 OPEN 0.00 236.00 us 29.00 us 443.00 us 2 READDIRP 0.00 42.18 us 21.00 us 71.00 us 22 STAT 0.00 281.00 us 255.00 us 311.00 us 4 MKDIR 0.01 100.62 us 67.00 us 180.00 us 13 SETXATTR 0.01 60.27 us 29.00 us 126.00 us 26 INODELK 0.01 207.33 us 35.00 us 1171.00 us 9 OPENDIR 0.02 107.02 us 65.00 us 269.00 us 51 XATTROP 0.04 26.83 us 8.00 us 105.00 us 327 FLUSH 0.05 174.41 us 35.00 us 447.00 us 75 GETXATTR 0.12 44.34 us 18.00 us 123.00 us 652 FINODELK 0.25 47.97 us 16.00 us 229.00 us 1292 STATFS 0.32 117.19 us 39.00 us 270.00 us 658 FXATTROP 0.36 269.22 us 166.00 us 1256.00 us 327 CREATE 0.53 45.34 us 11.00 us 480.00 us 2836 ENTRYLK 0.55 54.27 us 12.00 us 783.00 us 2482 FSTAT 0.77 553.87 us 149.00 us 12336.00 us 339 UNLINK 1.90 108.45 us 22.00 us 1621.00 us 4278 READ 7.03 70.74 us 28.00 us 1578.00 us 24231 WRITE 88.02 159.88 us 30.00 us 57080.00 us 134187 LOOKUP Duration: 229 seconds Data Read: 541364650 bytes Data Written: 370965933 bytes storage.owner-gid: 8080 diagnostics.latency-measurement: on diagnostics.count-fop-hits: on performance.low-prio-threads: 64 performance.normal-prio-threads: 64 performance.high-prio-threads: 64 performance.io-thread-count: 64 server.outstanding-rpc-limit: 128 network.inode-lru-limit: 32000 server.event-threads: 8 cluster.entry-self-heal: off cluster.metadata-self-heal: off cluster.data-self-heal: off