+++ This bug was initially created as a clone of Bug #1436739 +++ Description of problem: As per Sanjay Rao's inputs, there was a performance drop in random reads fio workload when run through vms hosted on sharded volumes. Volume profile indicated a big difference between the number of lookups sent by FUSE and number of lookups received by individual bricks. Through code reading, it was found that there is a performance bug in shard which was causing the translator to trigger unusually high number of lookups for cache invalidation even when there was no modification to the file. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: --- Additional comment from Worker Ant on 2017-03-28 10:23:33 EDT --- REVIEW: https://review.gluster.org/16961 (features/shard: Pass the correct iatt for cache invalidation) posted (#1) for review on master by Krutika Dhananjay (kdhananj) --- Additional comment from Worker Ant on 2017-03-30 01:48:38 EDT --- REVIEW: https://review.gluster.org/16961 (features/shard: Pass the correct iatt for cache invalidation) posted (#2) for review on master by Krutika Dhananjay (kdhananj) --- Additional comment from Worker Ant on 2017-03-30 02:15:59 EDT --- COMMIT: https://review.gluster.org/16961 committed in master by Pranith Kumar Karampuri (pkarampu) ------ commit dd5ada1f11d76b4c55c7c55d23718617f11a6c12 Author: Krutika Dhananjay <kdhananj> Date: Tue Mar 28 19:26:41 2017 +0530 features/shard: Pass the correct iatt for cache invalidation This fixes a performance issue with shard which was causing the translator to trigger unusually high number of lookups for cache invalidation even when there was no modification to the file. In shard_common_stat_cbk(), it is local->prebuf that contains the aggregated size and block count as opposed to buf which only holds the attributes for the physical copy of base shard. Passing buf for inode_ctx invalidation would always set refresh to true since the file size in inode ctx contains the aggregated size and would never be same as @buf->ia_size. This was leading to every write/read being preceded by a lookup on the base shard even when the file underwent no modification. Change-Id: Ib0349291d2d01f3782d6d0bdd90c6db5e0609210 BUG: 1436739 Signed-off-by: Krutika Dhananjay <kdhananj> Reviewed-on: https://review.gluster.org/16961 NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Smoke: Gluster Build System <jenkins.org> Reviewed-by: Pranith Kumar Karampuri <pkarampu>
https://code.engineering.redhat.com/gerrit/#/c/102390/
Had HC installed with glusterfs 3.2 bits and observed that lookups sent on the brick were greater than the one sent on fuse mount. Used fio tools and ran 100 random read workload. Lookup on fuse mount : 741 Lookup sent over the brick : 15540 Updated 3.2 bits to 3.3 and ran 100 read workload on the vms. I have run the workload thrice and i have got similar results twice but the first time lookups sent on the brick were greater. First run: ===================== === Interval 0 stats === Duration : 1021 secs BytesRead : 149981696 BytesWritten : 49664 Block Size : 512B+ 1024B+ 2048B+ Read Count : 29 0 2 Write Count : 59 3 0 Block Size : 4096B+ 8192B+ 16384B+ Read Count : 110 12007 2 Write Count : 0 0 1 Block Size : 32768B+ 65536B+ 131072B+ Read Count : 2 11 384 Write Count : 0 0 0 Fop Call Count Avg-Latency Min-Latency Max-Latency --- ---------- ----------- ----------- ----------- STAT 11987 569.88 us 190.00 us 31860.00 us OPEN 31 52.48 us 27.00 us 104.00 us READ 12103 12023.84 us 341.00 us 209385.00 us WRITE 15 7500.80 us 1334.00 us 78435.00 us STATFS 43 535.09 us 341.00 us 4031.00 us FLUSH 48 31.56 us 12.00 us 68.00 us FSYNC 18 912.50 us 423.00 us 1979.00 us REMOVEXATTR 7 961.29 us 894.00 us 1103.00 us OPENDIR 31 618.26 us 492.00 us 1432.00 us FSTAT 42 1106.86 us 256.00 us 29577.00 us LOOKUP 269 2146.18 us 482.00 us 87316.00 us SETATTR 7 856.14 us 749.00 us 1078.00 us READDIRP 62 1465.13 us 16.00 us 43345.00 us RELEASE 186 0 us 0 us 0 us RELEASEDIR 189 0 us 0 us 0 us ------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- === Interval 0 stats === Duration : 1022 secs BytesRead : 842203648 BytesWritten : 5270016 Block Size : 512B+ 1024B+ 2048B+ Read Count : 321 0 3 Write Count : 63 3 0 Block Size : 4096B+ 8192B+ 16384B+ Read Count : 409 12008 10 Write Count : 3 0 1 Block Size : 32768B+ 65536B+ 131072B+ Read Count : 6 507 5250 Write Count : 1 7 34 Fop Call Count Avg-Latency Min-Latency Max-Latency --- ---------- ----------- ----------- ----------- STAT 12012 688.10 us 191.00 us 29323.00 us OPEN 177 46.47 us 22.00 us 92.00 us READ 13041 10628.98 us 315.00 us 279819.00 us WRITE 14 3515.00 us 1525.00 us 22023.00 us STATFS 41 467.46 us 297.00 us 782.00 us FLUSH 280 32.73 us 12.00 us 72.00 us FSYNC 18 894.11 us 447.00 us 1864.00 us REMOVEXATTR 7 905.86 us 826.00 us 994.00 us OPENDIR 33 789.85 us 430.00 us 5189.00 us FSTAT 59 521.12 us 224.00 us 8359.00 us LOOKUP 542 3737.25 us 487.00 us 59769.00 us SETATTR 7 823.14 us 649.00 us 998.00 us READDIRP 66 857.58 us 15.00 us 36928.00 us RELEASE 1051 0 us 0 us 0 us RELEASEDIR 204 0 us 0 us 0 us ------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- === Interval 0 stats === Duration : 1022 secs BytesRead : 151019520 BytesWritten : 2146816 Block Size : 512B+ 1024B+ 2048B+ Read Count : 76 0 1 Write Count : 59 3 0 Block Size : 4096B+ 8192B+ 16384B+ Read Count : 249 12013 5 Write Count : 3 0 1 Block Size : 32768B+ 65536B+ 131072B+ Read Count : 3 13 384 Write Count : 1 4 12 Fop Call Count Avg-Latency Min-Latency Max-Latency --- ---------- ----------- ----------- ----------- STAT 11985 678.82 us 198.00 us 24093.00 us OPEN 29 45.83 us 28.00 us 69.00 us READ 12099 10853.84 us 330.00 us 218767.00 us WRITE 15 5502.40 us 1318.00 us 53345.00 us STATFS 42 444.74 us 349.00 us 537.00 us FLUSH 46 33.83 us 14.00 us 61.00 us FSYNC 18 750.39 us 422.00 us 1604.00 us REMOVEXATTR 6 917.33 us 854.00 us 978.00 us OPENDIR 30 673.97 us 461.00 us 4076.00 us FSTAT 56 435.66 us 296.00 us 2610.00 us LOOKUP 265 1870.12 us 473.00 us 25081.00 us SETATTR 6 802.50 us 741.00 us 883.00 us READDIRP 60 1277.52 us 18.00 us 35013.00 us RELEASE 184 0 us 0 us 0 us RELEASEDIR 191 0 us 0 us 0 us ------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- vol_profile_output: =========================== Interval 0 Stats: Block Size: 256b+ 512b+ 1024b+ No. of Reads: 19 0 0 No. of Writes: 0 30 9 Block Size: 8192b+ 16384b+ 65536b+ No. of Reads: 12007 0 6 No. of Writes: 0 3 0 Block Size: 131072b+ No. of Reads: 64 No. of Writes: 0 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 93 RELEASEDIR 0.00 108.00 us 108.00 us 108.00 us 1 OPEN 0.00 37.50 us 22.00 us 73.00 us 4 READDIR 0.00 57.42 us 28.00 us 104.00 us 84 FINODELK 0.00 73.95 us 2.00 us 117.00 us 93 OPENDIR 0.01 66.79 us 26.00 us 118.00 us 124 STATFS 0.01 309.07 us 62.00 us 824.00 us 54 FSYNC 0.02 763.58 us 35.00 us 29177.00 us 43 FSTAT 0.04 614.07 us 117.00 us 16973.00 us 84 FXATTROP 0.04 1365.17 us 162.00 us 24131.00 us 42 WRITE 0.06 1575.48 us 36.00 us 43256.00 us 52 READDIRP 0.10 165.39 us 22.00 us 11109.00 us 830 SETATTR 0.13 231.57 us 42.00 us 36982.00 us 811 SETXATTR 0.14 248.70 us 41.00 us 21395.00 us 830 REMOVEXATTR 0.27 328.41 us 22.00 us 37087.00 us 1167 GETXATTR 0.37 456.10 us 113.00 us 30304.00 us 1163 XATTROP 0.79 94.76 us 35.00 us 31630.00 us 11987 STAT 1.01 604.74 us 18.00 us 64453.00 us 2411 INODELK 3.10 1000.71 us 24.00 us 63019.00 us 4462 LOOKUP 93.92 11193.88 us 142.00 us 209114.00 us 12096 READ Duration: 173 seconds Data Read: 107151813 bytes Data Written: 73728 bytes Lookup sent on fuse mount : 1076 Lookup sent over brick : 4462 Second Run: ==================== === Interval 1 stats === Duration : 404 secs BytesRead : 119500800 BytesWritten : 10240 Block Size : 512B+ 4096B+ 8192B+ Read Count : 0 41 12007 Write Count : 20 0 0 Block Size : 131072B+ Read Count : 160 Write Count : 0 Fop Call Count Avg-Latency Min-Latency Max-Latency --- ---------- ----------- ----------- ----------- STAT 11988 554.58 us 204.00 us 1794.00 us OPEN 31 46.52 us 28.00 us 75.00 us READ 12107 10435.64 us 344.00 us 415124.00 us WRITE 10 3082.40 us 2172.00 us 7542.00 us STATFS 44 509.98 us 380.00 us 1104.00 us FLUSH 50 28.78 us 11.00 us 60.00 us REMOVEXATTR 6 2358.50 us 887.00 us 9236.00 us OPENDIR 33 1843.15 us 508.00 us 14872.00 us FSTAT 46 421.09 us 233.00 us 1191.00 us LOOKUP 277 2001.56 us 514.00 us 30868.00 us SETATTR 6 912.17 us 837.00 us 953.00 us READDIRP 66 766.29 us 14.00 us 29581.00 us RELEASE 68 0 us 0 us 0 us RELEASEDIR 71 0 us 0 us 0 us ------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- === Interval 1 stats === Duration : 404 secs BytesRead : 395698176 BytesWritten : 10240 Block Size : 512B+ 4096B+ 8192B+ Read Count : 72 41 12002 Write Count : 20 0 0 Block Size : 65536B+ 131072B+ Read Count : 196 2108 Write Count : 0 0 Fop Call Count Avg-Latency Min-Latency Max-Latency --- ---------- ----------- ----------- ----------- STAT 12014 717.24 us 202.00 us 36196.00 us OPEN 188 48.29 us 23.00 us 109.00 us READ 13119 10006.90 us 275.00 us 269229.00 us WRITE 9 2375.11 us 2191.00 us 2555.00 us STATFS 44 1183.20 us 376.00 us 19794.00 us FLUSH 298 33.81 us 11.00 us 76.00 us REMOVEXATTR 6 991.00 us 864.00 us 1263.00 us OPENDIR 31 634.19 us 481.00 us 1094.00 us FSTAT 60 406.83 us 200.00 us 770.00 us LOOKUP 583 3492.63 us 504.00 us 32151.00 us SETATTR 6 809.17 us 754.00 us 917.00 us READDIRP 62 311.15 us 16.00 us 1003.00 us RELEASE 408 0 us 0 us 0 us RELEASEDIR 69 0 us 0 us 0 us ------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- === Interval 1 stats === Duration : 404 secs BytesRead : 119484416 BytesWritten : 10240 Block Size : 512B+ 4096B+ 8192B+ Read Count : 0 41 12005 Write Count : 20 0 0 Block Size : 131072B+ Read Count : 160 Write Count : 0 Fop Call Count Avg-Latency Min-Latency Max-Latency --- ---------- ----------- ----------- ----------- STAT 11979 686.02 us 195.00 us 33833.00 us OPEN 31 38.81 us 25.00 us 70.00 us READ 12105 10773.69 us 342.00 us 213621.00 us WRITE 10 2432.90 us 2245.00 us 2622.00 us STATFS 44 426.73 us 335.00 us 674.00 us FLUSH 50 31.84 us 12.00 us 60.00 us REMOVEXATTR 6 857.00 us 803.00 us 932.00 us OPENDIR 32 573.72 us 443.00 us 1296.00 us FSTAT 64 484.00 us 276.00 us 3078.00 us LOOKUP 269 1648.59 us 488.00 us 33870.00 us SETATTR 6 750.33 us 703.00 us 840.00 us READDIRP 64 1220.69 us 15.00 us 59162.00 us RELEASE 68 0 us 0 us 0 us RELEASEDIR 71 0 us 0 us 0 us ------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- volume profile: ============================= Interval 0 Stats: Block Size: 256b+ 512b+ 8192b+ No. of Reads: 18 0 12007 No. of Writes: 0 24 0 Block Size: 131072b+ No. of Reads: 64 No. of Writes: 0 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 86 RELEASEDIR 0.00 377.33 us 253.00 us 724.00 us 6 READDIR 0.00 58.69 us 29.00 us 93.00 us 48 FINODELK 0.00 74.67 us 38.00 us 144.00 us 40 FSTAT 0.01 73.49 us 3.00 us 143.00 us 86 OPENDIR 0.01 67.92 us 33.00 us 118.00 us 116 STATFS 0.01 168.48 us 103.00 us 234.00 us 48 FXATTROP 0.01 107.80 us 69.00 us 153.00 us 164 REMOVEXATTR 0.02 132.43 us 89.00 us 201.00 us 148 SETXATTR 0.02 122.78 us 31.00 us 193.00 us 164 SETATTR 0.02 1087.46 us 769.00 us 5997.00 us 24 WRITE 0.02 65.09 us 24.00 us 113.00 us 404 INODELK 0.03 178.39 us 24.00 us 457.00 us 175 GETXATTR 0.03 215.85 us 160.00 us 521.00 us 170 XATTROP 0.03 760.12 us 37.00 us 29202.00 us 51 READDIRP 0.21 188.61 us 36.00 us 15933.00 us 1412 LOOKUP 0.76 78.78 us 29.00 us 723.00 us 11987 STAT 98.82 10132.91 us 135.00 us 414844.00 us 12089 READ Duration: 167 seconds Data Read: 106758142 bytes Data Written: 12288 bytes Lookup sent over fuse : 1129 Lookup sent over brick : 1412 Third run: ============================== === Interval 2 stats === Duration : 892 secs BytesRead : 143814656 BytesWritten : 22016 Block Size : 512B+ 4096B+ 8192B+ Read Count : 0 91 12006 Write Count : 43 0 0 Block Size : 131072B+ Read Count : 344 Write Count : 0 Fop Call Count Avg-Latency Min-Latency Max-Latency --- ---------- ----------- ----------- ----------- STAT 11981 557.32 us 209.00 us 1811.00 us OPEN 34 43.24 us 22.00 us 71.00 us READ 12099 11715.72 us 379.00 us 218366.00 us WRITE 9 2543.67 us 2284.00 us 2803.00 us STATFS 45 497.62 us 357.00 us 713.00 us FLUSH 54 32.30 us 13.00 us 66.00 us REMOVEXATTR 7 1033.14 us 822.00 us 1228.00 us OPENDIR 33 616.15 us 513.00 us 741.00 us FSTAT 61 416.98 us 230.00 us 1320.00 us LOOKUP 278 1807.43 us 532.00 us 12786.00 us SETATTR 7 969.14 us 771.00 us 1368.00 us READDIRP 66 303.11 us 15.00 us 820.00 us RELEASE 152 0 us 0 us 0 us RELEASEDIR 155 0 us 0 us 0 us ------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- === Interval 2 stats === Duration : 892 secs BytesRead : 752958464 BytesWritten : 22016 Block Size : 512B+ 4096B+ 8192B+ Read Count : 158 90 12006 Write Count : 43 0 0 Block Size : 65536B+ 131072B+ Read Count : 433 4639 Write Count : 0 0 Fop Call Count Avg-Latency Min-Latency Max-Latency --- ---------- ----------- ----------- ----------- STAT 12017 550.86 us 201.00 us 12317.00 us OPEN 190 45.58 us 22.00 us 129.00 us READ 13130 10153.46 us 270.00 us 260762.00 us WRITE 9 3066.78 us 2346.00 us 7145.00 us STATFS 45 535.87 us 354.00 us 2304.00 us FLUSH 302 34.14 us 11.00 us 71.00 us REMOVEXATTR 7 996.14 us 848.00 us 1119.00 us OPENDIR 36 685.11 us 488.00 us 1982.00 us FSTAT 63 385.44 us 230.00 us 635.00 us LOOKUP 578 3320.98 us 489.00 us 36146.00 us SETATTR 7 878.71 us 615.00 us 1027.00 us READDIRP 72 297.44 us 17.00 us 895.00 us RELEASE 902 0 us 0 us 0 us RELEASEDIR 160 0 us 0 us 0 us ------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- === Interval 2 stats === Duration : 892 secs BytesRead : 143806464 BytesWritten : 22016 Block Size : 512B+ 4096B+ 8192B+ Read Count : 0 91 12005 Write Count : 43 0 0 Block Size : 131072B+ Read Count : 344 Write Count : 0 Fop Call Count Avg-Latency Min-Latency Max-Latency --- ---------- ----------- ----------- ----------- STAT 11977 714.93 us 193.00 us 63057.00 us OPEN 32 39.69 us 22.00 us 71.00 us READ 12098 10851.17 us 352.00 us 268565.00 us WRITE 9 2406.11 us 2243.00 us 2701.00 us STATFS 44 435.57 us 303.00 us 1176.00 us FLUSH 52 29.65 us 12.00 us 49.00 us REMOVEXATTR 6 843.67 us 762.00 us 912.00 us OPENDIR 31 865.48 us 458.00 us 9475.00 us FSTAT 61 409.67 us 257.00 us 627.00 us LOOKUP 282 1754.85 us 460.00 us 25927.00 us SETATTR 6 749.17 us 638.00 us 847.00 us READDIRP 62 304.05 us 14.00 us 776.00 us RELEASE 154 0 us 0 us 0 us RELEASEDIR 155 0 us 0 us 0 us ------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- volume profile: ======================= Interval 0 Stats: Block Size: 256b+ 512b+ 8192b+ No. of Reads: 18 0 12006 No. of Writes: 0 24 0 Block Size: 131072b+ No. of Reads: 64 No. of Writes: 0 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 92 RELEASEDIR 0.00 37.75 us 21.00 us 76.00 us 4 READDIR 0.00 62.33 us 35.00 us 93.00 us 48 FINODELK 0.00 79.50 us 33.00 us 119.00 us 58 FSTAT 0.00 74.51 us 3.00 us 133.00 us 92 OPENDIR 0.01 64.51 us 25.00 us 410.00 us 121 STATFS 0.01 176.33 us 132.00 us 233.00 us 48 FXATTROP 0.01 169.34 us 40.00 us 416.00 us 56 READDIRP 0.01 103.52 us 69.00 us 147.00 us 173 REMOVEXATTR 0.01 124.14 us 83.00 us 185.00 us 155 SETXATTR 0.02 121.36 us 31.00 us 189.00 us 173 SETATTR 0.02 899.67 us 747.00 us 1118.00 us 24 WRITE 0.02 68.47 us 24.00 us 2359.00 us 435 INODELK 0.02 175.49 us 21.00 us 270.00 us 185 GETXATTR 0.03 210.42 us 154.00 us 281.00 us 181 XATTROP 0.17 165.59 us 24.00 us 580.00 us 1451 LOOKUP 0.67 78.49 us 38.00 us 442.00 us 11981 STAT 98.99 11416.08 us 130.00 us 218011.00 us 12088 READ Duration: 169 seconds Data Read: 106749950 bytes Data Written: 12288 bytes Lookup sent over fuse mount : 1138 Lookup sent over brick : 1451 Any idea why this was more first time? I ran same workload which is 100% random read.
(In reply to RamaKasturi from comment #7) > Had HC installed with glusterfs 3.2 bits and observed that lookups sent on > the brick were greater than the one sent on fuse mount. Used fio tools and > ran 100 random read workload. > > Lookup on fuse mount : 741 > Lookup sent over the brick : 15540 > > Updated 3.2 bits to 3.3 and ran 100 read workload on the vms. I have run the > workload thrice and i have got similar results twice but the first time > lookups sent on the brick were greater. > > First run: > ===================== > > === Interval 0 stats === > Duration : 1021 secs > BytesRead : 149981696 > BytesWritten : 49664 > > Block Size : 512B+ 1024B+ 2048B+ > Read Count : 29 0 2 > Write Count : 59 3 0 > > Block Size : 4096B+ 8192B+ 16384B+ > Read Count : 110 12007 2 > Write Count : 0 0 1 > > Block Size : 32768B+ 65536B+ 131072B+ > Read Count : 2 11 384 > Write Count : 0 0 0 > > Fop Call Count Avg-Latency Min-Latency Max-Latency > --- ---------- ----------- ----------- ----------- > STAT 11987 569.88 us 190.00 us 31860.00 us > OPEN 31 52.48 us 27.00 us 104.00 us > READ 12103 12023.84 us 341.00 us 209385.00 us > WRITE 15 7500.80 us 1334.00 us 78435.00 us > STATFS 43 535.09 us 341.00 us 4031.00 us > FLUSH 48 31.56 us 12.00 us 68.00 us > FSYNC 18 912.50 us 423.00 us 1979.00 us > REMOVEXATTR 7 961.29 us 894.00 us 1103.00 us > OPENDIR 31 618.26 us 492.00 us 1432.00 us > FSTAT 42 1106.86 us 256.00 us 29577.00 us > LOOKUP 269 2146.18 us 482.00 us 87316.00 us > SETATTR 7 856.14 us 749.00 us 1078.00 us > READDIRP 62 1465.13 us 16.00 us 43345.00 us > RELEASE 186 0 us 0 us 0 us > RELEASEDIR 189 0 us 0 us 0 us > ------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- > > === Interval 0 stats === > Duration : 1022 secs > BytesRead : 842203648 > BytesWritten : 5270016 > > Block Size : 512B+ 1024B+ 2048B+ > Read Count : 321 0 3 > Write Count : 63 3 0 > > Block Size : 4096B+ 8192B+ 16384B+ > Read Count : 409 12008 10 > Write Count : 3 0 1 > > Block Size : 32768B+ 65536B+ 131072B+ > Read Count : 6 507 5250 > Write Count : 1 7 34 > > Fop Call Count Avg-Latency Min-Latency Max-Latency > --- ---------- ----------- ----------- ----------- > STAT 12012 688.10 us 191.00 us 29323.00 us > OPEN 177 46.47 us 22.00 us 92.00 us > READ 13041 10628.98 us 315.00 us 279819.00 us > WRITE 14 3515.00 us 1525.00 us 22023.00 us > STATFS 41 467.46 us 297.00 us 782.00 us > FLUSH 280 32.73 us 12.00 us 72.00 us > FSYNC 18 894.11 us 447.00 us 1864.00 us > REMOVEXATTR 7 905.86 us 826.00 us 994.00 us > OPENDIR 33 789.85 us 430.00 us 5189.00 us > FSTAT 59 521.12 us 224.00 us 8359.00 us > LOOKUP 542 3737.25 us 487.00 us 59769.00 us > SETATTR 7 823.14 us 649.00 us 998.00 us > READDIRP 66 857.58 us 15.00 us 36928.00 us > RELEASE 1051 0 us 0 us 0 us > RELEASEDIR 204 0 us 0 us 0 us > ------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- > > === Interval 0 stats === > Duration : 1022 secs > BytesRead : 151019520 > BytesWritten : 2146816 > > Block Size : 512B+ 1024B+ 2048B+ > Read Count : 76 0 1 > Write Count : 59 3 0 > > Block Size : 4096B+ 8192B+ 16384B+ > Read Count : 249 12013 5 > Write Count : 3 0 1 > > Block Size : 32768B+ 65536B+ 131072B+ > Read Count : 3 13 384 > Write Count : 1 4 12 > > Fop Call Count Avg-Latency Min-Latency Max-Latency > --- ---------- ----------- ----------- ----------- > STAT 11985 678.82 us 198.00 us 24093.00 us > OPEN 29 45.83 us 28.00 us 69.00 us > READ 12099 10853.84 us 330.00 us 218767.00 us > WRITE 15 5502.40 us 1318.00 us 53345.00 us > STATFS 42 444.74 us 349.00 us 537.00 us > FLUSH 46 33.83 us 14.00 us 61.00 us > FSYNC 18 750.39 us 422.00 us 1604.00 us > REMOVEXATTR 6 917.33 us 854.00 us 978.00 us > OPENDIR 30 673.97 us 461.00 us 4076.00 us > FSTAT 56 435.66 us 296.00 us 2610.00 us > LOOKUP 265 1870.12 us 473.00 us 25081.00 us > SETATTR 6 802.50 us 741.00 us 883.00 us > READDIRP 60 1277.52 us 18.00 us 35013.00 us > RELEASE 184 0 us 0 us 0 us > RELEASEDIR 191 0 us 0 us 0 us > ------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- > vol_profile_output: > =========================== > Interval 0 Stats: > Block Size: 256b+ 512b+ > 1024b+ > No. of Reads: 19 0 > 0 > No. of Writes: 0 30 > 9 > > Block Size: 8192b+ 16384b+ > 65536b+ > No. of Reads: 12007 0 > 6 > No. of Writes: 0 3 > 0 > > Block Size: 131072b+ > No. of Reads: 64 > No. of Writes: 0 > %-latency Avg-latency Min-Latency Max-Latency No. of calls > Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 93 > RELEASEDIR > 0.00 108.00 us 108.00 us 108.00 us 1 > OPEN > 0.00 37.50 us 22.00 us 73.00 us 4 > READDIR > 0.00 57.42 us 28.00 us 104.00 us 84 > FINODELK > 0.00 73.95 us 2.00 us 117.00 us 93 > OPENDIR > 0.01 66.79 us 26.00 us 118.00 us 124 > STATFS > 0.01 309.07 us 62.00 us 824.00 us 54 > FSYNC > 0.02 763.58 us 35.00 us 29177.00 us 43 > FSTAT > 0.04 614.07 us 117.00 us 16973.00 us 84 > FXATTROP > 0.04 1365.17 us 162.00 us 24131.00 us 42 > WRITE > 0.06 1575.48 us 36.00 us 43256.00 us 52 > READDIRP > 0.10 165.39 us 22.00 us 11109.00 us 830 > SETATTR > 0.13 231.57 us 42.00 us 36982.00 us 811 > SETXATTR > 0.14 248.70 us 41.00 us 21395.00 us 830 > REMOVEXATTR > 0.27 328.41 us 22.00 us 37087.00 us 1167 > GETXATTR > 0.37 456.10 us 113.00 us 30304.00 us 1163 > XATTROP > 0.79 94.76 us 35.00 us 31630.00 us 11987 > STAT > 1.01 604.74 us 18.00 us 64453.00 us 2411 > INODELK > 3.10 1000.71 us 24.00 us 63019.00 us 4462 > LOOKUP > 93.92 11193.88 us 142.00 us 209114.00 us 12096 > READ > > Duration: 173 seconds > Data Read: 107151813 bytes > Data Written: 73728 bytes > > Lookup sent on fuse mount : 1076 > Lookup sent over brick : 4462 > > Second Run: > ==================== > === Interval 1 stats === > Duration : 404 secs > BytesRead : 119500800 > BytesWritten : 10240 > > Block Size : 512B+ 4096B+ 8192B+ > Read Count : 0 41 12007 > Write Count : 20 0 0 > > Block Size : 131072B+ > Read Count : 160 > Write Count : 0 > > Fop Call Count Avg-Latency Min-Latency Max-Latency > --- ---------- ----------- ----------- ----------- > STAT 11988 554.58 us 204.00 us 1794.00 us > OPEN 31 46.52 us 28.00 us 75.00 us > READ 12107 10435.64 us 344.00 us 415124.00 us > WRITE 10 3082.40 us 2172.00 us 7542.00 us > STATFS 44 509.98 us 380.00 us 1104.00 us > FLUSH 50 28.78 us 11.00 us 60.00 us > REMOVEXATTR 6 2358.50 us 887.00 us 9236.00 us > OPENDIR 33 1843.15 us 508.00 us 14872.00 us > FSTAT 46 421.09 us 233.00 us 1191.00 us > LOOKUP 277 2001.56 us 514.00 us 30868.00 us > SETATTR 6 912.17 us 837.00 us 953.00 us > READDIRP 66 766.29 us 14.00 us 29581.00 us > RELEASE 68 0 us 0 us 0 us > RELEASEDIR 71 0 us 0 us 0 us > ------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- > > === Interval 1 stats === > Duration : 404 secs > BytesRead : 395698176 > BytesWritten : 10240 > > Block Size : 512B+ 4096B+ 8192B+ > Read Count : 72 41 12002 > Write Count : 20 0 0 > > Block Size : 65536B+ 131072B+ > Read Count : 196 2108 > Write Count : 0 0 > > Fop Call Count Avg-Latency Min-Latency Max-Latency > --- ---------- ----------- ----------- ----------- > STAT 12014 717.24 us 202.00 us 36196.00 us > OPEN 188 48.29 us 23.00 us 109.00 us > READ 13119 10006.90 us 275.00 us 269229.00 us > WRITE 9 2375.11 us 2191.00 us 2555.00 us > STATFS 44 1183.20 us 376.00 us 19794.00 us > FLUSH 298 33.81 us 11.00 us 76.00 us > REMOVEXATTR 6 991.00 us 864.00 us 1263.00 us > OPENDIR 31 634.19 us 481.00 us 1094.00 us > FSTAT 60 406.83 us 200.00 us 770.00 us > LOOKUP 583 3492.63 us 504.00 us 32151.00 us > SETATTR 6 809.17 us 754.00 us 917.00 us > READDIRP 62 311.15 us 16.00 us 1003.00 us > RELEASE 408 0 us 0 us 0 us > RELEASEDIR 69 0 us 0 us 0 us > ------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- > > === Interval 1 stats === > Duration : 404 secs > BytesRead : 119484416 > BytesWritten : 10240 > > Block Size : 512B+ 4096B+ 8192B+ > Read Count : 0 41 12005 > Write Count : 20 0 0 > > Block Size : 131072B+ > Read Count : 160 > Write Count : 0 > > Fop Call Count Avg-Latency Min-Latency Max-Latency > --- ---------- ----------- ----------- ----------- > STAT 11979 686.02 us 195.00 us 33833.00 us > OPEN 31 38.81 us 25.00 us 70.00 us > READ 12105 10773.69 us 342.00 us 213621.00 us > WRITE 10 2432.90 us 2245.00 us 2622.00 us > STATFS 44 426.73 us 335.00 us 674.00 us > FLUSH 50 31.84 us 12.00 us 60.00 us > REMOVEXATTR 6 857.00 us 803.00 us 932.00 us > OPENDIR 32 573.72 us 443.00 us 1296.00 us > FSTAT 64 484.00 us 276.00 us 3078.00 us > LOOKUP 269 1648.59 us 488.00 us 33870.00 us > SETATTR 6 750.33 us 703.00 us 840.00 us > READDIRP 64 1220.69 us 15.00 us 59162.00 us > RELEASE 68 0 us 0 us 0 us > RELEASEDIR 71 0 us 0 us 0 us > ------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- > > volume profile: > ============================= > Interval 0 Stats: > Block Size: 256b+ 512b+ > 8192b+ > No. of Reads: 18 0 > 12007 > No. of Writes: 0 24 > 0 > > Block Size: 131072b+ > No. of Reads: 64 > No. of Writes: 0 > %-latency Avg-latency Min-Latency Max-Latency No. of calls > Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 86 > RELEASEDIR > 0.00 377.33 us 253.00 us 724.00 us 6 > READDIR > 0.00 58.69 us 29.00 us 93.00 us 48 > FINODELK > 0.00 74.67 us 38.00 us 144.00 us 40 > FSTAT > 0.01 73.49 us 3.00 us 143.00 us 86 > OPENDIR > 0.01 67.92 us 33.00 us 118.00 us 116 > STATFS > 0.01 168.48 us 103.00 us 234.00 us 48 > FXATTROP > 0.01 107.80 us 69.00 us 153.00 us 164 > REMOVEXATTR > 0.02 132.43 us 89.00 us 201.00 us 148 > SETXATTR > 0.02 122.78 us 31.00 us 193.00 us 164 > SETATTR > 0.02 1087.46 us 769.00 us 5997.00 us 24 > WRITE > 0.02 65.09 us 24.00 us 113.00 us 404 > INODELK > 0.03 178.39 us 24.00 us 457.00 us 175 > GETXATTR > 0.03 215.85 us 160.00 us 521.00 us 170 > XATTROP > 0.03 760.12 us 37.00 us 29202.00 us 51 > READDIRP > 0.21 188.61 us 36.00 us 15933.00 us 1412 > LOOKUP > 0.76 78.78 us 29.00 us 723.00 us 11987 > STAT > 98.82 10132.91 us 135.00 us 414844.00 us 12089 > READ > > Duration: 167 seconds > Data Read: 106758142 bytes > Data Written: 12288 bytes > > > Lookup sent over fuse : 1129 > Lookup sent over brick : 1412 > > Third run: > ============================== > === Interval 2 stats === > Duration : 892 secs > BytesRead : 143814656 > BytesWritten : 22016 > > Block Size : 512B+ 4096B+ 8192B+ > Read Count : 0 91 12006 > Write Count : 43 0 0 > > Block Size : 131072B+ > Read Count : 344 > Write Count : 0 > > Fop Call Count Avg-Latency Min-Latency Max-Latency > --- ---------- ----------- ----------- ----------- > STAT 11981 557.32 us 209.00 us 1811.00 us > OPEN 34 43.24 us 22.00 us 71.00 us > READ 12099 11715.72 us 379.00 us 218366.00 us > WRITE 9 2543.67 us 2284.00 us 2803.00 us > STATFS 45 497.62 us 357.00 us 713.00 us > FLUSH 54 32.30 us 13.00 us 66.00 us > REMOVEXATTR 7 1033.14 us 822.00 us 1228.00 us > OPENDIR 33 616.15 us 513.00 us 741.00 us > FSTAT 61 416.98 us 230.00 us 1320.00 us > LOOKUP 278 1807.43 us 532.00 us 12786.00 us > SETATTR 7 969.14 us 771.00 us 1368.00 us > READDIRP 66 303.11 us 15.00 us 820.00 us > RELEASE 152 0 us 0 us 0 us > RELEASEDIR 155 0 us 0 us 0 us > ------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- > > === Interval 2 stats === > Duration : 892 secs > BytesRead : 752958464 > BytesWritten : 22016 > > Block Size : 512B+ 4096B+ 8192B+ > Read Count : 158 90 12006 > Write Count : 43 0 0 > > Block Size : 65536B+ 131072B+ > Read Count : 433 4639 > Write Count : 0 0 > > Fop Call Count Avg-Latency Min-Latency Max-Latency > --- ---------- ----------- ----------- ----------- > STAT 12017 550.86 us 201.00 us 12317.00 us > OPEN 190 45.58 us 22.00 us 129.00 us > READ 13130 10153.46 us 270.00 us 260762.00 us > WRITE 9 3066.78 us 2346.00 us 7145.00 us > STATFS 45 535.87 us 354.00 us 2304.00 us > FLUSH 302 34.14 us 11.00 us 71.00 us > REMOVEXATTR 7 996.14 us 848.00 us 1119.00 us > OPENDIR 36 685.11 us 488.00 us 1982.00 us > FSTAT 63 385.44 us 230.00 us 635.00 us > LOOKUP 578 3320.98 us 489.00 us 36146.00 us > SETATTR 7 878.71 us 615.00 us 1027.00 us > READDIRP 72 297.44 us 17.00 us 895.00 us > RELEASE 902 0 us 0 us 0 us > RELEASEDIR 160 0 us 0 us 0 us > ------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- > > === Interval 2 stats === > Duration : 892 secs > BytesRead : 143806464 > BytesWritten : 22016 > > Block Size : 512B+ 4096B+ 8192B+ > Read Count : 0 91 12005 > Write Count : 43 0 0 > > Block Size : 131072B+ > Read Count : 344 > Write Count : 0 > > Fop Call Count Avg-Latency Min-Latency Max-Latency > --- ---------- ----------- ----------- ----------- > STAT 11977 714.93 us 193.00 us 63057.00 us > OPEN 32 39.69 us 22.00 us 71.00 us > READ 12098 10851.17 us 352.00 us 268565.00 us > WRITE 9 2406.11 us 2243.00 us 2701.00 us > STATFS 44 435.57 us 303.00 us 1176.00 us > FLUSH 52 29.65 us 12.00 us 49.00 us > REMOVEXATTR 6 843.67 us 762.00 us 912.00 us > OPENDIR 31 865.48 us 458.00 us 9475.00 us > FSTAT 61 409.67 us 257.00 us 627.00 us > LOOKUP 282 1754.85 us 460.00 us 25927.00 us > SETATTR 6 749.17 us 638.00 us 847.00 us > READDIRP 62 304.05 us 14.00 us 776.00 us > RELEASE 154 0 us 0 us 0 us > RELEASEDIR 155 0 us 0 us 0 us > ------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- > > volume profile: > ======================= > Interval 0 Stats: > Block Size: 256b+ 512b+ > 8192b+ > No. of Reads: 18 0 > 12006 > No. of Writes: 0 24 > 0 > > Block Size: 131072b+ > No. of Reads: 64 > No. of Writes: 0 > %-latency Avg-latency Min-Latency Max-Latency No. of calls > Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 92 > RELEASEDIR > 0.00 37.75 us 21.00 us 76.00 us 4 > READDIR > 0.00 62.33 us 35.00 us 93.00 us 48 > FINODELK > 0.00 79.50 us 33.00 us 119.00 us 58 > FSTAT > 0.00 74.51 us 3.00 us 133.00 us 92 > OPENDIR > 0.01 64.51 us 25.00 us 410.00 us 121 > STATFS > 0.01 176.33 us 132.00 us 233.00 us 48 > FXATTROP > 0.01 169.34 us 40.00 us 416.00 us 56 > READDIRP > 0.01 103.52 us 69.00 us 147.00 us 173 > REMOVEXATTR > 0.01 124.14 us 83.00 us 185.00 us 155 > SETXATTR > 0.02 121.36 us 31.00 us 189.00 us 173 > SETATTR > 0.02 899.67 us 747.00 us 1118.00 us 24 > WRITE > 0.02 68.47 us 24.00 us 2359.00 us 435 > INODELK > 0.02 175.49 us 21.00 us 270.00 us 185 > GETXATTR > 0.03 210.42 us 154.00 us 281.00 us 181 > XATTROP > 0.17 165.59 us 24.00 us 580.00 us 1451 > LOOKUP > 0.67 78.49 us 38.00 us 442.00 us 11981 > STAT > 98.99 11416.08 us 130.00 us 218011.00 us 12088 > READ > > Duration: 169 seconds > Data Read: 106749950 bytes > Data Written: 12288 bytes > > Lookup sent over fuse mount : 1138 > Lookup sent over brick : 1451 > > Any idea why this was more first time? I ran same workload which is 100% > random read. Assuming the first run was following a fresh mount of the volume, my guess is that shard has looked up some shards before reading from them. Earlier it used to be sending MKNODs before LOOKUP. But ever since we found the add-brick + vm corruption issue, we had to change the order in this patch - https://review.gluster.org/17010 Note that this is a one-time lookup. Upon first lookup, shard xl remembers the shard-to-inode mapping in memory. Subsequent reads/writes in that region won't require this lookup. -Krutika
Thanks krutika. Based on my observations and krutikas input on why lookups were more first time marking this bug to verified state as everything works as expected.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2774