Bug 1408639
Summary: | [Perf] : Sequential Writes are off target by 12% on EC backed volumes over FUSE | |||
---|---|---|---|---|
Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Ambarish <asoman> | |
Component: | disperse | Assignee: | Ashish Pandey <aspandey> | |
Status: | CLOSED ERRATA | QA Contact: | Ambarish <asoman> | |
Severity: | high | Docs Contact: | ||
Priority: | unspecified | |||
Version: | rhgs-3.2 | CC: | amukherj, asoman, bturner, rcyriac, rhinduja, rhs-bugs, storage-qa-internal | |
Target Milestone: | --- | Keywords: | Regression | |
Target Release: | RHGS 3.2.0 | |||
Hardware: | x86_64 | |||
OS: | Linux | |||
Whiteboard: | ||||
Fixed In Version: | glusterfs-3.8.4-13 | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | ||
Clone Of: | ||||
: | 1409191 1419549 (view as bug list) | Environment: | ||
Last Closed: | 2017-03-23 06:00:15 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | 1409191, 1415160 | |||
Bug Blocks: | 1351528 |
Description
Ambarish
2016-12-26 07:28:17 UTC
After setting io-threads to off,I do not see any improvement(unlike on a *2 deployment). Seq Writes(io-threads off) : 2581707.91 kB/sec Random Writes(io-threads off) : 485993.3 kB/sec I tried on my laptop and could see that there is a regression of around 22% for seq writes using dd command. ====================================== master - [root@apandey testvol]# time dd if=/dev/urandom of=file count=1024000 bs=1024 conv=fdatasync 1024000+0 records in 1024000+0 records out 1048576000 bytes (1.0 GB, 1000 MiB) copied, 117.527 s, 8.9 MB/s real 1m57.591s user 0m0.680s sys 0m18.235s ====================================== 3.1.3 - [root@apandey glusterfs]# [root@apandey glusterfs]# cd /mnt/testvol/ [root@apandey testvol]# time dd if=/dev/urandom of=file count=1024000 bs=1024 conv=fdatasync 1024000+0 records in 1024000+0 records out 1048576000 bytes (1.0 GB, 1000 MiB) copied, 92.1061 s, 11.4 MB/s real 1m32.116s user 0m0.602s sys 0m16.018s ====================================== profile for master - Block Size: 32768b+ 65536b+ No. of Reads: 0 0 No. of Writes: 19179 8 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 3 FORGET 0.00 0.00 us 0.00 us 0.00 us 27 RELEASE 0.00 0.00 us 0.00 us 0.00 us 50 RELEASEDIR 0.00 2.17 us 2.00 us 3.00 us 6 OPENDIR 0.00 31.00 us 31.00 us 31.00 us 1 STATFS 0.00 42.67 us 28.00 us 58.00 us 3 FLUSH 0.00 102.00 us 61.00 us 143.00 us 2 XATTROP 0.00 270.00 us 270.00 us 270.00 us 1 CREATE 0.00 30.08 us 23.00 us 45.00 us 12 FSTAT 0.00 143.46 us 66.00 us 800.00 us 13 OPEN 0.00 94.31 us 18.00 us 163.00 us 26 GETXATTR 0.01 141.17 us 19.00 us 573.00 us 36 LOOKUP 0.01 100.73 us 35.00 us 658.00 us 62 FXATTROP 0.02 811.00 us 508.00 us 2052.00 us 12 READDIR 0.10 58962.00 us 58962.00 us 58962.00 us 1 TRUNCATE 0.63 2355.08 us 20.00 us 30162.00 us 158 INODELK 5.82 51.46 us 31.00 us 5417.00 us 66437 READ 18.37 51.48 us 28.00 us 433140.00 us 209666 WRITE 75.03 22041646.50 us 4844215.00 us 39239078.00 us 2 FSYNC Profile for 3.1.3 - Block Size: 32768b+ No. of Reads: 0 No. of Writes: 81 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 1 RELEASE 0.00 0.00 us 0.00 us 0.00 us 3 RELEASEDIR 0.00 2.00 us 2.00 us 2.00 us 2 OPENDIR 0.00 35.00 us 35.00 us 35.00 us 1 STATFS 0.00 52.00 us 52.00 us 52.00 us 1 XATTROP 0.00 39.00 us 33.00 us 45.00 us 2 FLUSH 0.00 42.75 us 27.00 us 51.00 us 4 INODELK 0.00 88.50 us 40.00 us 137.00 us 2 FXATTROP 0.00 48.25 us 19.00 us 96.00 us 4 READDIR 0.00 349.00 us 349.00 us 349.00 us 1 CREATE 0.00 103.25 us 25.00 us 238.00 us 4 GETXATTR 0.00 102.45 us 9.00 us 291.00 us 11 LOOKUP 5.51 46.41 us 28.00 us 5835.00 us 41210 READ 23.69 8223891.00 us 8223891.00 us 8223891.00 us 1 FSYNC 70.79 196.77 us 25.00 us 19247646.00 us 124899 WRITE Duration: 159 seconds Data Read: 21098496 bytes Data Written: 293858304 bytes Clearly we can see the number of inodelk and fxattrop and other fop has increased considerably. Possible RCA - After implementing patch http://review.gluster.org/#/c/13733/, before writing on a file we set dirty flag and at the end we remove this flag. This creates an index entry in .glusterfs/indices/xattrop/ . which remains there through out write fop. every 60 seconds shd will come up and scan this entry and starts heal, Heal in turn takes a lot of locks to FIND and heal the file. Which raises the number of inodelk fop count and could be a possible culprit. I disabled the shd and wrote a file - time dd if=/dev/urandom of=a1 count=1024 bs=1M conv=fdatasync Profile shows only 4 calls inodelk. Brick: apandey:/brick/gluster/testvol-6 --------------------------------------- Cumulative Stats: Block Size: 32768b+ 65536b+ No. of Reads: 0 0 No. of Writes: 8188 2 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 1 RELEASE 0.00 47.00 us 47.00 us 47.00 us 1 STATFS 0.00 49.50 us 46.00 us 53.00 us 2 FLUSH 0.00 38.00 us 26.00 us 52.00 us 4 INODELK 0.00 92.50 us 85.00 us 100.00 us 2 XATTROP 0.00 305.00 us 305.00 us 305.00 us 1 CREATE 0.00 138.00 us 32.00 us 395.00 us 4 FXATTROP 0.00 164.14 us 119.00 us 212.00 us 7 LOOKUP 0.92 72.73 us 43.00 us 8431.00 us 8190 WRITE 99.08 64142355.00 us 64142355.00 us 64142355.00 us 1 FSYNC With shd enable it is around 54- Brick: apandey:/brick/gluster/testvol-1 --------------------------------------- Cumulative Stats: Block Size: 32768b+ 65536b+ No. of Reads: 0 0 No. of Writes: 8190 1 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 7 RELEASE 0.00 0.00 us 0.00 us 0.00 us 21 RELEASEDIR 0.00 30.00 us 30.00 us 30.00 us 1 STATFS 0.00 5.76 us 2.00 us 9.00 us 21 OPENDIR 0.00 64.50 us 30.00 us 99.00 us 2 FLUSH 0.00 23.17 us 20.00 us 27.00 us 6 FSTAT 0.00 95.50 us 89.00 us 102.00 us 2 XATTROP 0.00 272.00 us 272.00 us 272.00 us 1 CREATE 0.00 61.67 us 42.00 us 85.00 us 6 OPEN 0.00 98.94 us 31.00 us 428.00 us 16 FXATTROP 0.00 79.92 us 22.00 us 190.00 us 38 LOOKUP 0.12 2379.48 us 1376.00 us 4600.00 us 42 READDIR 0.74 74.70 us 42.00 us 49556.00 us 8191 WRITE 10.29 163490.19 us 19.00 us 1405941.00 us 52 INODELK 19.02 320668.04 us 26.00 us 15705174.00 us 49 GETXATTR 69.83 57700430.00 us 57700430.00 us 57700430.00 us 1 FSY ------- downstream patch : https://code.engineering.redhat.com/gerrit/#/c/95870/ Sequential Write on 3.8.4-13 - 2709973.61 kB/sec,which is well within the Perf Regression threshold,Verified. Raised a separate BZ (https://bugzilla.redhat.com/show_bug.cgi?id=1419549) for Rand Write regression post discussion with Atin,Ashish and Rahul. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2017-0486.html |