Bug 1535780
| Summary: | Poor Gluster Block Performance for PostgreSQL in CNS environment | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Shekhar Berry <shberry> |
| Component: | gluster-block | Assignee: | Prasanna Kumar Kalever <prasanna.kalever> |
| Status: | CLOSED ERRATA | QA Contact: | krishnaram Karthick <kramdoss> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | cns-3.7 | CC: | ekuric, kramdoss, mpillai, pkarampu, prasanna.kalever, psuriset, rhs-bugs, rsussman, sanandpa, shberry, storage-qa-internal, vinug, xiubli |
| Target Milestone: | --- | Keywords: | Performance |
| Target Release: | CNS 3.10 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | gluster-block-0.2.1-18.el7rhgs | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-09-12 09:25:26 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1559746, 1563298, 1564082, 1565063 | ||
| Bug Blocks: | 1568861 | ||
|
Description
Shekhar Berry
2018-01-18 05:04:26 UTC
Just cross my mind one possible limitation that the Ring Buffer in Kernel for now is fixed to 8M(If using the 3.10 kernel without the dynamic growth feature).This will also slow down the performance too in high IOPS case. Pranith, Shekhar, The following are my test output: Before changing of the Ring Buffer size(1M as default): file: [postgres@gprfs025 root]$ pgbench -c 10 -j 2 -t 4000 test tps = 81.652434 (including connections establishing) tps = 81.659641 (excluding connections establishing) block: bash-4.2$ pgbench -c 10 -j 2 -t 4000 test tps = 105.612957 (including connections establishing) tps = 105.620169 (excluding connections establishing) I'm not sure why the block's tps is higher than the file's when I first got the machines, and I have test this many times were still the same. ======================= After changing of the Ring Buffer size to 64M. 1, Replace the kernel to 3.10.0-693.5.2.3.el7.bclinux.x86_64, one of my old test kernel with LIO patches backported base 3.10.0-693.5.2, because the current kernel doesn't include the Ring Buffer resize patches. 2, Replace the gluster-block and configshell to add the ringbuffer size scaleable. file: postgres@gprfs025 ~]$ pgbench -c 10 -j 2 -t 100 test tps = 96.613083 (including connections establishing) tps = 97.094510 (excluding connections establishing) block: -bash-4.2$ pgbench -c 10 -j 2 -t 4000 test tps = 583.529217 (including connections establishing) tps = 583.750357 (excluding connections establishing) May it work ? (In reply to Xiubo Li from comment #11) > Pranith, Shekhar, > > The following are my test output: > > Before changing of the Ring Buffer size(1M as default): > > file: > [postgres@gprfs025 root]$ pgbench -c 10 -j 2 -t 4000 test > tps = 81.652434 (including connections establishing) > tps = 81.659641 (excluding connections establishing) > > block: > bash-4.2$ pgbench -c 10 -j 2 -t 4000 test > tps = 105.612957 (including connections establishing) > tps = 105.620169 (excluding connections establishing) > > I'm not sure why the block's tps is higher than the file's when I first got > the machines, and I have test this many times were still the same. Hi Xiubo, That was due to a node getting disconnected due to network failure. I have tested it with new volume after fixing the node issue. The tps is back to 150 tps for file. (In reply to Shekhar Berry from comment #12) > (In reply to Xiubo Li from comment #11) > > Pranith, Shekhar, > > > > The following are my test output: > > > > Before changing of the Ring Buffer size(1M as default): > > > > file: > > [postgres@gprfs025 root]$ pgbench -c 10 -j 2 -t 4000 test > > tps = 81.652434 (including connections establishing) > > tps = 81.659641 (excluding connections establishing) > > > > block: > > bash-4.2$ pgbench -c 10 -j 2 -t 4000 test > > tps = 105.612957 (including connections establishing) > > tps = 105.620169 (excluding connections establishing) > > > > I'm not sure why the block's tps is higher than the file's when I first got > > the machines, and I have test this many times were still the same. > > Hi Xiubo, > > That was due to a node getting disconnected due to network failure. I have > tested it with new volume after fixing the node issue. The tps is back to > 150 tps for file. Hi Shekhar, Okay, get it. BTW, have ever test the block currently using my changes? I have test it again today for both of them: file: [postgres@gprfs025 ~]$ pgbench -c 10 -j 2 -t 4000 test starting vacuum...end. transaction type: TPC-B (sort of) scaling factor: 3000 query mode: simple number of clients: 10 number of threads: 2 number of transactions per client: 4000 number of transactions actually processed: 40000/40000 tps = 65.080532 (including connections establishing) tps = 65.085644 (excluding connections establishing) block: -bash-4.2$ pgbench -c 10 -j 2 -t 4000 test starting vacuum...end. transaction type: TPC-B (sort of) scaling factor: 1 query mode: simple number of clients: 10 number of threads: 2 number of transactions per client: 4000 number of transactions actually processed: 40000/40000 tps = 494.927941 (including connections establishing) tps = 495.094597 (excluding connections establishing) -bash-4.2$ lsblk For the kernel, 3.10.0-799.el7.x86_64 the default one doesn't include the RingBuffer resize patches. I have cloned the RHEL git repository and it has already merged them, so we could update the kernel to a higher version of 799. Related Patch: https://github.com/gluster/gluster-block/pull/60 With newer builds we see that Gluster Block is performing well for postgresDB compared to gluster file. So this BZ can be closed. Moving the bug to verified based on comment 29. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:2691 |