Bug 1007866
| Summary: | sequential read performance not optimized for libgfapi | |||
|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Ben England <bengland> | |
| Component: | glusterd | Assignee: | Anand Avati <aavati> | |
| Status: | CLOSED ERRATA | QA Contact: | Ben England <bengland> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 2.1 | CC: | amarts, bturner, chrisw, grajaiya, kparthas, shaines, ssaha, vagarwal, vbellur | |
| Target Milestone: | --- | Keywords: | ZStream | |
| Target Release: | --- | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | glusterfs-3.4.0.34rhs | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1009134 (view as bug list) | Environment: | ||
| Last Closed: | 2013-11-27 15:37:54 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1009134 | |||
|
Description
Ben England
2013-09-13 13:17:48 UTC
I have collected data showing the impact of these changes in a spreadsheet at below URL: http://perf1.perf.lab.eng.bos.redhat.com/bengland/public/rhs/virt/kvm-libgfapi.ods This basically shows about a 60% improvement on reads by setting stat-prefetch volume parameter to on, and then for reads with multiple virtual block devices (1 iozone thread per device), another 40% improvement, allowing us to attain 10-GbE LINE SPEED with KVM/libgfapi on sequential reads with 2 virtual block devices, far better than FUSE. There is little variance in the read results. Avati's patch basically lowers the amount of CPU consumed by QEMU I/O threads. We observed that without the patch, single-thread reads caused one of QEMU threads, presumably an I/O thread, to consume 95% of a CPU core, so there was a "hot thread" CPU bottleneck caused by libgfapi (Avati verified this by using gdb). After the patch, there was still a CPU bottleneck on the single thread case, but the bottleneck moved from memset_sse() to memcpy(), a more reasonable bottleneck associated with actually moving data. I also measured write performance, but there was some variance in the test, so the small differences graphed should not be considered statistically significant. There is some variance between runs in the case of writes, may be due to NUMA+power-management+scheduler effects, etc. Upstream patch @ : http://review.gluster.org/5897 Fixed in version please. I think RHS 2.1 U2 contains the fix, I re-ran the same tests that I ran before and got the same improvement with libgfapi. Again I used "group virt" tuning, rhs-virtualization tuned profile, stat-prefetch on, guest readahead 8 MB. WARNING: I'm seeing a major self-heal operation when I run this test described in bz 1007948. Wish you had fixed that. This will cause alarm among sysadmins. However, it pulls itself together. This seems to affect writes more than reads. So I'm turning off self-heal-daemon for now so I can see what write performance would be if this was fixed. I used the RHS 2.1 U2 build dated Oct 27. top utility with "H" option shows that there are no hot threads blocking libgfapi performance on reads anymore. with 4 virtual block devices being read, I can see glusterfsd on server gprfs047 approaching 4.5 cores used, while server gprfs048, the other member of the replica pair, is idle. This can be corrected with gluster volume set my-vol read-hash-mode 2. I'm running rpms from build at baseurl=http://download.lab.bos.redhat.com/nightly/RHSS-2.1u2-20131027.n.0/2.1u2/RHS/x86_64/os on client: [root@gprfc093 ~]# rpm -qa | grep glusterfs glusterfs-3.4.0.35.1u2rhs-1.el6rhs.x86_64 glusterfs-fuse-3.4.0.35.1u2rhs-1.el6rhs.x86_64 glusterfs-api-3.4.0.35.1u2rhs-1.el6rhs.x86_64 glusterfs-libs-3.4.0.35.1u2rhs-1.el6rhs.x86_64 and on server: [root@gprfc093 ~]# ssh gprfs048 rpm -qa | grep glusterfs glusterfs-geo-replication-3.4.0.35.1u2rhs-1.el6rhs.x86_64 glusterfs-libs-3.4.0.35.1u2rhs-1.el6rhs.x86_64 glusterfs-fuse-3.4.0.35.1u2rhs-1.el6rhs.x86_64 glusterfs-server-3.4.0.35.1u2rhs-1.el6rhs.x86_64 glusterfs-api-3.4.0.35.1u2rhs-1.el6rhs.x86_64 glusterfs-3.4.0.35.1u2rhs-1.el6rhs.x86_64 glusterfs-rdma-3.4.0.35.1u2rhs-1.el6rhs.x86_64 Since libgfapi glfs.h no longer is in devel RPM, I have to pull it from the source RPM to compile my test program, which hasn't changed. That was dropped off by mistake, already added to errata a few days back I tested this on with the u1 rc(3.4.0.44) bits on the gqas server. I setup two different VMs, 1 back ended with fuse, 1 with libgfapi. In my tests libgfapi out performed fuse in single threaded sequential IOs and I didn't see any hot threads. Marking verified. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-1769.html |