Bug 1023124
| Summary: | optimize geo-replication changelog processing | |||
|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Venky Shankar <vshankar> | |
| Component: | geo-replication | Assignee: | Venky Shankar <vshankar> | |
| Status: | CLOSED ERRATA | QA Contact: | M S Vishwanath Bhat <vbhat> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | urgent | |||
| Version: | 2.1 | CC: | aavati, amarts, cbuissar, csaba, grajaiya, kcleveng, mzywusko, nsathyan, vbhat, vshankar | |
| Target Milestone: | --- | Keywords: | ZStream | |
| Target Release: | --- | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | glusterfs-3.4.0.42rhs-1 | Doc Type: | Bug Fix | |
| Doc Text: |
Previously, while syncing the changes from master cluster to slave cluster, using geo-replication, the processing of the changes used to take time due to extra stat() calls made to figure out type of the file created, mode of file, owner/group of file. The 'stat()' call was also made on a mount point, which caused further performance degradation. With this update, the changelog entries itself contains all the details required for creating/processing the file, hence no stat() calls are made on mount point, making the processing of changelog entries significantly faster.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1024472 (view as bug list) | Environment: | ||
| Last Closed: | 2013-11-27 15:44:01 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 1025967 | |||
| Bug Blocks: | 1022820, 1024472 | |||
|
Description
Venky Shankar
2013-10-24 16:46:44 UTC
I am seeing very bad performance of geo-rep with the small files. Started creating 101838 small files and immediately started geo-rep. It took 178m8.332s time to create small files. But even after 15 hours only 60475 files have been created. Volume set options rollover-time was set to 15 secs and fsync-interval was set to 5 secs as suggested by the perf team. There were no server reboots or downtime. (In reply to M S Vishwanath Bhat from comment #2) > I am seeing very bad performance of geo-rep with the small files. > > Started creating 101838 small files and immediately started geo-rep. It > took 178m8.332s time to create small files. But even after 15 hours only > 60475 files have been created. Volume set options rollover-time was set to > 15 secs and fsync-interval was set to 5 secs as suggested by the perf team. > There were no server reboots or downtime. As I asked before, did you get a chance to measure rsync times? If not, please let me know the hostnames of the master cluster. (In reply to Venky Shankar from comment #3) > (In reply to M S Vishwanath Bhat from comment #2) > > I am seeing very bad performance of geo-rep with the small files. > > > > Started creating 101838 small files and immediately started geo-rep. It > > took 178m8.332s time to create small files. But even after 15 hours only > > 60475 files have been created. Volume set options rollover-time was set to > > 15 secs and fsync-interval was set to 5 secs as suggested by the perf team. > > There were no server reboots or downtime. > > As I asked before, did you get a chance to measure rsync times? If not, > please let me know the hostnames of the master cluster. No I haven't measured it yet. I am running some other tests there right now. You can login into 10.70.42.192 and attach to a screen session to have access to the complete cluster. Since there was some issues in patch, more work required. Venky, Amar, How do I verify this? Just looking at the changelog and noting the extra information logged in there for mkdirs is fine? Or do I need to do something more to verify this? After having discussion with Venky, I found out that I can _conceptually_ verify this BZ, Neependra is anyway verifying the perf improvements using perf. tools and other things. So, conceptually, mkdir(), create() and mknod() have stat(2) [not all, but enough for entry creations] information _embedded_ in the CHANGELOG, which implies there is no need to do a stat() on the mount to get this information. parts of changelog: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CHANGELOG.1384195873:E 1763c452-d495-4f28-9a05-e7aedb3931ea MKDIR 16877 0 0 00000000-0000-0000-0000-000000000001%2Fetc.1 CHANGELOG.1384195873:E 8e927f9a-3dd9-4d6b-b6c0-a0e50dde5044 MKDIR 16877 0 0 1763c452-d495-4f28-9a05-e7aedb3931ea%2Fskel CHANGELOG.1384195873:E c17c7836-eb38-43f3-8317-ab2dfb2da5b8 CREATE 33188 0 0 8e927f9a-3dd9-4d6b-b6c0-a0e50dde5044%2F.emacs CHANGELOG.1384195873:E 39de31cb-ade5-49de-a9f7-b9247a10aca5 CREATE 33188 0 0 8e927f9a-3dd9-4d6b-b6c0-a0e50dde5044%2F.bash_logout CHANGELOG.1384195873:E feb75320-b3d2-4c7c-933d-54606a5e0244 MKDIR 16877 0 0 1763c452-d495-4f28-9a05-e7aedb3931ea%2Fcron.d CHANGELOG.1384195873:E 9d67a4ac-e58d-418d-90e0-2636490ecfa8 CREATE 33188 0 0 feb75320-b3d2-4c7c-933d-54606a5e0244%2F0hourly CHANGELOG.1384195873:E 9d775a44-8bab-42ac-bf25-9a61d81557ba CREATE 33188 0 0 feb75320-b3d2-4c7c-933d-54606a5e0244%2Fvdsm-libvirt-logrotate CHANGELOG.1384195873:E a187b7b5-c45d-4e9e-8ca1-b368ed418f34 MKDIR 16877 0 0 1763c452-d495-4f28-9a05-e7aedb3931ea%2Fsecurity CHANGELOG.1384195873:E e1c9ec7a-850f-45e9-8f9b-28e2fa451259 MKDIR 16877 0 0 a187b7b5-c45d-4e9e-8ca1-b368ed418f34%2Fnamespace.d CHANGELOG.1384195873:E 2223e86d-0d41-4748-8272-7b416009af00 CREATE 33188 0 0 a187b7b5-c45d-4e9e-8ca1-b368ed418f34%2Ftime.conf CHANGELOG.1384195873:E a2d94e7b-c618-4ef2-ba54-3c84e5d1ca60 MKDIR 16877 0 0 a187b7b5-c45d-4e9e-8ca1-b368ed418f34%2Flimits.d CHANGELOG.1384195873:E 9d31eae4-1c59-408a-8793-cdd961941984 CREATE 33188 0 0 a187b7b5-c45d-4e9e-8ca1-b368ed418f34%2Faccess.conf CHANGELOG.1384195873:E 280dd55c-fbba-43ec-8621-0bee4751aeb3 CREATE 33188 0 0 a187b7b5-c45d-4e9e-8ca1-b368ed418f34%2Flimits.conf CHANGELOG.1384195873:E a9bce17d-62d5-462c-911f-5eb30687df31 CREATE 33188 0 0 a187b7b5-c45d-4e9e-8ca1-b368ed418f34%2Fchroot.conf CHANGELOG.1384195873:E 23de9fdd-7c50-4e77-b780-8b90d77a6de8 CREATE 33261 0 0 a187b7b5-c45d-4e9e-8ca1-b368ed418f34%2Fnamespace.init >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, I'm marking this BZ as verified. Anything else related to changelog consumption performance will be raised by the perf team. [Clearing needinfo flag] Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-1769.html *** Bug 1078961 has been marked as a duplicate of this bug. *** |