Bug 1023124 - optimize geo-replication changelog processing
optimize geo-replication changelog processing
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: geo-replication (Show other bugs)
2.1
Unspecified Unspecified
urgent Severity urgent
: ---
: ---
Assigned To: Venky Shankar
M S Vishwanath Bhat
: ZStream
: 1078961 (view as bug list)
Depends On: 1025967
Blocks: 1022820 1024472
  Show dependency treegraph
 
Reported: 2013-10-24 12:46 EDT by Venky Shankar
Modified: 2016-05-31 21:56 EDT (History)
10 users (show)

See Also:
Fixed In Version: glusterfs-3.4.0.42rhs-1
Doc Type: Bug Fix
Doc Text:
Previously, while syncing the changes from master cluster to slave cluster, using geo-replication, the processing of the changes used to take time due to extra stat() calls made to figure out type of the file created, mode of file, owner/group of file. The 'stat()' call was also made on a mount point, which caused further performance degradation. With this update, the changelog entries itself contains all the details required for creating/processing the file, hence no stat() calls are made on mount point, making the processing of changelog entries significantly faster.
Story Points: ---
Clone Of:
: 1024472 (view as bug list)
Environment:
Last Closed: 2013-11-27 10:44:01 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Venky Shankar 2013-10-24 12:46:44 EDT
Geo-replication invokes stat() for all entry and data operations found in a changelog (and possibly will do it for metadata operations too). For mkdirs, creates and mknods this stat can be avoided by storing the relevant information in the changelog itself.

This would result in faster changelog processing rates and faster sync times.
Comment 2 M S Vishwanath Bhat 2013-11-01 08:06:57 EDT
I am seeing very bad performance of geo-rep with the small files.

Started creating  101838 small files and immediately  started geo-rep. It took 178m8.332s time to create small files. But even after 15 hours only 60475 files have been created. Volume set options rollover-time was set to 15 secs and fsync-interval was set to 5 secs as suggested by the perf team. There were no server reboots or downtime.
Comment 3 Venky Shankar 2013-11-02 05:00:51 EDT
(In reply to M S Vishwanath Bhat from comment #2)
> I am seeing very bad performance of geo-rep with the small files.
> 
> Started creating  101838 small files and immediately  started geo-rep. It
> took 178m8.332s time to create small files. But even after 15 hours only
> 60475 files have been created. Volume set options rollover-time was set to
> 15 secs and fsync-interval was set to 5 secs as suggested by the perf team.
> There were no server reboots or downtime.

As I asked before, did you get a chance to measure rsync times? If not, please let me know the hostnames of the master cluster.
Comment 4 M S Vishwanath Bhat 2013-11-02 05:12:33 EDT
(In reply to Venky Shankar from comment #3)
> (In reply to M S Vishwanath Bhat from comment #2)
> > I am seeing very bad performance of geo-rep with the small files.
> > 
> > Started creating  101838 small files and immediately  started geo-rep. It
> > took 178m8.332s time to create small files. But even after 15 hours only
> > 60475 files have been created. Volume set options rollover-time was set to
> > 15 secs and fsync-interval was set to 5 secs as suggested by the perf team.
> > There were no server reboots or downtime.
> 
> As I asked before, did you get a chance to measure rsync times? If not,
> please let me know the hostnames of the master cluster.

No I haven't measured it yet. I am running some other tests there right now.

You can login into 10.70.42.192 and attach to a screen session to have access to the complete cluster.
Comment 5 Nagaprasad Sathyanarayana 2013-11-05 07:17:27 EST
Since there was some issues in patch, more work required.
Comment 6 Amar Tumballi 2013-11-07 12:36:45 EST
https://code.engineering.redhat.com/gerrit/15354
Comment 7 M S Vishwanath Bhat 2013-11-12 04:01:09 EST
Venky, Amar,

How do I verify this? Just looking at the changelog and noting the extra information logged in there for mkdirs is fine? Or do I need to do something more to verify this?
Comment 8 M S Vishwanath Bhat 2013-11-12 06:23:12 EST
After having discussion with Venky, I found out that I can _conceptually_ verify this BZ, Neependra is anyway verifying the perf improvements using perf. tools and other things.

So, conceptually, mkdir(), create() and mknod() have stat(2) [not all, but enough for entry creations] information _embedded_ in the CHANGELOG, which implies there is no need to do a stat() on the mount to get this information.

parts of changelog:

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
CHANGELOG.1384195873:E 1763c452-d495-4f28-9a05-e7aedb3931ea MKDIR 16877 0 0 00000000-0000-0000-0000-000000000001%2Fetc.1
CHANGELOG.1384195873:E 8e927f9a-3dd9-4d6b-b6c0-a0e50dde5044 MKDIR 16877 0 0 1763c452-d495-4f28-9a05-e7aedb3931ea%2Fskel
CHANGELOG.1384195873:E c17c7836-eb38-43f3-8317-ab2dfb2da5b8 CREATE 33188 0 0 8e927f9a-3dd9-4d6b-b6c0-a0e50dde5044%2F.emacs
CHANGELOG.1384195873:E 39de31cb-ade5-49de-a9f7-b9247a10aca5 CREATE 33188 0 0 8e927f9a-3dd9-4d6b-b6c0-a0e50dde5044%2F.bash_logout
CHANGELOG.1384195873:E feb75320-b3d2-4c7c-933d-54606a5e0244 MKDIR 16877 0 0 1763c452-d495-4f28-9a05-e7aedb3931ea%2Fcron.d
CHANGELOG.1384195873:E 9d67a4ac-e58d-418d-90e0-2636490ecfa8 CREATE 33188 0 0 feb75320-b3d2-4c7c-933d-54606a5e0244%2F0hourly
CHANGELOG.1384195873:E 9d775a44-8bab-42ac-bf25-9a61d81557ba CREATE 33188 0 0 feb75320-b3d2-4c7c-933d-54606a5e0244%2Fvdsm-libvirt-logrotate
CHANGELOG.1384195873:E a187b7b5-c45d-4e9e-8ca1-b368ed418f34 MKDIR 16877 0 0 1763c452-d495-4f28-9a05-e7aedb3931ea%2Fsecurity
CHANGELOG.1384195873:E e1c9ec7a-850f-45e9-8f9b-28e2fa451259 MKDIR 16877 0 0 a187b7b5-c45d-4e9e-8ca1-b368ed418f34%2Fnamespace.d
CHANGELOG.1384195873:E 2223e86d-0d41-4748-8272-7b416009af00 CREATE 33188 0 0 a187b7b5-c45d-4e9e-8ca1-b368ed418f34%2Ftime.conf
CHANGELOG.1384195873:E a2d94e7b-c618-4ef2-ba54-3c84e5d1ca60 MKDIR 16877 0 0 a187b7b5-c45d-4e9e-8ca1-b368ed418f34%2Flimits.d
CHANGELOG.1384195873:E 9d31eae4-1c59-408a-8793-cdd961941984 CREATE 33188 0 0 a187b7b5-c45d-4e9e-8ca1-b368ed418f34%2Faccess.conf
CHANGELOG.1384195873:E 280dd55c-fbba-43ec-8621-0bee4751aeb3 CREATE 33188 0 0 a187b7b5-c45d-4e9e-8ca1-b368ed418f34%2Flimits.conf
CHANGELOG.1384195873:E a9bce17d-62d5-462c-911f-5eb30687df31 CREATE 33188 0 0 a187b7b5-c45d-4e9e-8ca1-b368ed418f34%2Fchroot.conf
CHANGELOG.1384195873:E 23de9fdd-7c50-4e77-b780-8b90d77a6de8 CREATE 33261 0 0 a187b7b5-c45d-4e9e-8ca1-b368ed418f34%2Fnamespace.init
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Therefore, I'm marking this BZ as verified. Anything else related to changelog consumption performance will be raised by the perf team.

[Clearing needinfo flag]
Comment 9 errata-xmlrpc 2013-11-27 10:44:01 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1769.html
Comment 10 Aravinda VK 2015-01-05 03:32:52 EST
*** Bug 1078961 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.