Geo-replication invokes stat() for all entry and data operations found in a changelog (and possibly will do it for metadata operations too). For mkdirs, creates and mknods this stat can be avoided by storing the relevant information in the changelog itself. This would result in faster changelog processing rates and faster sync times.
I am seeing very bad performance of geo-rep with the small files. Started creating 101838 small files and immediately started geo-rep. It took 178m8.332s time to create small files. But even after 15 hours only 60475 files have been created. Volume set options rollover-time was set to 15 secs and fsync-interval was set to 5 secs as suggested by the perf team. There were no server reboots or downtime.
(In reply to M S Vishwanath Bhat from comment #2) > I am seeing very bad performance of geo-rep with the small files. > > Started creating 101838 small files and immediately started geo-rep. It > took 178m8.332s time to create small files. But even after 15 hours only > 60475 files have been created. Volume set options rollover-time was set to > 15 secs and fsync-interval was set to 5 secs as suggested by the perf team. > There were no server reboots or downtime. As I asked before, did you get a chance to measure rsync times? If not, please let me know the hostnames of the master cluster.
(In reply to Venky Shankar from comment #3) > (In reply to M S Vishwanath Bhat from comment #2) > > I am seeing very bad performance of geo-rep with the small files. > > > > Started creating 101838 small files and immediately started geo-rep. It > > took 178m8.332s time to create small files. But even after 15 hours only > > 60475 files have been created. Volume set options rollover-time was set to > > 15 secs and fsync-interval was set to 5 secs as suggested by the perf team. > > There were no server reboots or downtime. > > As I asked before, did you get a chance to measure rsync times? If not, > please let me know the hostnames of the master cluster. No I haven't measured it yet. I am running some other tests there right now. You can login into 10.70.42.192 and attach to a screen session to have access to the complete cluster.
Since there was some issues in patch, more work required.
https://code.engineering.redhat.com/gerrit/15354
Venky, Amar, How do I verify this? Just looking at the changelog and noting the extra information logged in there for mkdirs is fine? Or do I need to do something more to verify this?
After having discussion with Venky, I found out that I can _conceptually_ verify this BZ, Neependra is anyway verifying the perf improvements using perf. tools and other things. So, conceptually, mkdir(), create() and mknod() have stat(2) [not all, but enough for entry creations] information _embedded_ in the CHANGELOG, which implies there is no need to do a stat() on the mount to get this information. parts of changelog: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CHANGELOG.1384195873:E 1763c452-d495-4f28-9a05-e7aedb3931ea MKDIR 16877 0 0 00000000-0000-0000-0000-000000000001%2Fetc.1 CHANGELOG.1384195873:E 8e927f9a-3dd9-4d6b-b6c0-a0e50dde5044 MKDIR 16877 0 0 1763c452-d495-4f28-9a05-e7aedb3931ea%2Fskel CHANGELOG.1384195873:E c17c7836-eb38-43f3-8317-ab2dfb2da5b8 CREATE 33188 0 0 8e927f9a-3dd9-4d6b-b6c0-a0e50dde5044%2F.emacs CHANGELOG.1384195873:E 39de31cb-ade5-49de-a9f7-b9247a10aca5 CREATE 33188 0 0 8e927f9a-3dd9-4d6b-b6c0-a0e50dde5044%2F.bash_logout CHANGELOG.1384195873:E feb75320-b3d2-4c7c-933d-54606a5e0244 MKDIR 16877 0 0 1763c452-d495-4f28-9a05-e7aedb3931ea%2Fcron.d CHANGELOG.1384195873:E 9d67a4ac-e58d-418d-90e0-2636490ecfa8 CREATE 33188 0 0 feb75320-b3d2-4c7c-933d-54606a5e0244%2F0hourly CHANGELOG.1384195873:E 9d775a44-8bab-42ac-bf25-9a61d81557ba CREATE 33188 0 0 feb75320-b3d2-4c7c-933d-54606a5e0244%2Fvdsm-libvirt-logrotate CHANGELOG.1384195873:E a187b7b5-c45d-4e9e-8ca1-b368ed418f34 MKDIR 16877 0 0 1763c452-d495-4f28-9a05-e7aedb3931ea%2Fsecurity CHANGELOG.1384195873:E e1c9ec7a-850f-45e9-8f9b-28e2fa451259 MKDIR 16877 0 0 a187b7b5-c45d-4e9e-8ca1-b368ed418f34%2Fnamespace.d CHANGELOG.1384195873:E 2223e86d-0d41-4748-8272-7b416009af00 CREATE 33188 0 0 a187b7b5-c45d-4e9e-8ca1-b368ed418f34%2Ftime.conf CHANGELOG.1384195873:E a2d94e7b-c618-4ef2-ba54-3c84e5d1ca60 MKDIR 16877 0 0 a187b7b5-c45d-4e9e-8ca1-b368ed418f34%2Flimits.d CHANGELOG.1384195873:E 9d31eae4-1c59-408a-8793-cdd961941984 CREATE 33188 0 0 a187b7b5-c45d-4e9e-8ca1-b368ed418f34%2Faccess.conf CHANGELOG.1384195873:E 280dd55c-fbba-43ec-8621-0bee4751aeb3 CREATE 33188 0 0 a187b7b5-c45d-4e9e-8ca1-b368ed418f34%2Flimits.conf CHANGELOG.1384195873:E a9bce17d-62d5-462c-911f-5eb30687df31 CREATE 33188 0 0 a187b7b5-c45d-4e9e-8ca1-b368ed418f34%2Fchroot.conf CHANGELOG.1384195873:E 23de9fdd-7c50-4e77-b780-8b90d77a6de8 CREATE 33261 0 0 a187b7b5-c45d-4e9e-8ca1-b368ed418f34%2Fnamespace.init >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Therefore, I'm marking this BZ as verified. Anything else related to changelog consumption performance will be raised by the perf team. [Clearing needinfo flag]
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-1769.html
*** Bug 1078961 has been marked as a duplicate of this bug. ***