With the new journal format, Geo-replication daemon needs to be enhanced to adjust with the new format. Other enhancements includes using tar+ssh as the data synchronization engine, parallel processing of changelogs and faster initial crawl crawling/processing.
REVIEW: http://review.gluster.org/6404 (gsyncd / geo-rep: geo-replication fixes) posted (#2) for review on master by ajeet jha (ajha)
COMMIT: http://review.gluster.org/6404 committed in master by Anand Avati (avati) ------ commit f999c17da5a5353196e68e7a68af64f91df6b902 Author: Ajeet Jha <ajha> Date: Mon Dec 2 12:37:34 2013 +0530 gsyncd / geo-rep: geo-replication fixes -> "threaded" hybrid crawl. -> Enabling metatadata synchronization. -> Handling EINVAL/ESTALE gracefully while syncing metadata. -> Improvments to changelog crawl code. -> Initial crawl changelog generation format. -> No gsyncd restart when checkpoint updated. -> Fix symlink handling in hybrid crawl. -> Slave's xtime key is 'stime'. -> tar+ssh as data synchronization. -> Instead of 'raise', just log in warning level for xtime missing cases. -> Fix for JSON object load failure -> Get new config value after config value reset. -> Skip already processed changelogs. -> Saving status of each individual worker thread. -> GFID fetch on slave for purges. -> Add tar ssh keys and config options. -> Fix nlink count when using backend. -> Include "data" operation for hardlink. -> Use changelog time prefix as slave's time. -> Process changelogs in parallel. Change-Id: I09fcbb2e2e418149a6d8435abd2ac6b2f015bb06 BUG: 1036539 Signed-off-by: Ajeet Jha <ajha> Reviewed-on: http://review.gluster.org/6404 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Anand Avati <avati>
REVIEW: http://review.gluster.org/6809 (gsyncd / geo-rep: geo-replication fixes) posted (#1) for review on release-3.5 by ajeet jha (ajha)
COMMIT: http://review.gluster.org/6809 committed in release-3.5 by Vijay Bellur (vbellur) ------ commit 30592e7f92515c5df620f300d6a3df6373ac6200 Author: Ajeet Jha <ajha> Date: Mon Dec 2 12:37:34 2013 +0530 gsyncd / geo-rep: geo-replication fixes -> "threaded" hybrid crawl. -> Enabling metatadata synchronization. -> Handling EINVAL/ESTALE gracefully while syncing metadata. -> Improvments to changelog crawl code. -> Initial crawl changelog generation format. -> No gsyncd restart when checkpoint updated. -> Fix symlink handling in hybrid crawl. -> Slave's xtime key is 'stime'. -> tar+ssh as data synchronization. -> Instead of 'raise', just log in warning level for xtime missing cases. -> Fix for JSON object load failure -> Get new config value after config value reset. -> Skip already processed changelogs. -> Saving status of each individual worker thread. -> GFID fetch on slave for purges. -> Add tar ssh keys and config options. -> Fix nlink count when using backend. -> Include "data" operation for hardlink. -> Use changelog time prefix as slave's time. -> Process changelogs in parallel. Change-Id: I09fcbb2e2e418149a6d8435abd2ac6b2f015bb06 BUG: 1036539 Signed-off-by: Ajeet Jha <ajha> Reviewed-on: http://review.gluster.org/6404 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Anand Avati <avati> Reviewed-on: http://review.gluster.org/6809 Reviewed-by: Vijay Bellur <vbellur>
REVIEW: http://review.gluster.org/6845 (geo-rep: get new config value after config value reset) posted (#2) for review on release-3.5 by Aravinda VK (avishwan)
REVIEW: http://review.gluster.org/6847 (geo-rep: optimizing update stime after directory synchronization) posted (#2) for review on release-3.5 by Aravinda VK (avishwan)
REVIEW: http://review.gluster.org/6858 (gsyncd / geo-rep: "patch" up missing stime) posted (#1) for review on release-3.5 by Venky Shankar (vshankar)
REVIEW: http://review.gluster.org/6859 (gsyncd / geo-rep: ignore DHTs sticky bit file during crawl) posted (#1) for review on release-3.5 by Venky Shankar (vshankar)
REVIEW: http://review.gluster.org/6860 (gsyncd / geo-rep: invoke changelog process() on non-empty change list) posted (#1) for review on release-3.5 by Venky Shankar (vshankar)
REVIEW: http://review.gluster.org/6861 (gsyncd / geo-rep: cleanup the "tar" process) posted (#1) for review on release-3.5 by Venky Shankar (vshankar)
REVIEW: http://review.gluster.org/6858 (gsyncd / geo-rep: "patch" up missing stime) posted (#2) for review on release-3.5 by Venky Shankar (vshankar)
REVIEW: http://review.gluster.org/6859 (gsyncd / geo-rep: ignore DHTs sticky bit file during crawl) posted (#2) for review on release-3.5 by Venky Shankar (vshankar)
REVIEW: http://review.gluster.org/6860 (gsyncd / geo-rep: invoke changelog process() on non-empty change list) posted (#2) for review on release-3.5 by Venky Shankar (vshankar)
REVIEW: http://review.gluster.org/6861 (gsyncd / geo-rep: cleanup the "tar" process) posted (#2) for review on release-3.5 by Venky Shankar (vshankar)
COMMIT: http://review.gluster.org/6847 committed in release-3.5 by Vijay Bellur (vbellur) ------ commit 540de26e2ef7c45eb0ef6c049844a864542bd38a Author: Aravinda VK <avishwan> Date: Wed Dec 18 00:02:58 2013 +0530 geo-rep: optimizing update stime after directory synchronization Since xsync crawl generates new changelog when number of entries reaches 8K or when directory is reached. If a directory has number of files less than 8K then respective changelog file will have less entries. Since xsync generated changelog files processed one after the other, so syncjobs are underutilized. hence low bandwidth utilization. With this patch, changelog will be generated for 8K entries only, but stime will be accumulated. Multiple dirs stime will be updated together since the generated changelog will have entries accross the dirs. BUG: 1036539 Change-Id: Ic4b9da6d8c1f11dd9ef7c13bcc7df80efe2c5caf Signed-off-by: Aravinda VK <avishwan> Reviewed-on: http://review.gluster.org/6744 Reviewed-on: http://review.gluster.org/6847 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Venky Shankar <vshankar> Reviewed-by: Vijay Bellur <vbellur>
COMMIT: http://review.gluster.org/6845 committed in release-3.5 by Vijay Bellur (vbellur) ------ commit ab55d60e2615bd47a9b7ad3da4e85ba1be4c3781 Author: Aravinda VK <avishwan> Date: Thu Nov 7 12:52:23 2013 +0530 geo-rep: get new config value after config value reset When config.read is called it preserves the previous values from the the previously opened config file. For example import ConfigParser config = ConfigParser.RawConfigParser() config.read("defaults.conf") config.read("preferences.conf") When change in config file is identified it will open new instance of config to avoid getting old config values. BUG: 1036539 Change-Id: Ie3226e4a75cbff9e69ad2d7cb5f4daf45c6ae0c8 Signed-off-by: Aravinda VK <avishwan> Reviewed-on: http://review.gluster.org/6747 Reviewed-on: http://review.gluster.org/6845 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Venky Shankar <vshankar>
REVIEW: http://review.gluster.org/6792 (gsyncd / geo-rep: ignore DHTs sticky bit file during crawl) posted (#3) for review on master by Venky Shankar (vshankar)
REVIEW: http://review.gluster.org/6793 (gsyncd / geo-rep: invoke changelog process() on non-empty change list) posted (#3) for review on master by Venky Shankar (vshankar)
REVIEW: http://review.gluster.org/6794 (gsyncd / geo-rep: cleanup the "tar" process) posted (#3) for review on master by Venky Shankar (vshankar)
REVIEW: http://review.gluster.org/6791 (gsyncd / geo-rep: "patch" up missing stime) posted (#3) for review on master by Venky Shankar (vshankar)
COMMIT: http://review.gluster.org/6794 committed in master by Vijay Bellur (vbellur) ------ commit 0035b37a2d19c1ba8502cda93d9f5debfdd35c45 Author: Venky Shankar <vshankar> Date: Mon Jan 20 19:38:44 2014 +0530 gsyncd / geo-rep: cleanup the "tar" process A missing cleanup for the "tar" process (when tar+ssh is used as the sync engine). Change-Id: Ib9599b43e7ec606c70b7c5598793417142be3c0b BUG: 1036539 Signed-off-by: Venky Shankar <vshankar> Reviewed-on: http://review.gluster.org/6794 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur>
COMMIT: http://review.gluster.org/6791 committed in master by Vijay Bellur (vbellur) ------ commit cbf6a264bd124c6b193709ad0e09c260c7d1845b Author: Venky Shankar <vshankar> Date: Mon Nov 18 19:13:27 2013 +0530 gsyncd / geo-rep: "patch" up missing stime In cases (mostly upgrade) of unavailability of "stime" key and availability of "xtime" (slave's xtime), introduce "stime" key on the fly by setting it to the value to "xtime". Change-Id: Iaa424662d838154c8abc2cf00830c7f9d6be45ac BUG: 1036539 Signed-off-by: Venky Shankar <vshankar> Reviewed-on: http://review.gluster.org/6791 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur>
COMMIT: http://review.gluster.org/6792 committed in master by Vijay Bellur (vbellur) ------ commit aa7b0c640818df25b4f0297b1931ff52a77121bf Author: Venky Shankar <vshankar> Date: Thu Dec 12 17:03:09 2013 +0530 gsyncd / geo-rep: ignore DHTs sticky bit file during crawl Change-Id: Ide927759c6a3d5301475eac9f6e785aa901d426e BUG: 1036539 Signed-off-by: Venky Shankar <vshankar> Reviewed-on: http://review.gluster.org/6792 Reviewed-by: Kotresh HR <khiremat> Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur>
COMMIT: http://review.gluster.org/6793 committed in master by Vijay Bellur (vbellur) ------ commit 4d7764a73f4c504ebd369c61f46e569ebc62ecc9 Author: Venky Shankar <vshankar> Date: Wed Jan 8 10:22:48 2014 +0530 gsyncd / geo-rep: invoke changelog process() on non-empty change list Change-Id: Ida4890abdc90d683a4a83127a1573bbb3829ea23 BUG: 1036539 Signed-off-by: Venky Shankar <vshankar> Reviewed-on: http://review.gluster.org/6793 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur>
COMMIT: http://review.gluster.org/6858 committed in release-3.5 by Vijay Bellur (vbellur) ------ commit 5c926a14d8479b2a9aba6aa50478625d444bf42b Author: Venky Shankar <vshankar> Date: Mon Nov 18 19:13:27 2013 +0530 gsyncd / geo-rep: "patch" up missing stime In cases (mostly upgrade) of unavailability of "stime" key and availability of "xtime" (slave's xtime), introduce "stime" key on the fly by setting it to the value to "xtime". Change-Id: Iaa424662d838154c8abc2cf00830c7f9d6be45ac BUG: 1036539 Signed-off-by: Venky Shankar <vshankar> Reviewed-on: http://review.gluster.org/6791 Reviewed-on: http://review.gluster.org/6858 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur>
COMMIT: http://review.gluster.org/6859 committed in release-3.5 by Vijay Bellur (vbellur) ------ commit 7289fa908b4355a148adb610534aceee8761a547 Author: Venky Shankar <vshankar> Date: Wed Jan 29 20:27:12 2014 +0530 gsyncd / geo-rep: ignore DHTs sticky bit file during crawl Change-Id: I0a6eeee8ae192d086b34d316ee1b82c7d76634da BUG: 1036539 Signed-off-by: Venky Shankar <vshankar> Reviewed-on: http://review.gluster.org/6792 Reviewed-on: http://review.gluster.org/6859 Reviewed-by: Kotresh HR <khiremat> Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Aravinda VK <avishwan>
COMMIT: http://review.gluster.org/6860 committed in release-3.5 by Vijay Bellur (vbellur) ------ commit a2355deb62f647e257f3ffc8a33f9d831f211964 Author: Venky Shankar <vshankar> Date: Wed Jan 8 10:22:48 2014 +0530 gsyncd / geo-rep: invoke changelog process() on non-empty change list Change-Id: Ida4890abdc90d683a4a83127a1573bbb3829ea23 BUG: 1036539 Signed-off-by: Venky Shankar <vshankar> Reviewed-on: http://review.gluster.org/6793 Reviewed-on: http://review.gluster.org/6860 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Aravinda VK <avishwan>
COMMIT: http://review.gluster.org/6861 committed in release-3.5 by Vijay Bellur (vbellur) ------ commit 692202e1ef7a5d009970e96e4933edb3231716b5 Author: Venky Shankar <vshankar> Date: Mon Jan 20 19:38:44 2014 +0530 gsyncd / geo-rep: cleanup the "tar" process A missing cleanup for the "tar" process (when tar+ssh is used as the sync engine). Change-Id: Ib9599b43e7ec606c70b7c5598793417142be3c0b BUG: 1036539 Signed-off-by: Venky Shankar <vshankar> Reviewed-on: http://review.gluster.org/6794 Reviewed-on: http://review.gluster.org/6861 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur>
REVIEW: http://review.gluster.org/6926 (spec: geo-rep: Include hooks and upgrade scripts) posted (#2) for review on master by Aravinda VK (avishwan)
REVIEW: http://review.gluster.org/6927 (spec: geo-rep: Include hooks and upgrade scripts) posted (#1) for review on release-3.5 by Aravinda VK (avishwan)
REVIEW: http://review.gluster.org/6934 (geo-rep: Config file upgrade) posted (#1) for review on master by Aravinda VK (avishwan)
REVIEW: http://review.gluster.org/6935 (geo-rep: Config file upgrade) posted (#1) for review on release-3.5 by Aravinda VK (avishwan)
REVIEW: http://review.gluster.org/6927 (spec: geo-rep: Include hooks and upgrade scripts) posted (#2) for review on release-3.5 by Aravinda VK (avishwan)
REVIEW: http://review.gluster.org/6939 (gverify: Correcting variable names) posted (#1) for review on master by Avra Sengupta (asengupt)
COMMIT: http://review.gluster.org/6939 committed in master by Vijay Bellur (vbellur) ------ commit 7903c55a0d488f93790f5e80105f5252b88f32af Author: Avra Sengupta <asengupt> Date: Fri Feb 7 06:04:23 2014 +0000 gverify: Correcting variable names Change-Id: I851a50e0ebb7554b821fb591375bb6ebd6240ade BUG: 1036539 Signed-off-by: Avra Sengupta <asengupt> Reviewed-on: http://review.gluster.org/6939 Reviewed-by: Venky Shankar <vshankar> Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur>
REVIEW: http://review.gluster.org/6926 (spec: geo-rep: Include hooks and upgrade scripts) posted (#3) for review on master by Aravinda VK (avishwan)
REVIEW: http://review.gluster.org/6927 (spec: geo-rep: Include hooks and upgrade scripts) posted (#3) for review on release-3.5 by Aravinda VK (avishwan)
COMMIT: http://review.gluster.org/6926 committed in master by Vijay Bellur (vbellur) ------ commit d4e7dd1d2fe2a76318ddd0a7200a9d4fa51dbaf7 Author: Aravinda VK <avishwan> Date: Thu Feb 6 17:16:20 2014 +0530 spec: geo-rep: Include hooks and upgrade scripts Added hook script S56glusterd-geo-rep-create-post.sh to %{_sharedstatedir}/glusterd/hooks/1/gsync-create/post/ and upgrade scripts added to %{_datadir}/glusterfs/scripts/ BUG: 1036539 Change-Id: Id114b827824e55516181ab8f564d419c258be3e2 Signed-off-by: Aravinda VK <avishwan> Reviewed-on: http://review.gluster.org/6926 Reviewed-by: Niels de Vos <ndevos> Tested-by: Niels de Vos <ndevos> Tested-by: Gluster Build System <jenkins.com>
COMMIT: http://review.gluster.org/6934 committed in master by Vijay Bellur (vbellur) ------ commit b000b934aff4b452cf1c35c42272482a7738506e Author: Aravinda VK <avishwan> Date: Mon Jan 20 12:23:35 2014 +0530 geo-rep: Config file upgrade When old config file is used with new geo-rep, config item like 'georep_session_working_dir' was missing in old config file. With this patch geo-rep sets the default value for new items. Following config options supported: - georep_session_working_dir - gluster_params - ssh_command_tar BUG: 1036539 Change-Id: I389c62e749f3b567f9ecf96d4b41367ef962c025 Signed-off-by: Aravinda VK <avishwan> Reviewed-on: http://review.gluster.org/6934 Reviewed-by: Venky Shankar <vshankar> Tested-by: Gluster Build System <jenkins.com>
COMMIT: http://review.gluster.org/6935 committed in release-3.5 by Vijay Bellur (vbellur) ------ commit 8014bf281294e155dfbc7041cfef5151ae4f19e5 Author: Aravinda VK <avishwan> Date: Mon Jan 20 12:23:35 2014 +0530 geo-rep: Config file upgrade When old config file is used with new geo-rep, config item like 'georep_session_working_dir' was missing in old config file. With this patch geo-rep sets the default value for new items. Following config options supported: - georep_session_working_dir - gluster_params - ssh_command_tar BUG: 1036539 Change-Id: If277a14d21f4caee568908b73a2988352c08dcc8 Reviewed-on: http://review.gluster.org/6934 Signed-off-by: Aravinda VK <avishwan> Reviewed-on: http://review.gluster.org/6935 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur>
COMMIT: http://review.gluster.org/6927 committed in release-3.5 by Vijay Bellur (vbellur) ------ commit fbf509dddf0745d360fe560a99dc593c542cfe61 Author: Aravinda VK <avishwan> Date: Thu Feb 6 17:48:19 2014 +0530 spec: geo-rep: Include hooks and upgrade scripts Added hook script S56glusterd-geo-rep-create-post.sh to %{_sharedstatedir}/glusterd/hooks/1/gsync-create/post/ and upgrade scripts added to %{_datadir}/glusterfs/scripts/ BUG: 1036539 Change-Id: I53f6984671feecbc2890fed59d8fc7f4545c50a3 Reviewed-on: http://review.gluster.org/6926 Signed-off-by: Aravinda VK <avishwan> Reviewed-on: http://review.gluster.org/6927 Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Vijay Bellur <vbellur>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.5.0, please reopen this bug report. glusterfs-3.5.0 has been announced on the Gluster Developers mailinglist [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/6137 [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user
*** Bug 1022992 has been marked as a duplicate of this bug. ***
*** Bug 1023998 has been marked as a duplicate of this bug. ***
*** Bug 1024472 has been marked as a duplicate of this bug. ***