Description of problem: By "asymptotic synchronization" we mean that any particular change on master side gets synchronized to slave at some time. The geo-rep model theoretically delivers asymptotic synchronization, but it's not robust: if the gsyncd worker is interrupted more frequently than the time needed for a complete crawl (eg. due to network failures, panicky slave, or aux glusterfs leaking up to triggering the OOM killer), then some files will never get to the slave end, due to the deterministic order of walking through the file tree. Solution is to randomize the walk. Version-Release number of selected component (if applicable): How reproducible: Well reproducible, but deeming if the issue appears is not easy to automate. Steps to Reproduce: 1. create a file tree in some volume that's bigger than being possible to sync in a minute 2. start geo-rep with the above volume as master and empty slave 3. in each minute, stop and re-start geo-rep Actual results: Some files of master never appear on slave side. Expected results: Eventually all files of master should appear on slave side. Additional info:
To add, the above description, while gives a good insight to the issue at hand, is a bit of oversimplification. Assuming that the worker is always interrupted early: - If we have a static file tree to sync over (as in the repro instructions), it will be synced over even with a deterministic traversal. - If there are ongoing changes in the tree, there might be deep locations in the file tree where the synchronization activity never reaches (or with very low probability), even with randomized traversal. Regardless, randomized traversal will provide a more even distribution of synchronization, tending to a broader coverage, something like this: Sync coverage with deterministic traversal: /\ /**\ /** \ /** \ /* \ /* \ Sync coverage with random traversal: /\ /**\ /****\ /** ** \ / * * \ / \ So in fact, the whole thing can be investigated only heuristically, for which purpose the actual scenario where this came to picture is quite suitable -- ie. in a case where aux glusterfs always leaks till OOM, untar a bunch of kernel trees under geo-rep.
CHANGE: http://review.gluster.com/3079 (geo-rep / gsyncd: shuffle directory entries in crawl) merged in master by Vijay Bellur (vijay)