Description of problem: Race conditions can occur while processing changelogs(by geo-replication) in Tiering based volume, where rebalanace can cause frequent movement of files between hot and cold tiers. Also, rebalance operations are internal operations which needs to be avoided. (Deletion of the file and Creation of the file while carrying out rebalance). Version-Release number of selected component (if applicable): How reproducible: Following is one such example: ================================== Brick1 Brick2 ================================== Create file (file moved due to rebalance). Data file Delete file ================================== If Brick2 changelogs processed first followed by Brick1, file may be created. But, we expect the file to be deleted (as per the last operation) Steps to Reproduce: 1. 2. 3. Actual results: Files are not synced properly. Expected results: Files are synced as expected. Additional info:
Patches : http://review.gluster.org/#/c/12239/ http://review.gluster.org/#/c/12326/ Ignoring cold brick namespace related fops is pending.
REVIEW: http://review.gluster.org/12355 (geo-rep: Avoid cold tier bricks during ENTRY operation) posted (#1) for review on master by Saravanakumar Arumugam (sarumuga)
REVIEW: http://review.gluster.org/12326 (geo-rep: Add data operation if mknod with tier attribute) posted (#2) for review on master by Saravanakumar Arumugam (sarumuga)
REVIEW: http://review.gluster.org/12326 (geo-rep: Add data operation if mknod with tier attribute) posted (#3) for review on master by Saravanakumar Arumugam (sarumuga)
REVIEW: http://review.gluster.org/12239 (geo-rep: ignore recording tiering rebalance fops) posted (#4) for review on master by Saravanakumar Arumugam (sarumuga)
REVIEW: http://review.gluster.org/12355 (geo-rep: Avoid cold tier bricks during ENTRY operation) posted (#2) for review on master by Saravanakumar Arumugam (sarumuga)
REVIEW: http://review.gluster.org/12326 (geo-rep: Add data operation if mknod with tier attribute) posted (#4) for review on master by Saravanakumar Arumugam (sarumuga)
REVIEW: http://review.gluster.org/12239 (geo-rep: ignore recording tiering rebalance fops) posted (#5) for review on master by Saravanakumar Arumugam (sarumuga)
REVIEW: http://review.gluster.org/12326 (geo-rep: Add data operation if mknod with tier attribute) posted (#5) for review on master by Saravanakumar Arumugam (sarumuga)
REVIEW: http://review.gluster.org/12326 (geo-rep: Add data operation if mknod with tier attribute) posted (#6) for review on master by Saravanakumar Arumugam (sarumuga)
REVIEW: http://review.gluster.org/12326 (geo-rep: Add data operation if mknod with tier attribute) posted (#7) for review on master by Saravanakumar Arumugam (sarumuga)
REVIEW: http://review.gluster.org/12326 (geo-rep: Add data operation if mknod with tier attribute) posted (#8) for review on master by Saravanakumar Arumugam (sarumuga)
REVIEW: http://review.gluster.org/12239 (features/changelog: ignore recording tiering rebalance fops) posted (#6) for review on master by Saravanakumar Arumugam (sarumuga)
REVIEW: http://review.gluster.org/12239 (features/changelog: ignore recording tiering rebalance fops) posted (#7) for review on master by Saravanakumar Arumugam (sarumuga)
REVIEW: http://review.gluster.org/12417 ( features/changelog: Add data operation if mknod with tier attribute) posted (#1) for review on master by Saravanakumar Arumugam (sarumuga)
REVIEW: http://review.gluster.org/12326 (geo-rep: Add data operation if mknod with tier attribute) posted (#9) for review on master by Saravanakumar Arumugam (sarumuga)
REVIEW: http://review.gluster.org/12417 (features/changelog: Add data operation if mknod with tier attribute) posted (#2) for review on master by Saravanakumar Arumugam (sarumuga)
REVIEW: http://review.gluster.org/12355 (geo-rep: Avoid cold tier bricks during ENTRY operation) posted (#3) for review on master by Saravanakumar Arumugam (sarumuga)
REVIEW: http://review.gluster.org/12417 (features/changelog: Add data operation if mknod with tier attribute) posted (#3) for review on master by Saravanakumar Arumugam (sarumuga)
REVIEW: http://review.gluster.org/12417 (features/changelog: Add data operation if mknod with tier attribute) posted (#4) for review on master by Saravanakumar Arumugam (sarumuga)
REVIEW: http://review.gluster.org/12239 (features/changelog: ignore recording tiering rebalance fops) posted (#8) for review on master by Saravanakumar Arumugam (sarumuga)
REVIEW: http://review.gluster.org/12417 (features/changelog: Add data operation if mknod with tier attribute) posted (#5) for review on master by Saravanakumar Arumugam (sarumuga)
REVIEW: http://review.gluster.org/12417 (features/changelog: record mknod if tier-dht linkto is set) posted (#6) for review on master by Saravanakumar Arumugam (sarumuga)
REVIEW: http://review.gluster.org/12425 (features/changelog: ignore recording tiering rebalance fops) posted (#1) for review on release-3.7 by Saravanakumar Arumugam (sarumuga)
COMMIT: http://review.gluster.org/12326 committed in master by Venky Shankar (vshankar) ------ commit ffc39c9d8807464b5c78959bc43dc12b22f5a37b Author: Saravanakumar Arumugam <sarumuga> Date: Fri Oct 9 20:29:30 2015 +0530 geo-rep: Add data operation if mknod with tier attribute This is a series of patches which aims to fix geo-replication in a Tiering Volume. Problem: Consider, a file is placed in volume initially and then hot tier is attached. During any operation on the file, due to lookup a linkto file is created in hot tier. Now, any namespace operation carried out on the file is recorded in both cold and hot tier. There is a room for races when both changelogs are replayed. Solution: So, We are going to replay (namespace related)operations only in the hot tier. Why? a. If the file is directly placed in Hot tier, all fops will be recorded in HOT tier. b. If the file is already present in Cold tier, and if any fop is carried out, it creates linkto file in Hot tier. Now, operations like UNLINK, RENAME are captured in Hot tier(by means of linkto file). This way, we can get both tier's operation in HOT tier itself. But, We may miss initial Data sync immediately after creating the file as it is only recording MKNOD. So, if MKNOD encountered with sticky bit set, queue DATA operation for the corresponding gfid. (This is addressed here in this patch) So, If tier-gfid linkto is set, we need to record the corresponding MKNOD. Earlier this was avoided as it was set as INTERNAL fop. (This changelog related changes are addressed in the patch: - http://review.gluster.org/12417) Change-Id: I2fa84cfa2b0f86506c3d15d484138ab9651e4f83 BUG: 1266875 Signed-off-by: Saravanakumar Arumugam <sarumuga> Reviewed-on: http://review.gluster.org/12326 Tested-by: NetBSD Build System <jenkins.org> Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Kotresh HR <khiremat> Reviewed-by: Aravinda VK <avishwan>
REVIEW: http://review.gluster.org/12417 (features/changelog: record mknod if tier-dht linkto is set) posted (#7) for review on master by Saravanakumar Arumugam (sarumuga)
REVIEW: http://review.gluster.org/12427 (geo-rep: Add data operation if mknod with tier attribute) posted (#1) for review on release-3.7 by Saravanakumar Arumugam (sarumuga)
COMMIT: http://review.gluster.org/12355 committed in master by Venky Shankar (vshankar) ------ commit 6188b5fcebc56b3d8af1956beeec9988f3e8f268 Author: Saravanakumar Arumugam <sarumuga> Date: Wed Oct 14 11:49:49 2015 +0530 geo-rep: Avoid cold tier bricks during ENTRY operation This is a series of patch which aims to fix geo-replication in a Tiering Volume. Problem: Consider, a file is placed in volume initially and then hot tier is attached. During any operation on the file, due to lookup a linkto file is created in hot tier. Now, any namespace operation carried out on the file is recorded in both cold and hot tier. There is a room for races when both changelogs are replayed. Solution: So, We are going to replay (namespace related)operations only in the hot tier. Why? a. If the file is directly placed in Hot tier , all fops will be recorded in HOT tier. b. If the file is already present in Cold tier, and if any fop is carried out, it creates linkto file in Hot tier. Now, operations like UNLINK, RENAME are captured in Hot tier(by means of linkto file). This way, we can get both tier's operation in HOT tier itself. Now, once the file is demoted to COLD tier, any namespace operation carried out on the cold tier can be avoided as we directly RECORD the same in HOT tier. How? 1. Check whether the brick is cold tier and skip ENTRY operation. 2. Also, if it is cold tier brick, use Xsync(which is used during initial run). This will help in getting all cold tier bricks changes using File System crawl and helps in avoiding races with hot tier brick(which can happen if historychangelog used in cold tier brick). Dependent patches: 1. http://review.gluster.org/12239 2. http://review.gluster.org/12326 Change-Id: I7692b1dbb8813a7e253451bca02f8f09a5782dde BUG: 1266875 Signed-off-by: Saravanakumar Arumugam <sarumuga> Reviewed-on: http://review.gluster.org/12355 Tested-by: NetBSD Build System <jenkins.org> Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Aravinda VK <avishwan>
REVIEW: http://review.gluster.org/12428 (features/changelog: record mknod if tier-dht linkto is set) posted (#1) for review on release-3.7 by Saravanakumar Arumugam (sarumuga)
REVIEW: http://review.gluster.org/12429 (geo-rep: Avoid cold tier bricks during ENTRY operation) posted (#1) for review on release-3.7 by Saravanakumar Arumugam (sarumuga)
REVIEW: http://review.gluster.org/12417 (features/changelog: record mknod if tier-dht linkto is set) posted (#8) for review on master by Saravanakumar Arumugam (sarumuga)
REVIEW: http://review.gluster.org/12239 (features/changelog: ignore recording tiering rebalance fops) posted (#9) for review on master by Venky Shankar (vshankar)
REVIEW: http://review.gluster.org/12417 (features/changelog: record mknod if tier-dht linkto is set) posted (#9) for review on master by Saravanakumar Arumugam (sarumuga)
REVIEW: http://review.gluster.org/12417 (features/changelog: record mknod if tier-dht linkto is set) posted (#10) for review on master by Saravanakumar Arumugam (sarumuga)
REVIEW: http://review.gluster.org/12239 (features/changelog: ignore recording tiering rebalance fops) posted (#10) for review on master by Saravanakumar Arumugam (sarumuga)
COMMIT: http://review.gluster.org/12239 committed in master by Venky Shankar (vshankar) ------ commit 66b0caa639a179cfd699616d1fcae01c26ae6425 Author: Saravanakumar Arumugam <sarumuga> Date: Mon Sep 28 16:31:54 2015 +0530 features/changelog: ignore recording tiering rebalance fops Recording of tiering rebalance process's fops like Creation and Deletion of file must be avoided. Ignore the fops using corresponding pid. Change-Id: Ifdc7765598d04d033f93e6339e9b188f7566cb65 BUG: 1266875 Signed-off-by: Saravanakumar Arumugam <sarumuga> Reviewed-on: http://review.gluster.org/12239 Reviewed-by: Aravinda VK <avishwan> Reviewed-by: Venky Shankar <vshankar>
COMMIT: http://review.gluster.org/12417 committed in master by Venky Shankar (vshankar) ------ commit 17e95cb81776650a2f68be00298c4f85b41e4242 Author: Saravanakumar Arumugam <sarumuga> Date: Fri Oct 23 11:57:42 2015 +0530 features/changelog: record mknod if tier-dht linkto is set This is a series of patches which aims to fix geo-replication in a Tiering Volume. Problem: Consider, a file is placed in volume initially and then hot tier is attached. During any operation on the file, due to lookup a linkto file is created in hot tier. Now, any namespace operation carried out on the file is recorded in both cold and hot tier. There is a room for races when both changelogs are replayed. Solution: So, We are going to replay (namespace related)operations only in the hot tier. Why? a. If the file is directly placed in Hot tier, all fops will be recorded in HOT tier. b. If the file is already present in Cold tier, and if any fop is carried out, it creates linkto file in Hot tier. Now, operations like UNLINK, RENAME are captured in Hot tier(by means of linkto file). This way, we can get both tier's operation in HOT tier itself. But, We may miss initial Data sync immediately after creating the file as it is only recording MKNOD. So, if MKNOD encountered with sticky bit set, queue DATA operation for the corresponding gfid. ( This geo-rep related changes are addressed in this patch: http://review.gluster.org/12326/ ) So, If tier-dht linkto is set, we need to record the corresponding MKNOD. Earlier this was avoided as it was set as INTERNAL fop. (This is addressed here in this patch) Change-Id: I25514fe3e25f68592a8d6361507f8c8a4fcb70b1 BUG: 1266875 Signed-off-by: Saravanakumar Arumugam <sarumuga> Reviewed-on: http://review.gluster.org/12417 Reviewed-by: Aravinda VK <avishwan> Reviewed-by: Kotresh HR <khiremat> Tested-by: Gluster Build System <jenkins.com> Reviewed-by: Venky Shankar <vshankar>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report. glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/ [2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user