Description of problem: Here's one such failure run: https://build.gluster.org/job/centos6-regression/2290/consoleFull So turns out the entry changelog expected is 00000001 but sometimes it becomes 00000002 on $BRICK/a/b/c. And whenever this happens, the entry 'c' is marked as a newentry. So the only possibility is for 'c' to have undergone two entry post-ops - once through afr_mark_new_entry_changelog() and once as part of the creation of the file named 'file' under it, making the xattr value 00000002. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1.Run the test in the bug summary in a loop 2. 3. Actual results: Expected results: Additional info:
And whenever this issue is hit, the names a,b and c are healed into the bad brick, which probably means that name heal kicked in somewhere and did the new entry marking. [root@krutika-1 glusterfs-3.10dev]# cd /d/backends/ [root@krutika-1 backends]# ls * patchy0: a patchy1: a patchy2: a [root@krutika-1 backends]# ls */a patchy0/a: b patchy1/a: b patchy2/a: b [root@krutika-1 backends]# ls */a/b patchy0/a/b: c patchy1/a/b: c patchy2/a/b: c [root@krutika-1 backends]# ls */a/b/c patchy0/a/b/c: file patchy1/a/b/c: file patchy2/a/b/c: Will confirm the same by printing log messages everytime name heal happens.
Indeed it is name heal that is doing the new entry marking, as is evident from the following: [2016-12-19 06:41:28.916995]:++++++++++ G_LOG:tests/bugs/replicate/bug-1402730.t: TEST: 31 touch file ++++++++++ [2016-12-19 06:41:29.049544] I [MSGID: 108031] [afr-common.c:2335:afr_local_discovery_cbk] 0-patchy-replicate-0: selecting local read_child patchy-client-0 [2016-12-19 06:41:29.050808] I [MSGID: 108031] [afr-common.c:2335:afr_local_discovery_cbk] 0-patchy-replicate-0: selecting local read_child patchy-client-1 [2016-12-19 06:41:29.051916] I [MSGID: 108031] [afr-common.c:2335:afr_local_discovery_cbk] 0-patchy-replicate-0: selecting local read_child patchy-client-2 [2016-12-19 06:41:29.053024] I [afr-common.c:2508:afr_lookup_selfheal_wrap] 0-patchy-replicate-0: About to name heal a [2016-12-19 06:41:29.069815] I [afr-common.c:2508:afr_lookup_selfheal_wrap] 0-patchy-replicate-0: About to name heal b [2016-12-19 06:41:29.085733] I [afr-common.c:2508:afr_lookup_selfheal_wrap] 0-patchy-replicate-0: About to name heal c
REVIEW: http://review.gluster.org/16193 (tests: Fix spurious failure in tests/bugs/replicate/bug-1402730.t) posted (#1) for review on master by Krutika Dhananjay (kdhananj)
REVIEW: http://review.gluster.org/16193 (tests: Fix spurious failure in tests/bugs/replicate/bug-1402730.t) posted (#2) for review on master by Krutika Dhananjay (kdhananj)
COMMIT: http://review.gluster.org/16193 committed in master by Pranith Kumar Karampuri (pkarampu) ------ commit 8094e5005fbe9016ae039003bcc617d1172385b3 Author: Krutika Dhananjay <kdhananj> Date: Mon Dec 19 12:22:29 2016 +0530 tests: Fix spurious failure in tests/bugs/replicate/bug-1402730.t Replace the EXPECT '00000001' with EXPECT_NOT '00000000'. This is because occasionally a name-heal is performing new-entry marking on 'c' causing the pending entry changelog on it to become '00000002'. Change-Id: I30916e6266534d18899cfa5771c892db8c51ad9a BUG: 1405902 Signed-off-by: Krutika Dhananjay <kdhananj> Reviewed-on: http://review.gluster.org/16193 NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Smoke: Gluster Build System <jenkins.org> Reviewed-by: Pranith Kumar Karampuri <pkarampu>
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.10.0, please open a new bug report. glusterfs-3.10.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://lists.gluster.org/pipermail/gluster-users/2017-February/030119.html [2] https://www.gluster.org/pipermail/gluster-users/