Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1614654

Summary: Potential fixes for tests/basic/afr/add-brick-self-heal.t failure
Product: [Community] GlusterFS Reporter: Ravishankar N <ravishankar>
Component: testsAssignee: Ravishankar N <ravishankar>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: unspecified Docs Contact:
Priority: medium    
Version: mainlineCC: bugs
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-5.0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-10-23 15:16:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ravishankar N 2018-08-10 07:14:01 UTC 
Description of problem:

From the runs captured at https://review.gluster.org/#/c/20637/, I saw that the latest runs where this particular .t failed were at https://build.gluster.org/job/line-coverage/415 and https://build.gluster.org/job/line-coverage/421/.
In both of these runs, there are no gluster 'regression' logs available at https://build.gluster.org/job/line-coverage/<build-number>/artifact. I have raised BZ 1613721 for it.

Jenkins output is:
20:37:43 ./tests/basic/afr/add-brick-self-heal.t .. 
20:37:43 1..34
20:37:43 ok 1, LINENUM:6
20:37:43 ok 2, LINENUM:7
20:37:43 ok 3, LINENUM:8
20:37:43 ok 4, LINENUM:9
20:37:43 ok 5, LINENUM:10
20:37:43 ok 6, LINENUM:11
20:37:43 ok 7, LINENUM:12
20:37:43 ok 8, LINENUM:14
20:37:43 ok 9, LINENUM:15
20:37:43 ok 10, LINENUM:24
20:37:43 ok 11, LINENUM:27
20:37:43 ok 12, LINENUM:30
20:37:43 ok 13, LINENUM:31
20:37:43 ok 14, LINENUM:34
20:37:43 ok 15, LINENUM:35
20:37:43 ok 16, LINENUM:36
20:37:43 ok 17, LINENUM:38
20:37:43 ok 18, LINENUM:39
20:37:43 ok 19, LINENUM:40
20:37:43 ok 20, LINENUM:42
20:37:43 ok 21, LINENUM:43
20:37:43 ok 22, LINENUM:44
20:37:43 ok 23, LINENUM:45
20:37:43 ok 24, LINENUM:46
20:37:43 ok 25, LINENUM:47
20:37:43 not ok 26 Got "10" instead of "^0$", LINENUM:50
20:37:43 FAILED COMMAND: ^0$ get_pending_heal_count patchy
20:37:43 ok 27, LINENUM:53
20:37:43 ok 28, LINENUM:54
20:37:43 not ok 29 , LINENUM:57
20:37:43 FAILED COMMAND: diff /d/backends/patchy0/file1.txt /d/backends/patchy2/file1.txt
20:37:43 not ok 30 Got "" instead of "qwerty", LINENUM:60
20:37:43 FAILED COMMAND: qwerty get_text_xattr user.test /d/backends/patchy2/file5.txt
20:37:43 ok 31, LINENUM:61
20:37:43 ok 32, LINENUM:63
20:37:43 ok 33, LINENUM:64
20:37:43 ok 34, LINENUM:65
20:37:43 Failed 3/34 subtests 
20:37:43 


So self-heal has failed to heal the entries. This could be due to bricks not being up/ shd not able to connect to the bricks/ lock contention and wait for $HEAL_TIMEOUT as described in BZ 1613807.

Hence modifying the .t to check for brick up status and reducing heal timeout to 5 seconds.


Note:
After the above said runs, the test has not failed again, up to the latest revision 8 of https://review.gluster.org/#/c/2063. So sending the patch with whatever can be theoretically fixed in the .t
Comment 1 Worker Ant 2018-08-10 07:17:21 UTC 
REVIEW: https://review.gluster.org/20696 (tests: potential fixes for tests/basic/afr/add-brick-self-heal.t) posted (#1) for review on master by Ravishankar N
Comment 2 Worker Ant 2018-08-13 04:56:51 UTC 
COMMIT: https://review.gluster.org/20696 committed in master by "Atin Mukherjee" <amukherj> with a commit message- tests: potential fixes for tests/basic/afr/add-brick-self-heal.t

Please see bug description for details.

Change-Id: Ieb6bce6d1d5c4c31f1878dd1a1c3d007d8ff81d5
fixes: bz#1614654
Signed-off-by: Ravishankar N <ravishankar>
Comment 3 Shyamsundar 2018-10-23 15:16:52 UTC 
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-5.0, please open a new bug report.

glusterfs-5.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] https://lists.gluster.org/pipermail/announce/2018-October/000115.html
[2] https://www.gluster.org/pipermail/gluster-users/