Bug 1332054

Adding the 'tracking' keyword so that our bug-status-check-script does not triple over it. Please remove the keyword when progress on this bug is made.

REVIEW: http://review.gluster.org/14138 (disperse: mark bug-1236065.t as bad_test) posted (#1) for review on master by Niels de Vos (ndevos)

I'm unable to reproduce the problem, however logs seem to indicate that healing operations are still running after a successful completion of test 'EXPECT_WITHIN $HEAL_TIMEOUT "0" get_pending_heal_count $V0'. Since additional bricks are killed after this test finishes, some files might get damaged as more that redundancy bricks will be bad, causing the I/O errors.

Most probably the root cause is that EXPECT_WITHIN uses a regular expression and a simple "0" matches many values, for example "10". This means that if exactly 10 files still need to be healed when the test is run, the test will finish successfully, but self-healing won't have finished yet.

I'll post a patch to solve this problem.

REVIEW: http://review.gluster.org/14145 (cluster/ec: Fix spurious failure of test bug-1236065.t) posted (#1) for review on master by Xavier Hernandez (xhernandez)

COMMIT: http://review.gluster.org/14138 committed in master by Jeff Darcy (jdarcy) 
------
commit 70a889489d79c41edfed52fdbdfa6869869906aa
Author: Niels de Vos <ndevos>
Date:   Sun May 1 22:49:57 2016 +0200

    disperse: mark bug-1236065.t as bad_test
    
    tests/bugs/disperse/bug-1236065.t failed several times on different
    Jenkins slaves:
    
    * https://build.gluster.org/job/rackspace-regression-2GB-triggered/20316/console
    * https://build.gluster.org/job/rackspace-regression-2GB-triggered/20320/console
    * https://build.gluster.org/job/rackspace-regression-2GB-triggered/20321/console
    
    BUG: 1332054
    Change-Id: Ie1934f09f843c2089c187e9295288c16c01913d2
    Signed-off-by: Niels de Vos <ndevos>
    Reviewed-on: http://review.gluster.org/14138
    Reviewed-by: Susant Palai <spalai>
    Smoke: Gluster Build System <jenkins.com>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Reviewed-by: Vijay Bellur <vbellur>
    CentOS-regression: Gluster Build System <jenkins.com>

(In reply to Xavier Hernandez from comment #3)
> I'm unable to reproduce the problem, however logs seem to indicate that
> healing operations are still running after a successful completion of test
> 'EXPECT_WITHIN $HEAL_TIMEOUT "0" get_pending_heal_count $V0'. Since
> additional bricks are killed after this test finishes, some files might get
> damaged as more that redundancy bricks will be bad, causing the I/O errors.
> 
> Most probably the root cause is that EXPECT_WITHIN uses a regular expression
> and a simple "0" matches many values, for example "10". This means that if
> exactly 10 files still need to be healed when the test is run, the test will
> finish successfully, but self-healing won't have finished yet.
> 
> I'll post a patch to solve this problem.

Good catch!, it could very well be this issue.

REVIEW: http://review.gluster.org/14985 (tests: Fix pending-heal-count checks) posted (#1) for review on master by Pranith Kumar Karampuri (pkarampu)

COMMIT: http://review.gluster.org/14985 committed in master by Jeff Darcy (jdarcy) 
------
commit c5bf5d98594a4237a72cf0d3c72925d5a5aa0f69
Author: Pranith Kumar K <pkarampu>
Date:   Fri Jul 22 13:58:22 2016 +0530

    tests: Fix pending-heal-count checks
    
    EXPECT_WITHIN takes regular expression to match the count,
    so even when there are say 10 entries to heal, it would
    think that the heal is complete. Fixed checking
    pending heal count with correct regex.
    
    Thanks to Xavi for finding this problem.
    
    Change-Id: Ic593d22468b2b586bfca864962ffa0eda96b1d1f
    BUG: 1332054
    Signed-off-by: Pranith Kumar K <pkarampu>
    Reviewed-on: http://review.gluster.org/14985
    Smoke: Gluster Build System <jenkins.org>
    Reviewed-by: Xavier Hernandez <xhernandez>
    NetBSD-regression: NetBSD Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>

REVIEW: http://review.gluster.org/15006 (tests: Fix get_pending_heal_count check in ec) posted (#1) for review on master by Ravishankar N (ravishankar)

REVIEW: http://review.gluster.org/15006 (tests: Fix get_pending_heal_count check in ec) posted (#2) for review on master by Ravishankar N (ravishankar)

COMMIT: http://review.gluster.org/15006 committed in master by Xavier Hernandez (xhernandez) 
------
commit 6c43efbb6b01726e450b71d274c3b45b56cc7916
Author: Ravishankar N <ravishankar>
Date:   Mon Jul 25 19:58:01 2016 +0530

    tests: Fix get_pending_heal_count check in ec
    
    Continuation of http://review.gluster.org/#/c/14985.
    Also renamed tests/bugs/disperse to tests/bugs/ec for a better
    correlation to tests/basic/ec and xlators/cluster/ec
    
    Change-Id: I662b3477c12af8a0b94597769e8f00f354b1168c
    BUG: 1332054
    Signed-off-by: Ravishankar N <ravishankar>
    Reviewed-on: http://review.gluster.org/15006
    NetBSD-regression: NetBSD Build System <jenkins.org>
    Smoke: Gluster Build System <jenkins.org>
    CentOS-regression: Gluster Build System <jenkins.org>
    Reviewed-by: Pranith Kumar Karampuri <pkarampu>
    Reviewed-by: Xavier Hernandez <xhernandez>