Bug 1206461

Hi zhoushicheng,

So the test that is failing is:
EXPECT "1" has_holes $B0/${V0}0/big2bigger

The has_holes function returns true if(`stat -c '%b*%B-%s' file_name`)) -lt 0). I think this means that XFS has not freed the preallocated blocks despite the fd being closed by the time we reach this line in the testcase. Could you run the test lowering the speculative_prealloc_lifetime ? 
(See http://xfs.org/index.php/XFS_FAQ#Q:_How_can_I_speed_up_or_avoid_delayed_removal_of_speculative_preallocation.3F)

(In reply to Ravishankar N from comment #1)
> Hi zhoushicheng,
> 
> So the test that is failing is:
> EXPECT "1" has_holes $B0/${V0}0/big2bigger
> 
> The has_holes function returns true if(`stat -c '%b*%B-%s' file_name`)) -lt
> 0). I think this means that XFS has not freed the preallocated blocks
> despite the fd being closed by the time we reach this line in the testcase.
> Could you run the test lowering the speculative_prealloc_lifetime ? 
> (See
> http://xfs.org/index.php/XFS_FAQ#Q:
> _How_can_I_speed_up_or_avoid_delayed_removal_of_speculative_preallocation.3F)

This feature is said to be introduced by linux 3.8 (and later), but my linux version is 2.6.32-358.el6.x86_64. I cannot find speculative_prealloc_lifetime under /proc/sys/fs/xfs/, it seems that lowering the speculative_prealloc_lifetime is not an option under this condition.

Well then you could try adding a sleep in the testcase or doing a drop_caches before `EXPECT "1" has_holes $B0/${V0}0/big2bigger` is hit.

You could also try the testcase manually so that you can observe the ll -lh, du -h and md5sum of the file in both bricks of the replica. They must be equal.

After adding echo 3 > /proc/sys/vm/drop_caches before `EXPECT "1" has_holes $B0/${V0}0/big2bigger`, the test has passed.

And despite whether big2bigger was sparse or not after self heal, the checksum was the same.

Hmm, so this is not a bug per se. It is  just that XFS takes more time to free the prealloc'd blocks.

By the way, in your bug description, you say "With both full and diff algorithm, file big2bigger would become sparse file after self heal was done." This is the expected behaviour because the test creates big2bigger as a sparse file:

TEST dd if=/dev/urandom of=$M0/big2bigger count=1 bs=1024k 
TEST truncate -s 2M $M0/big2bigger

Thanks, big2bigger should become sparse file after self heal, sorry for my wrong description.

(In reply to zhoushicheng from comment #4)
> After adding echo 3 > /proc/sys/vm/drop_caches before `EXPECT "1" has_holes
> $B0/${V0}0/big2bigger`, the test has passed.
> 
Would you like to send a patch to add this line to the test case describing why it is needed? The how-to is here:http://www.gluster.org/community/documentation/index.php/Development_Work_Flow.You can use this bug-id for the patch. Or if you would rather not, I can do it for you.

(In reply to Ravishankar N from comment #7)
> (In reply to zhoushicheng from comment #4)
> > After adding echo 3 > /proc/sys/vm/drop_caches before `EXPECT "1" has_holes
> > $B0/${V0}0/big2bigger`, the test has passed.
> > 
> Would you like to send a patch to add this line to the test case describing
> why it is needed? The how-to is
> here:http://www.gluster.org/community/documentation/index.php/
> Development_Work_Flow.You can use this bug-id for the patch. Or if you would
> rather not, I can do it for you.

I will send a patch :)

REVIEW: http://review.gluster.org/10185 (test: Fix sparse file self heal test) posted (#1) for review on master by 仕成 周 (madaozhou)

http://review.gluster.org/#/c/10185/

REVIEW: http://review.gluster.org/10185 (test: Fix sparse file self heal test) posted (#2) for review on master by zhoushicheng (madaozhou)

REVIEW: http://review.gluster.org/10253 (test: Fix sparse file self heal test) posted (#1) for review on master by zhoushicheng (madaozhou)

REVIEW: http://review.gluster.org/10253 (test: Fix sparse file self heal test) posted (#2) for review on master by zhoushicheng (madaozhou)

REVIEW: http://review.gluster.org/10253 (test: Fix sparse file self heal test) posted (#3) for review on master by zhoushicheng (madaozhou)

REVIEW: http://review.gluster.org/10253 (test: Fix sparse file self heal test) posted (#5) for review on master by zhoushicheng (madaozhou)

COMMIT: http://review.gluster.org/10253 committed in master by Vijay Bellur (vbellur) 
------
commit 8f788528e64c4c13e16f7ad2d9f667a3813e08cc
Author: zhoushicheng <madaozhou>
Date:   Fri Apr 10 12:10:26 2015 +0800

    test: Fix sparse file self heal test
    
    This patch solves problems caused by XFS with speculative preallocation feature on :
    Test EXPECT "1" has_holes $B0/${V0}0/big2bigger would fall when XFS has not freed the preallocated blocks.
    It is caused by XFS speculative preallocation feature. The test would pass if this feature is disabled.
    Speculative preallocation can speed up under linux 3.8(and later).
    Otherwise, the test would pass by dropping cache manually to speed up speculative preallocation.
    
    As in http://review.gluster.org/#/c/10411/, using "( cd $M0 ; umount $M0 )" to drop caches, which is
    better than "echo 3 > /proc/sys/vm/drop_caches".
    
    drop caches operation was added in test:
    tests/basic/afr/sparse-file-self-heal.t
    
    BUG: 1206461
    Change-Id: Ie2c9d1b92fa8307c44498752fdd100eb86f9689c
    Signed-off-by: zhoushicheng <madaozhou>
    Reviewed-on: http://review.gluster.org/10253
    Tested-by: Gluster Build System <jenkins.com>
    Reviewed-by: Jeff Darcy <jdarcy>
    Reviewed-by: Ravishankar N <ravishankar>
    Reviewed-by: Vijay Bellur <vbellur>

Moving BZ to modified since the patch has been merged.

This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report.

glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user