Bug 1262408

Summary: Data Tieirng:Detach tier status shows number of failures even when all files are migrated successfully
Product: [Community] GlusterFS Reporter: Nag Pavan Chilakam <nchilaka>
Component: tieringAssignee: Dan Lambright <dlambrig>
Status: CLOSED CURRENTRELEASE QA Contact: bugs <bugs>
Severity: high Docs Contact:
Priority: urgent    
Version: 3.7.4CC: bugs, dlambrig, rkavunga
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.7.5 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1263724 (view as bug list) Environment:
Last Closed: 2015-10-14 10:29:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1260923, 1263724, 1263726    

Description Nag Pavan Chilakam 2015-09-11 14:45:25 UTC
Description of problem:
=========================
When we have many files, say about 5000 files in hot tier, and when detach tier is issued, we can see many files as failed in detach tier status, in this case about 2000 files. But all files are migrated, if we check the backend bricks.

So, is this spurious errors or a serious miss?



[root@zod glusterfs]# gluster v detach-tier vol100 status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost                0        0Bytes             0             0             0          in progress               0.00
                                  yarrow                0        0Bytes             0             0             0          in progress               0.00
[root@zod glusterfs]# gluster v detach-tier vol100 status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost             1157         2.5GB          4736          1814             0            completed              98.00
                                  yarrow             1897       526.1MB          3056          1156             0            completed              97.00

Version-Release number of selected component (if applicable):
================================================================
[root@zod ~]# rpm -qa|grep gluster
glusterfs-client-xlators-3.7.4-0.27.git143f0f9.el7.centos.x86_64
glusterfs-cli-3.7.4-0.27.git143f0f9.el7.centos.x86_64
glusterfs-libs-3.7.4-0.27.git143f0f9.el7.centos.x86_64
glusterfs-api-3.7.4-0.27.git143f0f9.el7.centos.x86_64
glusterfs-fuse-3.7.4-0.27.git143f0f9.el7.centos.x86_64
glusterfs-debuginfo-3.7.4-0.27.git143f0f9.el7.centos.x86_64
glusterfs-server-3.7.4-0.27.git143f0f9.el7.centos.x86_64
glusterfs-3.7.4-0.27.git143f0f9.el7.centos.x86_64
[root@zod ~]# gluster --version
glusterfs 3.7.4 built on Sep  9 2015 01:30:47
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.
[root@zod ~]# 



Steps to Reproduce:
==================
1.have a tier volume and create thousands of files such that they are in hot tier
2.now issue a detach tier start and wait for it to complete
3.now check the detach tier status, which will show as complete for each node.
But it can also be seen many failures

[root@zod glusterfs]# gluster v detach-tier vol100 status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost                0        0Bytes             0             0             0          in progress               0.00
                                  yarrow                0        0Bytes             0             0             0          in progress               0.00
[root@zod glusterfs]# gluster v detach-tier vol100 status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost             1157         2.5GB          4736          1814             0            completed              98.00
                                  yarrow             1897       526.1MB          3056          1156             0            completed              97.00

Comment 1 Vijay Bellur 2015-09-16 14:53:19 UTC
REVIEW: http://review.gluster.org/12186 (cluster/tier do not flag migration error on already migrated file) posted (#2) for review on release-3.7 by Dan Lambright (dlambrig)

Comment 2 Vijay Bellur 2015-09-16 17:32:10 UTC
COMMIT: http://review.gluster.org/12186 committed in release-3.7 by Dan Lambright (dlambrig) 
------
commit ef231e41af79803a0e1e8829920b67e736578109
Author: Dan Lambright <dlambrig>
Date:   Wed Sep 16 10:26:44 2015 -0400

    cluster/tier do not flag migration error on already migrated file
    
    In some cases a brick will try to migrate a file that has already
    been migrated. This is a legal case, e.g. when both bricks
    are replica pairs.
    
    Change-Id: If2578b947014cbbdfb3c6591db9044d6b1d92774
    BUG: 1262408
    Signed-off-by: Dan Lambright <dlambrig>
    Reviewed-on: http://review.gluster.org/12186
    Tested-by: NetBSD Build System <jenkins.org>
    Reviewed-by: Joseph Fernandes
    Tested-by: Gluster Build System <jenkins.com>

Comment 3 Pranith Kumar K 2015-10-14 10:29:07 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-glusterfs-3.7.5, please open a new bug report.

glusterfs-glusterfs-3.7.5 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://www.gluster.org/pipermail/gluster-users/2015-October/023968.html
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user

Comment 4 Pranith Kumar K 2015-10-14 10:38:10 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.7.5, please open a new bug report.

glusterfs-3.7.5 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://www.gluster.org/pipermail/gluster-users/2015-October/023968.html
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user