Bug 1263726

Summary: Data Tieirng:Detach tier status shows number of failures even when all files are migrated successfully
Product: [Community] GlusterFS Reporter: Dan Lambright <dlambrig>
Component: tieringAssignee: Dan Lambright <dlambrig>
Status: CLOSED CURRENTRELEASE QA Contact: bugs <bugs>
Severity: high Docs Contact:
Priority: urgent    
Version: mainlineCC: bugs, dlambrig, nchilaka, rkavunga
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.8rc2 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1263724 Environment:
Last Closed: 2016-06-16 13:37:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1262408, 1263724    
Bug Blocks: 1260923    

Description Dan Lambright 2015-09-16 14:16:35 UTC 
+++ This bug was initially created as a clone of Bug #1263724 +++

+++ This bug was initially created as a clone of Bug #1262408 +++

Description of problem:
=========================
When we have many files, say about 5000 files in hot tier, and when detach tier is issued, we can see many files as failed in detach tier status, in this case about 2000 files. But all files are migrated, if we check the backend bricks.

So, is this spurious errors or a serious miss?



[root@zod glusterfs]# gluster v detach-tier vol100 status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost                0        0Bytes             0             0             0          in progress               0.00
                                  yarrow                0        0Bytes             0             0             0          in progress               0.00
[root@zod glusterfs]# gluster v detach-tier vol100 status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost             1157         2.5GB          4736          1814             0            completed              98.00
                                  yarrow             1897       526.1MB          3056          1156             0            completed              97.00

Version-Release number of selected component (if applicable):
================================================================
[root@zod ~]# rpm -qa|grep gluster
glusterfs-client-xlators-3.7.4-0.27.git143f0f9.el7.centos.x86_64
glusterfs-cli-3.7.4-0.27.git143f0f9.el7.centos.x86_64
glusterfs-libs-3.7.4-0.27.git143f0f9.el7.centos.x86_64
glusterfs-api-3.7.4-0.27.git143f0f9.el7.centos.x86_64
glusterfs-fuse-3.7.4-0.27.git143f0f9.el7.centos.x86_64
glusterfs-debuginfo-3.7.4-0.27.git143f0f9.el7.centos.x86_64
glusterfs-server-3.7.4-0.27.git143f0f9.el7.centos.x86_64
glusterfs-3.7.4-0.27.git143f0f9.el7.centos.x86_64
[root@zod ~]# gluster --version
glusterfs 3.7.4 built on Sep  9 2015 01:30:47
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.
[root@zod ~]# 



Steps to Reproduce:
==================
1.have a tier volume and create thousands of files such that they are in hot tier
2.now issue a detach tier start and wait for it to complete
3.now check the detach tier status, which will show as complete for each node.
But it can also be seen many failures

[root@zod glusterfs]# gluster v detach-tier vol100 status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost                0        0Bytes             0             0             0          in progress               0.00
                                  yarrow                0        0Bytes             0             0             0          in progress               0.00
[root@zod glusterfs]# gluster v detach-tier vol100 status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost             1157         2.5GB          4736          1814             0            completed              98.00
                                  yarrow             1897       526.1MB          3056          1156             0            completed              97.00
Comment 1 Vijay Bellur 2015-09-16 14:28:15 UTC 
REVIEW: http://review.gluster.org/12185 (cluster/tier do not flag migration error on already migrated file) posted (#1) for review on master by Dan Lambright (dlambrig)
Comment 2 Vijay Bellur 2015-09-16 14:37:37 UTC 
REVIEW: http://review.gluster.org/12186 (cluster/tier do not flag migration error on already migrated file) posted (#1) for review on release-3.7 by Dan Lambright (dlambrig)
Comment 3 Dan Lambright 2015-09-16 14:59:26 UTC 
*** Bug 1263724 has been marked as a duplicate of this bug. ***
Comment 4 Vijay Bellur 2015-09-16 17:33:39 UTC 
COMMIT: http://review.gluster.org/12185 committed in master by Dan Lambright (dlambrig) 
------
commit a9bb3dcb1d22c5bfb527b9b85ad238b0137d3282
Author: Dan Lambright <dlambrig>
Date:   Wed Sep 16 10:26:44 2015 -0400

    cluster/tier do not flag migration error on already migrated file
    
    In some cases a brick will try to migrate a file that has already
    been migrated. This is a legal case, e.g. when both bricks
    are replica pairs.
    
    Change-Id: If2578b947014cbbdfb3c6591db9044d6b1d92774
    BUG: 1263726
    Signed-off-by: Dan Lambright <dlambrig>
    Reviewed-on: http://review.gluster.org/12185
    Tested-by: NetBSD Build System <jenkins.org>
    Reviewed-by: Joseph Fernandes
    Tested-by: Gluster Build System <jenkins.com>
Comment 5 Niels de Vos 2016-06-16 13:37:36 UTC 
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.8.0, please open a new bug report.

glusterfs-3.8.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] http://blog.gluster.org/2016/06/glusterfs-3-8-released/
[2] http://thread.gmane.org/gmane.comp.file-systems.gluster.user