Bug 1283957 - Data Tiering:tier volume status shows as in-progress on all nodes of a cluster even if the node is not part of volume
Data Tiering:tier volume status shows as in-progress on all nodes of a cluste...
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: tier (Show other bugs)
unspecified
Unspecified Unspecified
high Severity low
: ---
: RHGS 3.1.3
Assigned To: hari gowtham
krishnaram Karthick
: ZStream
Depends On:
Blocks: 1268895 1299184 1315666 1316808 1347509
  Show dependency treegraph
 
Reported: 2015-11-20 06:40 EST by nchilaka
Modified: 2016-09-17 11:44 EDT (History)
9 users (show)

See Also:
Fixed In Version: glusterfs-3.7.9-4
Doc Type: Bug Fix
Doc Text:
If tiering is enabled for a volume, during volume restart the status of the tier daemon was incorrectly set to 'in progress' for all nodes. This meant that when status was requested for that volume, the tier daemon appeared to be running on all nodes, regardless of node type. A check has been added so that the tiering daemon only runs, and only appears to be running, on tiered volumes, so the status displayed for the volumes is now correct.
Story Points: ---
Clone Of:
: 1315666 (view as bug list)
Environment:
Last Closed: 2016-06-23 00:57:05 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description nchilaka 2015-11-20 06:40:01 EST
Description of problem:
======================
gluster v tier <vname> status and gluster v rebal vname tier status shows the tier as inprogress in all nodes of a cluster even when the volume has no bricks on certain nodes:
EG: In below node, I have both the hot and cold brick on the remote node(yarrow) only. That means promote/demote can happen only on that node. But it shows unncessarily for the localhost too , which is not part of this volume

Same with rebal too


[root@zod distrep]# gluster v status olala
Status of volume: olala
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Hot Bricks:
Brick yarrow:/rhs/brick1/olala_hot          49324     0          Y       14178
Cold Bricks:
Brick yarrow:/rhs/brick1/olala              49325     0          Y       14202
NFS Server on localhost                     2049      0          Y       31537
NFS Server on yarrow                        N/A       N/A        N       N/A  
 
Task Status of Volume olala
------------------------------------------------------------------------------
Task                 : Tier migration      
ID                   : 3daf6d99-63f9-412c-9dd3-cec168f87665
Status               : in progress         
 
[root@zod distrep]# gluster v tier olala status
Node                 Promoted files       Demoted files        Status              
---------            ---------            ---------            ---------           
localhost            0                    0                    in progress         
yarrow               0                    0                    in progress         
Tiering Migration Functionality: olala: success
[root@zod distrep]# gluster v rebal olala tier  status
Node                 Promoted files       Demoted files        Status              
---------            ---------            ---------            ---------           
localhost            0                    0                    in progress         
yarrow               0                    0                    in progress         
Tiering Migration Functionality: olala: success
[root@zod distrep]# gluster v rebal olala   status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost                0        0Bytes             0             0             0          in progress               0.00
                                  yarrow                0        0Bytes             0             0             0          in progress             335.00
volume rebalance: olala: success
[root@zod distrep]# gluster v info olala
 
Volume Name: olala
Type: Tier
Volume ID: 177962eb-d514-4488-a6e3-643b2c1c03c4
Status: Started
Number of Bricks: 2
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distribute
Number of Bricks: 1
Brick1: yarrow:/rhs/brick1/olala_hot
Cold Tier:
Cold Tier Type : Distribute
Number of Bricks: 1
Brick2: yarrow:/rhs/brick1/olala
Options Reconfigured:
features.ctr-enabled: on
performance.readdir-ahead: on
[root@zod distrep]# 


Version-Release number of selected component (if applicable):
=====================================
[root@zod distrep]# gluster v status olala
Status of volume: olala
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Hot Bricks:
Brick yarrow:/rhs/brick1/olala_hot          49324     0          Y       14178
Cold Bricks:
Brick yarrow:/rhs/brick1/olala              49325     0          Y       14202
NFS Server on localhost                     2049      0          Y       31537
NFS Server on yarrow                        N/A       N/A        N       N/A  
 
Task Status of Volume olala
------------------------------------------------------------------------------
Task                 : Tier migration      
ID                   : 3daf6d99-63f9-412c-9dd3-cec168f87665
Status               : in progress         
 
[root@zod distrep]# gluster v tier olala status
Node                 Promoted files       Demoted files        Status              
---------            ---------            ---------            ---------           
localhost            0                    0                    in progress         
yarrow               0                    0                    in progress         
Tiering Migration Functionality: olala: success
[root@zod distrep]# gluster v rebal olala tier  status
Node                 Promoted files       Demoted files        Status              
---------            ---------            ---------            ---------           
localhost            0                    0                    in progress         
yarrow               0                    0                    in progress         
Tiering Migration Functionality: olala: success
[root@zod distrep]# gluster v rebal olala   status
                                    Node Rebalanced-files          size       scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
                               localhost                0        0Bytes             0             0             0          in progress               0.00
                                  yarrow                0        0Bytes             0             0             0          in progress             335.00
volume rebalance: olala: success
[root@zod distrep]# gluster v info olala
 
Volume Name: olala
Type: Tier
Volume ID: 177962eb-d514-4488-a6e3-643b2c1c03c4
Status: Started
Number of Bricks: 2
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distribute
Number of Bricks: 1
Brick1: yarrow:/rhs/brick1/olala_hot
Cold Tier:
Cold Tier Type : Distribute
Number of Bricks: 1
Brick2: yarrow:/rhs/brick1/olala
Options Reconfigured:
features.ctr-enabled: on
performance.readdir-ahead: on
[root@zod distrep]# 




Steps to Reproduce:

1.have a 3 node cluster A,b,C
2.create a voluem with bricks on only node A and B
3.Now attach tier with brciks only from A and B
4. now check the vol tier status or vol rebal status and it will show that process is in progress even on the node C.
But in practice there is no tier deamon on node C as there are no bricks
Comment 5 Joseph Elwin Fernandes 2016-02-09 05:25:38 EST
Correction inline :

When volume status or volume tier status is requested for a tiered volume, the status of all nodes in the trusted storage pool is listed as in progress, even when a node is not part of the tiered volume.

Reason:

Tier daemon for every volume in the trusted storage pool runs on all the nodes of trusted storage pool and thus you see this.
Comment 8 Joseph Elwin Fernandes 2016-04-06 07:49:43 EDT
https://code.engineering.redhat.com/gerrit/#/c/71524/
Comment 10 krishnaram Karthick 2016-04-21 03:10:37 EDT
Issue is still seen with build - glusterfs-server-3.7.9-2.el7rhgs.x86_64

node 'dhcp-47-90' isn't part of the volume, but we still see tier migration is progress on the node. 

Moving the bug to assigned. sosreports shall be attached.

[root@dhcp47-90 yum.repos.d]# gluster v status
Status of volume: tier-test
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Hot Bricks:
Brick 10.70.47.9:/bricks/brick0/l1          49161     0          Y       28170
Cold Bricks:
Brick 10.70.47.90:/bricks/brick0/l1         49162     0          Y       8523 
Brick 10.70.47.105:/bricks/brick0/l1        49162     0          Y       32168
NFS Server on localhost                     2049      0          Y       8543 
NFS Server on 10.70.46.94                   2049      0          Y       1937 
NFS Server on 10.70.47.9                    2049      0          Y       28190
NFS Server on 10.70.47.105                  2049      0          Y       32188
 
Task Status of Volume tier-test
------------------------------------------------------------------------------
Task                 : Tier migration      
ID                   : d4654e28-88fa-40e7-965d-9525a2bbe67d
Status               : in progress         
 
[root@dhcp47-105 yum.repos.d]# gluster v status
Status of volume: tier-test
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Hot Bricks:
Brick 10.70.47.9:/bricks/brick0/l1          49161     0          Y       28170
Cold Bricks:
Brick 10.70.47.90:/bricks/brick0/l1         49162     0          Y       8523 
Brick 10.70.47.105:/bricks/brick0/l1        49162     0          Y       32168
NFS Server on localhost                     2049      0          Y       32188
NFS Server on 10.70.47.9                    2049      0          Y       28190
NFS Server on 10.70.46.94                   2049      0          Y       1937 
NFS Server on dhcp47-90.lab.eng.blr.redhat.
com                                         2049      0          Y       8543 
 
Task Status of Volume tier-test
------------------------------------------------------------------------------
Task                 : Tier migration      
ID                   : d4654e28-88fa-40e7-965d-9525a2bbe67d
Status               : in progress         


[root@dhcp47-9 yum.repos.d]# gluster v status
Status of volume: tier-test
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Hot Bricks:
Brick 10.70.47.9:/bricks/brick0/l1          49161     0          Y       28170
Cold Bricks:
Brick 10.70.47.90:/bricks/brick0/l1         49162     0          Y       8523 
Brick 10.70.47.105:/bricks/brick0/l1        49162     0          Y       32168
NFS Server on localhost                     2049      0          Y       28190
NFS Server on dhcp47-90.lab.eng.blr.redhat.
com                                         2049      0          Y       8543 
NFS Server on 10.70.46.94                   2049      0          Y       1937 
NFS Server on 10.70.47.105                  2049      0          Y       32188
 
Task Status of Volume tier-test
------------------------------------------------------------------------------
Task                 : Tier migration      
ID                   : d4654e28-88fa-40e7-965d-9525a2bbe67d
Status               : in progress         


[root@dhcp46-94 yum.repos.d]# gluster v status
Status of volume: tier-test
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Hot Bricks:
Brick 10.70.47.9:/bricks/brick0/l1          49161     0          Y       28170
Cold Bricks:
Brick 10.70.47.90:/bricks/brick0/l1         49162     0          Y       8523 
Brick 10.70.47.105:/bricks/brick0/l1        49162     0          Y       32168
NFS Server on localhost                     2049      0          Y       1937 
NFS Server on dhcp47-90.lab.eng.blr.redhat.
com                                         2049      0          Y       8543 
NFS Server on 10.70.47.9                    2049      0          Y       28190
NFS Server on 10.70.47.105                  2049      0          Y       32188
 
Task Status of Volume tier-test
------------------------------------------------------------------------------
Task                 : Tier migration      
ID                   : d4654e28-88fa-40e7-965d-9525a2bbe67d
Status               : in progress         


[root@dhcp47-90 yum.repos.d]# gluster v tier tier-test status
Node                 Promoted files       Demoted files        Status              
---------            ---------            ---------            ---------           
localhost            0                    0                    in progress         
10.70.47.105         0                    0                    in progress         
10.70.47.9           0                    0                    in progress         
10.70.46.94          0                    0                    in progress         
Tiering Migration Functionality: tier-test: success
Comment 11 krishnaram Karthick 2016-04-21 05:11:57 EDT
The fix works fine for volumes created on a system which already has the fix. However, when a tiered volume is already present in a system and upgraded to the build which has the fix, tier status continues to show the status on all nodes.
Comment 12 Mohammed Rafi KC 2016-05-03 00:50:24 EDT
upstream master patch : http://review.gluster.org/#/c/14106/
Comment 13 hari gowtham 2016-05-06 01:34:01 EDT
patch on master : http://review.gluster.org/#/c/14106/
patch on 3.7 : http://review.gluster.org/#/c/14229/
Comment 14 hari gowtham 2016-05-06 01:41:17 EDT
patch on downstream : https://code.engineering.redhat.com/gerrit/#/c/73782/
Comment 16 hari gowtham 2016-05-12 01:29:06 EDT
Skipping the status belongs to another bug. this fix and that one don't pass the same code path. the above mentioned issue will be fixed on the bug https://bugzilla.redhat.com/show_bug.cgi?id=1322695. so this bug is moved back to ON_QA .
Comment 17 krishnaram Karthick 2016-05-12 01:33:55 EDT
'detach tier status' and 'tier status' commands skip to update the status of nodes which are down. Fix for bz#1322695 would fix the issue in both these commands.

Moving this bug to verified as the actual issue reported in this bz is addressed and verified.
Comment 24 errata-xmlrpc 2016-06-23 00:57:05 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1240

Note You need to log in before you can comment on or make changes to this bug.