Description of problem: ====================== gluster v tier <vname> status and gluster v rebal vname tier status shows the tier as inprogress in all nodes of a cluster even when the volume has no bricks on certain nodes: EG: In below node, I have both the hot and cold brick on the remote node(yarrow) only. That means promote/demote can happen only on that node. But it shows unncessarily for the localhost too , which is not part of this volume Same with rebal too [root@zod distrep]# gluster v status olala Status of volume: olala Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Hot Bricks: Brick yarrow:/rhs/brick1/olala_hot 49324 0 Y 14178 Cold Bricks: Brick yarrow:/rhs/brick1/olala 49325 0 Y 14202 NFS Server on localhost 2049 0 Y 31537 NFS Server on yarrow N/A N/A N N/A Task Status of Volume olala ------------------------------------------------------------------------------ Task : Tier migration ID : 3daf6d99-63f9-412c-9dd3-cec168f87665 Status : in progress [root@zod distrep]# gluster v tier olala status Node Promoted files Demoted files Status --------- --------- --------- --------- localhost 0 0 in progress yarrow 0 0 in progress Tiering Migration Functionality: olala: success [root@zod distrep]# gluster v rebal olala tier status Node Promoted files Demoted files Status --------- --------- --------- --------- localhost 0 0 in progress yarrow 0 0 in progress Tiering Migration Functionality: olala: success [root@zod distrep]# gluster v rebal olala status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 0 0Bytes 0 0 0 in progress 0.00 yarrow 0 0Bytes 0 0 0 in progress 335.00 volume rebalance: olala: success [root@zod distrep]# gluster v info olala Volume Name: olala Type: Tier Volume ID: 177962eb-d514-4488-a6e3-643b2c1c03c4 Status: Started Number of Bricks: 2 Transport-type: tcp Hot Tier : Hot Tier Type : Distribute Number of Bricks: 1 Brick1: yarrow:/rhs/brick1/olala_hot Cold Tier: Cold Tier Type : Distribute Number of Bricks: 1 Brick2: yarrow:/rhs/brick1/olala Options Reconfigured: features.ctr-enabled: on performance.readdir-ahead: on [root@zod distrep]# Version-Release number of selected component (if applicable): ===================================== [root@zod distrep]# gluster v status olala Status of volume: olala Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Hot Bricks: Brick yarrow:/rhs/brick1/olala_hot 49324 0 Y 14178 Cold Bricks: Brick yarrow:/rhs/brick1/olala 49325 0 Y 14202 NFS Server on localhost 2049 0 Y 31537 NFS Server on yarrow N/A N/A N N/A Task Status of Volume olala ------------------------------------------------------------------------------ Task : Tier migration ID : 3daf6d99-63f9-412c-9dd3-cec168f87665 Status : in progress [root@zod distrep]# gluster v tier olala status Node Promoted files Demoted files Status --------- --------- --------- --------- localhost 0 0 in progress yarrow 0 0 in progress Tiering Migration Functionality: olala: success [root@zod distrep]# gluster v rebal olala tier status Node Promoted files Demoted files Status --------- --------- --------- --------- localhost 0 0 in progress yarrow 0 0 in progress Tiering Migration Functionality: olala: success [root@zod distrep]# gluster v rebal olala status Node Rebalanced-files size scanned failures skipped status run time in secs --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- localhost 0 0Bytes 0 0 0 in progress 0.00 yarrow 0 0Bytes 0 0 0 in progress 335.00 volume rebalance: olala: success [root@zod distrep]# gluster v info olala Volume Name: olala Type: Tier Volume ID: 177962eb-d514-4488-a6e3-643b2c1c03c4 Status: Started Number of Bricks: 2 Transport-type: tcp Hot Tier : Hot Tier Type : Distribute Number of Bricks: 1 Brick1: yarrow:/rhs/brick1/olala_hot Cold Tier: Cold Tier Type : Distribute Number of Bricks: 1 Brick2: yarrow:/rhs/brick1/olala Options Reconfigured: features.ctr-enabled: on performance.readdir-ahead: on [root@zod distrep]# Steps to Reproduce: 1.have a 3 node cluster A,b,C 2.create a voluem with bricks on only node A and B 3.Now attach tier with brciks only from A and B 4. now check the vol tier status or vol rebal status and it will show that process is in progress even on the node C. But in practice there is no tier deamon on node C as there are no bricks
Correction inline : When volume status or volume tier status is requested for a tiered volume, the status of all nodes in the trusted storage pool is listed as in progress, even when a node is not part of the tiered volume. Reason: Tier daemon for every volume in the trusted storage pool runs on all the nodes of trusted storage pool and thus you see this.
https://code.engineering.redhat.com/gerrit/#/c/71524/
Issue is still seen with build - glusterfs-server-3.7.9-2.el7rhgs.x86_64 node 'dhcp-47-90' isn't part of the volume, but we still see tier migration is progress on the node. Moving the bug to assigned. sosreports shall be attached. [root@dhcp47-90 yum.repos.d]# gluster v status Status of volume: tier-test Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Hot Bricks: Brick 10.70.47.9:/bricks/brick0/l1 49161 0 Y 28170 Cold Bricks: Brick 10.70.47.90:/bricks/brick0/l1 49162 0 Y 8523 Brick 10.70.47.105:/bricks/brick0/l1 49162 0 Y 32168 NFS Server on localhost 2049 0 Y 8543 NFS Server on 10.70.46.94 2049 0 Y 1937 NFS Server on 10.70.47.9 2049 0 Y 28190 NFS Server on 10.70.47.105 2049 0 Y 32188 Task Status of Volume tier-test ------------------------------------------------------------------------------ Task : Tier migration ID : d4654e28-88fa-40e7-965d-9525a2bbe67d Status : in progress [root@dhcp47-105 yum.repos.d]# gluster v status Status of volume: tier-test Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Hot Bricks: Brick 10.70.47.9:/bricks/brick0/l1 49161 0 Y 28170 Cold Bricks: Brick 10.70.47.90:/bricks/brick0/l1 49162 0 Y 8523 Brick 10.70.47.105:/bricks/brick0/l1 49162 0 Y 32168 NFS Server on localhost 2049 0 Y 32188 NFS Server on 10.70.47.9 2049 0 Y 28190 NFS Server on 10.70.46.94 2049 0 Y 1937 NFS Server on dhcp47-90.lab.eng.blr.redhat. com 2049 0 Y 8543 Task Status of Volume tier-test ------------------------------------------------------------------------------ Task : Tier migration ID : d4654e28-88fa-40e7-965d-9525a2bbe67d Status : in progress [root@dhcp47-9 yum.repos.d]# gluster v status Status of volume: tier-test Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Hot Bricks: Brick 10.70.47.9:/bricks/brick0/l1 49161 0 Y 28170 Cold Bricks: Brick 10.70.47.90:/bricks/brick0/l1 49162 0 Y 8523 Brick 10.70.47.105:/bricks/brick0/l1 49162 0 Y 32168 NFS Server on localhost 2049 0 Y 28190 NFS Server on dhcp47-90.lab.eng.blr.redhat. com 2049 0 Y 8543 NFS Server on 10.70.46.94 2049 0 Y 1937 NFS Server on 10.70.47.105 2049 0 Y 32188 Task Status of Volume tier-test ------------------------------------------------------------------------------ Task : Tier migration ID : d4654e28-88fa-40e7-965d-9525a2bbe67d Status : in progress [root@dhcp46-94 yum.repos.d]# gluster v status Status of volume: tier-test Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Hot Bricks: Brick 10.70.47.9:/bricks/brick0/l1 49161 0 Y 28170 Cold Bricks: Brick 10.70.47.90:/bricks/brick0/l1 49162 0 Y 8523 Brick 10.70.47.105:/bricks/brick0/l1 49162 0 Y 32168 NFS Server on localhost 2049 0 Y 1937 NFS Server on dhcp47-90.lab.eng.blr.redhat. com 2049 0 Y 8543 NFS Server on 10.70.47.9 2049 0 Y 28190 NFS Server on 10.70.47.105 2049 0 Y 32188 Task Status of Volume tier-test ------------------------------------------------------------------------------ Task : Tier migration ID : d4654e28-88fa-40e7-965d-9525a2bbe67d Status : in progress [root@dhcp47-90 yum.repos.d]# gluster v tier tier-test status Node Promoted files Demoted files Status --------- --------- --------- --------- localhost 0 0 in progress 10.70.47.105 0 0 in progress 10.70.47.9 0 0 in progress 10.70.46.94 0 0 in progress Tiering Migration Functionality: tier-test: success
The fix works fine for volumes created on a system which already has the fix. However, when a tiered volume is already present in a system and upgraded to the build which has the fix, tier status continues to show the status on all nodes.
upstream master patch : http://review.gluster.org/#/c/14106/
patch on master : http://review.gluster.org/#/c/14106/ patch on 3.7 : http://review.gluster.org/#/c/14229/
patch on downstream : https://code.engineering.redhat.com/gerrit/#/c/73782/
Skipping the status belongs to another bug. this fix and that one don't pass the same code path. the above mentioned issue will be fixed on the bug https://bugzilla.redhat.com/show_bug.cgi?id=1322695. so this bug is moved back to ON_QA .
'detach tier status' and 'tier status' commands skip to update the status of nodes which are down. Fix for bz#1322695 would fix the issue in both these commands. Moving this bug to verified as the actual issue reported in this bz is addressed and verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1240