Bug 1534327

Summary:	After reboot a node which has hot brick, it's Tier Daemon TCP port will lost.
Product:	[Community] GlusterFS	Reporter:	suxianxin <suxianxinshuai>
Component:	tiering	Assignee:	bugs <bugs>
Status:	CLOSED WONTFIX	QA Contact:	bugs <bugs>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	mainline	CC:	bugs, dwojslaw
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-11-02 08:13:45 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description suxianxin 2018-01-15 03:08:09 UTC

Description of problem:
Create a volume with hot tiering. After reboot a node which has hot brick, it's Tier Daemon TCP port will changer to N/A.

Version-Release number of selected component (if applicable):
[root@tnode5 ~]$gluster --version
glusterfs 3.11.0
Repository revision: git://git.gluster.org/glusterfs.git
Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.

How reproducible:


Steps to Reproduce:
1. Create a volume. Eg. Distributed-Disperse volume 2 x (3 + 2) = 10 
2. Add a tiering on the volume. Distributed-Replicate 2 x 2 =4

[root@tnode4 ~]$gluster peer status
Number of Peers: 4

Hostname: tnode3
Uuid: 219d0c3a-a0fe-4a6a-85b2-fb609d6c01a2
State: Peer in Cluster (Connected)

Hostname: tnode5
Uuid: e32bcad3-0636-4094-88e6-91989756cf91
State: Peer in Cluster (Connected)

Hostname: tnode2
Uuid: 01acaa72-5fb5-46d1-963f-25213c059643
State: Peer in Cluster (Connected)

Hostname: tnode1
Uuid: de21d0b7-daef-4257-a91c-58e4cc545897
State: Peer in Cluster (Connected)



Status of volume: data
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Hot Bricks:
Brick tnode2:/disks/3DpaDTww/brick          49153     49155      Y       86355
Brick tnode3:/disks/N295uyEC/brick          49153     49155      Y       73028
Brick tnode4:/disks/dwjzBgAu/brick          49153     49155      Y       892  
Brick tnode5:/disks/8ByHbfgi/brick          49153     49155      Y       38300
Cold Bricks:
Brick tnode4:/disks/0dgP2lr5/brick          49154     49156      Y       126757
Brick tnode5:/disks/ygytBDRD/brick          49154     49156      Y       33405
Brick tnode2:/disks/GVs528b8/brick          49154     49156      Y       81032
Brick tnode3:/disks/OxsK9G4j/brick          49154     49156      Y       67980
Brick tnode1:/disks/V8a0QEVv/brick          49154     49156      Y       126639
Brick tnode3:/disks/qiXDTXa7/brick          49157     49158      Y       68023
Brick tnode1:/disks/XWJxNayc/brick          49157     49158      Y       126671
Brick tnode2:/disks/PO9OetR5/brick          49157     49158      Y       81066
Brick tnode5:/disks/B6jDSa2H/brick          49157     49158      Y       33434
Brick tnode4:/disks/ve7Qzq5M/brick          49157     49158      Y       126792
Tier Daemon on localhost                    49159     0          Y       38654
Self-heal Daemon on localhost               N/A       N/A        Y       38350
Quota Daemon on localhost                   N/A       N/A        Y       38423
Tier Daemon on tnode3                       49159     0          Y       73358
Self-heal Daemon on tnode3                  N/A       N/A        Y       73101
Quota Daemon on tnode3                      N/A       N/A        Y       73124
Tier Daemon on tnode1                       49153     0          Y       3271 
Self-heal Daemon on tnode1                  N/A       N/A        Y       2730 
Quota Daemon on tnode1                      N/A       N/A        Y       2898 
Tier Daemon on tnode2                       49159     0          Y       86815
Self-heal Daemon on tnode2                  N/A       N/A        Y       86480
Quota Daemon on tnode2                      N/A       N/A        Y       86502
Tier Daemon on tnode4                       49159     0          Y       1416 
Self-heal Daemon on tnode4                  N/A       N/A        Y       947  
Quota Daemon on tnode4                      N/A       N/A        Y       991 


The tier daemon tcp port on each node:

Tier Daemon on localhost                    49159     0          Y       38654
Tier Daemon on tnode3                       49159     0          Y       73358
Tier Daemon on tnode5                       49159     0          Y       38654
Tier Daemon on tnode1                       49153     0          Y       3271 
Tier Daemon on tnode2                       49159     0          Y       86815


3. Reboot localhost. After the node bootup check the volume status
Status of volume: data
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Hot Bricks:
Brick tnode2:/disks/3DpaDTww/brick          49153     49155      Y       86355
Brick tnode3:/disks/N295uyEC/brick          49153     49155      Y       73028
Brick tnode4:/disks/dwjzBgAu/brick          49153     49154      Y       4132 
Brick tnode5:/disks/8ByHbfgi/brick          49153     49155      Y       38300
Cold Bricks:
Brick tnode4:/disks/0dgP2lr5/brick          49155     49156      Y       4142 
Brick tnode5:/disks/ygytBDRD/brick          49154     49156      Y       33405
Brick tnode2:/disks/GVs528b8/brick          49154     49156      Y       81032
Brick tnode3:/disks/OxsK9G4j/brick          49154     49156      Y       67980
Brick tnode1:/disks/V8a0QEVv/brick          49154     49156      Y       126639
Brick tnode3:/disks/qiXDTXa7/brick          49157     49158      Y       68023
Brick tnode1:/disks/XWJxNayc/brick          49157     49158      Y       126671
Brick tnode2:/disks/PO9OetR5/brick          49157     49158      Y       81066
Brick tnode5:/disks/B6jDSa2H/brick          49157     49158      Y       33434
Brick tnode4:/disks/ve7Qzq5M/brick          49157     49158      Y       4153 
Tier Daemon on localhost                    N/A       N/A        Y       1626 
Self-heal Daemon on localhost               N/A       N/A        Y       4490 
Quota Daemon on localhost                   N/A       N/A        Y       4532 
Tier Daemon on tnode1                       49153     0          Y       3271 
Self-heal Daemon on tnode1                  N/A       N/A        Y       13201
Quota Daemon on tnode1                      N/A       N/A        Y       13223
Tier Daemon on tnode5                       49159     0          Y       38654
Self-heal Daemon on tnode5                  N/A       N/A        Y       47827
Quota Daemon on tnode5                      N/A       N/A        Y       47852
Tier Daemon on tnode2                       49159     0          Y       86815
Self-heal Daemon on tnode2                  N/A       N/A        Y       97044
Quota Daemon on tnode2                      N/A       N/A        Y       97083
Tier Daemon on tnode3                       49159     0          Y       73358
Self-heal Daemon on tnode3                  N/A       N/A        Y       82684
Quota Daemon on tnode3                      N/A       N/A        Y       82727
 
Task Status of Volume data

The tier daemon tcp port on each node:
Tier Daemon on localhost                    N/A       N/A        Y       1626 
Tier Daemon on tnode3                       49159     0          Y       73358
Tier Daemon on tnode1                       49153     0          Y       3271 
Tier Daemon on tnode2                       49159     0          Y       86815
Tier Daemon on tnode5                       49159     0          Y       38654

Comment 1 suxianxin 2018-01-15 03:23:20 UTC

Volume Name: data
Type: Tier
Volume ID: 85b10003-403e-4f3d-aa79-4757d7f1f449
Status: Started
Snapshot Count: 0
Number of Bricks: 14
Transport-type: tcp,rdma
Hot Tier :
Hot Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick1: tnode2:/disks/3DpaDTww/brick
Brick2: tnode3:/disks/N295uyEC/brick
Brick3: tnode4:/disks/dwjzBgAu/brick
Brick4: tnode5:/disks/8ByHbfgi/brick
Cold Tier:
Cold Tier Type : Distributed-Disperse
Number of Bricks: 2 x (3 + 2) = 10
Brick5: tnode4:/disks/0dgP2lr5/brick
Brick6: tnode5:/disks/ygytBDRD/brick
Brick7: tnode2:/disks/GVs528b8/brick
Brick8: tnode3:/disks/OxsK9G4j/brick
Brick9: tnode1:/disks/V8a0QEVv/brick
Brick10: tnode3:/disks/qiXDTXa7/brick
Brick11: tnode1:/disks/XWJxNayc/brick
Brick12: tnode2:/disks/PO9OetR5/brick
Brick13: tnode5:/disks/B6jDSa2H/brick
Brick14: tnode4:/disks/ve7Qzq5M/brick
Options Reconfigured:
cluster.tier-mode: cache
features.ctr-enabled: on
features.quota-deem-statfs: on
features.inode-quota: on
features.quota: on
disperse.eager-lock: off
features.trash-internal-op: off
features.trash: off
diagnostics.stats-dump-interval: 10
server.event-threads: 8
client.event-threads: 8
server.keepalive-count: 1
server.keepalive-interval: 1
server.keepalive-time: 2
transport.keepalive: 1
client.keepalive-count: 1
client.keepalive-interval: 1
client.keepalive-time: 2
features.cache-invalidation: on
network.ping-timeout: 3
snap-activate-on-create: enable

Comment 2 Amar Tumballi 2018-11-02 08:13:45 UTC

Patch https://review.gluster.org/#/c/glusterfs/+/21331/ removes tier functionality from GlusterFS. 

https://bugzilla.redhat.com/show_bug.cgi?id=1642807 is used as the tracking bug for this. Recommendation is to convert your tier volume to regular volume (either replicate, ec, or plain distribute) with "tier detach" command before upgrade, and use backend features like dm-cache etc to utilize the caching from backend to provide better performance and functionality.