Description of problem: Create a volume with hot tiering. After reboot a node which has hot brick, it's Tier Daemon TCP port will changer to N/A. Version-Release number of selected component (if applicable): [root@tnode5 ~]$gluster --version glusterfs 3.11.0 Repository revision: git://git.gluster.org/glusterfs.git Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/> GlusterFS comes with ABSOLUTELY NO WARRANTY. It is licensed to you under your choice of the GNU Lesser General Public License, version 3 or any later version (LGPLv3 or later), or the GNU General Public License, version 2 (GPLv2), in all cases as published by the Free Software Foundation. How reproducible: Steps to Reproduce: 1. Create a volume. Eg. Distributed-Disperse volume 2 x (3 + 2) = 10 2. Add a tiering on the volume. Distributed-Replicate 2 x 2 =4 [root@tnode4 ~]$gluster peer status Number of Peers: 4 Hostname: tnode3 Uuid: 219d0c3a-a0fe-4a6a-85b2-fb609d6c01a2 State: Peer in Cluster (Connected) Hostname: tnode5 Uuid: e32bcad3-0636-4094-88e6-91989756cf91 State: Peer in Cluster (Connected) Hostname: tnode2 Uuid: 01acaa72-5fb5-46d1-963f-25213c059643 State: Peer in Cluster (Connected) Hostname: tnode1 Uuid: de21d0b7-daef-4257-a91c-58e4cc545897 State: Peer in Cluster (Connected) Status of volume: data Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Hot Bricks: Brick tnode2:/disks/3DpaDTww/brick 49153 49155 Y 86355 Brick tnode3:/disks/N295uyEC/brick 49153 49155 Y 73028 Brick tnode4:/disks/dwjzBgAu/brick 49153 49155 Y 892 Brick tnode5:/disks/8ByHbfgi/brick 49153 49155 Y 38300 Cold Bricks: Brick tnode4:/disks/0dgP2lr5/brick 49154 49156 Y 126757 Brick tnode5:/disks/ygytBDRD/brick 49154 49156 Y 33405 Brick tnode2:/disks/GVs528b8/brick 49154 49156 Y 81032 Brick tnode3:/disks/OxsK9G4j/brick 49154 49156 Y 67980 Brick tnode1:/disks/V8a0QEVv/brick 49154 49156 Y 126639 Brick tnode3:/disks/qiXDTXa7/brick 49157 49158 Y 68023 Brick tnode1:/disks/XWJxNayc/brick 49157 49158 Y 126671 Brick tnode2:/disks/PO9OetR5/brick 49157 49158 Y 81066 Brick tnode5:/disks/B6jDSa2H/brick 49157 49158 Y 33434 Brick tnode4:/disks/ve7Qzq5M/brick 49157 49158 Y 126792 Tier Daemon on localhost 49159 0 Y 38654 Self-heal Daemon on localhost N/A N/A Y 38350 Quota Daemon on localhost N/A N/A Y 38423 Tier Daemon on tnode3 49159 0 Y 73358 Self-heal Daemon on tnode3 N/A N/A Y 73101 Quota Daemon on tnode3 N/A N/A Y 73124 Tier Daemon on tnode1 49153 0 Y 3271 Self-heal Daemon on tnode1 N/A N/A Y 2730 Quota Daemon on tnode1 N/A N/A Y 2898 Tier Daemon on tnode2 49159 0 Y 86815 Self-heal Daemon on tnode2 N/A N/A Y 86480 Quota Daemon on tnode2 N/A N/A Y 86502 Tier Daemon on tnode4 49159 0 Y 1416 Self-heal Daemon on tnode4 N/A N/A Y 947 Quota Daemon on tnode4 N/A N/A Y 991 The tier daemon tcp port on each node: Tier Daemon on localhost 49159 0 Y 38654 Tier Daemon on tnode3 49159 0 Y 73358 Tier Daemon on tnode5 49159 0 Y 38654 Tier Daemon on tnode1 49153 0 Y 3271 Tier Daemon on tnode2 49159 0 Y 86815 3. Reboot localhost. After the node bootup check the volume status Status of volume: data Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Hot Bricks: Brick tnode2:/disks/3DpaDTww/brick 49153 49155 Y 86355 Brick tnode3:/disks/N295uyEC/brick 49153 49155 Y 73028 Brick tnode4:/disks/dwjzBgAu/brick 49153 49154 Y 4132 Brick tnode5:/disks/8ByHbfgi/brick 49153 49155 Y 38300 Cold Bricks: Brick tnode4:/disks/0dgP2lr5/brick 49155 49156 Y 4142 Brick tnode5:/disks/ygytBDRD/brick 49154 49156 Y 33405 Brick tnode2:/disks/GVs528b8/brick 49154 49156 Y 81032 Brick tnode3:/disks/OxsK9G4j/brick 49154 49156 Y 67980 Brick tnode1:/disks/V8a0QEVv/brick 49154 49156 Y 126639 Brick tnode3:/disks/qiXDTXa7/brick 49157 49158 Y 68023 Brick tnode1:/disks/XWJxNayc/brick 49157 49158 Y 126671 Brick tnode2:/disks/PO9OetR5/brick 49157 49158 Y 81066 Brick tnode5:/disks/B6jDSa2H/brick 49157 49158 Y 33434 Brick tnode4:/disks/ve7Qzq5M/brick 49157 49158 Y 4153 Tier Daemon on localhost N/A N/A Y 1626 Self-heal Daemon on localhost N/A N/A Y 4490 Quota Daemon on localhost N/A N/A Y 4532 Tier Daemon on tnode1 49153 0 Y 3271 Self-heal Daemon on tnode1 N/A N/A Y 13201 Quota Daemon on tnode1 N/A N/A Y 13223 Tier Daemon on tnode5 49159 0 Y 38654 Self-heal Daemon on tnode5 N/A N/A Y 47827 Quota Daemon on tnode5 N/A N/A Y 47852 Tier Daemon on tnode2 49159 0 Y 86815 Self-heal Daemon on tnode2 N/A N/A Y 97044 Quota Daemon on tnode2 N/A N/A Y 97083 Tier Daemon on tnode3 49159 0 Y 73358 Self-heal Daemon on tnode3 N/A N/A Y 82684 Quota Daemon on tnode3 N/A N/A Y 82727 Task Status of Volume data The tier daemon tcp port on each node: Tier Daemon on localhost N/A N/A Y 1626 Tier Daemon on tnode3 49159 0 Y 73358 Tier Daemon on tnode1 49153 0 Y 3271 Tier Daemon on tnode2 49159 0 Y 86815 Tier Daemon on tnode5 49159 0 Y 38654
Volume Name: data Type: Tier Volume ID: 85b10003-403e-4f3d-aa79-4757d7f1f449 Status: Started Snapshot Count: 0 Number of Bricks: 14 Transport-type: tcp,rdma Hot Tier : Hot Tier Type : Distributed-Replicate Number of Bricks: 2 x 2 = 4 Brick1: tnode2:/disks/3DpaDTww/brick Brick2: tnode3:/disks/N295uyEC/brick Brick3: tnode4:/disks/dwjzBgAu/brick Brick4: tnode5:/disks/8ByHbfgi/brick Cold Tier: Cold Tier Type : Distributed-Disperse Number of Bricks: 2 x (3 + 2) = 10 Brick5: tnode4:/disks/0dgP2lr5/brick Brick6: tnode5:/disks/ygytBDRD/brick Brick7: tnode2:/disks/GVs528b8/brick Brick8: tnode3:/disks/OxsK9G4j/brick Brick9: tnode1:/disks/V8a0QEVv/brick Brick10: tnode3:/disks/qiXDTXa7/brick Brick11: tnode1:/disks/XWJxNayc/brick Brick12: tnode2:/disks/PO9OetR5/brick Brick13: tnode5:/disks/B6jDSa2H/brick Brick14: tnode4:/disks/ve7Qzq5M/brick Options Reconfigured: cluster.tier-mode: cache features.ctr-enabled: on features.quota-deem-statfs: on features.inode-quota: on features.quota: on disperse.eager-lock: off features.trash-internal-op: off features.trash: off diagnostics.stats-dump-interval: 10 server.event-threads: 8 client.event-threads: 8 server.keepalive-count: 1 server.keepalive-interval: 1 server.keepalive-time: 2 transport.keepalive: 1 client.keepalive-count: 1 client.keepalive-interval: 1 client.keepalive-time: 2 features.cache-invalidation: on network.ping-timeout: 3 snap-activate-on-create: enable
Patch https://review.gluster.org/#/c/glusterfs/+/21331/ removes tier functionality from GlusterFS. https://bugzilla.redhat.com/show_bug.cgi?id=1642807 is used as the tracking bug for this. Recommendation is to convert your tier volume to regular volume (either replicate, ec, or plain distribute) with "tier detach" command before upgrade, and use backend features like dm-cache etc to utilize the caching from backend to provide better performance and functionality.