We have test cluster, with tiering on SSD and sharding. If i create dispersed volume with only SSD, i have 6-10K IOPS on random write. If i create volume with tiering, i have 6-1200 IOPS on random write. Network in cluster - 10Gbe with jumboframes. OS: Debian 4.9.144-3.1 With tiering volume, i have log messages, like that: [2019-04-23 11:38:00.844369] I [MSGID: 109038] [tier.c:1122:tier_migrate_using_query_file] 0-freezer-tier-dht: Promotion failed for a0afd3e3-0109-49b7-9b74-ba77bf653aba.9173(gfid:a7c0d9b3-f680-4f2a-a41c-98326840cb1d) [2019-04-23 11:38:01.055004] W [MSGID: 109023] [dht-rebalance.c:2058:dht_migrate_file] 0-freezer-tier-dht: Migrate file failed:/.shard/a0afd3e3-0109-49b7-9b74-ba77bf653aba.23351: failed to get xattr from freezer-cold-dht [No data available] I think,for some reason, chunks cannot promotion. root@dtln-ceph01:/# gluster volume status Status of volume: freezer Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Hot Bricks: Brick dtln-ceph03:/data/ssds/sdo/brick 49152 0 Y 3619528 Brick dtln-ceph02:/data/ssds/sdo/brick 49152 0 Y 3598217 Brick dtln-ceph01:/data/ssds/sdj/brick 49152 0 Y 141971 Cold Bricks: Brick dtln-ceph01:/data/disks/sdd/brick 49153 0 Y 141992 Brick dtln-ceph01:/data/disks/sde/brick 49154 0 Y 142013 Brick dtln-ceph01:/data/disks/sdg/brick 49155 0 Y 142034 Brick dtln-ceph01:/data/disks/sda/brick 49156 0 Y 142055 Brick dtln-ceph01:/data/disks/sdb/brick 49157 0 Y 142076 Brick dtln-ceph01:/data/disks/sdc/brick 49158 0 Y 142109 Brick dtln-ceph01:/data/disks/sdl/brick 49159 0 Y 142130 Brick dtln-ceph01:/data/disks/sdm/brick 49160 0 Y 142151 Brick dtln-ceph01:/data/disks/sdn/brick 49161 0 Y 142179 Brick dtln-ceph01:/data/disks/sdh/brick 49162 0 Y 142200 Brick dtln-ceph01:/data/disks/sdi/brick 49163 0 Y 142221 Brick dtln-ceph01:/data/disks/sdk/brick 49164 0 Y 142242 Brick dtln-ceph02:/data/disks/sdd/brick 49153 0 Y 3598238 Brick dtln-ceph02:/data/disks/sde/brick 49154 0 Y 3598259 Brick dtln-ceph02:/data/disks/sdf/brick 49155 0 Y 3598293 Brick dtln-ceph02:/data/disks/sda/brick 49156 0 Y 3598314 Brick dtln-ceph02:/data/disks/sdb/brick 49157 0 Y 3598341 Brick dtln-ceph02:/data/disks/sdc/brick 49158 0 Y 3598363 Brick dtln-ceph02:/data/disks/sdl/brick 49159 0 Y 3598384 Brick dtln-ceph02:/data/disks/sdm/brick 49160 0 Y 3598411 Brick dtln-ceph02:/data/disks/sdn/brick 49161 0 Y 3598432 Brick dtln-ceph02:/data/disks/sdh/brick 49162 0 Y 3598453 Brick dtln-ceph02:/data/disks/sdi/brick 49163 0 Y 3598474 Brick dtln-ceph02:/data/disks/sdk/brick 49164 0 Y 3598502 Brick dtln-ceph03:/data/disks/sdd/brick 49153 0 Y 3619549 Brick dtln-ceph03:/data/disks/sde/brick 49154 0 Y 3619570 Brick dtln-ceph03:/data/disks/sdf/brick 49155 0 Y 3619603 Brick dtln-ceph03:/data/disks/sda/brick 49156 0 Y 3619631 Brick dtln-ceph03:/data/disks/sdb/brick 49157 0 Y 3619652 Brick dtln-ceph03:/data/disks/sdc/brick 49158 0 Y 3619673 Brick dtln-ceph03:/data/disks/sdl/brick 49159 0 Y 3619694 Brick dtln-ceph03:/data/disks/sdm/brick 49160 0 Y 3619715 Brick dtln-ceph03:/data/disks/sdn/brick 49161 0 Y 3619737 Brick dtln-ceph03:/data/disks/sdh/brick 49162 0 Y 3619758 Brick dtln-ceph03:/data/disks/sdi/brick 49163 0 Y 3619779 Brick dtln-ceph03:/data/disks/sdk/brick 49164 0 Y 3619806 Tier Daemon on localhost N/A N/A Y 142368 Self-heal Daemon on localhost N/A N/A Y 142264 Tier Daemon on dtln-ceph02 N/A N/A Y 3598624 Self-heal Daemon on dtln-ceph02 N/A N/A Y 3598531 Tier Daemon on dtln-ceph03 N/A N/A Y 3619940 Self-heal Daemon on dtln-ceph03 N/A N/A Y 3619840 Task Status of Volume freezer ------------------------------------------------------------------------------ There are no active volume tasks root@dtln-ceph01:/# gluster volume tier freezer status Node Promoted files Demoted files Status run time in h:m:s --------- --------- --------- --------- --------- localhost 94 0 in progress 116:55:27 dtln-ceph03 79 0 in progress 116:55:27 dtln-ceph02 83 0 in progress 116:55:27 root@dtln-ceph01:/# glusterfs --version glusterfs 3.13.2 Repository revision: git://git.gluster.org/glusterfs.git
Hi Petr, We would like to let you know that we are not actively working on tiering feature, and for your info, the feature is already deprecated in our latest 6.x versions. We recommend you to use dmcache or similar tiering options on brick itself to gain better performance using glusterfs on top of it.
With Comment#1, closing the issue as EOL (as the component itself is no more in current builds).