Bug 1702289

Summary:	Promotion failed for a0afd3e3-0109-49b7-9b74-ba77bf653aba.11229
Product:	[Community] GlusterFS	Reporter:	Petr <p.stukalov>
Component:	tiering	Assignee:	bugs <bugs>
Status:	CLOSED EOL	QA Contact:	bugs <bugs>
Severity:	unspecified	Docs Contact:
Priority:	low
Version:	mainline	CC:	atumball, bugs, pasik
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-07-15 05:09:17 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Petr 2019-04-23 12:26:57 UTC

We have test cluster, with tiering on SSD and sharding.
If i create dispersed volume with only SSD, i have 6-10K IOPS on random write.
If i create volume with tiering, i have 6-1200 IOPS on random write.
Network in cluster - 10Gbe with jumboframes.
OS: Debian 4.9.144-3.1

With tiering volume, i have log messages, like that:
[2019-04-23 11:38:00.844369] I [MSGID: 109038] [tier.c:1122:tier_migrate_using_query_file] 0-freezer-tier-dht: Promotion failed for a0afd3e3-0109-49b7-9b74-ba77bf653aba.9173(gfid:a7c0d9b3-f680-4f2a-a41c-98326840cb1d)
[2019-04-23 11:38:01.055004] W [MSGID: 109023] [dht-rebalance.c:2058:dht_migrate_file] 0-freezer-tier-dht: Migrate file failed:/.shard/a0afd3e3-0109-49b7-9b74-ba77bf653aba.23351: failed to get xattr from freezer-cold-dht [No data available]

I think,for some reason, chunks cannot promotion.

root@dtln-ceph01:/# gluster volume status
Status of volume: freezer
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Hot Bricks:
Brick dtln-ceph03:/data/ssds/sdo/brick      49152     0          Y       3619528
Brick dtln-ceph02:/data/ssds/sdo/brick      49152     0          Y       3598217
Brick dtln-ceph01:/data/ssds/sdj/brick      49152     0          Y       141971
Cold Bricks:
Brick dtln-ceph01:/data/disks/sdd/brick     49153     0          Y       141992
Brick dtln-ceph01:/data/disks/sde/brick     49154     0          Y       142013
Brick dtln-ceph01:/data/disks/sdg/brick     49155     0          Y       142034
Brick dtln-ceph01:/data/disks/sda/brick     49156     0          Y       142055
Brick dtln-ceph01:/data/disks/sdb/brick     49157     0          Y       142076
Brick dtln-ceph01:/data/disks/sdc/brick     49158     0          Y       142109
Brick dtln-ceph01:/data/disks/sdl/brick     49159     0          Y       142130
Brick dtln-ceph01:/data/disks/sdm/brick     49160     0          Y       142151
Brick dtln-ceph01:/data/disks/sdn/brick     49161     0          Y       142179
Brick dtln-ceph01:/data/disks/sdh/brick     49162     0          Y       142200
Brick dtln-ceph01:/data/disks/sdi/brick     49163     0          Y       142221
Brick dtln-ceph01:/data/disks/sdk/brick     49164     0          Y       142242
Brick dtln-ceph02:/data/disks/sdd/brick     49153     0          Y       3598238
Brick dtln-ceph02:/data/disks/sde/brick     49154     0          Y       3598259
Brick dtln-ceph02:/data/disks/sdf/brick     49155     0          Y       3598293
Brick dtln-ceph02:/data/disks/sda/brick     49156     0          Y       3598314
Brick dtln-ceph02:/data/disks/sdb/brick     49157     0          Y       3598341
Brick dtln-ceph02:/data/disks/sdc/brick     49158     0          Y       3598363
Brick dtln-ceph02:/data/disks/sdl/brick     49159     0          Y       3598384
Brick dtln-ceph02:/data/disks/sdm/brick     49160     0          Y       3598411
Brick dtln-ceph02:/data/disks/sdn/brick     49161     0          Y       3598432
Brick dtln-ceph02:/data/disks/sdh/brick     49162     0          Y       3598453
Brick dtln-ceph02:/data/disks/sdi/brick     49163     0          Y       3598474
Brick dtln-ceph02:/data/disks/sdk/brick     49164     0          Y       3598502
Brick dtln-ceph03:/data/disks/sdd/brick     49153     0          Y       3619549
Brick dtln-ceph03:/data/disks/sde/brick     49154     0          Y       3619570
Brick dtln-ceph03:/data/disks/sdf/brick     49155     0          Y       3619603
Brick dtln-ceph03:/data/disks/sda/brick     49156     0          Y       3619631
Brick dtln-ceph03:/data/disks/sdb/brick     49157     0          Y       3619652
Brick dtln-ceph03:/data/disks/sdc/brick     49158     0          Y       3619673
Brick dtln-ceph03:/data/disks/sdl/brick     49159     0          Y       3619694
Brick dtln-ceph03:/data/disks/sdm/brick     49160     0          Y       3619715
Brick dtln-ceph03:/data/disks/sdn/brick     49161     0          Y       3619737
Brick dtln-ceph03:/data/disks/sdh/brick     49162     0          Y       3619758
Brick dtln-ceph03:/data/disks/sdi/brick     49163     0          Y       3619779
Brick dtln-ceph03:/data/disks/sdk/brick     49164     0          Y       3619806
Tier Daemon on localhost                    N/A       N/A        Y       142368
Self-heal Daemon on localhost               N/A       N/A        Y       142264
Tier Daemon on dtln-ceph02                  N/A       N/A        Y       3598624
Self-heal Daemon on dtln-ceph02             N/A       N/A        Y       3598531
Tier Daemon on dtln-ceph03                  N/A       N/A        Y       3619940
Self-heal Daemon on dtln-ceph03             N/A       N/A        Y       3619840
 
Task Status of Volume freezer
------------------------------------------------------------------------------
There are no active volume tasks

root@dtln-ceph01:/# gluster volume tier freezer status
Node                 Promoted files       Demoted files        Status               run time in h:m:s   
---------            ---------            ---------            ---------            ---------           
localhost            94                   0                    in progress          116:55:27
dtln-ceph03          79                   0                    in progress          116:55:27
dtln-ceph02          83                   0                    in progress          116:55:27

root@dtln-ceph01:/# glusterfs --version
glusterfs 3.13.2
Repository revision: git://git.gluster.org/glusterfs.git

Comment 1 Amar Tumballi 2019-05-17 05:13:46 UTC

Hi Petr,

We would like to let you know that we are not actively working on tiering feature, and for your info, the feature is already deprecated in our latest 6.x versions.

We recommend you to use dmcache or similar tiering options on brick itself to gain better performance using glusterfs on top of it.

Comment 2 Amar Tumballi 2019-07-15 05:09:17 UTC

With Comment#1, closing the issue as EOL (as the component itself is no more in current builds).