Bug 1296908

Summary: Tier: Actual files are not demoted and keep on trying to demoted deleted files
Product: Red Hat Gluster Storage Reporter: RajeshReddy <rmekala>
Component: tierAssignee: hari gowtham <hgowtham>
Status: CLOSED WONTFIX QA Contact: Nag Pavan Chilakam <nchilaka>
Severity: high Docs Contact:
Priority: unspecified    
Version: rhgs-3.1CC: hgowtham, kramdoss, rhs-bugs
Target Milestone: ---Keywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: tier-migration
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1311178 (view as bug list) Environment:
Last Closed: 2018-11-08 18:38:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1311178, 1317366    

Description RajeshReddy 2016-01-08 11:51:38 UTC 
Description of problem:
==================
Tier: Actual files are not demoted and keep on trying to demoted deleted files  

Version-Release number of selected component (if applicable):
================
glusterfs-server-3.7.5-15


How reproducible:


Steps to Reproduce:
============
1. Create 3x2 volume and attach 1x2 hot tier to the volume 
2. Create script to untar the linux kernel and delete the the all files and run the script for 2 days and make sure no files under linux folder 
3. Create directory and create around 200 files and 71 files are on hot tier and remaining on cold tier even after waiting for one day no file from hot tier not demoted though 94% of hot tier is full 

Actual results:


Expected results:
=========
Files should be demoted 


Additional info:
================
In the tier log able to see lot of stale file handle 

16-01-08 10:24:02.228648] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 64493a52-f1b5-4383-a646-3fb676eb7819 [Stale file handle]
[2016-01-08 10:24:02.233600] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 2195e82d-3025-4f47-a998-cb7f734ca163 [Stale file handle]
[2016-01-08 10:24:02.234412] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 8aff6946-2f83-4d9b-95b5-de292e8067e6 [Stale file handle]
[2016-01-08 10:24:02.235233] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 576cacfe-f8db-4138-9fad-0497bcc5c518 [Stale file handle]
[2016-01-08 10:24:02.236092] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for a53b83ab-2c8a-4437-9d2e-bb63ffc71bb4 [Stale file handle]
[2016-01-08 10:24:02.236901] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 8fde8e3d-164a-45ae-9363-b09bc996cf47 [Stale file handle]
[2016-01-08 10:24:02.237702] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 68873bab-eaf3-4455-a556-0972c5dbaf9a [Stale file handle]
[2016-01-08 10:24:02.238685] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for d4ecbdf7-3149-4690-8c23-61baf8c8515a [Stale file handle]
[2016-01-08 10:24:02.239453] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 8974f414-2bd4-4409-a1c9-c04e6a5318d7 [Stale file handle]
[2016-01-08 10:24:02.240216] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for d7a79559-a1af-498f-88f1-65d0b052c766 [Stale file handle]
[2016-01-08 10:24:02.241117] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for f676e331-1381-48a1-82d2-1f896fab0d88 [Stale file handle]
[2016-01-08 10:24:02.242045] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 227f0b56-23bb-4f3b-a9f5-96da815b2efe [Stale file handle]
[2016-01-08 10:24:02.242784] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 9f7eca6b-aadd-443d-a3b9-db4232aa5ff9 [Stale file handle]
[2016-01-08 10:24:02.243612] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for e969e1d3-b7da-465e-ba15-2d58db45cf7c [Stale file handle]
[2016-01-08 10:24:02.250122] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for e0fdce38-5ae4-4175-963e-507ddb7b71c3 [Stale file handle]
[2016-01-08 10:24:02.250971] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for a6ee01ab-6654-473c-861b-d2072d30bc87 [Stale file handle]
[2016-01-08 10:24:02.251846] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 12ab9ba8-d9bf-4b17-9e4b-47cf55ebc074 [Stale file handle]
[2016-01-08 10:24:02.252659] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for e880079a-6bfd-4d51-9594-7184f58563dd [Stale file handle]
[2016-01-08 10:24:02.254160] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 923f1037-4ebd-4212-85f7-ba2fe313832e [Stale file handle]
[2016-01-08 10:24:02.255069] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 7e692b4c-367e-4a77-bce0-0cf171cd35c1 [Stale file handle]
[2016-01-08 10:24:02.256556] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 3cba1be1-121b-4294-958b-5543899ae9d2 [Stale file handle]
[2016-01-08 10:24:02.257460] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 44d525e2-ae56-4d00-bf40-e73a181a9cff [Stale file handle]
[2016-01-08 10:24:02.258321] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for d1248c74-40b0-4c55-8e4c-975e7d8df021 [Stale file handle]
[2016-01-08 10:24:02.259246] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 563435f3-50b0-412b-9e38-b82c8a648cec [Stale file handle]
[2016-01-08 10:24:02.259997] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 33572160-72b0-4ece-bce5-931d44365029 [Stale file handle]
[2016-01-08 10:24:02.260778] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for b23aff0a-003b-4d92-af53-d95d666fdd76 [Stale file handle]
[2016-01-08 10:24:02.261660] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 27657369-c673-445a-bd0f-2f835032fffb [Stale file handle]
[2016-01-08 10:24:02.262424] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 93f9a30b-4d31-491b-a3b6-a9cd59e33159 [Stale file handle]
[2016-01-08 10:24:02.263710] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 272a5c36-43db-425d-b9dc-834e4f635b75 [Stale file handle]
[2016-01-08 10:24:02.264499] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for d967d537-4e33-4337-98a0-5ea74fd900df [Stale file handle]
[2016-01-08 10:24:02.265386] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for abb7b52c-b7e5-4bac-ad02-fea27c8066f7 [Stale file handle]
[2016-01-08 10:24:02.266268] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 110fe0ca-ff69-4e64-8071-e5eb3d946da7 [Stale file handle]
[2016-01-08 10:24:02.267052] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 90167141-e392-4a4d-aec6-3ecb5ed297d4 [Stale file handle]
[2016-01-08 10:24:02.267875] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 89644039-ed4c-48e2-aa20-612058b1f014 [Stale file handle]
[2016-01-08 10:24:02.268696] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for afbae7d2-3145-47a1-a8a1-1188b7a88a15 [Stale file handle]
[2016-01-08 10:24:02.269432] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for ab7b55f2-f4f8-4a9e-8899-9fbb61048c52 [Stale file handle]
[2016-01-08 10:24:02.270150] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 9f16ba11-322e-402a-be75-61a6a12cde1a [Stale file handle]
[2016-01-08 10:24:02.270916] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 789a07f0-a05b-4ccc-9291-6d28d2891bee [Stale file handle]
[2016-01-08 10:24:02.271619] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 92dcf4e1-46aa-4745-a947-26a6830e2b49 [Stale file handle]
[2016-01-08 10:24:02.272397] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 91cd92b3-528d-41e2-852a-576a0e95abc6 [Stale file handle]
[2016-01-08 10:24:02.273114] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for ae8a92f3-444f-4736-969b-0bb49e610152 [Stale file handle]
[2016-01-08 10:24:02.273914] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 11e00c25-ca27-4e1f-8f10-87aa95d2dcae [Stale file handle]
[2016-01-08 10:24:02.274687] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 2ee5b15d-e62c-469e-9b1d-83f6b623bf03 [Stale file handle]
[2016-01-08 10:24:02.275738] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 490c1f00-f75c-47a9-9a46-aac6ea00f402 [Stale file handle]
[2016-01-08 10:24:02.276598] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for a33e9cb9-9429-4b25-be64-9c88a1852cfb [Stale file handle]
[2016-01-08 10:24:02.277369] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for c8fca829-4b40-42be-9821-71b22aae6d49 [Stale file handle]
[2016-01-08 10:24:02.278209] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 82155227-ab05-41a7-bc4a-d1f2b0724500 [Stale file handle]
[2016-01-08 10:24:02.705150] W [MSGID: 109009] [dht-common.c:1885:dht_lookup_linkfile_cbk] 0-system_rajesh-tier-dht: /data/dir1/dir2/file14247: gfid different on data file on system_rajesh-hot-dht, gfid local = 033df9cd-d94e-4dc5-8bc4-9a3eb590b517, gfid node = 033df9cd-d94e-4dc5-8bc4-9a3eb590b517
[2016-01-08 10:24:02.705711] W [MSGID: 109009] [dht-common.c:1629:dht_lookup_everywhere_cbk] 0-system_rajesh-tier-dht: /data/dir1/dir2/file14247: gfid differs on subvolume system_rajesh-hot-dht, gfid local = 42c184d7-3cad-4a84-bc31-a28ef177d654, gfid node = 033df9cd-d94e-4dc5-8bc4-9a3eb590b517
[2016-01-08 10:24:02.706392] W [MSGID: 109009] [dht-common.c:1629:dht_lookup_everywhere_cbk] 0-system_rajesh-tier-dht: /data/dir1/dir2/file14247: gfid differs on subvolume system_rajesh-cold-dht, gfid local = 033df9cd-d94e-4dc5-8bc4-9a3eb590b517, gfid node = 42c184d7-3cad-4a84-bc31-a28ef177d654
[2016-01-08 10:24:02.706492] E [MSGID: 109037] [tier.c:462:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Failed to lookup file file14247
 [Stale file handle]
[2016-01-08 10:24:02.708251] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 07e347e2-7431-47e8-a673-7e15cade2166 [Stale file handle]
[2016-01-08 10:24:02.713230] E [MSGID: 109037] [tier.c:1531:tier_start] 0-system_rajesh-tier-dht: Demotion failed

Looks like DB has entries of deleted files and even DB is not synced across two bricks 

[root@tettnang system_rajesh-tier-dht]# gluster vol info system_rajesh
 
Volume Name: system_rajesh
Type: Tier
Volume ID: fac2f70a-8a9e-4d09-8a0a-1b73be4cc137
Status: Started
Number of Bricks: 8
Transport-type: tcp
Hot Tier :
Hot Tier Type : Replicate
Number of Bricks: 1 x 2 = 2
Brick1: tettnang.lab.eng.blr.redhat.com:/rhs/brick2/hot2
Brick2: tettnang.lab.eng.blr.redhat.com:/rhs/brick2/hot1
Cold Tier:
Cold Tier Type : Distributed-Replicate
Number of Bricks: 3 x 2 = 6
Brick3: rhs-client19.lab.eng.blr.redhat.com:/rhs/brick1/system_rajesh
Brick4: rhs-client18.lab.eng.blr.redhat.com:/rhs/brick1/system_rajesh
Brick5: vertigo.lab.eng.blr.redhat.com:/rhs/brick1/system_rajesh
Brick6: ninja.lab.eng.blr.redhat.com:/rhs/brick1/system_rajesh
Brick7: transformers.lab.eng.blr.redhat.com:/rhs/brick1/system_rajesh
Brick8: interstellar.lab.eng.blr.redhat.com:/rhs/brick1/system_rajesh
Options Reconfigured:
performance.readdir-ahead: on
features.ctr-enabled: on
cluster.tier-mode: cache
Comment 2 RajeshReddy 2016-01-08 13:22:57 UTC 
DB and log file are available @ /home/repo/sosreports/bug.1296908 on rhsqe-repo.lab.eng.blr.redhat.com
Comment 3 hari gowtham 2016-06-06 08:16:35 UTC 
*** Bug 1303291 has been marked as a duplicate of this bug. ***
Comment 6 hari gowtham 2017-09-06 12:02:54 UTC 
patch on master: https://review.gluster.org/#/c/13501/ 
patch on 3.7: https://review.gluster.org/#/c/13692/
Comment 8 hari gowtham 2018-11-08 18:38:40 UTC 
As tier is not being actively developed, I'm closing this bug. Feel free to open it if necessary.