Description of problem: ================== Tier: Actual files are not demoted and keep on trying to demoted deleted files Version-Release number of selected component (if applicable): ================ glusterfs-server-3.7.5-15 How reproducible: Steps to Reproduce: ============ 1. Create 3x2 volume and attach 1x2 hot tier to the volume 2. Create script to untar the linux kernel and delete the the all files and run the script for 2 days and make sure no files under linux folder 3. Create directory and create around 200 files and 71 files are on hot tier and remaining on cold tier even after waiting for one day no file from hot tier not demoted though 94% of hot tier is full Actual results: Expected results: ========= Files should be demoted Additional info: ================ In the tier log able to see lot of stale file handle 16-01-08 10:24:02.228648] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 64493a52-f1b5-4383-a646-3fb676eb7819 [Stale file handle] [2016-01-08 10:24:02.233600] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 2195e82d-3025-4f47-a998-cb7f734ca163 [Stale file handle] [2016-01-08 10:24:02.234412] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 8aff6946-2f83-4d9b-95b5-de292e8067e6 [Stale file handle] [2016-01-08 10:24:02.235233] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 576cacfe-f8db-4138-9fad-0497bcc5c518 [Stale file handle] [2016-01-08 10:24:02.236092] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for a53b83ab-2c8a-4437-9d2e-bb63ffc71bb4 [Stale file handle] [2016-01-08 10:24:02.236901] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 8fde8e3d-164a-45ae-9363-b09bc996cf47 [Stale file handle] [2016-01-08 10:24:02.237702] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 68873bab-eaf3-4455-a556-0972c5dbaf9a [Stale file handle] [2016-01-08 10:24:02.238685] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for d4ecbdf7-3149-4690-8c23-61baf8c8515a [Stale file handle] [2016-01-08 10:24:02.239453] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 8974f414-2bd4-4409-a1c9-c04e6a5318d7 [Stale file handle] [2016-01-08 10:24:02.240216] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for d7a79559-a1af-498f-88f1-65d0b052c766 [Stale file handle] [2016-01-08 10:24:02.241117] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for f676e331-1381-48a1-82d2-1f896fab0d88 [Stale file handle] [2016-01-08 10:24:02.242045] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 227f0b56-23bb-4f3b-a9f5-96da815b2efe [Stale file handle] [2016-01-08 10:24:02.242784] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 9f7eca6b-aadd-443d-a3b9-db4232aa5ff9 [Stale file handle] [2016-01-08 10:24:02.243612] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for e969e1d3-b7da-465e-ba15-2d58db45cf7c [Stale file handle] [2016-01-08 10:24:02.250122] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for e0fdce38-5ae4-4175-963e-507ddb7b71c3 [Stale file handle] [2016-01-08 10:24:02.250971] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for a6ee01ab-6654-473c-861b-d2072d30bc87 [Stale file handle] [2016-01-08 10:24:02.251846] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 12ab9ba8-d9bf-4b17-9e4b-47cf55ebc074 [Stale file handle] [2016-01-08 10:24:02.252659] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for e880079a-6bfd-4d51-9594-7184f58563dd [Stale file handle] [2016-01-08 10:24:02.254160] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 923f1037-4ebd-4212-85f7-ba2fe313832e [Stale file handle] [2016-01-08 10:24:02.255069] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 7e692b4c-367e-4a77-bce0-0cf171cd35c1 [Stale file handle] [2016-01-08 10:24:02.256556] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 3cba1be1-121b-4294-958b-5543899ae9d2 [Stale file handle] [2016-01-08 10:24:02.257460] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 44d525e2-ae56-4d00-bf40-e73a181a9cff [Stale file handle] [2016-01-08 10:24:02.258321] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for d1248c74-40b0-4c55-8e4c-975e7d8df021 [Stale file handle] [2016-01-08 10:24:02.259246] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 563435f3-50b0-412b-9e38-b82c8a648cec [Stale file handle] [2016-01-08 10:24:02.259997] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 33572160-72b0-4ece-bce5-931d44365029 [Stale file handle] [2016-01-08 10:24:02.260778] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for b23aff0a-003b-4d92-af53-d95d666fdd76 [Stale file handle] [2016-01-08 10:24:02.261660] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 27657369-c673-445a-bd0f-2f835032fffb [Stale file handle] [2016-01-08 10:24:02.262424] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 93f9a30b-4d31-491b-a3b6-a9cd59e33159 [Stale file handle] [2016-01-08 10:24:02.263710] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 272a5c36-43db-425d-b9dc-834e4f635b75 [Stale file handle] [2016-01-08 10:24:02.264499] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for d967d537-4e33-4337-98a0-5ea74fd900df [Stale file handle] [2016-01-08 10:24:02.265386] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for abb7b52c-b7e5-4bac-ad02-fea27c8066f7 [Stale file handle] [2016-01-08 10:24:02.266268] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 110fe0ca-ff69-4e64-8071-e5eb3d946da7 [Stale file handle] [2016-01-08 10:24:02.267052] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 90167141-e392-4a4d-aec6-3ecb5ed297d4 [Stale file handle] [2016-01-08 10:24:02.267875] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 89644039-ed4c-48e2-aa20-612058b1f014 [Stale file handle] [2016-01-08 10:24:02.268696] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for afbae7d2-3145-47a1-a8a1-1188b7a88a15 [Stale file handle] [2016-01-08 10:24:02.269432] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for ab7b55f2-f4f8-4a9e-8899-9fbb61048c52 [Stale file handle] [2016-01-08 10:24:02.270150] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 9f16ba11-322e-402a-be75-61a6a12cde1a [Stale file handle] [2016-01-08 10:24:02.270916] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 789a07f0-a05b-4ccc-9291-6d28d2891bee [Stale file handle] [2016-01-08 10:24:02.271619] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 92dcf4e1-46aa-4745-a947-26a6830e2b49 [Stale file handle] [2016-01-08 10:24:02.272397] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 91cd92b3-528d-41e2-852a-576a0e95abc6 [Stale file handle] [2016-01-08 10:24:02.273114] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for ae8a92f3-444f-4736-969b-0bb49e610152 [Stale file handle] [2016-01-08 10:24:02.273914] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 11e00c25-ca27-4e1f-8f10-87aa95d2dcae [Stale file handle] [2016-01-08 10:24:02.274687] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 2ee5b15d-e62c-469e-9b1d-83f6b623bf03 [Stale file handle] [2016-01-08 10:24:02.275738] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 490c1f00-f75c-47a9-9a46-aac6ea00f402 [Stale file handle] [2016-01-08 10:24:02.276598] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for a33e9cb9-9429-4b25-be64-9c88a1852cfb [Stale file handle] [2016-01-08 10:24:02.277369] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for c8fca829-4b40-42be-9821-71b22aae6d49 [Stale file handle] [2016-01-08 10:24:02.278209] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 82155227-ab05-41a7-bc4a-d1f2b0724500 [Stale file handle] [2016-01-08 10:24:02.705150] W [MSGID: 109009] [dht-common.c:1885:dht_lookup_linkfile_cbk] 0-system_rajesh-tier-dht: /data/dir1/dir2/file14247: gfid different on data file on system_rajesh-hot-dht, gfid local = 033df9cd-d94e-4dc5-8bc4-9a3eb590b517, gfid node = 033df9cd-d94e-4dc5-8bc4-9a3eb590b517 [2016-01-08 10:24:02.705711] W [MSGID: 109009] [dht-common.c:1629:dht_lookup_everywhere_cbk] 0-system_rajesh-tier-dht: /data/dir1/dir2/file14247: gfid differs on subvolume system_rajesh-hot-dht, gfid local = 42c184d7-3cad-4a84-bc31-a28ef177d654, gfid node = 033df9cd-d94e-4dc5-8bc4-9a3eb590b517 [2016-01-08 10:24:02.706392] W [MSGID: 109009] [dht-common.c:1629:dht_lookup_everywhere_cbk] 0-system_rajesh-tier-dht: /data/dir1/dir2/file14247: gfid differs on subvolume system_rajesh-cold-dht, gfid local = 033df9cd-d94e-4dc5-8bc4-9a3eb590b517, gfid node = 42c184d7-3cad-4a84-bc31-a28ef177d654 [2016-01-08 10:24:02.706492] E [MSGID: 109037] [tier.c:462:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Failed to lookup file file14247 [Stale file handle] [2016-01-08 10:24:02.708251] E [MSGID: 109037] [tier.c:403:tier_migrate_using_query_file] 0-system_rajesh-tier-dht: Error in parent lookup for 07e347e2-7431-47e8-a673-7e15cade2166 [Stale file handle] [2016-01-08 10:24:02.713230] E [MSGID: 109037] [tier.c:1531:tier_start] 0-system_rajesh-tier-dht: Demotion failed Looks like DB has entries of deleted files and even DB is not synced across two bricks [root@tettnang system_rajesh-tier-dht]# gluster vol info system_rajesh Volume Name: system_rajesh Type: Tier Volume ID: fac2f70a-8a9e-4d09-8a0a-1b73be4cc137 Status: Started Number of Bricks: 8 Transport-type: tcp Hot Tier : Hot Tier Type : Replicate Number of Bricks: 1 x 2 = 2 Brick1: tettnang.lab.eng.blr.redhat.com:/rhs/brick2/hot2 Brick2: tettnang.lab.eng.blr.redhat.com:/rhs/brick2/hot1 Cold Tier: Cold Tier Type : Distributed-Replicate Number of Bricks: 3 x 2 = 6 Brick3: rhs-client19.lab.eng.blr.redhat.com:/rhs/brick1/system_rajesh Brick4: rhs-client18.lab.eng.blr.redhat.com:/rhs/brick1/system_rajesh Brick5: vertigo.lab.eng.blr.redhat.com:/rhs/brick1/system_rajesh Brick6: ninja.lab.eng.blr.redhat.com:/rhs/brick1/system_rajesh Brick7: transformers.lab.eng.blr.redhat.com:/rhs/brick1/system_rajesh Brick8: interstellar.lab.eng.blr.redhat.com:/rhs/brick1/system_rajesh Options Reconfigured: performance.readdir-ahead: on features.ctr-enabled: on cluster.tier-mode: cache
DB and log file are available @ /home/repo/sosreports/bug.1296908 on rhsqe-repo.lab.eng.blr.redhat.com
*** Bug 1303291 has been marked as a duplicate of this bug. ***
patch on master: https://review.gluster.org/#/c/13501/ patch on 3.7: https://review.gluster.org/#/c/13692/
As tier is not being actively developed, I'm closing this bug. Feel free to open it if necessary.