Bug 1262885

Summary: Data Tieiring: legacy files not at all getting promoted(inspite of many modifications)
Product: [Community] GlusterFS Reporter: Nag Pavan Chilakam <nchilaka>
Component: tieringAssignee: Joseph Elwin Fernandes <josferna>
Status: CLOSED DUPLICATE QA Contact: bugs <bugs>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 3.7.4CC: bugs, sankarshan, vagarwal
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.7.6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-02-14 07:05:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1260923    

Description Nag Pavan Chilakam 2015-09-14 14:40:52 UTC 
Description of problem:
=======================
I had a tier volume with cold as 2 x (4 + 2) = 12 and hot as 2x2




Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.create a volume(in my case an ec vol of 2 x (4 + 2))
2.now create some files say c1,c2,c3
3.Now attach a hot tier 
4. set ctr on (features.ctr-enabled: on)
5. set promote freq to say 10sec
cluster.tier-demote-frequency: 10

6. Now touch the files repeatedly and check the db.
It can be seen that the files are not at all getting promoted.
Even after doing "ls", this also didnt help

7. Now create a new file, which will go to hot tier.
8. Now set demote freq to say 20sec and keep the vol idle for about 1min, it can be seen that the file gets demoted.
9. Now access all the files which are all in cold tier now/
10. only the files created post attach tier are heated/cooled




=========CLI========================

[2015-09-14 13:59:30.504847] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 13:59:40.505916] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 13:59:50.506991] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
The message "I [MSGID: 109103] [dht-shared.c:468:dht_reconfigure] 0-DHT: conf->dthrottle: normal, conf->defrag->recon_thread_count: 4" repeated 2 times between [2015-09-14 13:59:17.713835] and [2015-09-14 13:59:17.714437]
[2015-09-14 13:59:55.065408] I [MSGID: 109028] [dht-rebalance.c:3075:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 58.00 secs
[2015-09-14 13:59:55.065469] I [MSGID: 109028] [dht-rebalance.c:3079:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 0, failures: 0, skipped: 0
[2015-09-14 13:59:55.087425] I [MSGID: 109028] [dht-rebalance.c:3075:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 58.00 secs
[2015-09-14 13:59:55.087457] I [MSGID: 109028] [dht-rebalance.c:3079:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 0, failures: 0, skipped: 0
[2015-09-14 14:00:00.507956] I [MSGID: 109038] [tier.c:588:tier_build_migration_qfile] 0-yahoo-tier-dht: Failed to remove /var/run/gluster/yahoo-tier-dht/promotequeryfile-yahoo-tier-dht
[2015-09-14 14:00:00.508022] I [MSGID: 109038] [tier.c:588:tier_build_migration_qfile] 0-yahoo-tier-dht: Failed to remove /var/run/gluster/yahoo-tier-dht/demotequeryfile-yahoo-tier-dht
[2015-09-14 14:00:09.121740] I [MSGID: 109028] [dht-rebalance.c:3075:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 72.00 secs
[2015-09-14 14:00:09.121785] I [MSGID: 109028] [dht-rebalance.c:3079:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 0, failures: 0, skipped: 0
[2015-09-14 14:00:09.146200] I [MSGID: 109028] [dht-rebalance.c:3075:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 72.00 secs
[2015-09-14 14:00:09.146256] I [MSGID: 109028] [dht-rebalance.c:3079:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 0, failures: 0, skipped: 0
[2015-09-14 14:00:10.515121] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:00:20.516131] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:00:30.517991] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:00:40.519041] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:00:50.520078] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:01:00.521109] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:01:10.522109] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:01:20.523135] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:01:30.524236] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:01:40.525293] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:01:50.526343] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]

[2015-09-14 14:02:10.533963] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:02:20.534976] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:02:30.536098] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:02:40.537123] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:02:50.538222] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:03:00.539298] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:03:10.540469] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:03:15.621425] I [MSGID: 109028] [dht-rebalance.c:3075:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 258.00 secs
[2015-09-14 14:03:15.621469] I [MSGID: 109028] [dht-rebalance.c:3079:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 0, failures: 0, skipped: 0
[2015-09-14 14:03:20.541455] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
=============
[2015-09-14 14:03:30.542780] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:03:40.543837] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:03:50.544899] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:04:10.552114] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:04:17.678132] I [MSGID: 109028] [dht-rebalance.c:3075:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 320.00 secs
[2015-09-14 14:04:18.283069] I [MSGID: 109028] [dht-rebalance.c:3075:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 321.00 secs
[2015-09-14 14:04:20.553145] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:04:30.554187] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:04:40.555232] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:03:15.646729] I [MSGID: 109028] [dht-rebalance.c:3075:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 258.00 secs
[2015-09-14 14:04:17.702656] I [MSGID: 109028] [dht-rebalance.c:3075:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 320.00 secs
The message "I [MSGID: 109028] [dht-rebalance.c:3075:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 321.00 secs" repeated 3 times between [2015-09-14 14:04:18.283069] and [2015-09-14 14:04:18.730483]
The message "I [MSGID: 109028] [dht-rebalance.c:3079:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 0, failures: 0, skipped: 0" repeated 7 times between [2015-09-14 14:03:15.621469] and [2015-09-14 14:04:18.730485]
[2015-09-14 14:04:50.556275] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:05:00.557435] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:05:10.558562] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:05:20.559632] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:05:30.560660] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:05:40.561716] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:05:50.562722] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
============================ after setting demote===================
[2015-09-14 14:06:10.570311] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:06:20.571336] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:06:30.572397] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:06:40.573430] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:06:50.574580] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:06:55.033667] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2015-09-14 14:06:55.545986] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2015-09-14 14:06:55.569533] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-tier-dht: adding option 'commit-hash' for volume 'yahoo-tier-dht' with value '2947978428'
[2015-09-14 14:06:55.569555] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-tier-dht: adding option 'node-uuid' for volume 'yahoo-tier-dht' with value '0712f242-24fe-4fac-9ba5-1396b11ecc5f'
[2015-09-14 14:06:55.569567] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-tier-dht: adding option 'rebalance-cmd' for volume 'yahoo-tier-dht' with value '6'
[2015-09-14 14:06:55.569576] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-tier-dht: adding option 'xattr-name' for volume 'yahoo-tier-dht' with value 'trusted.tier-gfid'
[2015-09-14 14:06:55.569585] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-tier-dht: adding option 'readdir-optimize' for volume 'yahoo-tier-dht' with value 'on'
[2015-09-14 14:06:55.569596] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-tier-dht: adding option 'assert-no-child-down' for volume 'yahoo-tier-dht' with value 'yes'
[2015-09-14 14:06:55.569650] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-tier-dht: adding option 'lookup-unhashed' for volume 'yahoo-tier-dht' with value 'yes'
[2015-09-14 14:06:55.569664] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-tier-dht: adding option 'use-readdirp' for volume 'yahoo-tier-dht' with value 'yes'
[2015-09-14 14:06:55.569673] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-hot-dht: adding option 'commit-hash' for volume 'yahoo-hot-dht' with value '2947978428'
[2015-09-14 14:06:55.569694] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-hot-dht: adding option 'node-uuid' for volume 'yahoo-hot-dht' with value '0712f242-24fe-4fac-9ba5-1396b11ecc5f'
[2015-09-14 14:06:55.569704] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-hot-dht: adding option 'rebalance-cmd' for volume 'yahoo-hot-dht' with value '6'
[2015-09-14 14:06:55.569715] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-hot-dht: adding option 'readdir-optimize' for volume 'yahoo-hot-dht' with value 'on'
[2015-09-14 14:06:55.569760] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-hot-dht: adding option 'assert-no-child-down' for volume 'yahoo-hot-dht' with value 'yes'
[2015-09-14 14:06:55.569771] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-hot-dht: adding option 'lookup-unhashed' for volume 'yahoo-hot-dht' with value 'yes'
[2015-09-14 14:06:55.569780] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-hot-dht: adding option 'use-readdirp' for volume 'yahoo-hot-dht' with value 'yes'
[2015-09-14 14:06:55.569793] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-hot-replicate-1: adding option 'entry-self-heal' for volume 'yahoo-hot-replicate-1' with value 'off'
[2015-09-14 14:06:55.569801] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-hot-replicate-1: adding option 'metadata-self-heal' for volume 'yahoo-hot-replicate-1' with value 'off'
[2015-09-14 14:06:55.569809] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-hot-replicate-1: adding option 'data-self-heal' for volume 'yahoo-hot-replicate-1' with value 'off'
[2015-09-14 14:06:55.569818] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-hot-replicate-0: adding option 'entry-self-heal' for volume 'yahoo-hot-replicate-0' with value 'off'
[2015-09-14 14:06:55.569825] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-hot-replicate-0: adding option 'metadata-self-heal' for volume 'yahoo-hot-replicate-0' with value 'off'
[2015-09-14 14:06:55.569834] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-hot-replicate-0: adding option 'data-self-heal' for volume 'yahoo-hot-replicate-0' with value 'off'
[2015-09-14 14:06:55.569842] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-cold-dht: adding option 'commit-hash' for volume 'yahoo-cold-dht' with value '2947978428'
[2015-09-14 14:06:55.569853] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-cold-dht: adding option 'node-uuid' for volume 'yahoo-cold-dht' with value '0712f242-24fe-4fac-9ba5-1396b11ecc5f'
[2015-09-14 14:06:55.569896] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-cold-dht: adding option 'rebalance-cmd' for volume 'yahoo-cold-dht' with value '6'
[2015-09-14 14:06:55.569906] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-cold-dht: adding option 'readdir-optimize' for volume 'yahoo-cold-dht' with value 'on'
[2015-09-14 14:06:55.569917] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-cold-dht: adding option 'assert-no-child-down' for volume 'yahoo-cold-dht' with value 'yes'
[2015-09-14 14:06:55.569926] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-cold-dht: adding option 'lookup-unhashed' for volume 'yahoo-cold-dht' with value 'yes'
[2015-09-14 14:06:55.569937] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-cold-dht: adding option 'use-readdirp' for volume 'yahoo-cold-dht' with value 'yes'
[2015-09-14 14:06:55.570675] I [MSGID: 109103] [dht-shared.c:468:dht_reconfigure] 0-DHT: conf->dthrottle: normal, conf->defrag->recon_thread_count: 4
[2015-09-14 14:06:55.593289] I [glusterfsd-mgmt.c:1512:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2015-09-14 14:06:58.399777] I [MSGID: 109028] [dht-rebalance.c:3075:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 481.00 secs
[2015-09-14 14:06:58.399822] I [MSGID: 109028] [dht-rebalance.c:3079:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 0, failures: 0, skipped: 0
==========================


new hot files get immediately demoted
Comment 1 Joseph Elwin Fernandes 2016-02-14 07:05:27 UTC 
http://review.gluster.org/12331

*** This bug has been marked as a duplicate of bug 1261146 ***