Description of problem:
=======================
I had a tier volume with cold as 2 x (4 + 2) = 12 and hot as 2x2




Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.create a volume(in my case an ec vol of 2 x (4 + 2))
2.now create some files say c1,c2,c3
3.Now attach a hot tier 
4. set ctr on (features.ctr-enabled: on)
5. set promote freq to say 10sec
cluster.tier-demote-frequency: 10

6. Now touch the files repeatedly and check the db.
It can be seen that the files are not at all getting promoted.
Even after doing "ls", this also didnt help

7. Now create a new file, which will go to hot tier.
8. Now set demote freq to say 20sec and keep the vol idle for about 1min, it can be seen that the file gets demoted.
9. Now access all the files which are all in cold tier now/
10. only the files created post attach tier are heated/cooled




=========CLI========================

[2015-09-14 13:59:30.504847] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 13:59:40.505916] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 13:59:50.506991] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
The message "I [MSGID: 109103] [dht-shared.c:468:dht_reconfigure] 0-DHT: conf->dthrottle: normal, conf->defrag->recon_thread_count: 4" repeated 2 times between [2015-09-14 13:59:17.713835] and [2015-09-14 13:59:17.714437]
[2015-09-14 13:59:55.065408] I [MSGID: 109028] [dht-rebalance.c:3075:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 58.00 secs
[2015-09-14 13:59:55.065469] I [MSGID: 109028] [dht-rebalance.c:3079:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 0, failures: 0, skipped: 0
[2015-09-14 13:59:55.087425] I [MSGID: 109028] [dht-rebalance.c:3075:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 58.00 secs
[2015-09-14 13:59:55.087457] I [MSGID: 109028] [dht-rebalance.c:3079:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 0, failures: 0, skipped: 0
[2015-09-14 14:00:00.507956] I [MSGID: 109038] [tier.c:588:tier_build_migration_qfile] 0-yahoo-tier-dht: Failed to remove /var/run/gluster/yahoo-tier-dht/promotequeryfile-yahoo-tier-dht
[2015-09-14 14:00:00.508022] I [MSGID: 109038] [tier.c:588:tier_build_migration_qfile] 0-yahoo-tier-dht: Failed to remove /var/run/gluster/yahoo-tier-dht/demotequeryfile-yahoo-tier-dht
[2015-09-14 14:00:09.121740] I [MSGID: 109028] [dht-rebalance.c:3075:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 72.00 secs
[2015-09-14 14:00:09.121785] I [MSGID: 109028] [dht-rebalance.c:3079:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 0, failures: 0, skipped: 0
[2015-09-14 14:00:09.146200] I [MSGID: 109028] [dht-rebalance.c:3075:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 72.00 secs
[2015-09-14 14:00:09.146256] I [MSGID: 109028] [dht-rebalance.c:3079:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 0, failures: 0, skipped: 0
[2015-09-14 14:00:10.515121] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:00:20.516131] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:00:30.517991] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:00:40.519041] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:00:50.520078] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:01:00.521109] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:01:10.522109] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:01:20.523135] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:01:30.524236] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:01:40.525293] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:01:50.526343] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]

[2015-09-14 14:02:10.533963] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:02:20.534976] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:02:30.536098] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:02:40.537123] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:02:50.538222] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:03:00.539298] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:03:10.540469] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:03:15.621425] I [MSGID: 109028] [dht-rebalance.c:3075:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 258.00 secs
[2015-09-14 14:03:15.621469] I [MSGID: 109028] [dht-rebalance.c:3079:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 0, failures: 0, skipped: 0
[2015-09-14 14:03:20.541455] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
=============
[2015-09-14 14:03:30.542780] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:03:40.543837] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:03:50.544899] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:04:10.552114] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:04:17.678132] I [MSGID: 109028] [dht-rebalance.c:3075:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 320.00 secs
[2015-09-14 14:04:18.283069] I [MSGID: 109028] [dht-rebalance.c:3075:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 321.00 secs
[2015-09-14 14:04:20.553145] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:04:30.554187] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:04:40.555232] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:03:15.646729] I [MSGID: 109028] [dht-rebalance.c:3075:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 258.00 secs
[2015-09-14 14:04:17.702656] I [MSGID: 109028] [dht-rebalance.c:3075:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 320.00 secs
The message "I [MSGID: 109028] [dht-rebalance.c:3075:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 321.00 secs" repeated 3 times between [2015-09-14 14:04:18.283069] and [2015-09-14 14:04:18.730483]
The message "I [MSGID: 109028] [dht-rebalance.c:3079:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 0, failures: 0, skipped: 0" repeated 7 times between [2015-09-14 14:03:15.621469] and [2015-09-14 14:04:18.730485]
[2015-09-14 14:04:50.556275] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:05:00.557435] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:05:10.558562] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:05:20.559632] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:05:30.560660] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:05:40.561716] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:05:50.562722] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
============================ after setting demote===================
[2015-09-14 14:06:10.570311] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:06:20.571336] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:06:30.572397] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:06:40.573430] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:06:50.574580] E [tier.c:693:tier_promote] (-->/lib64/libc.so.6(clone+0x6d) [0x7ffab26041ad] -->/lib64/libpthread.so.0(+0x7df5) [0x7ffab2cbddf5] -->/usr/lib64/glusterfs/3.7.4/xlator/cluster/tier.so(+0x56b48) [0x7ffaa59feb48] ) 0-tier: invalid argument: promotion_args->this [Invalid argument]
[2015-09-14 14:06:55.033667] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2015-09-14 14:06:55.545986] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2015-09-14 14:06:55.569533] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-tier-dht: adding option 'commit-hash' for volume 'yahoo-tier-dht' with value '2947978428'
[2015-09-14 14:06:55.569555] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-tier-dht: adding option 'node-uuid' for volume 'yahoo-tier-dht' with value '0712f242-24fe-4fac-9ba5-1396b11ecc5f'
[2015-09-14 14:06:55.569567] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-tier-dht: adding option 'rebalance-cmd' for volume 'yahoo-tier-dht' with value '6'
[2015-09-14 14:06:55.569576] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-tier-dht: adding option 'xattr-name' for volume 'yahoo-tier-dht' with value 'trusted.tier-gfid'
[2015-09-14 14:06:55.569585] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-tier-dht: adding option 'readdir-optimize' for volume 'yahoo-tier-dht' with value 'on'
[2015-09-14 14:06:55.569596] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-tier-dht: adding option 'assert-no-child-down' for volume 'yahoo-tier-dht' with value 'yes'
[2015-09-14 14:06:55.569650] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-tier-dht: adding option 'lookup-unhashed' for volume 'yahoo-tier-dht' with value 'yes'
[2015-09-14 14:06:55.569664] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-tier-dht: adding option 'use-readdirp' for volume 'yahoo-tier-dht' with value 'yes'
[2015-09-14 14:06:55.569673] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-hot-dht: adding option 'commit-hash' for volume 'yahoo-hot-dht' with value '2947978428'
[2015-09-14 14:06:55.569694] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-hot-dht: adding option 'node-uuid' for volume 'yahoo-hot-dht' with value '0712f242-24fe-4fac-9ba5-1396b11ecc5f'
[2015-09-14 14:06:55.569704] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-hot-dht: adding option 'rebalance-cmd' for volume 'yahoo-hot-dht' with value '6'
[2015-09-14 14:06:55.569715] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-hot-dht: adding option 'readdir-optimize' for volume 'yahoo-hot-dht' with value 'on'
[2015-09-14 14:06:55.569760] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-hot-dht: adding option 'assert-no-child-down' for volume 'yahoo-hot-dht' with value 'yes'
[2015-09-14 14:06:55.569771] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-hot-dht: adding option 'lookup-unhashed' for volume 'yahoo-hot-dht' with value 'yes'
[2015-09-14 14:06:55.569780] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-hot-dht: adding option 'use-readdirp' for volume 'yahoo-hot-dht' with value 'yes'
[2015-09-14 14:06:55.569793] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-hot-replicate-1: adding option 'entry-self-heal' for volume 'yahoo-hot-replicate-1' with value 'off'
[2015-09-14 14:06:55.569801] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-hot-replicate-1: adding option 'metadata-self-heal' for volume 'yahoo-hot-replicate-1' with value 'off'
[2015-09-14 14:06:55.569809] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-hot-replicate-1: adding option 'data-self-heal' for volume 'yahoo-hot-replicate-1' with value 'off'
[2015-09-14 14:06:55.569818] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-hot-replicate-0: adding option 'entry-self-heal' for volume 'yahoo-hot-replicate-0' with value 'off'
[2015-09-14 14:06:55.569825] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-hot-replicate-0: adding option 'metadata-self-heal' for volume 'yahoo-hot-replicate-0' with value 'off'
[2015-09-14 14:06:55.569834] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-hot-replicate-0: adding option 'data-self-heal' for volume 'yahoo-hot-replicate-0' with value 'off'
[2015-09-14 14:06:55.569842] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-cold-dht: adding option 'commit-hash' for volume 'yahoo-cold-dht' with value '2947978428'
[2015-09-14 14:06:55.569853] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-cold-dht: adding option 'node-uuid' for volume 'yahoo-cold-dht' with value '0712f242-24fe-4fac-9ba5-1396b11ecc5f'
[2015-09-14 14:06:55.569896] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-cold-dht: adding option 'rebalance-cmd' for volume 'yahoo-cold-dht' with value '6'
[2015-09-14 14:06:55.569906] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-cold-dht: adding option 'readdir-optimize' for volume 'yahoo-cold-dht' with value 'on'
[2015-09-14 14:06:55.569917] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-cold-dht: adding option 'assert-no-child-down' for volume 'yahoo-cold-dht' with value 'yes'
[2015-09-14 14:06:55.569926] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-cold-dht: adding option 'lookup-unhashed' for volume 'yahoo-cold-dht' with value 'yes'
[2015-09-14 14:06:55.569937] I [graph.c:269:gf_add_cmdline_options] 0-yahoo-cold-dht: adding option 'use-readdirp' for volume 'yahoo-cold-dht' with value 'yes'
[2015-09-14 14:06:55.570675] I [MSGID: 109103] [dht-shared.c:468:dht_reconfigure] 0-DHT: conf->dthrottle: normal, conf->defrag->recon_thread_count: 4
[2015-09-14 14:06:55.593289] I [glusterfsd-mgmt.c:1512:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2015-09-14 14:06:58.399777] I [MSGID: 109028] [dht-rebalance.c:3075:gf_defrag_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 481.00 secs
[2015-09-14 14:06:58.399822] I [MSGID: 109028] [dht-rebalance.c:3079:gf_defrag_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 0, failures: 0, skipped: 0
==========================


new hot files get immediately demoted