Bug 1238944

Description of problem:
=======================
I have a volume, where i had created some 5 zerobyte files and about 3 non-empty files.
I later attached a tier, which means all the files reside in the cold tier, by default.
After enabling ctr, I performed a read of all files one after one(and even tried parallelly issuing cat *).
I observed that only about two zerobyte files were getting promoted and not all the files.
This problem was seen consistently.




Version-Release number of selected component (if applicable):
============================================================
[root@tettnang glusterfs]# gluster --version
glusterfs 3.7.1 built on Jun 28 2015 11:01:14
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.
[root@tettnang glusterfs]# rpm -qa|grep gluster
glusterfs-cli-3.7.1-6.el7rhgs.x86_64
glusterfs-client-xlators-3.7.1-6.el7rhgs.x86_64
glusterfs-fuse-3.7.1-6.el7rhgs.x86_64
glusterfs-geo-replication-3.7.1-6.el7rhgs.x86_64
glusterfs-libs-3.7.1-6.el7rhgs.x86_64
glusterfs-api-3.7.1-6.el7rhgs.x86_64
glusterfs-rdma-3.7.1-6.el7rhgs.x86_64
glusterfs-3.7.1-6.el7rhgs.x86_64
glusterfs-debuginfo-3.7.1-6.el7rhgs.x86_64
samba-vfs-glusterfs-4.1.17-7.el7rhgs.x86_64
glusterfs-server-3.7.1-6.el7rhgs.x86_64


Steps to Reproduce:
===================
1.create a dist-rep volume
2.start the volume and turn off following
performance.io-cache: off
performance.quick-read: off

3. mount volume and create some empty and non-empty files
4. Attach a distributed hot tier 
5. Turn on features.ctr-enabled 
6. do a ls -l(this will create T files of all cold files in hot tier for which bug#1238549 was raised)
7. Now read file contents of all files using cat, one after another without any time gap. Also you can do a parallel read by issing cat *


Actual results:
==============
It can be seen that only a couple of files are getting promoted(and in my case the files with data never got promoted)


Expected results:
===================
All files should get promoted, if accessed once atleast



Sos and cli logs are attached