Bug 1264831

Description of problem:
========================
while  writes are going to a file if user issue a detach-tier, then the IOs are getting missed to the file.
For eg, I mounted a tier vol on fuse and issued file creates of size 100Mb as below:
[root@localhost lanka]# for i in {1..100};do dd if=/dev/urandom of=file.$i bs=1024 count=100000;done

Now after about 2 files were created and when third file was in progress and about 70% of writes(70MB) was created, I issued a detach-tier start. The writes to the file stopped there and detach-tier went on. After detach tier start completed, it started to create the 4th file on cold tier. That means the 3rd file was incomplete and writes were missed

See brick logs:
[root@zod ~]# ###############no destach start when file3 is creating##########
[root@zod ~]# ll /rhs/brick*/lank*
/rhs/brick1/lanka:
total 335244
-rw-r--r--. 2 root root 102400000 Sep 21 15:26 file.1
-rw-r--r--. 2 root root 102400000 Sep 21 15:27 file.2
-rw-r--r--. 2 root root  71508992 Sep 21 15:27 file.3(writes stopped to this file as soon as detach-tier start was issued)(also files were migrated to cold brick as here)
-rw-r--r--. 2 root root  57340928 Sep 21 15:28 file.4(new file create post detach tier start completed)

==========Mount point fuse error========
[root@localhost lanka]# for i in {1..100};do dd if=/dev/urandom of=file.$i bs=1024 count=100000;done
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 53.8811 s, 1.9 MB/s
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 55.6703 s, 1.8 MB/s
dd: error writing ‘file.3’: Input/output error
69858+0 records in
69857+0 records out
71533568 bytes (72 MB) copied, 41.3222 s, 1.7 MB/s
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 59.3975 s, 1.7 MB/s
100000+0 records in
100000+0 records out





Version-Release number of selected component (if applicable):
===============================================================

[root@zod ~]# gluster --version
glusterfs 3.7.4 built on Sep 19 2015 01:30:43
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General Public License.
[root@zod ~]# rpm -qa|grep gluster
glusterfs-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-fuse-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-debuginfo-3.7.4-0.33.git1d02d4b.el7.centos.x86_64
glusterfs-api-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-client-xlators-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-server-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-cli-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-libs-3.7.4-0.43.gitf139283.el7.centos.x86_64
[root@zod ~]# 


[root@zod ~]# gluster v info
 
Volume Name: lanka
Type: Tier
Volume ID: 258a9a07-43e8-417e-8152-880ca5186f53
Status: Started
Number of Bricks: 10
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick1: yarrow:/rhs/brick7/lanka_hot
Brick2: zod:/rhs/brick7/lanka_hot
Brick3: yarrow:/rhs/brick6/lanka_hot
Brick4: zod:/rhs/brick6/lanka_hot
Cold Tier:
Cold Tier Type : Distributed-Replicate
Number of Bricks: 3 x 2 = 6
Brick5: zod:/rhs/brick1/lanka
Brick6: yarrow:/rhs/brick1/lanka
Brick7: zod:/rhs/brick2/lanka
Brick8: yarrow:/rhs/brick2/lanka
Brick9: zod:/rhs/brick3/lanka
Brick10: yarrow:/rhs/brick3/lanka
Options Reconfigured:
features.quota-deem-statfs: on
features.inode-quota: on
features.quota: on
performance.readdir-ahead: on
[root@zod ~]# gluster v status
Status of volume: lanka
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Hot Bricks:
Brick yarrow:/rhs/brick7/lanka_hot          49261     0          Y       23253
Brick zod:/rhs/brick7/lanka_hot             49278     0          Y       8213 
Brick yarrow:/rhs/brick6/lanka_hot          49260     0          Y       23234
Brick zod:/rhs/brick6/lanka_hot             49277     0          Y       8195 
Cold Bricks:
Brick zod:/rhs/brick1/lanka                 49274     0          Y       8015 
Brick yarrow:/rhs/brick1/lanka              49257     0          Y       22961
Brick zod:/rhs/brick2/lanka                 49275     0          Y       8033 
Brick yarrow:/rhs/brick2/lanka              49258     0          Y       22981
Brick zod:/rhs/brick3/lanka                 49276     0          Y       8051 
Brick yarrow:/rhs/brick3/lanka              49259     0          Y       22999
NFS Server on localhost                     2049      0          Y       8232 
Quota Daemon on localhost                   N/A       N/A        Y       8246 
NFS Server on yarrow                        2049      0          Y       23294
Quota Daemon on yarrow                      N/A       N/A        Y       23324
 
Task Status of Volume lanka
------------------------------------------------------------------------------
Task                 : Rebalance           
ID                   : 4306a687-1a83-4df8-8890-bdec702820c0
Status               : in progress         
 



Steps to Reproduce:
====================
1.attach tier layer to a volume with quota enabled
2.enable ctr
3.now fuse mount the volume
4. Now start file creates for about 15 files  in loop using dd command, of say 100MB, 
5)Now, after two files are created completely, and while 3rd file create has started, issue a detach tier start
6)after detach tier start is completed, it can be seen that the 3rd file create abrubptbly ends and 4th file is created on cold layer, with IO error for the 3rd file on mount

Expected results:
===================
seemless detach-tier is required