Bug 1275751

Description of problem:
=========================
When attach tier during a file create using dd command of say about 500MB, the file create terminates due to following error:

dd: error writing ‘file2’: Input/output error
dd: closing output file ‘file2’: Input/output error

Tier log shows following error:

[root@rhel7-autofuseclient glusterfs]# cat mnt-tin.log|grep " E "
[2015-10-27 16:56:02.136132] E [MSGID: 108006] [afr-common.c:3881:afr_notify] 0-tin-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
[2015-10-27 16:56:02.136370] E [MSGID: 108006] [afr-common.c:3881:afr_notify] 0-tin-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.
[2015-10-27 16:57:05.123426] E [MSGID: 109040] [dht-helper.c:1020:dht_migration_complete_check_task] 2-tin-cold-dht: (null): failed to lookup the file on tin-cold-dht [Stale file handle]
[2015-10-27 16:57:05.124695] E [MSGID: 108008] [afr-transaction.c:1975:afr_transaction] 2-tin-replicate-1: Failing WRITE on gfid 81fbd264-4e2d-45b7-9554-cfd030764f36: split-brain observed. [Input/output error]
[root@rhel7-autofuseclient glusterfs]# less mnt-tin.log 



How reproducible:
===================
not consistent but quite occasionally seen

Steps to Reproduce:
====================
1.create a dist-rep or ec volume and start it followed by enabling quota
2.now mount the volume and use dd command to create say 10 files of atleast 700MB each " for i in {1..10};do dd if=/dev/urandom of=file$i bs=1024 count=700000;echo $?;done"

3.Now wait for one file create to get completed and when file2 create is in progress attach a dist-rep tier and enable ctr immediately




Actual results:
==============
error can be seen as below on fuse mount:
[root@rhel7-autofuseclient tin]# for i in {1..10};do dd if=/dev/urandom of=file$i bs=1024 count=700000;echo $?;done
700000+0 records in
700000+0 records out
716800000 bytes (717 MB) copied, 408.87 s, 1.8 MB/s
0
dd: error writing ‘file2’: Input/output error
dd: closing output file ‘file2’: Input/output error
1
700000+0 records in
700000+0 records out
716800000 bytes (717 MB) copied, 345.21 s, 2.1 MB/s
0
^C301502+0 records in
301501+0 records out
308737024 bytes (309 MB) copied, 142.061 s, 2.2 MB/s


LOGS:
====


[root@zod ~]#  gluster v create tin rep 2 zod:/rhs/brick1/tin yarrow:/rhs/brick1/tin zod:/rhs/brick2/tin yarrow:/rhs/brick2/tin
volume create: tin: success: please start the volume to access data
[root@zod ~]# gluster v start bell
volume start: bell: failed: Volume bell already started
[root@zod ~]# gluster v start tin
volume start: tin: success
[root@zod ~]# gluster v quota tin enable
volume quota : success
[root@zod ~]# gluster v attach-tier tin rep 2 zod:/rhs/brick7/tin_hot yarrow:/rhs/brick7/tin_hot zod:/rhs/brick6/tin_hot yarrow:/rhs/brick6/tin_hot; 
Attach tier is recommended only for testing purposes in this release. Do you want to continue? (y/n) y
volume attach-tier: success
volume rebalance: tin: success: Rebalance on tin has been started successfully. Use rebalance status command to check status of the rebalance process.
ID: 3161efbb-36b7-4b78-a314-7e8aa5d320fa

[root@zod ~]# gluster v set tin features.ctr-enabled on
volume set: success
[root@zod ~]# gluster v status tin
Status of volume: tin
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Hot Bricks:
Brick yarrow:/rhs/brick6/tin_hot            49183     0          Y       24462
Brick zod:/rhs/brick6/tin_hot               49183     0          Y       26303
Brick yarrow:/rhs/brick7/tin_hot            49182     0          Y       24438
Brick zod:/rhs/brick7/tin_hot               49182     0          Y       26285
Cold Bricks:
Brick zod:/rhs/brick1/tin                   49180     0          Y       25646
Brick yarrow:/rhs/brick1/tin                49180     0          Y       23911
Brick zod:/rhs/brick2/tin                   49181     0          Y       25664
Brick yarrow:/rhs/brick2/tin                49181     0          Y       23935
NFS Server on localhost                     2049      0          Y       26322
Self-heal Daemon on localhost               N/A       N/A        Y       26336
Quota Daemon on localhost                   N/A       N/A        Y       26352
NFS Server on yarrow                        2049      0          Y       24506
Self-heal Daemon on yarrow                  N/A       N/A        Y       24524
Quota Daemon on yarrow                      N/A       N/A        Y       24532
 
Task Status of Volume tin
------------------------------------------------------------------------------
Task                 : Tier migration      
ID                   : 3161efbb-36b7-4b78-a314-7e8aa5d320fa
Status               : in progress         
 
[root@zod ~]# gluster v info tin
 
Volume Name: tin
Type: Tier
Volume ID: c56894ad-d33e-4124-8600-da830891e0c1
Status: Started
Number of Bricks: 8
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick1: yarrow:/rhs/brick6/tin_hot
Brick2: zod:/rhs/brick6/tin_hot
Brick3: yarrow:/rhs/brick7/tin_hot
Brick4: zod:/rhs/brick7/tin_hot
Cold Tier:
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick5: zod:/rhs/brick1/tin
Brick6: yarrow:/rhs/brick1/tin
Brick7: zod:/rhs/brick2/tin
Brick8: yarrow:/rhs/brick2/tin
Options Reconfigured:
features.ctr-enabled: on
features.quota-deem-statfs: on
features.inode-quota: on
features.quota: on
performance.readdir-ahead: on
[root@zod ~]#