Bug 1319634

The issue is still seen, however this time with detach tier and with NFS mount.

steps followed:
1) create a dist-rep and start it followed by enabling quota
2) now nfs mount the volume and use dd command to create say 5 files of atleast 700MB each " for i in {1..5};do dd if=/dev/urandom of=file$i bs=1024 count=700000;echo $?;done"
3) Now while dd is in progress, perform an attach tier operation
4) After attach tier is successful, Perform detach tier start --> This is when dd throws IO error


[root@dhcp46-9 mnt]# while true; do for i in {1..5};do dd if=/dev/urandom of=file$i bs=1024 count=700000;echo $?;done; echo 'end of cycle'; done
700000+0 records in
700000+0 records out
716800000 bytes (717 MB) copied, 73.3324 s, 9.8 MB/s
0
700000+0 records in
700000+0 records out
716800000 bytes (717 MB) copied, 71.0725 s, 10.1 MB/s
0
dd: error writing ‘file3’: Input/output error
600027+0 records in
600026+0 records out
614426624 bytes (614 MB) copied, 70.7233 s, 8.7 MB/s
1
700000+0 records in
700000+0 records out
716800000 bytes (717 MB) copied, 75.3172 s, 9.5 MB/s
0
700000+0 records in
700000+0 records out
716800000 bytes (717 MB) copied, 73.2562 s, 9.8 MB/s
0
end of cycle

[2016-04-12 01:43:39.423991] E [MSGID: 108008] [afr-transaction.c:1981:afr_transaction] 0-testvol-replicate-4: Failing WRITE on gfid 250d586b-3591-470b-a3ce-99fe52bb453d: split-brain observed. [Input/output error]
[2016-04-12 01:43:39.424838] E [MSGID: 108008] [afr-transaction.c:1981:afr_transaction] 0-testvol-replicate-4: Failing WRITE on gfid 250d586b-3591-470b-a3ce-99fe52bb453d: split-brain observed. [Input/output error]
[2016-04-12 01:43:39.425705] E [MSGID: 108008] [afr-transaction.c:1981:afr_transaction] 0-testvol-replicate-4: Failing WRITE on gfid 250d586b-3591-470b-a3ce-99fe52bb453d: split-brain observed. [Input/output error]
[2016-04-12 01:43:39.429049] E [MSGID: 108008] [afr-transaction.c:1981:afr_transaction] 0-testvol-replicate-4: Failing WRITE on gfid 250d586b-3591-470b-a3ce-99fe52bb453d: split-brain observed. [Input/output error]
[2016-04-12 01:43:39.430226] E [MSGID: 108008] [afr-transaction.c:1981:afr_transaction] 0-testvol-replicate-4: Failing WRITE on gfid 250d586b-3591-470b-a3ce-99fe52bb453d: split-brain observed. [Input/output error]

[root@dhcp47-105 ~]# gluster v info
 
Volume Name: testvol
Type: Tier
Volume ID: 02427025-adcf-48a2-ac58-ae494839e9f8
Status: Started
Number of Bricks: 12
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick1: 10.70.46.94:/bricks/brick3/leg1
Brick2: 10.70.47.9:/bricks/brick3/leg1
Brick3: 10.70.47.105:/bricks/brick3/leg1
Brick4: 10.70.47.90:/bricks/brick3/leg1
Cold Tier:
Cold Tier Type : Distributed-Replicate
Number of Bricks: 4 x 2 = 8
Brick5: 10.70.47.90:/bricks/brick0/ct
Brick6: 10.70.47.105:/bricks/brick0/ct
Brick7: 10.70.47.9:/bricks/brick0/ct
Brick8: 10.70.46.94:/bricks/brick0/ct
Brick9: 10.70.47.90:/bricks/brick1/ct
Brick10: 10.70.47.105:/bricks/brick1/ct
Brick11: 10.70.47.9:/bricks/brick1/ct
Brick12: 10.70.46.94:/bricks/brick1/ct
Options Reconfigured:
cluster.tier-mode: cache
features.ctr-enabled: on
performance.readdir-ahead: on

sosreports are available here --> http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/1319634/

Marking this as verified as no issue is seen with attach tier. A different bug for detach tier and large file writes shall be raised.

https://bugzilla.redhat.com/show_bug.cgi?id=1326248 is raised to track the IO error issue with detach tier operation.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1240