Bug 1588452

sorry i press enter by mistake. please find details here.

Description of problem: gluster is allowing me write meta data in arbiter bricks when my data bricks are full and one data node is down.


Version-Release number of selected component (if applicable):3.3.1-15


How reproducible:


Steps to Reproduce:
1. bring down the one arbiter node.
2. bring down the one data node.
3. bring up arbiter node.
4. write data until data devices are not getting full.

Actual results:
meta data is going in arbiter bricks when data bricks are full.

Expected results:
it should not allow to write meta data in arbiter brick when data bricks are full.

Created attachment 1448707 [details]
sosreports and logs

Setup details

I had 3 nodes in gluster setup i which 
- first node is arbiter node
- second node is data node
- third node is data node

I created volume of 10gb.

- i wrote some files in that volume.
- after that i brought first and second node down.
- only one data node is up now, so we can not write data.
- i brought arbiter node up.
- after bringing arbiter node up, new files were written into the volume
- after some time data brick got full. And IOs failed with the error "no sufficient space" at mount point. However, at the back-end it is still writing meta data in arbiter bricks.
- i was doing IOs with dd.

Initially, in my set up there are 3 nodes in which 2 are data nodes and 1 arbiter node. 
Step 1- I wrote 4 files in the nodes. Named, file1, file2, file3, file4.
Step 2- I got the arbiter and one of the data node in down state. 
Step 3- I got the arbiter node back to up state and wrote 11 new files. Named, file5, file6, file7, file8, file9, file10, file11, file12, file13, file14, file15.
Step 4- As there was space available to write 6 files. Files till file number 10 were successfully created and their metadata has been successfully written in the arbiter brick. 
Step 5- Files 11 to file 15 were not successfully created yet their metadata was written in the arbiter brick. This should NOT have taken place. 
Step 6- I next got the data node in up state. Self healing process started.
Step 7- The data brick now has data (files 5 to 10) and the meta data for files 11 to 15 from the arbiter were also written in the data brick while the self healing was in process.

Brick 10.70.46.49:/var/lib/heketi/mounts/vg_7bd0718ec9b17e44856735b60d662d17/brick_bd90b029a07f7a66a6844fb614b51100/brick
Status: Connected
Number of entries: 0

Brick 10.70.47.43:/var/lib/heketi/mounts/vg_bf140e043bce089871f1c08cd2d3b4e5/brick_ec4f4b9fae005127f48aad84618ecfe5/brick
/file11 
/file12 
/file13 
/file14 
/file15 
Status: Connected
Number of entries: 5

Brick 10.70.47.69:/var/lib/heketi/mounts/vg_5dbabacaae2fd8bf759d266308fd89b6/brick_dcbf8386608c3b142a6c45e9dac78e5a/brick
/file11 
/file12 
/file13 
/file14 
/file15 
Status: Connected
Number of entries: 5

Node Id: deeaf902d09002db7127e9e0d06198be
State: online
Cluster Id: dc1c772ff615f0b094c13f822ebffd71
Zone: 1
Management Hostname: dhcp47-69.lab.eng.blr.redhat.com
Storage Hostname: 10.70.47.69
Tags:
  arbiter: required
Devices:
Id:4503a77d87b1e93744156ca3df5c310e   Name:/dev/sdd            State:online    Size (GiB):1023    Used (GiB):0       Free (GiB):1023    Bricks:0       
Id:5dbabacaae2fd8bf759d266308fd89b6   Name:/dev/sdf            State:online    Size (GiB):1023    Used (GiB):0       Free (GiB):1023    Bricks:1       
Id:9977ada7baef55bb75743bcb87975d7d   Name:/dev/sde            State:online    Size (GiB):1023    Used (GiB):2       Free (GiB):1021    Bricks:1       


Node Id: f36b1b77767851acbe5ac8ab35fd1e7a
State: online
Cluster Id: dc1c772ff615f0b094c13f822ebffd71
Zone: 1
Management Hostname: dhcp47-43.lab.eng.blr.redhat.com
Storage Hostname: 10.70.47.43
Tags:
  arbiter: disabled
Devices:
Id:15f30088d3d9de55024e037521a7ec03   Name:/dev/sde            State:online    Size (GiB):1023    Used (GiB):0       Free (GiB):1023    Bricks:0       
Id:a32145bbf8ae443376a244c2d9efd23a   Name:/dev/sdf            State:online    Size (GiB):1023    Used (GiB):0       Free (GiB):1023    Bricks:0       
Id:bf140e043bce089871f1c08cd2d3b4e5   Name:/dev/sdd            State:online    Size (GiB):1023    Used (GiB):3       Free (GiB):1020    Bricks:2       


Node Id: 27a9fd92e5a6ae3f132f2191c84c41e1
State: online
Cluster Id: dc1c772ff615f0b094c13f822ebffd71
Zone: 1
Management Hostname: dhcp46-49.lab.eng.blr.redhat.com
Storage Hostname: 10.70.46.49
Tags:
  arbiter: disabled
Devices:
Id:6f0a700b5c2e1da9d0ee96b49ff05320   Name:/dev/sde            State:online    Size (GiB):1023    Used (GiB):0       Free (GiB):1023    Bricks:0       
Id:7bd0718ec9b17e44856735b60d662d17   Name:/dev/sdd            State:online    Size (GiB):1023    Used (GiB):1       Free (GiB):1022    Bricks:1       
Id:ccf19f43c82142a3208661f7b5ed625c   Name:/dev/sdf            State:online    Size (GiB):1023    Used (GiB):2       Free (GiB):1021    Bricks:1