Created attachment 1476753 [details]
gfapi log

Description of problem:

We have recently upgraded glusterfs from 3.7.6 to  3.10.12, sometimes creating file using gfapi in a new directory fail (errno 5), but no error after retrying.


Version-Release number of selected component (if applicable):

glusterfs-3.10.12


How reproducible:

application log (timezone GMT+8):
------------------------------------------------

2018-08-18 18:05:05,333 DEBUG -52399312- Create new directory [FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m] on cluster [4] ((unknown file):0)(0036d13e7d3a93ee)
2018-08-18 18:05:05,371 CRITICAL -58525808- Failed to open cluster [4] object [FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a] with mode [w]: [[Errno 5] Input/output error] (app:1461)(0036d13e7d3a93ee)
2018-08-18 18:05:05,703 DEBUG -58525808- Open cluster [4] file [FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a] with mode [w] (app:1461)(0036d13e7d3a93ee)
2018-08-18 18:05:05,943 INFO -58525808- Rename file: cluster=4 src=FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a dst=FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003 (app:1461)(0036d13e7d3a93ee)


error in gfapi
----------------------

[2018-08-18 10:05:05.338722] E [MSGID: 109011] [dht-common.c:7248:dht_create] 0-vol0-dht: no subvolume in layout for path=/FS/rt/mbXx/service-log_0/1161baf2416c3b608ba06db7055b0e7b_171632304626615382.m/0003_2df00662a2ce11e8850b246e9631871a


Steps to Reproduce:
1. create new directory
2. create file under the directory

Actual results:

IO error happends

Expected results:

No error when creating file


Additional info:
-----------------
 
Volume Name: vol0
Type: Distributed-Replicate
Volume ID: 18e1c05d-570a-4c97-aa91-ef984881c4f2
Status: Started
Snapshot Count: 0
Number of Bricks: 36 x 3 = 108
Transport-type: tcp

Options Reconfigured:
locks.trace: false
client.event-threads: 6
cluster.self-heal-daemon: enable
performance.write-behind: True
transport.keepalive: True
cluster.rebal-throttle: lazy
server.event-threads: 4
performance.io-cache: False
nfs.disable: True
cluster.quorum-type: auto
network.ping-timeout: 120
features.cache-invalidation: False
performance.read-ahead: False
performance.client-io-threads: True
cluster.server-quorum-type: none
performance.md-cache-timeout: 0
performance.readdir-ahead: True